Image processing method and device, equipment and storage medium

Abstract

The invention provides an image processing method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a first original image; scaling the first original image into a second original image based on the scaling coefficient; obtaining a first target object mask image based on the second original image; according to the first target object mask image and a diffusion model, generating a background image except the second target object mask image to obtain a first image; based on the obtained edge mask image of the first target mask image, repairing the edge image of the target in the first image to obtain a second image; on the basis of a detail mask image in the obtained second original image, repairing a detail image of the target object in the second image to obtain a third image; and judging whether the resolution of the third image is greater than that of the first original image, and repairing the resolution according to a judgment result to obtain a target image. According to the method, the background can be efficiently and naturally replaced, and meanwhile, detail restoration is performed to obtain a high-resolution image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer processing technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] With the continuous development of AI Generated Content (AIGC) technology, more and more image generation models are being applied to fashion, clothing and other fields.

[0003] However, in practical applications, existing image generation technologies still face some problems, especially in image background replacement. Traditional background replacement methods, such as Photoshop's mapping method, have disadvantages such as unnaturalness and low manual efficiency, and cannot achieve efficient and natural effects. In addition, existing image processing algorithms can usually only process low-resolution images, and when performing image synthesis, obvious boundary effects often occur, such as poor fusion of hair and character edges, and even image artifacts. Especially in the image generation process of clothing models, clothing details are easily lost, and the perspective relationship between the character and the background is confusing, which further affects the naturalness and realism of the generated image. Summary of the invention

[0004] The present disclosure provides an image processing method, device, equipment and storage medium to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, the method comprising:

[0006] Acquire a first original image, wherein the first original image includes an image of a target object;

[0007] Based on the scaling factor, scaling the first original image to a second original image;

[0008] Based on the second original image, obtaining a first target object mask image;

[0009] Generate a background image other than the second target object mask image according to the first target object mask image and a diffusion model to obtain a first image;

[0010] Based on the acquired edge mask image of the first target object mask image, repairing the edge image of the target object in the first image to obtain a second image;

[0011] Based on the acquired detail mask image in the second original image, repairing the detail image of the target object in the second image to obtain a third image;

[0012] It is determined whether the resolution of the third image is greater than that of the first original image, and resolution restoration is performed according to the determination result to obtain a target image.

[0013] In one possible implementation manner, acquiring a first target object mask image based on the second original image includes:

[0014] A mask image of the target object in the second original image is obtained as an initial target object mask image; and an edge of the initial target object mask image is reduced according to a preset size to obtain a first target object mask image.

[0015] In one possible implementation, the diffusion model includes a depth preprocessor and a depth control network, and the generating of a background image outside the second target object mask image according to the first target object mask image and the diffusion model to obtain the first image includes:

[0016] The first target object mask image is subjected to Gaussian processing to obtain a second target object mask image; the depth information of the first original image is obtained through a depth preprocessor; based on the depth control network, the depth information and the second target object mask image, a background image area outside the second target object mask image is preliminarily generated to obtain a first intermediate image; based on the first intermediate image and a diffusion model, the background image area outside the second target object mask image is redrawn to obtain a first image.

[0017] In one possible implementation manner, obtaining the second image based on the edge mask image acquired from the first object mask image includes:

[0018] Acquire an edge mask image of the first target object mask image; based on the edge mask image, acquire an edge image in the second original image; paste the edge image to the first image to obtain a second intermediate image; based on the edge mask image and a diffusion model, perform detail restoration on an edge image region in the second intermediate image to obtain a second image.

[0019] In one possible implementation manner, the step of restoring the detail image of the target object in the second image based on the detail mask image in the acquired second original image to obtain the third image includes:

[0020] Based on the image segmentation model, the detail image in the second original image is processed to obtain a detail mask image; based on the detail mask image and the diffusion model, detail restoration is performed on the detail area in the second image to obtain a third image.

[0021] In one possible implementation manner, determining whether the resolution of the third image is greater than that of the first original image, and performing resolution restoration according to the determination result to obtain the target image includes:

[0022] In response to the resolution of the third image being greater than that of the first original image, the third image is scaled based on the scaling factor to obtain a target image; or, in response to the resolution of the third image being equal to that of the first original image, the third image is used as the target image; in response to the resolution of the third image being less than that of the first original image, the resolution of the third image is repaired to obtain a target image.

[0023] In one possible implementation manner, performing resolution restoration on the third image to obtain a target image includes:

[0024] The third image is amplified based on a bilinear interpolation algorithm to obtain a fourth image; the fourth image is processed based on a super-resolution algorithm to obtain a fifth image; the first target mask image is amplified based on a scaling factor to obtain an amplified first target mask image; the first original image is cut out based on the amplified first target mask image to obtain an original target image; and the original target image is pasted to the fifth image to obtain a target image.

[0025] According to a second aspect of the present disclosure, there is provided an image processing device, the device comprising:

[0026] A first acquisition module, used to acquire a first original image, wherein the first original image includes an image of a target object;

[0027] A second acquisition module, configured to scale the first original image to a second original image based on a scaling factor;

[0028] A third acquisition module, configured to acquire a first target object mask image based on the second original image;

[0029] A background generation module, configured to generate a background image other than the second target object mask image according to the first target object mask image and a diffusion model, so as to obtain a first image;

[0030] an edge generation module, configured to repair an edge image of the target object in the first image based on the edge mask image of the first target object mask image obtained, so as to obtain a second image;

[0031] A detail restoration module, configured to restore a detail image of a target object in the second original image based on the detail mask image acquired in the second original image, so as to obtain a third image;

[0032] The resolution restoration module is used to determine whether the resolution of the third image is greater than that of the first original image, and perform resolution restoration according to the determination result to obtain a target image.

[0033] In one possible implementation, the third acquisition module is specifically used to:

[0034] A mask image of the target object in the second original image is obtained as an initial target object mask image; and an edge of the initial target object mask image is reduced according to a preset size to obtain a first target object mask image.

[0035] In one possible implementation, the diffusion model includes a depth preprocessor and a depth control network, and the background generation module is specifically used to:

[0036] The first target object mask image is subjected to Gaussian processing to obtain a second target object mask image; the depth information of the first original image is obtained through a depth preprocessor; based on the depth control network, the depth information and the second target object mask image, a background image area outside the second target object mask image is preliminarily generated to obtain a first intermediate image; based on the first intermediate image and a diffusion model, the background image area outside the second target object mask image is redrawn to obtain a first image.

[0037] In one possible implementation, the edge generation module is specifically used to:

[0038] Acquire an edge mask image of the first target object mask image; based on the edge mask image, acquire an edge image in the second original image; paste the edge image to the first image to obtain a second intermediate image; based on the edge mask image and a diffusion model, perform detail restoration on an edge image region in the second intermediate image to obtain a second image.

[0039] In one possible implementation, the detail repair module is specifically used to:

[0040] Based on the image segmentation model, the detail image in the second original image is processed to obtain a detail mask image; based on the detail mask image and the diffusion model, detail restoration is performed on the detail area in the second image to obtain a third image.

[0041] In one possible implementation manner, the resolution restoration module is specifically used to:

[0042] In response to the resolution of the third image being greater than that of the first original image, the third image is scaled based on the scaling factor to obtain a target image; or, in response to the resolution of the third image being equal to that of the first original image, the third image is used as the target image; in response to the resolution of the third image being less than that of the first original image, the resolution of the third image is repaired to obtain a target image.

[0043] In one possible implementation manner, the resolution restoration module is specifically used to:

[0044] The third image is amplified based on a bilinear interpolation algorithm to obtain a fourth image; the fourth image is processed based on a super-resolution algorithm to obtain a fifth image; the first target mask image is amplified based on a scaling factor to obtain an amplified first target mask image; the first original image is cut out based on the amplified first target mask image to obtain an original target image; and the original target image is pasted to the fifth image to obtain a target image.

[0045] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0046] at least one processor; and

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0049] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0050] The image processing method, device, equipment and storage medium disclosed in the present invention obtain a first original image, wherein the first original image includes a target image; based on a scaling factor, scale the first original image to a second original image; based on the second original image, obtain a first target mask image; generate a background image outside the second target mask image according to the first target mask image and a diffusion model to obtain a first image; based on the edge mask image of the first target mask image obtained, repair the edge image of the target in the first image to obtain a second image; based on the detail mask image in the second original image obtained, repair the detail image of the target in the second image to obtain a third image; determine whether the resolution of the third image is greater than that of the first original image, and perform resolution repair according to the determination result to obtain a target image. The present application can efficiently and naturally replace the background, and perform detail repair at the same time to obtain a high-resolution image.

[0051] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0053] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0054] Figure 1 A schematic diagram of an implementation flow of an image processing method provided by an embodiment of the present disclosure is shown;

[0055] Figure 2 A schematic diagram of an implementation flow of an exemplary image processing method provided by an embodiment of the present disclosure is shown;

[0056] Figure 3 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure is shown;

[0057] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0059] Figure 1 This is a flowchart of an image processing method provided by an embodiment of the present disclosure. The method can be executed by an image processing device provided by an embodiment of the present disclosure. The device can be implemented in software and / or hardware. The method specifically includes:

[0060] S110: Acquire a first original image, wherein the first original image includes an image of a target object.

[0061] The first original image is an image containing a target object, and the target object may be a person, an animal, or any other object that needs to replace the background. The target object image is only an image containing a target object. Specifically, the first original image obtained in this embodiment may be a color image, a grayscale image, or an image in any form.

[0062] S120: Scale the first original image into a second original image based on a scaling factor.

[0063] The scaling factor is determined according to the target resolution and the resolution of the first original image, and the target resolution is determined according to the resolution supported by the diffusion model and user requirements. The second original image is the image scaled according to the scaling factor.

[0064] Specifically, the scaling factor is calculated based on the target resolution and the resolution of the first original image. For example, if the target resolution for scaling the first original image is W×H=1024×1536, the formulas for calculating the scaling factors a1 and a2 are as follows: Formula (1) and Formula (2):

[0065] a1= w / W (1),

[0066] a2 = h / H (2)

[0067] Wherein, w×h is the width and height of the first original image, and W×H is the width and height of the target resolution. Specifically, after obtaining the scaling coefficients of the width and height, this embodiment selects a smaller scaling coefficient a=min(a1,a2), and obtains the scaling coefficient a to scale the first original image in the same proportion to ensure that the image does not exceed the target resolution after scaling, and obtains the scaled image as the second original image.

[0068] S130: Acquire a first target object mask image based on the second original image.

[0069] The first target object mask image is a mask image that only includes a portion of the target object.

[0070] In the embodiment of the present disclosure, the method of obtaining a first target object mask image based on the second original image includes: obtaining a mask image of the target object in the second original image as an initial target object mask image; and reducing the edge of the initial target object mask image according to a preset size to obtain a first target object mask image.

[0071] The preset size may be any size set according to the type of the target object, for example, the preset size may be 2 pixels.

[0072] Specifically, this embodiment uses any image segmentation model (Segment Anything Model, SAM) model in the prior art to cut out the target object in the second original image to generate a preliminary mask image, that is, a first target object mask image. Exemplarily, if the target object is a person, the first target object mask image is a mask of the person area.

[0073] Then, this embodiment can reduce the edge of the initial target mask image according to a preset size to obtain a first target mask image. Exemplarily, this embodiment can reduce the mask inner area of ​​the above-mentioned person mask area by 2 pixels to obtain a more refined mask image.

[0074] Since the SAM model may not be accurate when performing cutout on the target object in the second original image, and some residual details may exist, this embodiment can remove some background details that may be residual after cutout by reducing the edge of the initial target object mask image.

[0075] S140: Generate a background image other than the second target object mask image according to the first target object mask image and a diffusion model to obtain a first image.

[0076] The diffusion model is an image generation method based on deep learning, which generates high-quality images by gradually adding noise (diffusion process) and denoising (inverse diffusion process). The second target mask image is the image of the first target mask image after Gaussian processing. The first image is the initial image after the target replaces the background.

[0077] In an embodiment of the present disclosure, the diffusion model includes a depth preprocessor and a depth control network. The process of generating a background image outside the second target object mask image according to the first target object mask image and the diffusion model to obtain a first image includes: performing Gaussian processing on the first target object mask image to obtain a second target object mask image; obtaining depth information of the first original image through the depth preprocessor; preliminarily generating a background image area outside the second target object mask image based on the depth control network, the depth information and the second target object mask image to obtain a first intermediate image; and redrawing the background image area outside the second target object mask image based on the first intermediate image and the diffusion model to obtain a first image.

[0078] The depth preprocessor and the depth control network can be plug-ins in the diffusion model or neural network models independent of the diffusion model. The depth preprocessor is used to obtain the depth information of the target object in the first original image. The depth control network is used to generate a preliminary background image according to the key points input by the user.

[0079] Specifically, in this embodiment, the first target mask image is subjected to Gaussian processing to obtain the second target mask image to smooth the edge, so as to improve the fusion degree between the target edge area and the background. The Gaussian formula is as follows (3):

[0080]

[0081] Where σ is the standard deviation of the Gaussian kernel, and (x, y) is the pixel coordinate of each pixel in the first target mask image. G(x, y) represents the weight value of the weight matrix of each pixel in the first target mask image. Then, the RGB value (r, g, b) of each pixel in the first target mask image is (r*G(x, y), g*G(x, y), b*G(x, y)) after Gaussian blur.

[0082] Specifically, after obtaining the second target object mask image, this embodiment obtains the depth information of the first original image through the depth preprocessor, and the perspective relationship of the first original image can be obtained through the depth information. Then, this embodiment can preliminarily generate the background image area outside the second target object mask image based on the depth control network and the depth information to obtain a first intermediate image. This embodiment can control the perspective relationship in image generation according to the depth information through the depth control network to ensure the naturalness and realism of the background generation, and dominate the perspective relationship of the image in the early stage of diffusion (0-0.3 process) through the depth control network.

[0083] Specifically, after obtaining the first intermediate image, this embodiment will redraw the background image area outside the second target mask image through the diffusion model to obtain the first image. Exemplarily, if the depth control network weight is a plug-in in the diffusion model, the weight of the depth control network is configured to be zero in the late expansion process (0.3-1), and the diffusion model is handed over to generate the background according to the prompt word, and the background image outside the second target mask image is precisely controlled to redraw the area to generate the image after the background is replaced, that is, the first image.

[0084] S150: Based on the acquired edge mask image of the first object mask image, repair the edge image of the object in the first image to obtain a second image.

[0085] The edge mask image is an image obtained by reducing the edge of the initial target mask image in step S130. The second image is an image obtained by repairing the target edge image region.

[0086] Since the first target object mask image is an image obtained by shrinking the pixels of a preset size within the initial target object mask image, some details may be lost. For example, if the target object is a person, the detailed features of the person's clothing may be lost. Therefore, this embodiment can obtain an edge mask image by subtracting the first target object mask image from the initial target object mask image.

[0087] In the disclosed embodiment, obtaining a second image based on the edge mask image of the first target object mask image includes: obtaining an edge mask image of the first target object mask image; obtaining an edge image in the second original image based on the edge mask image; pasting the edge image to the first image to obtain a second intermediate image; and performing detail repair on an edge image region in the second intermediate image based on the edge mask image and a diffusion model to obtain a second image.

[0088] Specifically, after obtaining the edge mask image, the present embodiment can cut out the scaled second original image with reference to the edge mask image to obtain the edge image in the second original image, and paste the edge image to the first image after replacing the background to obtain the second intermediate image, so as to restore the effect of the edge feature. Then, based on the edge mask image and the diffusion model, the present embodiment uses a subsequent diffusion process (0.3-1) to perform detail repair on the edge image area in the second intermediate image to obtain the second image.

[0089] This embodiment can add relatively small noise to the edge image region in the second intermediate image, thereby achieving edge detail restoration with a low redrawing amplitude, so as to improve the fusion degree of the edge and the background and restore the original features of the edge.

[0090] S160: Based on the acquired detail mask image in the second original image, repair the detail image of the target object in the second image to obtain a third image.

[0091] The detail mask image may be a mask image of the details of the target object in the second original image. For example, if the target object is a person, the detail mask image may be a mask image of the hair; for example, if the target object is an animal, the detail mask image may be a mask image of the hair. The third image is an image after the detail image area of ​​the target object is repaired.

[0092] In an embodiment of the present disclosure, the method of repairing the detail image of the target object in the second original image based on the acquired detail mask image to obtain a third image includes: processing the detail image in the second original image based on an image segmentation model to obtain a detail mask image; and repairing the details of the detail area in the second image based on the detail mask image and a diffusion model to obtain a third image.

[0093] Specifically, this embodiment can use the image segmentation model SAM to cut out the detail image in the second original image to obtain a detail mask image. Then, based on the detail mask image and the diffusion model, this embodiment uses a subsequent diffusion process (0.3-1) to perform detail restoration on the detail area in the second image to obtain a third image.

[0094] For example, if the target object is a person, the detail image may be the hair region of the person. In this embodiment, for the hair region, the SAM model may be used to perform a special cutout of the person's hair to generate a detail mask image of the hair, and a small noise adjustment is performed in the hair region through a subsequent diffusion process of the diffusion model to repair the hair edge and obtain the third image of the final repaired image.

[0095] S170: Determine whether the resolution of the third image is greater than that of the first original image, and perform resolution repair according to the determination result to obtain a target image.

[0096] Since the first original image is scaled at the beginning of the image processing in this embodiment, the third image is also a scaled image. Since the first original image is an image uploaded by the user and has a certain resolution, this embodiment determines whether the resolution of the third image is greater than that of the first original image. If the resolution of the third image is less than that of the first original image, it is necessary to perform resolution repair on it to obtain the target image, that is, the final ultra-clear image.

[0097] In the embodiment of the present disclosure, the determining whether the resolution of the third image is greater than that of the first original image and performing resolution repair according to the determination result to obtain a target image includes: in response to the resolution of the third image being greater than that of the first original image, scaling the third image based on the scaling factor to obtain a target image; or, in response to the resolution of the third image being equal to that of the first original image, using the third image as a target image; in response to the resolution of the third image being less than that of the first original image, performing resolution repair on the third image to obtain a target image.

[0098] Specifically, if it is determined that the resolution of the third image is greater than that of the first original image, it means that the resolution of the image after the scene in this embodiment is greater than that required by the user, so the present embodiment scales the third image by the scaling factor calculated above to obtain the image required by the user, that is, to obtain the target image. If it is determined that the resolution of the third image is equal to that of the first original image, it means that the resolution of the image after the scene in this embodiment is exactly what the user requires, that is, it means that the resolution of the second original image and the first original image in the above text are the same, so the present embodiment directly uses the third image as the target image. If it is determined that the resolution of the third image is less than that of the first original image, it means that the resolution of the image after the scene in this embodiment is less than that of the image originally uploaded by the user, so the present embodiment needs to perform resolution repair on the third image to obtain the target image, so that the resolution of the target image is the same as the resolution of the first original image originally uploaded by the user.

[0099] In the embodiment of the present disclosure, the resolution restoration of the third image to obtain the target image includes: enlarging the third image based on a bilinear interpolation algorithm to obtain a fourth image; processing the fourth image based on a super-resolution algorithm to obtain a fifth image; enlarging the first target object mask image based on a scaling factor to obtain an enlarged first target object mask image; performing a matting on the first original image based on the enlarged first target object mask image to obtain an original target object image; and pasting the original target object image to the fifth image to obtain a target image.

[0100] Among them, the bilinear difference algorithm and the super-resolution algorithm can be algorithms within the diffusion model or algorithms independent of the diffusion model. Super-resolution algorithms such as the SUPIR algorithm can generate high-quality high-definition images based on the diffusion model.

[0101] Specifically, this embodiment enlarges the third image by a bilinear difference algorithm, and enlarges the size of the third image to the same as the first original image to obtain a fourth image. Although the size of the fourth image is the same as the size of the first original image, the resolution will be low, and the image will appear blurry, so this embodiment will perform high-resolution restoration on the fourth image based on the super-resolution algorithm Supir algorithm to obtain a fifth image. The restored fifth image has higher clarity and details.

[0102] However, since the present embodiment replaces the background of the target object, the diffusion model also generates a background image, and the target object image is not changed, so the target object image in the fourth image is already a high-definition image. Therefore, if the target object image area is restored with high resolution, it may have an adverse effect on the target object image. Therefore, the present embodiment needs to use the target object image area originally in the first original image to replace the part of the image in the fifth image.

[0103] Specifically, this embodiment uses the first target object mask image reduced to a preset size, and enlarges the first target object mask image based on a scaling factor to obtain an enlarged first target object mask image; the first original image is cut out based on the enlarged first target object mask image to obtain the original target object image; the original target object image is pasted to the same position in the fifth image to obtain the original high-definition target object image, while also having the high-definition generated background image required by the user, that is, the target image.

[0104] Since most of the existing image background replacement methods are based on traditional image processing techniques, such as simple image stitching, mask synthesis, etc., these methods usually produce unnatural edges and lack of realism. Although AI algorithms can improve image quality to a certain extent, the processing results are often limited to low resolution and cannot effectively avoid the sense of disconnection between characters and backgrounds. In particular, for the restoration of complex clothing images and details, traditional methods are difficult to meet actual needs.

[0105] However, the method provided in this embodiment can solve the above problems and can replace the background efficiently and naturally. At the same time, it can process high-resolution images and restore character details, especially clothing details and hair edges, thereby achieving high-quality image background replacement and detail restoration.

[0106] Figure 2 This is a flowchart of an exemplary image processing method provided by an embodiment of the present disclosure. The method can be executed by an image processing device provided by an embodiment of the present disclosure, and the device can be implemented in software and / or hardware. The method specifically includes the following main steps:

[0107] Step 1: Image scaling: Scale the original image proportionally to ensure that the image does not exceed the target resolution (1024x1536). Calculate the scaling factor to obtain the optimal scaling ratio to avoid image distortion.

[0108] Step 2: Portrait cutout and background mask generation: Use SAM (Segment Anything Model) to cut out the portrait of the image and generate a preliminary mask. Then, further refine the mask area by reducing it to enhance the background replacement effect.

[0109] Step 3: Gaussian blur and edge fusion: Gaussian blur the mask edge to weaken the edge features, improve the image background fusion, and avoid boundary effects in the generated image.

[0110] Step 4: Depth map control and background generation by redrawing the model: The depth information of the original background is processed through the depth control network, and the perspective relationship of the image is controlled in the process of redrawing the diffusion model to generate the image, ensuring that the background generation conforms to the physical laws of the real world.

[0111] Step 5: Edge repair and detail restoration: Accurately repair the details of characters and clothing, especially the hair and edges of clothing, to ensure that the image is natural and seamless.

[0112] Step 6: Super-resolution restoration: Use the SUPIR algorithm for high-definition restoration to improve the clarity and details of the image, ensuring that the final output image reaches high resolution and excellent quality.

[0113] The present invention successfully solves the shortcomings of traditional background replacement methods by combining the AIGC diffusion model, the deep control network and the high-resolution restoration algorithm, and provides an efficient and natural clothing model background replacement solution that can process high-resolution images and restore details, especially the natural fusion of clothing, character edges and background.

[0114] Figure 3 : is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure, and the device specifically includes:

[0115] A first acquisition module 310, configured to acquire a first original image, wherein the first original image includes an image of a target object;

[0116] A second acquisition module 320, configured to scale the first original image to a second original image based on a scaling factor;

[0117] A third acquisition module 330 is used to acquire a first target object mask image based on the second original image;

[0118] A background generation module 340, configured to generate a background image other than the second target object mask image according to the first target object mask image and a diffusion model, to obtain a first image;

[0119] An edge generation module 350 is used to repair the edge image of the target object in the first image based on the edge mask image of the first target object mask image obtained to obtain a second image;

[0120] A detail restoration module 360 ​​is used to restore the detail image of the target object in the second original image based on the detail mask image obtained in the second original image to obtain a third image;

[0121] The resolution restoration module 370 is used to determine whether the resolution of the third image is greater than that of the first original image, and perform resolution restoration according to the determination result to obtain a target image.

[0122] In one possible implementation, the third acquisition module 330 is specifically configured to:

[0123] A mask image of the target object in the second original image is obtained as an initial target object mask image; and an edge of the initial target object mask image is reduced according to a preset size to obtain a first target object mask image.

[0124] In one embodiment, the diffusion model includes a depth preprocessor and a depth control network, and the background generation module 340 is specifically used to:

[0125] The first target object mask image is subjected to Gaussian processing to obtain a second target object mask image; the depth information of the first original image is obtained through a depth preprocessor; based on the depth control network, the depth information and the second target object mask image, a background image area outside the second target object mask image is preliminarily generated to obtain a first intermediate image; based on the first intermediate image and a diffusion model, the background image area outside the second target object mask image is redrawn to obtain a first image.

[0126] In one possible implementation, the edge generation module 350 is specifically configured to:

[0127] Acquire an edge mask image of the first target object mask image; based on the edge mask image, acquire an edge image in the second original image; paste the edge image to the first image to obtain a second intermediate image; based on the edge mask image and a diffusion model, perform detail restoration on an edge image region in the second intermediate image to obtain a second image.

[0128] In one possible implementation, the detail restoration module 360 ​​is specifically configured to:

[0129] Based on the image segmentation model, the detail image in the second original image is processed to obtain a detail mask image; based on the detail mask image and the diffusion model, detail restoration is performed on the detail area in the second image to obtain a third image.

[0130] In one possible implementation, the resolution restoration module 370 is specifically used to:

[0131] In response to the resolution of the third image being greater than that of the first original image, the third image is scaled based on the scaling factor to obtain a target image; or, in response to the resolution of the third image being equal to that of the first original image, the third image is used as the target image; in response to the resolution of the third image being less than that of the first original image, the resolution of the third image is repaired to obtain a target image.

[0132] In one possible implementation, the resolution restoration module 370 is specifically used to:

[0133] The third image is amplified based on a bilinear interpolation algorithm to obtain a fourth image; the fourth image is processed based on a super-resolution algorithm to obtain a fifth image; the first target mask image is amplified based on a scaling factor to obtain an amplified first target mask image; the first original image is cut out based on the amplified first target mask image to obtain an original target image; and the original target image is pasted to the fifth image to obtain a target image.

[0134] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0135] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0136] like Figure 4As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0137] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0138] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the sound effect correction method in any other appropriate manner (e.g., by means of firmware).

[0139] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0141] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0144] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0146] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0147] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first original image, wherein the first original image includes an image of a target object; Based on the scaling factor, scaling the first original image to a second original image; Based on the second original image, obtaining a first target object mask image; Generate a background image other than the second target object mask image according to the first target object mask image and the diffusion model to obtain a first image; Based on the acquired edge mask image of the first target object mask image, repairing the edge image of the target object in the first image to obtain a second image; Based on the acquired detail mask image in the second original image, repairing the detail image of the target object in the second image to obtain a third image; It is determined whether the resolution of the third image is greater than that of the first original image, and resolution restoration is performed according to the determination result to obtain a target image.

2. The method according to claim 1, characterized in that The step of acquiring a first target object mask image based on the second original image includes: Acquire a mask image of the target object in the second original image as an initial target object mask image; According to a preset size, the edge of the initial target object mask image is reduced to obtain a first target object mask image.

3. The method according to claim 1, wherein: The diffusion model includes a depth preprocessor and a depth control network. The process of generating a background image other than the second target object mask image according to the first target object mask image and the diffusion model to obtain a first image includes: Performing Gaussian processing on the first target object mask image to obtain a second target object mask image; Acquire depth information of the first original image through a depth preprocessor; Based on the depth control network, the depth information and the second target object mask image, preliminarily generate a background image area outside the second target object mask image to obtain a first intermediate image; Based on the first intermediate image and the diffusion model, the background image area outside the second target object mask image is redrawn to obtain a first image.

4. The method according to claim 1, characterized in that: The step of obtaining a second image based on the edge mask image of the first object mask image comprises: Acquire an edge mask image of the first target object mask image; Based on the edge mask image, acquiring an edge image in the second original image; Pasting the edge image onto the first image to obtain a second intermediate image; Based on the edge mask image and the diffusion model, detail restoration is performed on the edge image region in the second intermediate image to obtain a second image.

5. The method according to claim 1, characterized in that The method of repairing the detail image of the target object in the second image based on the detail mask image in the acquired second original image to obtain the third image includes: Based on the image segmentation model, the detail image in the second original image is processed to obtain a detail mask image; Based on the detail mask image and the diffusion model, detail restoration is performed on the detail area in the second image to obtain a third image.

6. The method according to claim 1, characterized in that The determining whether the resolution of the third image is greater than that of the first original image, and performing resolution restoration according to the determination result to obtain the target image, comprises: In response to the resolution of the third image being greater than that of the first original image, scaling the third image based on the scaling factor to obtain a target image; or, In response to the resolution of the third image being equal to the first original image, taking the third image as a target image; In response to the fact that the resolution of the third image is smaller than that of the first original image, resolution restoration is performed on the third image to obtain a target image.

7. The method according to claim 6, characterized in that The step of performing resolution restoration on the third image to obtain a target image includes: amplifying the third image based on a bilinear difference algorithm to obtain a fourth image; Processing the fourth image based on a super-resolution algorithm to obtain a fifth image; Amplifying the first target object mask image based on the scaling factor to obtain an amplified first target object mask image; Performing matting on the first original image based on the enlarged first target object mask image to obtain an original target object image; The original target object image is pasted onto the fifth image to obtain a target image.

8. An image processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a first original image, wherein the first original image includes an image of a target object; A second acquisition module, configured to scale the first original image to a second original image based on a scaling factor; A third acquisition module, configured to acquire a first target object mask image based on the second original image; A background generation module, configured to generate a background image other than the second target object mask image according to the first target object mask image and a diffusion model, so as to obtain a first image; an edge generation module, configured to repair an edge image of the target object in the first image based on the edge mask image of the first target object mask image obtained, so as to obtain a second image; A detail restoration module, configured to restore a detail image of a target object in the second original image based on the detail mask image acquired in the second original image, so as to obtain a third image; The resolution restoration module is used to determine whether the resolution of the third image is greater than that of the first original image, and perform resolution restoration according to the determination result to obtain a target image.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images