Image processing method, device, apparatus and storage medium

Abstract

The present disclosure relates to an image processing method, device, equipment and storage medium. The method comprises: obtaining a first face image to be processed, the first face image comprising a defect area and a non-defect area; performing defect removal processing on the first face image by using an image generation model to obtain a second face image; performing image segmentation processing on the first face image by using an image segmentation model to obtain a defect segmentation image; determining a weight value corresponding to each pixel point in the first face image and the second face image according to the defect segmentation image; and performing weighted fusion processing on the second face image and the first face image according to the weight value corresponding to each pixel point to generate a third face image. Based on the above method, the skin texture of the original image can be preserved, and the problems of image blurring and loss of realism can be avoided.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Technology

[0002] In related technologies, image processing software is generally used to process human images. In addition to removing spots, blemishes, or discoloration from the skin of people in the image, image processing software can also smooth the skin of the face, making the face more delicate and the contours clearer.

[0003] However, traditional image processing methods inevitably lose skin texture, resulting in blurry images and a loss of realism. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, device, and storage medium to at least solve the problems of blurring and loss of realism that occur when processing images in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0006] Obtain a first face image to be processed, the first face image including blemish areas and non-blemish areas;

[0007] The first face image is processed by an image generation model to remove blemishes, so as to obtain a second face image;

[0008] The first face image is segmented using an image segmentation model to obtain a defect segmentation image. The value of each pixel in the defect segmentation image represents the predicted defect level of the corresponding pixel in the first face image.

[0009] The weight value corresponding to each pixel in the first face image and the second face image is determined based on the defect segmentation image, wherein the larger the predicted defect level value corresponding to the pixel in the first face image, the larger the weight value corresponding to the pixel.

[0010] The second face image and the first face image are subjected to weighted fusion processing based on the weight value corresponding to each pixel to generate a third face image.

[0011] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0012] The acquisition module is used to acquire a first face image to be processed, wherein the first face image includes blemish areas and non-blemish areas;

[0013] The processing module is used to perform blemish removal processing on the first face image through an image generation model to obtain a second face image; and to perform image segmentation processing on the first face image through an image segmentation model to obtain a blemish segmentation image, wherein the value of each pixel in the blemish segmentation image represents the predicted blemish degree value corresponding to the corresponding pixel in the first face image.

[0014] The determination module is used to determine the weight value corresponding to each pixel in the first face image and the second face image based on the defect segmentation image, wherein the larger the predicted defect level value corresponding to the pixel in the first face image, the larger the weight value corresponding to the pixel.

[0015] The processing module is further configured to perform weighted fusion processing on the second face image and the first face image according to the weight value corresponding to each pixel, so as to generate a third face image.

[0016] According to a third aspect of the present disclosure, an image processing apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image processing method as described in the first aspect.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the computer-readable storage medium are executed by a processor of an image processing apparatus, the image processing apparatus is able to perform the image processing method as described in the first aspect.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method described in the first aspect.

[0019] According to a sixth aspect of the present disclosure, a model training method is provided, comprising:

[0020] Multiple training sample pairs are obtained, wherein each training sample pair includes an initial face sample image and a target face sample image, wherein the target face sample image is an image after the initial face sample image has been de-defected;

[0021] Determine the difference image between the initial face sample image and the target face sample image;

[0022] Based on the difference image and the set pixel value segmentation threshold, a binarized defect segmentation sample image corresponding to the corresponding training sample pair is generated;

[0023] Based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs, train the image generation model and the image segmentation model;

[0024] The step of training the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs includes:

[0025] Keeping the parameters of the first image generation model unchanged, the first image segmentation model is trained using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, so as to generate the second image segmentation model.

[0026] Using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, the first image generation model and the second image segmentation model are trained simultaneously to generate the second image generation model and the third image segmentation model. The parameter update frequency of the image generation model is higher than that of the image segmentation model.

[0027] Keeping the parameters of the second image generation model unchanged, the third image segmentation model is trained using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate the fourth image segmentation model;

[0028] Using the fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs, the second image generation model and the fourth image segmentation model are trained simultaneously to generate the trained image generation model and the image segmentation model. The parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

[0029] According to a seventh aspect of the present disclosure, a model training apparatus is provided, comprising:

[0030] The acquisition module is used to acquire multiple training sample pairs, wherein each training sample pair includes an initial face sample image and a target face sample image, wherein the target face sample image is an image after the initial face sample image has been de-defected;

[0031] The determination module is used to determine the difference image between the initial face sample image and the target face sample image;

[0032] The generation module is used to generate a binarized defect segmentation sample image corresponding to the corresponding training sample pair based on the difference image and the set pixel value segmentation threshold.

[0033] The training module is used to train the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs.

[0034] The step of training the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs includes:

[0035] Keeping the parameters of the first image generation model unchanged, the first image segmentation model is trained using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, so as to generate the second image segmentation model.

[0036] Using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, the first image generation model and the second image segmentation model are trained simultaneously to generate the second image generation model and the third image segmentation model. The parameter update frequency of the image generation model is higher than that of the image segmentation model.

[0037] Keeping the parameters of the second image generation model unchanged, the third image segmentation model is trained using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate the fourth image segmentation model;

[0038] Using the fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs, the second image generation model and the fourth image segmentation model are trained simultaneously to generate the trained image generation model and the image segmentation model. The parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

[0039] According to an eighth aspect of the present disclosure, a model training apparatus is provided, comprising:

[0040] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the model training method as described in the fifth aspect.

[0041] According to a ninth aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the computer-readable storage medium are executed by a processor of a model training apparatus, the model training apparatus is able to perform the model training method as described in the fifth aspect.

[0042] According to a tenth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the model training method described in the fifth aspect.

[0043] The technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: A second face image is obtained by removing blemishes from a first face image containing blemish and non-blemish regions using an image generation model. Simultaneously, the degree of blemishes corresponding to each pixel in the first face image is predicted using an image segmentation model to generate a blemish segmentation image that represents the predicted blemish degree values ​​corresponding to each pixel in the first face image. Thus, the blemish and non-blemish regions in the first face image can be distinguished using the blemish segmentation image, and the degree of blemishes in the blemish regions can be predicted. Finally, the weight value corresponding to each pixel in the first and second images can be determined using the blemish segmentation image. Based on the weight values ​​of each pixel in the first and second images, a weighted fusion process can be performed on the first and second images to generate a processed third face image. Based on image segmentation and image generation models, the first image is processed to obtain the second image. On the other hand, the flawed and non-flawed areas of the first image are distinguished, and the degree of flaw of each pixel is predicted to determine the weight value of each pixel in the first and second images. Finally, the first and second images are fused based on the weight values ​​of each pixel to obtain the processed image. While processing the flawed areas, the skin texture of the original image can be preserved, avoiding the problems of image blurring and loss of realism.

[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0046] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0047] Figure 2 This is a schematic diagram illustrating an image processing method according to an exemplary embodiment.

[0048] Figure 3a This is a schematic diagram of a first human face image according to an exemplary embodiment.

[0049] Figure 3b This is a schematic diagram illustrating a third face image according to an exemplary embodiment.

[0050] Figure 4 This is a schematic diagram illustrating the structure of an image processing model according to an exemplary embodiment.

[0051] Figure 5 This is a flowchart illustrating a training method for an image processing model according to an exemplary embodiment.

[0052] Figure 6 This is a schematic diagram illustrating a defect segmentation sample image according to an exemplary embodiment.

[0053] Figure 7 This is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment.

[0054] Figure 8 This is a schematic diagram of the structure of an image processing device according to an exemplary embodiment.

[0055] Figure 9 This is a schematic diagram of the structure of a model training device according to an exemplary embodiment.

[0056] Figure 10 This is a schematic diagram of the structure of a model training device according to an exemplary embodiment. Detailed Implementation

[0057] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0058] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0059] In portrait retouching, traditional skin smoothing methods inevitably lose skin texture. While they can even out skin tone, they also blur the image, failing to preserve skin texture and resulting in a loss of realism. To make processed images more in line with people's needs, portrait retouching has gradually transitioned from traditional methods to deep learning methods. For example, image processing is performed using networks like U-Net. However, U-Net's approach to image processing leans towards coarse-grained generation, such as changing image style, gender, and facial features. It cannot accurately reproduce and generate fine-grained features, such as skin texture.

[0060] To address the aforementioned issues, the core idea of ​​the image processing method provided in this disclosure is as follows: First, a first face image to be processed is acquired, comprising blemish and non-blemish regions; blemish removal is performed on the first face image using an image generation model to obtain a second face image; blemish segmentation is performed on the first face image using an image segmentation model to obtain a blemish segmentation image, where the value of each pixel in the blemish segmentation image represents the predicted blemish level corresponding to the corresponding pixel in the first face image; weight values ​​are determined for each pixel in the first and second face images based on the blemish segmentation image, wherein a larger predicted blemish level value corresponds to a pixel in the first face image, the larger the weight value corresponding to that pixel; weighted fusion processing is performed on the second and first face images based on the weight values ​​corresponding to each pixel to generate a third face image. Based on image segmentation and image generation models, the first image is processed to obtain the second image. On the other hand, the flawed and non-flawed areas of the first image are distinguished, and the degree of flaw of each pixel is predicted to determine the weight value of each pixel in the first and second images. Finally, the first and second images are fused based on the weight values ​​of each pixel to obtain the processed image. While processing the flawed areas, the skin texture of the original image can be preserved, avoiding the problems of image blurring and loss of realism.

[0061] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 1 As shown, the image processing method includes the following steps.

[0062] S101, Obtain the first face image to be processed.

[0063] S102, the first face image is processed to remove defects using an image generation model to obtain the second face image.

[0064] S103, the first face image is segmented using an image segmentation model to obtain a defect segmentation image.

[0065] S104, determine the weight value corresponding to each pixel in the first face image and the second face image based on the defect segmentation image.

[0066] S105, perform weighted fusion processing on the second face image and the first face image according to the weight value corresponding to each pixel to generate a third face image.

[0067] First, a first face image to be processed is acquired. In this embodiment, the first face image may include an image with complete facial features or an image with partial facial features. The first face image can be acquired by a processing terminal performing the image processing method, or it can be acquired from other electronic devices.

[0068] Generally, a first facial image contains areas with varying degrees of blemishes and areas without blemishes. In this embodiment, optionally, blemish areas may include skin areas with blemishes such as acne marks, moles, age spots, scars, and uneven skin tone. Conversely, non-blemish areas may include areas of normal skin. It should be noted that the skin tone of blemish areas and non-blemish areas can have significant differences or subtle differences.

[0069] Then, the first face image is processed using an image generation model to remove blemishes, resulting in a second face image. To make the image more aesthetically pleasing, the blemish areas in the first face image need to be processed. In this embodiment, the blemish removal process may optionally include image processing techniques such as skin smoothing and color correction. After blemish removal of the first face image, the skin tone in the blemish areas and the non-blemish areas becomes more uniform. For example, taking the blemish area in the first face image as a pigmented area, by performing skin smoothing on the pigmented area, the skin tone in that pigmented area can be lightened, making the skin tone in that pigmented area more uniform with the skin tone in the adjacent areas.

[0070] It should be noted that since the image generation model is only used to remove blemishes from the first face image, the skin tone in the blemish areas of the second face image is uniform with the skin tone in the non-blemish areas. However, after the blemish removal process described above, the image texture in the second face image cannot be preserved, resulting in a blurred image and a loss of realism. For example, after smoothing the skin in the blemish areas of the first face image, the processed areas in the resulting second face image will lack texture features.

[0071] To address the issue of image realism loss, while performing blemish removal on the first face image, an image segmentation model can also be used to segment the first face image, resulting in a blemish segmentation image representing the degree of blemishes at each pixel in the first face image. Specifically, the value of each pixel in the blemish segmentation image represents the predicted degree of blemishes corresponding to that pixel in the first face image.

[0072] In practical applications, optionally, the aforementioned defect segmentation image differs from the traditional binary mask image; that is, the defect segmentation image can be a fine-grained defect segmentation image. Specifically, each pixel in the defect segmentation image can take values ​​within a preset range, which can be set based on specific needs. For example, the preset range can include 0-1, 0-10, etc. Specifically, taking a preset range of 0-1 as an example, if the value of a pixel in the defect segmentation image is closer to 0, it indicates a lower degree of defect for that pixel; conversely, if the value of a pixel in the defect segmentation image is closer to 1, it indicates a higher degree of defect for that pixel.

[0073] Based on the predicted blemish severity value of each pixel in the blemish segmentation image, the weight value corresponding to each pixel in the first and second face images can be determined. Specifically, the larger the predicted blemish severity value of a pixel in the first face image, the larger the weight value of that pixel; conversely, the smaller the predicted blemish severity value of a pixel in the first face image, the smaller the weight value of that pixel.

[0074] Finally, the second face image and the first face image can be weighted and fused according to the weight value corresponding to each pixel to generate the third face image.

[0075] In this embodiment, the predicted defect level of each pixel in the defect segmentation image can characterize the defect level of each pixel in the first face image. Based on the defect level of each pixel, the weight value of each pixel in the first face image and the second face image is determined during fusion. Furthermore, weighted fusion processing can be performed based on the weight values ​​corresponding to each pixel in the first and second face images. For example, for a certain pixel, if the weight value of that pixel in the first face image is greater than its weight value in the second face image, then the information in the first face image is emphasized during image fusion; conversely, if the weight value of that pixel in the second face image is greater than its weight value in the first face image, then the information in the second face image is emphasized during image fusion.

[0076] Based on image segmentation and image generation models, the first image is processed to obtain the second image. On the other hand, the flawed and non-flawed areas of the first image are distinguished, and the degree of flaw of each pixel is predicted to determine the weight value of each pixel in the first and second images. Finally, the first and second images are fused based on the weight values ​​of each pixel to obtain the processed image. This method can preserve the skin texture of the original image while processing flawed areas, avoiding image blurring and loss of realism.

[0077] To facilitate understanding, the following will be combined with... Figure 2, Figure 3a , Figure 3b The specific embodiments further illustrate the image processing method provided in this disclosure.

[0078] like Figure 2 As shown, the first face image is used as... Figure 3a Taking the face image shown as an example, this image processing method specifically includes the following steps:

[0079] First, obtain the first face image, including both flawed and non-flawed areas.

[0080] like Figure 3a As shown, the facial image includes multiple blemishes such as acne scars and moles. Correspondingly, in this embodiment, areas other than the aforementioned blemishes such as acne scars and uneven skin tone are considered non-blemish areas.

[0081] The first face image is input into the image generation model. After the image generation model removes blemishes from the first face image, the second face image is obtained.

[0082] Because the first face image contains blemishes such as acne scars and moles, the skin tone in these blemish areas differs significantly from that in the non-blemish areas. Therefore, blemish removal processing is needed to make the skin tone more uniform across the blemish and non-blemish areas of the first face image. Specifically, after the image generation model obtains the first face image, it can perform skin smoothing and color correction to adjust the skin tone in the blemish areas, thus obtaining the second face image. In the second face image, the skin tone in the areas corresponding to blemishes is more uniform compared to the skin in the areas corresponding to non-blemishes.

[0083] As mentioned above, after the image generation model processes the first face image, the generated second face image usually fails to retain the image texture. For example, when the image generation model smooths the acne scar area in the first face image, it causes the skin area to lose texture features, resulting in a blurred image.

[0084] To improve image processing quality, skin color and texture features in the first face image can be separated and processed separately. Specifically, the first face image is input into an image segmentation model, which processes the image to obtain a blemish segmentation image. In this embodiment, the value of each pixel in the blemish segmentation image represents the predicted blemish severity value corresponding to the corresponding pixel in the first face image.

[0085] The first and second face images are weighted and fused by predicting the degree of defects of each pixel in the defect segmentation image.

[0086] Specifically, firstly, the weight values ​​of the first and second face images during fusion can be determined based on the predicted defect levels of each pixel in the defect segmentation image. Then, the second and first face images are subjected to pixel-weighted fusion processing based on the weight values ​​corresponding to each pixel to generate the third face image.

[0087] In this embodiment, optionally, the weight value of each pixel in the first face image can be determined based on the difference between a preset value and the predicted defect level of the corresponding pixel in the defect segmentation image, and the weight value of each pixel in the second face image can be determined based on the predicted defect level of the corresponding pixel in the defect segmentation image. The preset value may include the maximum value of the predicted defect level corresponding to the pixel in the defect segmentation image; that is, the preset value can determine the preset range of the predicted defect level of the pixel in the defect segmentation image. For example, if the preset value is 1, then the preset range of the corresponding predicted defect level is 0 to 1.

[0088] In the above embodiment, taking the preset value of the defect segmentation image as 1 and the preset value range as 0 to 1 as an example, based on the predicted value of the defect segmentation image, if the value of a certain pixel tends to 0, it indicates that the defect level of the pixel is lower, and in the fusion process, the information in the original face image can be used more. If the value of a certain pixel tends to 1, it indicates that the defect level of the pixel is higher, and in the fusion process, the information in the second face image after the defect removal process can be used more.

[0089] In practical applications, when performing weighted fusion processing on the second and first face images, the value of each pixel in the third face image can be generated according to the following formula (1):

[0090] output=tmp×mask+input*(1-mask) (1)

[0091] Where output represents the pixel value in the third face image, tmp represents the pixel value in the second face image, input represents the pixel value in the first face image, and mask represents the predicted defect level value corresponding to the pixel in the defect segmentation image.

[0092] Figure 3b This is a schematic diagram illustrating a third face image according to an exemplary embodiment. For example... Figure 3b As shown, it can be seen that by analyzing... Figure 3a The third face image obtained after processing the first face image shown can retain the texture features of the skin, thus preserving the image quality.

[0093] The image processing method provided in the above embodiments requires processing the first face image using an image generation model and an image segmentation model. The following section combines... Figure 4 The image generation model and image segmentation model provided in the embodiments of this disclosure are illustrated by way of example.

[0094] like Figure 4 As shown, the image generation model includes multiple first feature extraction layers cascaded from top to bottom, multiple first feature fusion layers cascaded from bottom to top, and a first convolutional output layer. In practical applications, the first feature extraction layer and the first feature fusion layer, corresponding to feature maps of the same scale, are connected.

[0095] Accordingly, the image segmentation model includes multiple cascaded second feature extraction layers from top to bottom, multiple cascaded second feature fusion layers from bottom to top, and a second convolutional output layer. In practical applications, the second feature extraction layers and the second feature fusion layers corresponding to feature maps of the same scale are connected.

[0096] For the image generation model, when processing the first face image to generate the second face image, firstly, the first face image is subjected to feature extraction and downsampling processing through multiple first feature extraction layers to obtain feature maps at multiple scales. In practical applications, the feature map extracted by the previous first feature extraction layer is downsampled and used as the input to the next first feature extraction layer, and the feature map extracted by the last first feature extraction layer is downsampled and input to the first first feature fusion layer.

[0097] Then, multiple first feature fusion layers are used to upsample and fuse feature maps at multiple scales to obtain the final fused target feature map. In practical applications, the previous first feature fusion layer upsamples the input feature map and then fuses it with feature maps of the same scale received from multiple first feature extraction layers. The feature fusion result is used as the input to the next first feature fusion layer.

[0098] Finally, the target feature map is input into the first convolutional output layer to obtain the second face image output by the first convolutional output layer.

[0099] For image segmentation models, when processing the first face image and generating a defect segmentation image, the first step involves multiple second feature extraction layers to extract and downsample features from the first face image, resulting in feature maps at multiple scales. In practical applications, the feature map extracted by the previous second feature extraction layer is downsampled and used as the input to the next second feature extraction layer, and the feature map extracted by the last second feature extraction layer is downsampled and input to the first second feature fusion layer.

[0100] Then, multiple second feature fusion layers are used to upsample and fuse feature maps at multiple scales to obtain the final fused target feature map. In practical applications, the previous second feature fusion layer upsamples the input feature map and then fuses it with feature maps of the same scale received from multiple second feature extraction layers. The feature fusion result is used as the input to the next second feature fusion layer.

[0101] Finally, the target feature map is input into the second convolutional output layer to obtain the defect segmentation image output by the second convolutional output layer.

[0102] After obtaining the second face image and the blemish segmentation image, the blemish segmentation image is input into the image generation model. In this way, the image generation model can fuse the first and second face images based on the blemish segmentation image to generate the processed third face image.

[0103] In the above embodiments, optionally, a preset number of feature extraction layers of multiple first feature extraction layers are shared with multiple second feature extraction layers. For example, such as Figure 4 As shown, the first two layers of multiple first feature extraction layers are shared with the first two layers of multiple second feature extraction layers. This means that the image generation model and the image segmentation model can share a preset number of feature extraction layers. It should be noted that the preset number can include one or more layers. Furthermore, depending on specific implementation needs, the image generation model and the image segmentation model can also use different feature extraction layers.

[0104] Before processing the first face image using the image generation and segmentation models, the image generation and segmentation models need to be trained. The following section combines... Figure 5 and Figure 6 The training methods for image generation and image segmentation models are explained.

[0105] Figure 5 This is a flowchart illustrating a training method for an image processing model according to an exemplary embodiment, such as... Figure 5 As shown, the method includes the following steps.

[0106] S501, obtain multiple training sample pairs.

[0107] S502, determine the difference image between the initial face sample image and the target face sample image.

[0108] S503: Based on the difference image and the set pixel value segmentation threshold, generate a binarized defect segmentation sample image corresponding to the corresponding training sample pair.

[0109] S504: Train the image generation model and the image segmentation model based on multiple training sample pairs and the defect segmentation sample images corresponding to each of the multiple training sample pairs.

[0110] Before training the image generation model and the image segmentation model, training sample images must first be obtained. Each training sample pair includes an initial face sample image and a target face sample image. The target face sample image is the image after the initial face sample image has been de-defected.

[0111] In this embodiment, the training samples may include a training sample pair of initial face sample images and target face sample images. The target face sample image is an image obtained by removing blemishes from the initial face sample image, where the blemishes and non-blemishes in the target face sample image have matching texture features. Optionally, the target face sample image may be an image obtained by processing the initial face sample image using image processing software such as Adobe Photoshop.

[0112] In practical applications, to increase the number of training pairs, image enhancement processing can be applied to the sample images contained in each training pair. For example, the initial face sample image and the target face sample image in the training pair can be processed simultaneously by random cropping, random scaling, random flipping, etc., to increase the richness of the training samples.

[0113] Then, the difference image between the initial face sample image and the target face sample image is determined. In this embodiment, the absolute value of the difference between each pixel in the face sample image and the target face sample image can be calculated, and then the difference image is generated based on the absolute value of the difference between each pixel.

[0114] After generating the difference image, a binarized defect segmentation sample image corresponding to the corresponding training sample pair can be generated based on the difference image and the set pixel value segmentation threshold.

[0115] The image processing methods described above involve blemish segmentation images that indicate the severity of blemishes. However, fine-grained blemish segmentation images cannot be directly obtained before model training. Therefore, binarized blemish segmentation sample images can be determined using initial face sample images and target face sample images, so that image generation and image segmentation models can be trained based on these binarized blemish segmentation sample images.

[0116] For ease of understanding, combined with Figure 6 The meaning of the defect segmentation sample image is illustrated by an example. For example... Figure 6 As shown, with Figure 6Taking the initial face sample image and target face sample image shown as examples, the target face image is obtained by smoothing the skin of the blemish areas in the initial face sample image. Correspondingly, in the blemish segmentation image, the white pixel area (RGB = (1,1,1)) corresponds to the blemish area, while the remaining black pixel area (RGB = (0,0,0)) corresponds to the non-blemish area in the face image. The generation method of the blemish segmentation image can refer to existing related technologies and will not be elaborated here. It should be noted that the size of the binarized blemish segmentation sample image is the same as the size of the first face sample image.

[0117] Finally, after obtaining multiple training sample pairs and their corresponding defect segmentation sample images, the image generation model and image segmentation model are trained.

[0118] In practical applications, to ensure the training effectiveness of the image generation and segmentation models, they can be trained thoroughly in multiple stages. Specifically:

[0119] Keeping the parameters of the first image generation model unchanged, the first image segmentation model is trained using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, so as to generate the second image segmentation model.

[0120] Since only binary defect segmentation sample images are available before training, to enable the image segmentation model to output more fine-grained defect segmentation images, the parameters of the image generation model need to be kept constant, and the image segmentation model needs to be trained to a certain extent first. In practical applications, a loss function is used to evaluate the similarity between the defect segmentation image output by the image segmentation model and the defect segmentation sample image. The first stage of training stops when the loss function converges or when all samples are used up.

[0121] Using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, the first image generation model and the second image segmentation model are trained simultaneously to generate the second image generation model and the third image segmentation model. The parameter update frequency of the image generation model is higher than that of the image segmentation model.

[0122] After the initial training of the image segmentation model is completed, the image generation model can be trained. In practical applications, the similarity between the third-party face image output by the image generation model and the target face sample image in the training sample pair can be evaluated using a loss function. The second stage of training stops when the loss function converges or when all samples are used up.

[0123] In practical applications, to further improve the fine-grainedness of the output images from the image segmentation model, the image segmentation model can be trained simultaneously during the second training phase. Specifically, the parameter update frequency of the image segmentation model can be lower than that of the image generation model. For example, the parameters of the image generation model can be updated once every four updates.

[0124] After the initial training of the image generation model is completed, the image segmentation model can be further trained. Specifically, keeping the parameters of the second image generation model unchanged, the third image segmentation model is trained using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs, in order to generate the fourth image segmentation model.

[0125] It should be noted that during the training phase described above, the target face sample image can be updated based on the parameters of the current image generation model. Correspondingly, the defect segmentation sample image can be updated based on the updated target face sample image.

[0126] By training the image segmentation model using the input and output of the image generation model under the current parameters, the finer granularity of the output image of the image segmentation model can be further increased.

[0127] After the image segmentation model is retrained, both the image generation model and the image segmentation model can be further trained simultaneously to improve the image processing model's image processing capabilities. Specifically, using the fourth set of training sample pairs and the corresponding defect segmentation sample images, the second image generation model and the fourth image segmentation model are trained simultaneously to generate the trained image generation model and image segmentation model. The parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

[0128] Figure 7 This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment. Figure 7 As shown, the image processing device 700 includes an acquisition module 701, a processing module 702, a determination module 703, and a training module 704.

[0129] The acquisition module 701 is used to acquire a first face image to be processed, which includes blemish areas and non-blemish areas.

[0130] The processing module 702 is used to perform blemish removal processing on the first face image through an image generation model to obtain a second face image; and to perform image segmentation processing on the first face image through an image segmentation model to obtain a blemish segmentation image, wherein the value of each pixel in the blemish segmentation image represents the predicted blemish degree value corresponding to the corresponding pixel in the first face image.

[0131] The determination module 703 is used to determine the weight value corresponding to each pixel in the first face image and the second face image based on the defect segmentation image, wherein the larger the defect degree prediction value corresponding to the pixel in the first face image, the larger the weight value corresponding to the pixel.

[0132] The processing module 702 is further configured to perform weighted fusion processing on the second face image and the first face image according to the weight value corresponding to each pixel, so as to generate a third face image.

[0133] According to an embodiment of this disclosure, the determining module 703 is further configured to determine the weight value of each pixel in the first face image based on the difference between a preset value and the predicted value of the degree of defect of the corresponding pixel in the defect segmentation image; and to determine the weight value of each pixel in the second face image based on the predicted value of the degree of defect of the corresponding pixel in the defect segmentation image.

[0134] According to embodiments of this disclosure, the acquisition module 701 is further configured to acquire multiple training sample pairs, wherein each training sample pair includes an initial face sample image and a target face sample image, the target face sample image being an image obtained by removing blemishes from the initial face sample image. The determination module 703 is further configured to determine a difference image between the initial face sample image and the target face sample image. The processing module 702 is further configured to generate a binarized blemish segmentation sample image corresponding to the corresponding training sample pair based on the difference image and a set pixel value segmentation threshold. Correspondingly, the training module 704 is configured to train an image generation model and an image segmentation model based on the multiple training sample pairs and the blemish segmentation sample images corresponding to each of the multiple training sample pairs.

[0135] According to embodiments of this disclosure, the training module 704 is further configured to: keep the parameters of the first image generation model unchanged; train the first image segmentation model using a first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs to generate a second image segmentation model; simultaneously train the first image generation model and the second image segmentation model using a second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs to generate a second image generation model and a third image segmentation model, wherein the parameter update frequency of the image generation model is higher than the parameter update frequency of the image segmentation model; keep the parameters of the second image generation model unchanged at this time; train the third image segmentation model using a third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate a fourth image segmentation model; simultaneously train the second image generation model and the fourth image segmentation model using a fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs to generate the trained image generation model and image segmentation model, wherein the parameter update frequency of the second image generation model is the same as the parameter update frequency of the fourth image segmentation model.

[0136] According to embodiments of this disclosure, the image generation model includes multiple first feature extraction layers cascaded from top to bottom, multiple first feature fusion layers cascaded from bottom to top, and a first convolutional output layer, wherein the first feature extraction layer and the first feature fusion layer corresponding to feature maps of the same scale are connected.

[0137] Correspondingly, the processing module 702 is further configured to perform feature extraction and downsampling processing on the first face image through multiple first feature extraction layers to obtain feature maps at multiple scales. The feature map extracted by the previous first feature extraction layer is downsampled and used as the input to the next first feature extraction layer, and the feature map extracted by the last first feature extraction layer is downsampled and input to the first first feature fusion layer. Multiple first feature fusion layers are used to perform upsampling and feature fusion processing on the feature maps at multiple scales to obtain the final fused target feature map. The previous first feature fusion layer upsamples the input feature map and then fuses it with feature maps of the same scale received from multiple first feature extraction layers. The feature fusion result is used as the input to the next first feature fusion layer. The target feature map is then input to the first convolutional output layer to obtain the second face image output by the first convolutional output layer.

[0138] According to embodiments of this disclosure, the image segmentation model includes multiple second feature extraction layers cascaded from top to bottom, multiple second feature fusion layers cascaded from bottom to top, and a second convolutional output layer, wherein the second feature extraction layers corresponding to feature maps of the same scale are connected to the second feature fusion layers.

[0139] Correspondingly, the processing module 702 is also used to perform feature extraction and downsampling processing on the first face image through multiple second feature extraction layers to obtain feature maps at multiple scales. The feature map extracted by the previous second feature extraction layer is downsampled and used as the input of the next second feature extraction layer, and the feature map extracted by the last second feature extraction layer is downsampled and input into the first second feature fusion layer. The feature maps at multiple scales are upsampled and fused through multiple second feature fusion layers to obtain the final fused target feature map. The previous second feature fusion layer is used to upsample the input feature map and then fuse it with the feature maps of the same scale received from multiple second feature extraction layers. The feature fusion result is used as the input of the next second feature fusion layer. The target feature map is input into the second convolutional output layer to obtain the defect segmentation image output by the second convolutional output layer.

[0140] According to embodiments of this disclosure, a preset number of feature extraction layers of multiple first feature extraction layers are shared with multiple second feature extraction layers.

[0141] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0142] In one possible design, the above Figure 7 The structure of the image processing device shown can be implemented as an image processing equipment, such as... Figure 8 As shown, the image processing device 800 may include a processor 801 and a memory 802. The memory 802 stores executable code, which, when executed by the processor 801, enables the processor 801 to at least perform the functions described above. Figure 1 The image processing method provided in the illustrated embodiment.

[0143] Optionally, the image processing device may also include a communication interface 803 for communicating with other devices.

[0144] Figure 9 This is a schematic diagram illustrating the structure of a model training device according to an exemplary embodiment. Figure 9 As shown, the model training device 900 includes an acquisition module 901, a determination module 902, a generation module 903, and a training module 904.

[0145] The acquisition module 901 is used to acquire multiple training sample pairs, wherein each training sample pair includes an initial face sample image and a target face sample image, and the target face sample image is the image after the initial face sample image has been de-defected.

[0146] The determination module 902 is used to determine the difference image between the initial face sample image and the target face sample image.

[0147] The generation module 903 is used to generate a binarized defect segmentation sample image corresponding to the corresponding training sample pair based on the difference image and the set pixel value segmentation threshold.

[0148] Training module 904 is used to train the image generation model and the image segmentation model based on multiple training sample pairs and the defect segmentation sample images corresponding to each of the multiple training sample pairs.

[0149] Specifically, the training module 904 is further configured to keep the parameters of the first image generation model unchanged, train the first image segmentation model using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, to generate the second image segmentation model; train the first image generation model and the second image segmentation model simultaneously using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, to generate the second image generation model and the third image segmentation model, wherein the parameter update frequency of the image generation model is higher than that of the image segmentation model; keep the parameters of the second image generation model unchanged at this time, train the third image segmentation model using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs, to generate the fourth image segmentation model; train the second image generation model and the fourth image segmentation model simultaneously using the fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs, to generate the trained image generation model and the image segmentation model, wherein the parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

[0150] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0151] In one possible design, the above Figure 9 The structure of the model training device shown can be implemented as a model training equipment, such as... Figure 10 As shown, the model training device 1000 may include a processor 1001 and a memory 1002. The memory 1002 stores executable code, which, when executed by the processor 1001, enables the processor 1001 to at least achieve the aforementioned functionality. Figure 6 The model training method provided in the illustrated embodiment.

[0152] Optionally, the electronic device may also include a communication interface 1003 for communicating with other devices.

[0153] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0154] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain a first face image to be processed, the first face image including blemish areas and non-blemish areas; The first face image is processed by an image generation model to remove blemishes, so as to obtain a second face image; The first face image is segmented using an image segmentation model to obtain a defect segmentation image. The value of each pixel in the defect segmentation image represents the predicted defect level of the corresponding pixel in the first face image. The weight value corresponding to each pixel in the first face image and the second face image is determined based on the defect segmentation image. The larger the predicted defect level value of the pixel in the first face image, the larger the weight value of the pixel. The weight value of each pixel in the second face image is determined based on the predicted defect level value of the corresponding pixel in the defect segmentation image. The second face image and the first face image are subjected to weighted fusion processing based on the weight value corresponding to each pixel to generate a third face image.

2. The method according to claim 1, characterized in that, The step of determining the weight value corresponding to each pixel in the first face image and the second face image based on the defect segmentation image includes: The weight value of each pixel in the first face image is determined based on the difference between a preset value and the predicted value of the defect severity of the corresponding pixel in the defect segmentation image; and The weight value of each pixel in the second face image is determined based on the predicted value of the degree of defect of the corresponding pixel in the defect segmentation image.

3. The method according to claim 1, characterized in that, The method further includes: Multiple training sample pairs are obtained, wherein each training sample pair includes an initial face sample image and a target face sample image, wherein the target face sample image is an image after the initial face sample image has been de-defected; Determine the difference image between the initial face sample image and the target face sample image; Based on the difference image and the set pixel value segmentation threshold, a binarized defect segmentation sample image corresponding to the corresponding training sample pair is generated; The image generation model and the image segmentation model are trained based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs.

4. The method according to claim 3, characterized in that, The step of training the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs includes: Keeping the parameters of the first image generation model unchanged, the first image segmentation model is trained using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, so as to generate the second image segmentation model. Using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, the first image generation model and the second image segmentation model are trained simultaneously to generate the second image generation model and the third image segmentation model. The parameter update frequency of the image generation model is higher than that of the image segmentation model. Keeping the parameters of the second image generation model unchanged, the third image segmentation model is trained using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate the fourth image segmentation model; Using the fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs, the second image generation model and the fourth image segmentation model are trained simultaneously to generate the trained image generation model and the image segmentation model. The parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

5. The method according to any one of claims 1 to 4, characterized in that, The image generation model includes multiple first feature extraction layers cascaded from top to bottom, multiple first feature fusion layers cascaded from bottom to top, and a first convolutional output layer, wherein the first feature extraction layer and the first feature fusion layer corresponding to feature maps of the same scale are connected. Generating the second face image using the image generation model includes: The first face image is subjected to feature extraction and downsampling processing through the multiple first feature extraction layers to obtain feature maps of multiple scales. The feature map extracted by the previous first feature extraction layer is downsampled and used as the input of the next first feature extraction layer. The feature map extracted by the last first feature extraction layer is downsampled and input into the first first feature fusion layer. The multiple first feature fusion layers perform upsampling and feature fusion processing on the feature maps at multiple scales to obtain the final fused target feature map. The previous first feature fusion layer is used to upsample the input feature map and then perform feature fusion with the feature map of the same scale received from the multiple first feature extraction layers. The feature fusion result is used as the input of the next first feature fusion layer. The target feature map is input into the first convolutional output layer to obtain the second face image output by the first convolutional output layer.

6. The method according to claim 5, characterized in that, The image segmentation model includes multiple second feature extraction layers cascaded from top to bottom, multiple second feature fusion layers cascaded from bottom to top, and a second convolutional output layer, wherein the second feature extraction layer and the second feature fusion layer corresponding to the feature map of the same scale are connected. Generating the defect segmentation image using the image segmentation model includes: The first face image is subjected to feature extraction and downsampling processing through the multiple second feature extraction layers to obtain feature maps of multiple scales. The feature map extracted by the previous second feature extraction layer is downsampled and used as the input of the next second feature extraction layer. The feature map extracted by the last second feature extraction layer is downsampled and input into the first second feature fusion layer. The multiple second feature fusion layers perform upsampling and feature fusion processing on the feature maps at multiple scales to obtain the final fused target feature map. The previous second feature fusion layer is used to upsample the input feature map and then perform feature fusion with the feature map of the same scale received from the multiple second feature extraction layers. The feature fusion result is used as the input of the next second feature fusion layer. The target feature map is input into the second convolutional output layer to obtain the defect segmentation image output by the second convolutional output layer.

7. The method according to claim 6, characterized in that, The preset number of feature extraction layers in the plurality of first feature extraction layers are shared with the plurality of second feature extraction layers.

8. A model training method, characterized in that, include: Multiple training sample pairs are obtained, wherein each training sample pair includes an initial face sample image and a target face sample image, wherein the target face sample image is an image after the initial face sample image has been de-defected; Determine the difference image between the initial face sample image and the target face sample image; Based on the difference image and the set pixel value segmentation threshold, a binarized defect segmentation sample image corresponding to the corresponding training sample pair is generated; Based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs, an image generation model and an image segmentation model are trained, and the image generation model and the image segmentation model are used to implement the method described in any one of claims 1-7; The step of training the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs includes: Keeping the parameters of the first image generation model unchanged, the first image segmentation model is trained using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, so as to generate the second image segmentation model. Using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, the first image generation model and the second image segmentation model are trained simultaneously to generate the second image generation model and the third image segmentation model. The parameter update frequency of the image generation model is higher than that of the image segmentation model. Keeping the parameters of the second image generation model unchanged, the third image segmentation model is trained using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate the fourth image segmentation model; Using the fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs, the second image generation model and the fourth image segmentation model are trained simultaneously to generate the trained image generation model and the image segmentation model. The parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

9. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire a first face image to be processed, wherein the first face image includes blemish areas and non-blemish areas; The processing module is used to perform blemish removal processing on the first face image through an image generation model to obtain a second face image; and to perform image segmentation processing on the first face image through an image segmentation model to obtain a blemish segmentation image, wherein the value of each pixel in the blemish segmentation image represents the predicted blemish degree value corresponding to the corresponding pixel in the first face image. The determining module is used to determine the weight value corresponding to each pixel in the first face image and the second face image based on the defect segmentation image, wherein the larger the predicted defect level value corresponding to the pixel in the first face image, the larger the weight value corresponding to the pixel; the weight value of each pixel in the second face image is determined based on the predicted defect level value of the corresponding pixel in the defect segmentation image. The processing module is further configured to perform weighted fusion processing on the second face image and the first face image according to the weight value corresponding to each pixel, so as to generate a third face image.

10. The apparatus according to claim 9, characterized in that, The determining module is further configured to determine the weight value of each pixel in the first face image based on the difference between the preset value and the predicted value of the degree of defect of the corresponding pixel in the defect segmentation image; and to determine the weight value of each pixel in the second face image based on the predicted value of the degree of defect of the corresponding pixel in the defect segmentation image.

11. The apparatus according to claim 9, characterized in that, It also includes a training module; The acquisition module is also used to acquire multiple training sample pairs, wherein each training sample pair includes an initial face sample image and a target face sample image, and the target face sample image is an image after the initial face sample image has been de-defected; The determining module is further configured to determine the difference image between the initial face sample image and the target face sample image; The processing module is also used to generate a binarized defect segmentation sample image corresponding to the corresponding training sample pair based on the difference image and the set pixel value segmentation threshold. The training module is used to train the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs.

12. The apparatus according to claim 11, characterized in that, The training module is further configured to: keep the parameters of the first image generation model unchanged; train the first image segmentation model using a first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs to generate a second image segmentation model; train the first image generation model and the second image segmentation model simultaneously using a second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs to generate a second image generation model and a third image segmentation model, wherein the parameter update frequency of the image generation model is higher than that of the image segmentation model; keep the parameters of the second image generation model unchanged at this time; train the third image segmentation model using a third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate a fourth image segmentation model; and train the second image generation model and the fourth image segmentation model simultaneously using a fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs to generate the trained image generation model and the image segmentation model, wherein the parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The image generation model includes multiple first feature extraction layers cascaded from top to bottom, multiple first feature fusion layers cascaded from bottom to top, and a first convolutional output layer, wherein the first feature extraction layer and the first feature fusion layer corresponding to feature maps of the same scale are connected. The processing module is further configured to perform feature extraction and downsampling processing on the first face image through the plurality of first feature extraction layers to obtain feature maps at multiple scales. The feature map extracted by the previous first feature extraction layer is downsampled and used as the input to the next first feature extraction layer, and the feature map extracted by the last first feature extraction layer is downsampled and input to the first first feature fusion layer. The module then performs upsampling and feature fusion processing on the feature maps at the multiple scales through the plurality of first feature fusion layers to obtain the final fused target feature map. The previous first feature fusion layer upsamples the input feature map and then fuses it with feature maps of the same scale received from the plurality of first feature extraction layers. The feature fusion result is used as the input to the next first feature fusion layer. Finally, the target feature map is input to the first convolutional output layer to obtain the second face image output by the first convolutional output layer.

14. The apparatus according to claim 13, characterized in that, The image segmentation model includes multiple second feature extraction layers cascaded from top to bottom, multiple second feature fusion layers cascaded from bottom to top, and a second convolutional output layer, wherein the second feature extraction layer and the second feature fusion layer corresponding to the feature map of the same scale are connected. The processing module is further configured to perform feature extraction and downsampling processing on the first face image through the plurality of second feature extraction layers to obtain feature maps at multiple scales. The feature map extracted by the previous second feature extraction layer is downsampled and used as the input to the next second feature extraction layer, and the feature map extracted by the last second feature extraction layer is downsampled and input to the first second feature fusion layer. The module then performs upsampling and feature fusion processing on the feature maps at the multiple scales through the plurality of second feature fusion layers to obtain the final fused target feature map. The previous second feature fusion layer upsamples the input feature map and then fuses it with feature maps of the same scale received from the plurality of second feature extraction layers. The feature fusion result is used as the input to the next second feature fusion layer. Finally, the target feature map is input to the second convolutional output layer to obtain the defect segmentation image output by the second convolutional output layer.

15. The apparatus according to claim 14, characterized in that, The preset number of feature extraction layers in the plurality of first feature extraction layers are shared with the plurality of second feature extraction layers.

16. A model training device, characterized in that, include: The acquisition module is used to acquire multiple training sample pairs, wherein each training sample pair includes an initial face sample image and a target face sample image, wherein the target face sample image is an image after the initial face sample image has been de-defected; The determination module is used to determine the difference image between the initial face sample image and the target face sample image; The generation module is used to generate a binarized defect segmentation sample image corresponding to the corresponding training sample pair based on the difference image and the set pixel value segmentation threshold. The training module is used to train an image generation model and an image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs, wherein the image generation model and the image segmentation model are used to implement the method described in any one of claims 1-7; The step of training the image generation model and the image segmentation model based on the plurality of training sample pairs and the defect segmentation sample images corresponding to each of the plurality of training sample pairs includes: Keeping the parameters of the first image generation model unchanged, the first image segmentation model is trained using the first set of training sample pairs and the defect segmentation sample images corresponding to the first set of training sample pairs, so as to generate the second image segmentation model. Using the second set of training sample pairs and the defect segmentation sample images corresponding to the second set of training sample pairs, the first image generation model and the second image segmentation model are trained simultaneously to generate the second image generation model and the third image segmentation model. The parameter update frequency of the image generation model is higher than that of the image segmentation model. Keeping the parameters of the second image generation model unchanged, the third image segmentation model is trained using the third set of training sample pairs and the defect segmentation sample images corresponding to the third set of training sample pairs to generate the fourth image segmentation model; Using the fourth set of training sample pairs and the defect segmentation sample images corresponding to the fourth set of training sample pairs, the second image generation model and the fourth image segmentation model are trained simultaneously to generate the trained image generation model and the image segmentation model. The parameter update frequency of the second image generation model is the same as that of the fourth image segmentation model.

17. An image processing device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the image processing method as described in any one of claims 1 to 7.

18. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an image processing apparatus, enable the image processing apparatus to perform the image processing method as claimed in any one of claims 1 to 7.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1 to 7.

20. A model training device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the model training method as described in claim 8.

21. A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of a model training apparatus, the model training apparatus is enabled to perform the model training method as described in claim 8.

22. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method of claim 8.

Images