Image processing method, device, electronic device and storage medium
By occluding the target image, extracting key points and masking, and generating skin fusion images with neural network, the problem of difficulty in accurately describing skin types in the description text is solved, and the authenticity of skin fusion effect and facial structure are improved .
Patent Information
- Application Number
- CN202411118959.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-15
AI Technical Summary
In the prior art, when controlling the skin adjustment of the face through description text, it is difficult to accurately describe the detailed characteristics of the skin type, resulting in poor skin type fusion effect.
Using image processing method, skin occlusion, key point extraction and masking processing of the target image, combined with neural network for skin generation, generate skin fusion images that fuse the skin types presented by the target skin image.
It improves the skin fusion effect, so that the generated skin fusion image can accurately present the desired skin type, and maintains the authenticity of the facial structure, avoiding distortion of the facial structure after changing the skin type.
Smart Images

Figure CN118644407B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and more specifically, to an image processing method, device, electronic device and storage medium. Background Art
[0002] Face skin fusion refers to adjusting the skin quality of a face in a given image including a face, so as to achieve the purpose of changing the skin quality of the face in the image.
[0003] Currently, a description text for describing skin quality can be obtained, and the description text is used as a condition to adjust the skin quality of the image to be adjusted through a neural network model to obtain a skin quality fusion image.
[0004] However, the control ability of the description text is weak, that is, the description text is difficult to accurately describe the detailed characteristics of the skin quality, which makes it difficult for the skin quality fusion image obtained by this method to present the desired skin quality, resulting in poor skin quality fusion effect. Summary of the invention
[0005] In view of this, embodiments of the present application provide an image processing method, device, electronic device, and storage medium.
[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: performing occlusion processing on a skin area in a target image to obtain a skin occlusion image; performing key point extraction on a face area in the target image to obtain a face key point map; the skin area includes a facial skin area in the face area; performing mask processing on the skin area in the target image to obtain a skin mask; performing skin generation on the skin occlusion image based on the target skin image, the face key point map and the skin mask to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image.
[0007] In a second aspect, an embodiment of the present application provides an image processing device, which includes: an occlusion module, which is used to perform occlusion processing on the skin area in the target image to obtain a skin occlusion image; an extraction module, which is used to extract key points from the face area in the target image to obtain a face key point map; the skin area includes a facial skin area in the face area; a mask module, which is used to perform mask processing on the skin area in the target image to obtain a skin mask; a skin generation module, which is used to perform skin generation on the skin occlusion image based on the target skin image, the face key point map and the skin mask to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image.
[0008] Optionally, the skin generation module is also used to encode the target skin image, the skin occlusion image and the facial key point map respectively to obtain a first feature map of the target skin image, a second feature map of the skin occlusion image and a third feature map of the facial key point map; based on the first feature map, the second feature map, the third feature map and the skin mask, perform back diffusion processing on the noise image to obtain a skin quality fusion feature map; decode the skin quality fusion feature map to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image.
[0009] Optionally, the skin generation module is also used to perform feature extraction on the first feature map through the first neural network to obtain a feature extraction result; using the second feature map, the third feature map and the skin mask as diffusion conditions, through the second neural network, perform back diffusion processing on the noise image based on the feature extraction result to obtain a skin quality fusion feature map.
[0010] Optionally, the first neural network includes K layers of first self-attention layers, and the second neural network includes K layers of second self-attention layers; the feature extraction result includes the first self-attention features input by each first self-attention layer; K is an integer greater than 1; the skin generation module is also used to use the second feature map, the third feature map and the skin mask as diffusion conditions, through the second neural network, based on the feature extraction result, to sequentially perform M rounds of denoising on the noisy image, and use the output feature map obtained by the Mth round of denoising as the skin quality fusion feature map; M is a positive integer greater than 1; wherein the i-th round of denoising is performed according to the following process: obtaining the input feature map of the i-th round of denoising, wherein the input feature map of the first round of denoising is the second round of denoising. The feature map, the third feature map, the skin mask and the noise feature map corresponding to the noise image are fused; when 1<i≤M, the input feature map of the i-th round of denoising is the output feature map of the i-1th round of denoising; in the j-th second self-attention layer, the attention calculation is performed based on the first self-attention feature input by the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising, and the fused feature map output by the j-th second self-attention layer for the i-th round of denoising is obtained; 1≤j≤K, j is a positive integer; the second self-attention feature of the first second self-attention layer for the i-th round of denoising is determined based on the input feature map of the i-th round of denoising; The fused feature map output by the j-th second self-attention layer is denoised to obtain the j-th intermediate denoised feature map for the i-th round of denoising; if j is less than K, based on the j-th intermediate denoised feature map for the i-th round of denoising, the second self-attention feature of the j+1-th second self-attention layer for the i-th round of denoising is determined, j is accumulated and added by 1, and the step of performing attention calculation in the j-th second self-attention layer based on the first self-attention feature input by the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising is returned to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising; if j=K, the j-th intermediate denoised feature map for the i-th round of denoising is used as the output feature map of the i-th round of denoising.
[0011] Optionally, the first self-attention feature includes a first key vector and a first value vector; the second self-attention feature includes a second key vector, a second value vector and a target query vector; the skin generation module is also used to perform the following processing in the j-th second self-attention layer: fuse the first key vector and the second key vector of the j-th second self-attention layer for the i-th round of denoising to obtain the target key vector of the j-th second self-attention layer for the i-th round of denoising; fuse the first value vector and the second value vector of the j-th second self-attention layer for the i-th round of denoising to obtain the target value vector of the j-th second self-attention layer for the i-th round of denoising; determine the attention score of the j-th second self-attention layer for the i-th round of denoising based on the target query vector and target key vector of the j-th second self-attention layer for the i-th round of denoising; weight the corresponding target value vector according to the attention score of the j-th second self-attention layer for the i-th round of denoising to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising.
[0012] Optionally, the image processing device further includes a training module for acquiring a sample skin image, a sample skin occlusion image, a sample skin mask, and a sample facial key point map; the sample skin image is determined based on a sample skin region in the first sample image, the sample skin occlusion image is obtained by occluding the sample skin region in the first sample image, the sample skin mask is a mask of the sample skin region, and the sample facial key point map is obtained by extracting key points from the sample facial region in the first sample image; the sample skin region includes a facial skin region in the sample facial region; The sample skin image, the sample skin occlusion image and the sample facial key point map are respectively encoded to obtain a first sample feature map of the sample skin image, a second sample feature map of the sample skin occlusion image and a third sample feature map of the sample facial key point map; the first sample feature map is feature extracted by a first neural network to obtain a sample feature extraction result; the second sample feature map, the third sample feature map and the sample skin mask are used as diffusion conditions, and the sample noise image is back-diffused based on the sample feature extraction result by a second neural network to obtain a sample skin quality fusion feature map; the sample skin quality fusion feature map is decoded to obtain a sample skin quality fusion image; based on the difference between the sample skin quality fusion image and the first sample image, the parameters of the first neural network and the second neural network are adjusted until the first training end condition is reached.
[0013] Optionally, the training module is also used to perform skin area segmentation on the first sample image to determine the sample skin area in the first sample image; perform mask processing on the sample skin area to obtain a sample skin mask; based on the sample skin mask, perform occlusion processing on the sample skin area in the first sample image to obtain a sample skin occlusion image; based on the sample skin mask, process the first sample image to obtain a sample skin image; perform key point extraction on the sample face area in the first sample image to obtain a sample face key point map.
[0014] Optionally, the training module is also used to normalize the first sample image to a first pixel value range to obtain a first intermediate sample image; normalize the sample skin mask to a second pixel value range to obtain a first sample skin mask, and the second pixel value range is within the first pixel value range; perform a dot product on the first intermediate sample image and the first sample skin mask to obtain a second intermediate image; and perform denormalization on the second intermediate image to obtain a sample skin occlusion image.
[0015] Optionally, the training module is further used to normalize the sample skin mask to a second pixel value range to obtain a second sample skin mask; and perform a dot multiplication on the second sample skin mask and the first sample image to obtain a sample skin image.
[0016] Optionally, the training module is also used to extract spatial detail features of the first sample image through the first branch network to obtain a sample spatial feature map; extract semantic features of the first sample image through the second branch network to obtain a sample semantic feature map; downsample the sample spatial feature map to obtain a downsampled spatial feature map; upsample the sample semantic feature map to obtain an upsampled semantic feature map; aggregate the sample semantic feature map with the downsampled spatial feature map to obtain a first sample aggregated feature map; aggregate the sample spatial feature map with the upsampled semantic feature map to obtain a second sample aggregated feature map; fuse the first sample aggregated feature map and the second sample aggregated feature map to obtain a sample feature map; and determine the sample skin area in the first sample image based on the sample feature map.
[0017] Optionally, the diffusion condition also includes an object description text generated by a text generation model based on the target image; the training module is also used to extract features of the second sample image through a visual encoder to obtain a first sample feature; extract features of the sample description text corresponding to the second sample image through a text encoder to obtain a second sample feature; perform content consistency judgment on the first sample feature and the sample description text through a visual text encoder to obtain a sample judgment result; generate a predicted sample description text based on the first sample encoding result and the sample description text through the text generation model; determine the text generation model loss based on the difference between the predicted sample description text and the sample description text, the difference between the first sample feature and the second sample feature, and the sample judgment result; adjust the parameters of the visual encoder, text encoder, visual text encoder and text generation model through the text generation model loss until the second training end condition is reached.
[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above method is implemented.
[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the above-mentioned method is implemented.
[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer-readable instructions, which implement the above method when executed by a processor.
[0021] The embodiments of the present application provide an image processing method, device, electronic device and storage medium. In the present application, skin generation is performed on a skin-occluded image based on a target skin image, a facial key point map and a skin mask, and a skin-quality fusion image that integrates the skin quality presented by the target skin image is obtained. Through the scheme of the present application, the skin quality in the target image is changed, that is, the skin quality in the target image is changed to the skin quality in the target skin image, thereby achieving skin replacement. Since the target skin image is highly concrete and can accurately describe the detailed features of the skin quality, the skin quality fusion image generated according to the target skin image can present the desired skin quality, thereby improving the skin quality fusion effect; at the same time, the facial key point map can accurately express the external structure of the facial area in the skin occlusion image, and the skin mask can accurately express the position of the skin area in the skin occlusion image. In this way, skin generation is performed on the skin occlusion image according to the target skin image, the facial key point map and the skin mask. On the one hand, based on the position where skin generation needs to be performed expressed by the skin mask and the skin quality presented by the target skin image, the skin quality presented by the skin quality fusion image can be highly consistent with the skin quality presented in the target skin image, and the non-skin area in the skin quality fusion image is consistent with the non-skin area in the skin occlusion image; on the other hand, since the facial key point map is added as a constraint, the structure of the facial area in the generated skin quality fusion image is consistent with the facial structure presented by the facial key point map, thereby avoiding the situation where the facial structure is distorted or changed after the skin quality is changed. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 A schematic diagram showing an application scenario applicable to an embodiment of the present application is shown;
[0024] Figure 2 A flowchart of an image processing method proposed in one embodiment of the present application is shown;
[0025] Figure 3 A schematic diagram of a skin segmentation model in an embodiment of the present application is shown;
[0026] Figure 4 A schematic diagram showing a fusion process of a target spatial feature map and a target semantic feature map in an embodiment of the present application is shown;
[0027] Figure 5 A schematic diagram showing a process of generating a facial key point map in an embodiment of the present application is shown;
[0028] Figure 6 Shows Figure 2 The flowchart of step S140 in the corresponding embodiment in one embodiment;
[0029] Figure 7 A schematic diagram showing a skin fusion process in an embodiment of the present application is shown;
[0030] Figure 8 Shows Figure 6 The flowchart of step S220 in the corresponding embodiment in one embodiment;
[0031] Fig. 9 A flowchart of the i-th round of denoising processing in one embodiment of the present application is shown;
[0032] Fig.10 A schematic diagram showing the attention calculation process in the i-th round of denoising processing in an embodiment of the present application is shown;
[0033] Fig.11 A schematic diagram showing a training process of a text generation model in an embodiment of the present application is shown;
[0034] Fig.12 A schematic diagram showing a process of obtaining an object description text in an embodiment of the present application;
[0035] Fig.13 Shows Figure 8 The steps before step S310 in the corresponding embodiment are a flowchart in an embodiment;
[0036] Fig.14 A schematic diagram showing training data in an embodiment of the present application is shown;
[0037] Fig.15 A schematic diagram showing a facial region recognition process in an embodiment of the present application;
[0038] Fig.16 A schematic diagram showing a training process of a skin-changing model according to an embodiment of the present application is shown;
[0039] Fig.17 A block diagram of an image processing device proposed in one embodiment of the present application is shown;
[0040] Fig.18 A structural block diagram of an electronic device for executing an image processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0042] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein are for the purpose of describing the embodiments of the present application only and are not intended to limit the present application. It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0044] It should be noted that in the embodiments of the present application, the acquisition of facial images involved (including target images, reference images, first sample images, and second sample images, etc.) requires the user's permission or consent, and the collection, use, processing, and storage of facial images must comply with the regulations of the region.
[0045] In addition, in an embodiment of the present application, the images including human faces involved in the drawings (such as a target image, a reference image, a first sample image, and a second sample image, etc.) can all be generated by the image generation model described below, and the human faces involved in the drawings are not real human faces.
[0046] The following is an explanation of the abbreviations or self-defined nouns involved in the embodiments of this application:
[0047] Character consistency is used to indicate the degree of change of the character's face, clothing, accessories and background of the image; the smaller the degree of change of the character's face, clothing, accessories and background of the image, the higher the character consistency; conversely, the greater the degree of change of the character's face, clothing, accessories and background of the image, the lower the task consistency.
[0048] See also Figure 1 , Figure 1A schematic diagram of an application scenario applicable to an embodiment of the present application is shown, the application scenario includes a terminal device 400 and a server 200, the terminal device 400 is connected to the server 200 via a network 300, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.
[0049] In some embodiments, the image processing method provided in the embodiments of the present application may be implemented by a server. The terminal device 400 may send a target image and a target skin image to the server 200 .
[0050] Afterwards, the server 200 can perform occlusion processing on the skin area in the target image to obtain a skin occlusion image; perform key point extraction on the face area in the target image to obtain a facial key point map; the skin area includes the facial skin area in the face area; perform mask processing on the skin area in the target image to obtain a skin mask, and then generate skin for the skin occlusion image based on the target skin image, the facial key point map and the skin mask to obtain a skin texture fusion image that incorporates the skin texture presented by the target skin image.
[0051] The user can select different skin images as target skin images on the interface of the terminal device 400 to obtain skin quality fusion images showing different skin qualities.
[0052] In some embodiments, the skin-changing model can be used to generate skin from the skin-occluded image according to the target skin image, the facial key point map, and the skin mask, and output a skin-quality fused image that combines the skin quality presented by the target skin image. To ensure the accuracy of the skin-quality fused image output by the skin-changing model, the skin-changing model can be trained in advance.
[0053] In some embodiments, the application scenario of the present application may further include a database 500, and the server 200 may obtain training data of the skin replacement model from the database 500, the training data including: a sample skin image, a sample skin occlusion image, a sample skin mask, a sample facial key point map, and a first sample image. The specific training process is described below.
[0054] In some embodiments, the image processing method provided by the embodiments of the present application can be used by the terminal device 400 to perform occlusion processing on the skin area in the target image to obtain a skin occlusion image; perform key point extraction on the face area in the target image to obtain a facial key point map; the skin area includes the facial skin area in the face area; mask processing is performed on the skin area in the target image to obtain a skin mask; skin generation is performed on the skin occlusion image based on the target skin image, the facial key point map and the skin mask to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image.
[0055] In some embodiments, the terminal device 400 or the server 200 can implement the image processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP, and the small program can be controlled by the user to run or close. In short, the above-mentioned computer program can be an application, module or plug-in in any form.
[0056] In some embodiments, server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0057] The terminal device 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, a smart TV, an intelligent service terminal, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0058] For the convenience of description, the present application uses an electronic device as the execution subject of the method of the present application for explanation.
[0059] See also Figure 2 , Figure 2 A flowchart of an image processing method proposed in one embodiment of the present application is shown. The method can be executed by an electronic device, such as an electronic device Figure 1 The server 200 or the terminal device 400 shown in the figure, in other embodiments, the method of the present application can also be implemented by the interaction between the server 200 and the terminal device 400, and the method can include S110-S140, which are described in detail as follows:
[0060] S110, performing occlusion processing on the skin area in the target image to obtain a skin occlusion image.
[0061] The target image refers to an image of a person whose skin quality is to be changed, and the target image at least presents the face area of the person.
[0062] The skin area in the target image may include the facial skin area in the face area of the target image. The skin area in the target image may also present skin of other body parts except the facial skin area. For example, the skin area may also include the neck skin area and the shoulder skin area in the neck area. It is worth mentioning that the facial area of the target image includes not only the facial skin area but also the area where the facial organs are located, such as the mouth, eyes, and eyebrows.
[0063] In some embodiments, the target image may be segmented to determine the skin area in the target image; the skin area may then be masked to obtain a skin mask; and the skin area in the target image may be occluded based on the skin mask to obtain a skin occluded image.
[0064] In some embodiments, the skin area in the target image can be segmented by a skin segmentation model, wherein the skin segmentation model is a neural network model constructed by one or more neural networks, and the skin segmentation model is used to identify the pixels representing the skin in the target image and the positions of the pixels representing the skin. The skin segmentation model can be obtained by training a model such as a fully convolutional network, a U-net network, etc.
[0065] In some embodiments, the skin segmentation model may include a first branch network and a second branch network. Accordingly, the segmentation process of the skin area of the target image includes: extracting spatial detail features of the target image through the first branch network to obtain a target spatial feature map; extracting semantic features of the target image through the second branch network to obtain a target semantic feature map; downsampling the target spatial feature map to obtain a downsampled target spatial feature map; upsampling the target semantic feature map to obtain an upsampled target semantic feature map; aggregating the target semantic feature map with the downsampled target spatial feature map to obtain a first target aggregate feature map; aggregating the target spatial feature map with the upsampled target semantic feature map to obtain a second target aggregate feature map; fusing the first target aggregate feature map and the second target aggregate feature map to obtain a target feature map; and determining the skin area in the target image based on the target feature map.
[0066] In the above embodiment, the first branch network is responsible for extracting the spatial detail features of the image, and the second branch network is responsible for extracting the semantic features of the image. Relatively speaking, the semantic features of the image are higher-level features, and the spatial detail features of the image are shallow features of the image. Therefore, the first branch network can be a network with a large number of channels but a small number of network layers. In this way, a large number of channels can encode rich spatial detail features. At the same time, because the first branch network focuses on shallow features, the stride of the first branch network can be set to be smaller.
[0067] Correspondingly, since the second branch network focuses on deep semantic features, the second branch network can be a neural network with a larger step size and more network layers to extract deep semantic features. At the same time, the second branch network does not need to pay attention to more spatial features, so the number of channels of the second branch network can be smaller.
[0068] In summary, the number of channels of the first branch network is greater than that of the second branch network, the number of network layers of the first branch network is smaller than that of the second branch network, and the step size of feature sampling between different network layers of the first branch network is smaller than that of the second branch network.
[0069] Figure 3 It is a structural schematic diagram of a skin segmentation model according to an embodiment of the present application, wherein the skin segmentation model includes a first branch network, a second branch network, an aggregation network and a target segmentation head.
[0070] like Figure 3 As shown, the first branch network includes three network layers, namely Figure 3 The first network layer 401, the second network layer 402 and the third network layer 403 are cascaded in sequence. The size of the feature map output by the first network layer 401 is 1 / 2 of the input of the first branch network, the size of the feature map output by the second network layer 402 is 1 / 4 of the input of the first branch network, and the size of the feature map output by the third network layer 403 is 1 / 8 of the input of the first branch network.
[0071] like Figure 3 As shown, the second branch network includes 6 network layers, which are the fourth network layer 404, the fifth network layer 405, the sixth network layer 406, the seventh network layer 407, the eighth network layer 408 and the ninth network layer 409, which are cascaded in sequence. The size of the feature map output by the fourth network layer 404 is 1 / 2 of the input of the second branch network, the size of the feature map output by the fifth network layer 405 is 1 / 4 of the input of the second branch network, the size of the feature map output by the sixth network layer 406 is 1 / 8 of the input of the second branch network, the size of the feature map output by the seventh network layer 407 is 1 / 16 of the input of the second branch network, the size of the feature map output by the eighth network layer 408 is 1 / 32 of the input of the second branch network, and the size of the feature map output by the ninth network layer 409 is 1 / 32 of the input of the second branch network.
[0072] like Figure 3After obtaining the target spatial feature map and the target semantic feature map, the target spatial feature map and the target semantic feature map are aggregated by an aggregation network, wherein the aggregation process includes: downsampling the target spatial feature map to obtain a downsampled target spatial feature map; upsampling the target semantic feature map to obtain an upsampled target semantic feature map; aggregating the target semantic feature map with the downsampled target spatial feature map to obtain a first target aggregated feature map; aggregating the target spatial feature map with the upsampled target semantic feature map to obtain a second target aggregated feature map; and fusing the first target aggregated feature map and the second target aggregated feature map to obtain a target feature map.
[0073] As mentioned above, since the first branch network extracts shallow spatial detail features, while the second branch network extracts deep semantic features, the sizes of the target spatial feature map output by the first branch network and the target semantic feature map output by the second branch network are inconsistent, specifically: the size of the target spatial feature map is larger than the size of the target semantic feature map. Therefore, in order to ensure the effect of feature fusion, the target spatial feature map is first downsampled to obtain a downsampled target spatial feature map; and the target semantic feature map is upsampled to obtain an upsampled target semantic feature map; the feature dimension (i.e., size) of the downsampled target spatial feature map is the same as the feature dimension of the target semantic feature map, and the feature dimension (i.e., size) of the upsampled target semantic feature map is the same as the feature dimension of the target spatial feature map.
[0074] Through the aggregation network, bilateral guided aggregation is achieved, that is, the complementary information of the two branches (the first branch network and the second branch network) is integrated.
[0075] Figure 4 FIG. 1 is a schematic diagram of an aggregation network according to an embodiment of the present application. Figure 4 As shown in the figure, the aggregation network includes: the first deep convolution layer (convolution kernel size 3×3), the first convolution layer (convolution kernel size 3×3, step size 2), the second convolution layer (convolution kernel size 3×3), the second deep convolution layer (convolution kernel size 3×3), the third convolution layer (convolution kernel size 1×1), the average pooling layer (pooling kernel size 3×3, step size 2), the upsampling layer (4×4 times upsampling) and the fourth convolution layer (convolution kernel size 1×1).
[0076] based on Figure 4In the aggregation network shown, the size of the target spatial feature map is H×W×C, the size of the target semantic feature map is H / 4×W / 4×C, and the downsampling process to obtain the downsampled target spatial feature map includes: performing convolution process on the target spatial feature map through a first convolution layer to obtain a first convolution result, normalizing the first convolution result to obtain a first normalized result, and then performing average pooling process on the first normalized result to obtain the average pooling result as the downsampled target spatial feature map. The size of the downsampled target spatial feature map is consistent with that of the target semantic feature map, both of which are H / 4×W / 4×C.
[0077] Continue to refer to Figure 4 , the upsampling process to obtain the upsampled target semantic feature map includes: performing convolution processing on the target semantic feature map through the second convolution layer to obtain a second convolution result, then normalizing the second convolution result to obtain a second normalized result, then upsampling the second normalized result through the upsampling layer to obtain an initial upsampling result, and activating the initial upsampling result to obtain the upsampled target semantic feature map. The size of the upsampled target semantic feature map is consistent with that of the target spatial feature map, both of which are H×W×C.
[0078] Continue to refer to Figure 4 The process of determining the first target aggregate feature map includes: performing convolution processing on the target spatial feature map through the first depth convolution layer to obtain the first depth convolution result, then normalizing the first depth convolution result to obtain the third normalized result, performing convolution processing on the first normalized result through the third convolution layer to obtain the third convolution result (the size of the third convolution result is consistent with that of the target spatial feature map), and then performing vector product of the third convolution result and the upsampled target semantic feature map, and the product result is used as the first target aggregate feature map. Since the size of the third convolution result and the upsampled target semantic feature map is consistent (both are H / 4×W / 4×C), the fusion of the third convolution result and the upsampled target semantic feature map can be achieved.
[0079] Continue to refer to Figure 4The process of determining the second target aggregate feature map includes: performing convolution processing on the target semantic feature map through the second depth convolution layer to obtain a second depth convolution result, then normalizing the second depth convolution result to obtain a fourth normalized result, then performing convolution processing on the fourth normalized result through the fourth convolution layer to obtain a fourth convolution result, then performing activation processing on the fourth convolution result, and then performing vector product of the result after activation processing (the size of the result after activation processing is consistent with that of the target semantic feature map) and the downsampled target spatial feature map, and the product result is used as the second target aggregate feature map. Since the size of the result after activation processing is consistent with that of the downsampled target spatial feature map (both are H×W×C), the fusion of the result after activation processing and the downsampled target spatial feature map can be achieved.
[0080] Continue to refer to Figure 4 The fusion process of the first target aggregate feature map and the second target aggregate feature map includes: summing the first target aggregate feature map and the second target aggregate feature map to obtain a summed skin quality fusion feature map, convolving the summed skin quality fusion feature map through the fifth convolution layer to obtain a fifth convolution result, and then normalizing the fifth convolution result to obtain a target feature map. The size of the obtained target feature map is consistent with that of the target space feature map, both of which are H×W×C.
[0081] After obtaining the target feature map, the target feature map is input into the target segmentation head in the skin segmentation model for segmentation processing, and the skin area segmentation result of the target image is output. In the skin area segmentation result, different pixel values can be used to represent the pixels belonging to the skin area and the pixels that do not belong to the skin area in the target image.
[0082] Among them, the sizes of the convolution kernels used in the aforementioned first convolutional layer, second convolutional layer, third convolutional layer, fourth convolutional layer, and fifth convolutional layer can also be other sizes, which are not limited in this application; the aforementioned activation processing can be implemented by the activation function Sigmoid, and the aforementioned normalization processing can be implemented by a BN (Batch Normalization) layer.
[0083] S120, extracting key points from the face area in the target image to obtain a face key point map.
[0084] The facial key point map presents the positions (i.e., pixel positions) of key points extracted from the face region of the target image. The facial key point map is used to constrain the structure of the face region in the target image. Key points extracted from the face region include contour points used to constrain the contour of the face region, corner points with significant changes (e.g., significant changes in pixel values) in the face region. Key points extracted from the face region include key points in the eyebrow region, eye region, nose region, mouth region, and face contour region.
[0085] In some embodiments, key point extraction can be performed through a key point extraction model to output a facial key point map. The key point extraction model can be through a convolutional neural network, a pooling neural network, a fully connected neural network, etc. The key point extraction model can be a cascaded convolutional neural network (Deep Convolutional Network, DCNN), a multi-task cascaded convolutional neural network (Multi-task Cascaded Convolutional Networks, MTCNN), a cascaded deep neural network (Deep Alignment Network, DAN), a face mesh model (Face Mesh model), etc., which are not specifically limited here.
[0086] In some embodiments, the key point extraction model may be a face mesh model. The face key point map generated by the face mesh model is a three-dimensional face key point map, which includes 468 key points, and the 468 key points may represent the surface contour and depth of the face area.
[0087] The process of generating facial key point map from facial mesh model is as follows Figure 5 As shown, the input target image is first processed and the target facial image is cropped. In this process, face detection can be performed on the target image first, and then the facial area is cropped. Thereafter, a standardized facial area image is obtained through a facial posture transformation matrix as the target facial image.
[0088] Afterwards, facial features are extracted from the target facial image through the feature extractor in the facial mesh model to segment the features of each facial part from the target facial image, such as eyes, irises, lips, noses, etc., and then a three-dimensional facial key point map is generated as the facial key point map through methods such as facial posture transformation matrix and triangular facial network.
[0089] S130, performing mask processing on the skin area in the target image to obtain a skin mask.
[0090] The pixel values of the pixels in the non-skin area of the skin mask are different from the pixel values of the pixels in the skin area, wherein the pixel values of different pixels in the non-skin area are the same, and the pixel values of different pixels in the skin area are the same. The non-skin area is the pixel area other than the skin area in the target image.
[0091] In some embodiments, after the skin area in the target image is determined according to the aforementioned process, the pixel values of the pixels in the skin area in the target image can be adjusted to a first pixel value (e.g., 0), and the pixel values of the pixels in the non-skin area in the target image can be adjusted to a second pixel value (e.g., 255), so as to perform mask processing on the skin area and obtain a skin mask. The first pixel value is different from the second pixel value, and the first pixel value and the second pixel value can be set according to actual needs, and are not specifically limited here.
[0092] In some embodiments, the skin area in the target image can be occluded with the help of a skin mask to obtain a skin occlusion image. The process may include: normalizing the target image to a first pixel value range to obtain a normalized image; normalizing the skin mask to a second pixel value range to obtain a first skin mask; performing a dot multiplication on the normalized image and the first skin mask to obtain a fused normalized image; and denormalizing the fused normalized image to obtain a skin occlusion image. The second pixel value range is within the first pixel value range, for example, the first pixel value range may be [-1,1], and the second pixel value range may be [0,1].
[0093] The target image can be normalized according to formula 1 to normalize the pixel values of the pixels in the target image to the first pixel value range [-1, 1] to obtain a normalized image. At this time, the size of the normalized image is consistent with that of the target image. Formula 1 is as follows:
[0094] (one)
[0095] Where I is the target image, is the normalized image.
[0096] The skin mask may be normalized according to Formula 2 to normalize the skin mask to the second pixel value range [0, 1] to obtain a first skin mask. At this time, the first skin mask is consistent with the skin mask size. Formula 2 is as follows:
[0097] (two)
[0098] Where M is the skin mask, For the first skin mask.
[0099] Then, the normalized image and the first skin mask are multiplied to obtain a fused normalized image K, that is, K = .
[0100] Afterwards, the fused normalized image is denormalized according to the following formula 3 to obtain the skin occlusion image. The formula 3 is as follows:
[0101] (three)
[0102] Where D is the skin occlusion image. It can be seen that the pixel values of the pixels in the skin occlusion image are within [0, 255].
[0103] use Normalize the target image I to [-1,1] to get the normalized image, and use The skin mask is normalized to [0,1] to obtain the first skin mask. If the pixels representing the skin area in the skin mask are black and the color of the pixels representing the non-skin area is white, the pixel values of the pixels representing the skin area in the first skin mask are 0 and the pixel values of the pixels representing the non-skin area are 1. Therefore, after the normalized image is dot-multiplied with the first skin mask, the pixel values of the pixels corresponding to the skin area in the first skin mask in the normalized image are changed to 0, and the pixel values of other pixels remain unchanged to obtain a fused normalized image. The fused normalized image is then denormalized to adjust the color of the pixels corresponding to the skin area in the first skin mask to gray, and the pixel values of the pixels corresponding to the non-skin area in the first skin mask remain unchanged, thereby achieving the purpose of deleting the skin, that is, the skin area is blocked to obtain a skin-blocked image.
[0104] S140, performing skin generation on the skin-occluded image according to the target skin image, the facial key point map and the skin mask, to obtain a skin-quality fused image that incorporates the skin quality presented by the target skin image.
[0105] The target skin image refers to a given skin image used to guide skin generation, and the target skin image presents the pixel values of the pixels in the skin area. In some embodiments, the target skin image only includes the skin area. In other embodiments, the target skin image may also include other pixel areas except the skin area, but the other pixel areas except the skin area are masked to the specified pixel values.
[0106] The target skin image and the target image may be from different persons or from the same person, which is not specifically limited here.
[0107] The target skin image can present the skin area of one part or the skin areas of multiple parts. For example, the target skin image can present only the skin area of the face (i.e., the facial skin area mentioned above), or only the skin area of the neck. The target skin image can also present at least two of the facial skin area, the neck skin area, and the shoulder skin area.
[0108] In some embodiments, a reference image may be acquired, and then the reference image may be segmented into skin regions to determine a reference skin region in the reference image; then, the pixel values of the non-skin region (i.e., other pixel regions except the reference skin region) in the reference image may be adjusted to preset pixel values to obtain a target skin image, wherein the preset pixel values are different from the pixel values of the pixels in the reference pixel region. In this way, it can be ensured that the pixel values of the pixels in the non-skin region in the target skin image are the same pixel values and are different from the pixel values of the pixels in the reference skin region.
[0109] In some embodiments, after performing skin region segmentation on the reference image and determining the reference skin region in the reference image, the reference skin region can be masked to obtain a reference skin mask; the reference skin mask is then normalized to a second pixel value range to obtain a second skin mask; and the second skin mask and the reference image are dot-multiplied to obtain a target skin image.
[0110] The process of determining the reference skin area and the reference skin mask by referring to the aforementioned process of determining the skin area and the skin mask of the target image is not described in detail.
[0111] In the present application, the reference skin mask may be subjected to a second normalization process according to Formula 4, so as to normalize the reference skin mask to a range of [0, 1] to obtain a second skin mask, wherein the second skin mask has the same size as the reference skin mask, and Formula 4 is as follows:
[0112] (Four)
[0113] in, For the reference skin mask, For the second skin mask.
[0114] Then, the second skin mask and the reference image can be dot-multiplied to achieve fusion of the second skin mask and the reference image to obtain the target skin image. At this time, the process of determining the target skin image is briefly described as shown in Figure 5, and Formula 5 is as follows:
[0115] (five)
[0116] Among them, S is the target skin image, is the reference image.
[0117] Here, use The reference skin mask is normalized to the range of [0,1] to obtain the second skin mask. If the pixels representing the skin area in the reference skin mask are black and the pixels representing the non-skin area are white, then the pixel values of the pixels representing the skin area in the second skin mask are 0, and the pixel values of the pixels representing the non-skin area are 1. Therefore, after dot multiplication of the reference image and the second skin mask, the pixel values of the pixels in the reference image corresponding to the skin area in the second skin mask are changed to 0, and the pixel values of other pixels remain unchanged to obtain the target skin image.
[0118] In the process of skin generation, the structure of the facial area is constrained by the facial key point map, the area where skin needs to be generated in the skin occlusion image is specified by the skin mask, and the skin quality of the skin to be generated is determined by the target skin image, so that the skin quality presented by the skin area in the generated skin quality fusion image is the same as the skin quality presented by the target skin image, and the non-skin area in the skin quality fusion image is consistent with the non-skin area in the skin occlusion image.
[0119] In some embodiments, a skin generation can be performed on a skin-blocking image according to a target skin image, a facial key point map, and a skin mask through a skin-changing model, so as to obtain a skin-quality fusion image that integrates the skin quality presented by the target skin image. The skin-changing model can be a neural network model with an image generation function, and the skin-changing model can extract features from the target skin image, the facial key point map, the skin mask, and the skin-blocking image, respectively, and then fuse the features of the four to obtain a target fusion feature map, and decode the target fusion feature map to output a skin-quality fusion image.
[0120] The image processing method of the present application can be integrated into a small program, through which the skin of the person in the target image can be replaced according to the method of the present application based on the input target image and target skin image.
[0121] In the present application, skin generation is performed on the skin occluded image based on the target skin image, the facial key point map and the skin mask, and a skin texture fusion image that integrates the skin texture presented by the target skin image is obtained. Through the scheme of the present application, the skin texture in the target image is changed, that is, the skin texture in the target image is changed to the skin texture in the target skin image, thereby achieving skin replacement. Since the target skin image is highly concrete and can accurately describe the detailed features of the skin quality, the skin quality fusion image generated according to the target skin image can present the desired skin quality, thereby improving the skin quality fusion effect; at the same time, the facial key point map can accurately express the external structure of the facial area in the skin occlusion image, and the skin mask can accurately express the position of the skin area in the skin occlusion image. In this way, skin generation is performed on the skin occlusion image according to the target skin image, the facial key point map and the skin mask. On the one hand, based on the position where skin generation needs to be performed expressed by the skin mask and the skin quality presented by the target skin image, the skin quality presented by the skin quality fusion image can be highly consistent with the skin quality presented in the target skin image, and the non-skin area in the skin quality fusion image is consistent with the non-skin area in the skin occlusion image; on the other hand, since the facial key point map is added as a constraint, the structure of the facial area in the generated skin quality fusion image is consistent with the facial structure presented by the facial key point map, thereby avoiding the situation where the facial structure is distorted or changed after the skin quality is changed.
[0122] Compared with describing skin texture through text, since the target skin image expresses the skin texture concretely, the skin texture presented by the target skin image of the skin area in the generated skin texture fusion image is highly consistent. This can effectively solve the problem in related technologies that the skin texture presented in the image obtained after skin resurfacing is different from the expected skin texture because the description text cannot accurately describe the expected skin texture, thereby effectively ensuring the skin resurfacing effect.
[0123] In some embodiments, the training process of the aforementioned skin segmentation model may include: obtaining a third sample image and a valid sample skin area of the third sample image, and then performing feature extraction on the third sample image through a parameter-initialized first branch network and a second branch network to obtain a training sample spatial feature map and a training sample semantic feature map, and then fusing the training sample spatial feature map and the training sample semantic feature map according to the aforementioned bilateral guided aggregation process to obtain a training sample feature map, and then processing the training sample feature map through a parameter-initialized target segmentation head to achieve skin area segmentation of the third sample image to obtain a predicted sample skin area, determine the difference between the predicted sample skin area and the valid sample skin area as the skin segmentation loss, and then adjust the network parameters of the first branch network, the second branch network and the target segmentation head through the skin segmentation loss until the third training end condition is reached to obtain a skin segmentation model (including a first neural network, a second neural network and a target segmentation head).
[0124] Among them, the third sample image may refer to an image including a human face, and the human face in the third sample image may be the face of a real person or the face of a virtual character. The effective sample skin area in the third sample image may be determined by manual marking. In addition, in the present application, the difference between the predicted sample skin area and the effective sample skin area may be determined by a cross entropy loss function, a mean square error loss function, etc. as the skin segmentation loss. Here, the third training end condition may include the skin segmentation loss being less than the first threshold, and / or the number of iterations reaching the first iteration number threshold, etc. The present application does not limit the first threshold and the first iteration number threshold.
[0125] Exemplarily, the training process of the skin segmentation model is as follows Figure 3 As shown, the training process introduces auxiliary branches including segmentation head 1, segmentation head 2, segmentation head 3 and segmentation head 4. The segmentation head 1 is used to perform semantic classification on the feature map output by the fifth network layer 405 to obtain a first auxiliary segmentation result for the third sample image (the first auxiliary segmentation result is used to indicate the predicted skin area predicted by the segmentation head 1). The segmentation head 2 is used to perform semantic classification on the feature map output by the sixth network layer 406 to obtain a second auxiliary segmentation result for the third sample image (the second auxiliary segmentation result is used to indicate the sample skin area predicted by the segmentation head 2). The segmentation head 3 is used to perform semantic classification on the feature map output by the seventh network layer 407 to obtain a first auxiliary segmentation result for the third sample image (the third auxiliary segmentation result is used to indicate the sample skin area predicted by the segmentation head 3). The segmentation head 4 is used to perform semantic classification on the feature map output by the eighth network layer 408 to obtain a fourth auxiliary segmentation result for the third sample image (the fourth auxiliary segmentation result is used to indicate the sample skin area predicted by the segmentation head 4).
[0126] Then, the difference between the first auxiliary segmentation result and the valid sample skin area, the difference between the second auxiliary segmentation result and the valid sample skin area, the difference between the third auxiliary segmentation result and the valid sample skin area, the difference between the fourth auxiliary segmentation result and the valid sample skin area, and the difference between the predicted sample skin area and the valid sample skin area are fused (summing up the differences, weighted summing up, and multiplying the differences) to obtain the skin segmentation loss. The network parameters of the first branch network, the second branch network, and the five segmentation heads are adjusted according to the skin segmentation loss until the third training end condition is reached to obtain the skin segmentation model. The skin segmentation model may include a first neural network, a second neural network, and a target segmentation head.
[0127] Among them, the structures of segmentation head 1, segmentation head 2, segmentation head 3 and segmentation head 4 can be different, which are similar to the structure of the target segmentation head. The difference is that the parameters of the network layer of each segmentation head are different. The specific parameters of each network in each segmentation head can be determined according to the size of the input data input to each segmentation head, which will not be repeated here.
[0128] In some embodiments, Figure 6 As shown, S140 may include the following S210-S230:
[0129] S210, encoding the target skin image, the skin occlusion image, and the facial key point map respectively to obtain a first feature map of the target skin image, a second feature map of the skin occlusion image, and a third feature map of the facial key point map.
[0130] The target skin image, the skin occlusion image and the facial key point map can be encoded by an encoder to obtain a first feature map of the target skin image, a second feature map of the skin occlusion image and a third feature map of the facial key point map.
[0131] In some embodiments, the encoder may be an encoder in a variational auto-encoder (VAE).
[0132] S220: Based on the first feature map, the second feature map, the third feature map and the skin mask, perform back diffusion processing on the noise image to obtain a skin quality fusion feature map.
[0133] As mentioned above, the sizes of the skin occlusion image, the skin mask and the facial key point map are consistent, so the first feature map, the second feature map and the third feature map obtained are also consistent in size. A random noise image with the same size as the first feature map can be randomly generated as the aforementioned noise image, and then based on the first feature map, the second feature map, the third feature map and the skin mask, the noise image is back-diffused to obtain a skin quality fusion feature map.
[0134] The process of back diffusion is to gradually denoise the noise image to restore the skin quality fusion feature map from the noise image. In the process of back diffusion, the first feature map, the second feature map, the third feature map and the skin mask are all used as conditions for back diffusion.
[0135] S230: Decode the skin quality fusion feature map to obtain a skin quality fusion image that is fused with the skin quality presented by the target skin image.
[0136] The skin quality fusion feature map can be decoded by a decoder having an image decoding function to output a skin quality fusion image. In some embodiments, the decoder can be a decoder in a variational self-encoder.
[0137] In some embodiments, the above S210-S230 can be implemented by a skinning model. The skinning model can integrate an encoder, an aggregation network that implements the above S220 process, and a decoder. Figure 7 As shown, the target skin image is obtained ( Figure 7 6011 and the second target skin image 6012), a skin occlusion image 602, a noise image 603, a skin mask 604, and a facial key point map 605. Figure 7 6011 and the second target skin image 6012 in the embodiment), the skin occlusion image 602, the noise image 603, the skin mask 604 and the facial key point map 605, and the skin quality is adjusted according to the process in the aforementioned embodiment using the skin replacement model to obtain a skin quality fusion image ( Figure 7 2 , two skin quality fusion images are exemplarily shown, namely a first skin quality fusion image 6061 fused with the skin quality presented by the first target skin image 6011 and a second skin quality fusion image 6062 fused with the skin quality presented by the second target skin image 6012 ).
[0138] In the above embodiment, the principle of back diffusion is used to recover the skin quality fusion feature map from the noise image, and then decode it to obtain the skin quality fusion image.
[0139] In some embodiments, the skin replacement model may include a first neural network and a second neural network, and the first neural network and the second neural network may perform back diffusion processing on the noise image based on the first feature map, the second feature map, the third feature map and the skin mask to obtain a skin quality fusion feature map. Figure 8 As shown, S220 may include:
[0140] S310: Perform feature extraction on the first feature map through a first neural network to obtain a feature extraction result.
[0141] S320, using the second feature map, the third feature map and the skin mask as diffusion conditions, performing reverse diffusion processing on the noise image based on the feature extraction result through the second neural network to obtain a skin quality fusion feature map.
[0142] The process of performing reverse diffusion processing on the noisy image through the second neural network is to perform multiple rounds of denoising processing on the noisy image through the second neural network, wherein the result of the previous round of denoising processing is used as the input of the next round of denoising processing.
[0143] In some embodiments, the feature extraction result can be used as the input of each round of denoising, that is, the input of each round of denoising includes the feature extraction result and the result of the previous round of denoising. Of course, the input of the first round of denoising includes the feature extraction result and the fusion result obtained by fusing the second feature map, the third feature map and the skin mask. In the above process, it is equivalent to sharing the feature extraction result in multiple rounds of denoising, so that the feature extraction result participates in multiple rounds of denoising. Since the feature extraction result reflects the skin quality characteristics of the skin to be generated, therefore, the feature extraction result is involved in multiple rounds of denoising, which can avoid some features in the skin quality characteristics being misidentified as noise in multiple rounds of denoising.
[0144] Among them, in each round of denoising processing, the feature extraction result and the result of the previous round of denoising processing can be fused based on the attention mechanism, and noise prediction is performed based on the attention mechanism fusion result, and denoising processing is performed based on the predicted noise.
[0145] In some embodiments, the first neural network and the second neural network may both include multiple network layers, and the structures of the first neural network and the second neural network are the same, and the feature extraction result includes the features extracted by each network layer in the first neural network for the first feature map. On this basis, in each round of denoising process, the features extracted by the mth network layer in the first neural network for the first feature map in the feature extraction result can be fused into the mth network layer in the second neural network, that is, the input of the mth network layer in the second neural network includes the output of the previous layer in the second neural network and the features extracted by the mth network layer in the first neural network for the first feature map. Wherein, m is a positive integer, and m does not exceed the total number of network layers of the first neural network.
[0146] It is understandable that in each round of denoising, if the results extracted from each network layer of the first neural network are fused into the corresponding network layer of the second neural network, the number of feature fusions is large, and multiple rounds of denoising are involved, so there are more features to be fused overall, resulting in a complex processing process and more processing resources occupied. Based on this, the features extracted from some network layers in the first neural network can be fused into the corresponding network layers in the second neural network.
[0147] In some embodiments, the first neural network includes K layers of first self-attention layers, and the second neural network includes K layers of second self-attention layers; the feature extraction result includes the first self-attention features input by each first self-attention layer; K is an integer greater than 1; S320 may include: using the second feature map, the third feature map and the skin mask as diffusion conditions, through the second neural network, sequentially performing M rounds of denoising on the noisy image based on the feature extraction result, and using the output feature map obtained by the Mth round of denoising as the skin quality fusion feature map; M is a positive integer greater than 1.
[0148] Among them, the denoising process of the i-th round is as follows: Fig. 9 As shown, including:
[0149] S410, obtaining an input feature map of the i-th round of denoising processing.
[0150] Among them, the input feature map of the first round of denoising processing is obtained by fusing the second feature map, the third feature map, the skin mask and the noise feature map corresponding to the noise image; when 1<i≤M, the input feature map of the i-th round of denoising processing is the output feature map of the i-1th round of denoising processing.
[0151] S420. In the j-th second self-attention layer, attention calculation is performed based on the first self-attention feature input from the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising processing, to obtain a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing.
[0152] Wherein, 1≤j≤K, j is a positive integer; the second self-attention feature of the second self-attention layer of the first layer for the i-th round of denoising processing is determined based on the input feature map of the i-th round of denoising processing.
[0153] S430, denoising the fused feature map output by the j-th second self-attention layer to obtain the j-th intermediate denoised feature map for the i-th round of denoising.
[0154] S440: If j is less than K, based on the jth intermediate denoising feature map for the i-th round of denoising, determine the second self-attention feature of the j+1th second self-attention layer for the i-th round of denoising, and add j to 1. Then, return to step S420.
[0155] S450: If j=K, the j-th intermediate denoising feature map for the i-th round of denoising is used as the output feature map of the i-th round of denoising.
[0156] The denoising round M can be set based on demand, for example, M is 50, 60, 80, 100, etc. The noise image can be encoded by the aforementioned encoder to obtain the encoded result as the noise feature map corresponding to the noise image.
[0157] In the above embodiment, the self-attention layer in the first neural network is called the first self-attention layer, and the self-attention layer in the second neural network is called the second self-attention layer. Among them, the first self-attention layer and the second self-attention layer both perform attention calculations according to the self-attention mechanism. The j-th second self-attention layer as above refers to the j-th second self-attention layer in the second neural network along the direction from input to output. Similarly, the j-th first self-attention layer refers to the j-th first self-attention layer in the first neural network along the direction from input to output.
[0158] It can be understood that the first self-attention feature input to each first self-attention layer (e.g., the jth first self-attention layer) is determined based on the output of the previous network layer of the jth first self-attention layer in the first neural network. Similarly, the second self-attention feature of the jth second self-attention layer for the i-th round of denoising is determined based on the output of the previous network layer of the jth second self-attention layer in the second neural network for the i-th round of denoising.
[0159] The second feature map, the third feature map, the skin mask, and the noise feature map corresponding to the noise image can be fused to obtain a second fused feature map, and the second fused feature map is obtained as the input feature map required for the first round of denoising. For each round of denoising, the output feature map obtained after the previous round of denoising is used as the input feature map required for the next round of denoising, until the Mth round of denoising is completed, and the output feature map obtained from the Mth round of denoising is used as the skin fused image, thereby achieving M rounds of denoising.
[0160] The feature extraction result obtained by the first neural network based on the first feature map includes the first self-attention features of each of the K first attention layers. For example, if K is 6, the feature extraction result includes the first self-attention features of each of the 6 first attention layers. However, the feature extraction result is only processed once, and there is no multiple rounds of processing. Therefore, the first self-attention features of each in the feature extraction result are reused in multiple rounds of denoising processing.
[0161] In some embodiments, the first self-attention feature includes a first key vector and a first value vector; the second self-attention feature includes a second key vector, a second value vector, and a target query vector; in the j-th second self-attention layer, attention calculation is performed based on the first self-attention feature input by the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising processing, and a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing is obtained, including:
[0162] In the j-th second self-attention layer, the following steps A1-A4 are performed:
[0163] A1, fuse the first key vector and the second key vector of the j-th second self-attention layer for the i-th round of denoising to obtain the target key vector of the j-th second self-attention layer for the i-th round of denoising;
[0164] A2, fuse the first value vector and the second value vector of the j-th second self-attention layer for the i-th round of denoising to obtain the target value vector of the j-th second self-attention layer for the i-th round of denoising;
[0165] A3, determining the attention score of the second self-attention layer of the jth layer for the i-th round of denoising according to the target query vector and the target key vector of the second self-attention layer of the jth layer for the i-th round of denoising;
[0166] A4, weight the corresponding target value vector according to the attention score of the second self-attention layer of the jth layer for the i-th round of denoising processing, and obtain the fused feature map output by the second self-attention layer of the jth layer for the i-th round of denoising processing.
[0167] Among them, the first key vector and the second key vector of the j-th second self-attention layer for the i-th round of denoising processing can be spliced to obtain the target key vector of the j-th second self-attention layer for the i-th round of denoising processing; similarly, the first value vector and the second value vector of the j-th second self-attention layer for the i-th round of denoising processing are spliced to obtain the target value vector of the j-th second self-attention layer for the i-th round of denoising processing.
[0168] After obtaining the target key vector, target value vector and target query vector of the j-th second self-attention layer for the i-th round of denoising, the attention score of the j-th second self-attention layer for the i-th round of denoising is determined according to the target query vector and target key vector of the j-th second self-attention layer for the i-th round of denoising; then, the corresponding target value vector is weighted according to the attention score of the j-th second self-attention layer for the i-th round of denoising to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising.
[0169] The attention operation process of the second attention layer of the jth layer for the i-th round of denoising can be briefly described as Formula 6, which is as follows:
[0170] (six)
[0171] Among them, y refers to the fused feature map of the second attention layer of the jth layer in the i-th round of denoising. is the target query vector for the denoising process of the second self-attention layer of the jth layer, is the first key vector of the first self-attention layer of the jth layer, is the first value vector of the first self-attention layer of the jth layer, is the second key vector of the second self-attention layer of the jth layer for the ith round of denoising, is the second value vector of the second self-attention layer of the jth layer for the i-th round of denoising processing, and cat is the concatenation processing.
[0172] In this embodiment, the attention operation process of the i-th round of denoising is as follows: Fig.10 As shown, the first self-attention feature of the first self-attention layer of the jth layer includes a first key vector K11 and a first value vector V11, the second self-attention feature of the second self-attention layer of the jth layer includes a second key vector K22, a second value vector V22 and a target query vector Q00, the first key vector K11 and the second key vector K22 are concatenated to obtain a target key vector, the first value vector V11 and the second value vector V22 are concatenated to obtain a target value vector, and then the target value vector, the target key vector and the target query vector Q00 are used to determine the fused feature map output by the second self-attention layer of the jth layer for the i-th round of denoising processing.
[0173] In some embodiments, the first neural network and the second neural network can be Unet networks with the same structure, wherein the Unet network includes an upsampling network and a downsampling network, wherein the upsampling network and the downsampling network both include multi-layer transformer networks, and each transformer network involves residual connections (ResNet) and self-attention layers. In this case, the j-th first self-attention layer above is the self-attention layer in the j-th transformer network in the first neural network, and similarly, the j-th second self-attention layer is the self-attention layer in the j-th transformer network in the second neural network.
[0174] It is worth mentioning that the number of input channels of the Unet network is 4, and in the present application, the input of the second neural network includes a 4-channel noise feature map (after encoding the 3-channel noise image through the encoder, the number of channels becomes 4), a 4-channel second feature map (after encoding the 3-channel skin occlusion image through the encoder, the number of channels becomes 4), a 4-channel third feature map (after encoding the 3-channel facial key points through the encoder, the number of channels becomes 4) and a 1-channel skin mask. At this time, the number of input channels of the second neural network is 13. Therefore, in the present application, it is also necessary to adjust the number of input channels of the Unet network based on the Unet network to obtain the second neural network.
[0175] Similarly, the input of the first neural network includes a 4-channel first feature map (after encoding the 3-channel target skin image through the encoder, the number of channels becomes 4). In this application, the parameter-initialized Unet network can be directly obtained as the first neural network.
[0176] It should be noted that the attention parameter accounts for a relatively small proportion in the entire skin-changing model (including the first neural network and the second neural network), but the attention parameter plays an important role in the skin-changing model. Therefore, in the present application, the self-attention feature determined by the first self-attention layer of the first neural network layer is passed to the second self-attention layer in the second neural network, and the second self-attention layer fuses the self-attention features determined by the self-attention layers in the first neural network and the second neural network. It is not necessary to fuse all the network layers in the first neural network and the second neural network, and efficient fusion of the feature information of the first neural network and the second neural network can be achieved.
[0177] In this embodiment, a skin quality fusion feature map is generated through multiple rounds of denoising processing, and in each round of denoising processing, the self-attention features corresponding to the first self-attention layer and the second self-attention layer are fused, so as to achieve accurate fusion of the output of the first neural network layer and the output of the second network layer in each round of denoising processing, so that the output feature map obtained by each round of denoising processing can restore the skin quality in the target skin image under the constraint of the target skin image, and retain the information of the area outside the skin area in the target image under the constraints of the facial key point map, skin mask and skin occlusion image in the output feature map obtained by each round of denoising processing, thereby making the skin quality fusion feature map obtained by multiple rounds of denoising processing more accurate, thereby improving the accuracy of the generated skin quality fusion image.
[0178] At the same time, the fusion of self-attention features corresponding to the first self-attention layer and the second self-attention layer does not require the fusion of data from all network layers in the first neural network layer and the second neural network layer, which greatly reduces the amount of data fusion and improves the skin quality fusion efficiency, thereby quickly obtaining a skin quality fusion image.
[0179] In some embodiments, the diffusion conditions in S320 may also include object description text for the target image, that is, S320 may include: using the object description text, the second feature map, the third feature map and the skin mask as diffusion conditions, through the second neural network, performing back diffusion processing on the noise image based on the feature extraction result to obtain a skin quality fusion feature map.
[0180] In this embodiment, the object description text of the target image is used to describe the object in the target image (i.e., the object to which the skin area belongs). The object description text can describe other information of the object in the target image except the skin. For example, the object description text can be "Asian woman with long hair wearing a blue coat".
[0181] In some embodiments, a technician may manually write text to describe the object in the target image according to the image content presented by the target image, as the object description text of the target image. In other embodiments, the object description text of the target image may also be generated by a text generation model, wherein the text generation model may be a neural network capable of generating text according to an image, which is not limited in this application.
[0182] When the diffusion condition can also include object description text for the target image, the input feature map of the first round of denoising processing is obtained by fusing the second feature map, the third feature map, the skin mask, the text encoding result of the object description text and the noise feature map corresponding to the noise image.
[0183] In the above embodiment, the information of the area other than the skin in the target image is further constrained by the object description text, so that the difference between the other areas other than the skin in the generated skin quality fusion image and the other areas other than the skin in the target image is smaller, and the skin quality fusion image and the target image have higher consistency in person. The training process of the second neural network is described in the following embodiment and will not be repeated here.
[0184] Specifically, the text encoding result, the second feature map, the third feature map, the skin mask and the noise feature map can be fused into the aforementioned second fused feature map, and then the second fused feature map is used as the input feature map of the first round of denoising processing, and the aforementioned M rounds of denoising processing are performed to obtain a skin fusion image, which will not be repeated here.
[0185] In some embodiments, the training process of the text generation model may include the following steps B1-B6:
[0186] B1, extract features of the second sample image through the visual encoder to obtain the first sample features;
[0187] B2, extracting features of the sample description text corresponding to the second sample image through a text encoder to obtain second sample features;
[0188] B3, performing content consistency determination on the first sample feature and the sample description text through a visual text encoder to obtain a sample determination result;
[0189] B4, generating a predicted sample description text according to the first sample encoding result and the sample description text through a text generation model;
[0190] B5, determining the text generation model loss based on the difference between the predicted sample description text and the sample description text, the difference between the first sample feature and the second sample feature, and the sample determination result;
[0191] B6, adjust the parameters of the visual encoder, text encoder, visual text encoder and text generation model through the text generation model loss until the second training end condition is reached.
[0192] The second sample image may be any type of image, and may be an image including or excluding a human face. The descriptive text used to describe the subject object in the second sample image serves as the sample descriptive text of the second sample image. For example, the second sample image is an image including a woman, and the sample descriptive text may be text used to describe the woman's clothing and appearance.
[0193] In the present application, the visual encoder may be the visual encoder in Blip (Bootstrapping Language—Image Pre-tarining, bootstrapping pre-trained language image multimodal model), the text encoder may be the text encoder in Blip, the visual text encoder may be the visual text encoder in Blip, and the text generation model may be the visual text decoder in Blip.
[0194] In the above embodiment, a text image pair can be pre-constructed, wherein a text image pair includes a second sample image and a sample description text, and a label of each text image pair is determined. The label of the text image pair is used to indicate whether the content of the second sample image in the text image pair and the sample description text in the text image pair are consistent. For example, if the content of the second sample image and the sample description text in the same text image pair are consistent, the label of the text image pair can be determined to be 1, otherwise, the label is determined to be 0. On this basis, in the above embodiment, the sample description text corresponding to the second sample image refers to the sample description text in the same text image pair as the second sample image.
[0195] The judgment loss can be determined based on the sample judgment result and the label of the text image pair; and the difference between the first sample feature and the second sample feature can be determined by a loss function such as a cross entropy loss function or a mean square error loss function as a picture-text encoding loss; and the difference between the predicted sample description text and the sample description text can be determined by a loss function such as a cross entropy loss function or a mean square error loss function as a text prediction loss.
[0196] Then, the judgment loss, image-text encoding loss, and text prediction loss are summed, weighted summed, and averaged to obtain the text generation model loss. The second training end condition as above may include the visual text encoder loss being less than a second threshold, and / or the number of iterations reaching a second iteration number threshold, etc. This application does not limit the second threshold and the second iteration number threshold.
[0197] For example, the training process of the text generation model is as follows Fig.11 As shown, the visual encoder is composed of Q first network modules, each of which includes a self-attention layer and a feedforward network. The visual encoder extracts features from the image encoding result of the second sample image to obtain the first sample features.
[0198] The text encoder includes P second network modules, each of which includes a dual self-attention layer, a cross-attention layer and a feedforward network. The text encoder performs feature extraction on the text encoding result of the sample description text corresponding to the second sample image to obtain the second sample feature.
[0199] The visual text encoder may include a bidirectional self-attention layer, a cross-attention layer, and a feedforward network. The visual text encoder performs consistency judgment on the first sample feature and the vector encoded by the sample description text to obtain a sample judgment result.
[0200] The text generation model may include a multi-head self-attention layer, a cross-attention layer, and a feedforward network. The text generation model generates a predicted sample description text based on the encoded vector of the sample description text and the first sample feature.
[0201] Then, based on the difference between the predicted sample description text and the sample description text, the difference between the first sample feature and the second sample feature, and the sample judgment result, the text generation model loss is determined, and then the parameters of the visual encoder, text encoder, visual text encoder and text generation model are adjusted according to the text generation model loss to obtain the text generation model.
[0202] When the text generation model is obtained, the object description text of the target image is determined based on the text generation model, such as Fig.12 As shown, the target image is input into the text generation model to generate object description text, and the obtained object description text is: a long-haired Asian woman carrying a backpack.
[0203] The object description text is extracted by the trained text generation model, and the object description text of the target image does not need to be manually edited, thereby improving the generation efficiency of the object description text.
[0204] In some embodiments, Fig.13 As shown, before step S310, the method may include:
[0205] S510, obtaining a sample skin image, a sample skin occlusion image, a sample skin mask, and a sample facial key point map.
[0206] Among them, the sample skin image is determined based on the sample skin area in the first sample image, the sample skin occlusion image is obtained by occluding the sample skin area in the first sample image, the sample skin mask is a mask of the sample skin area, and the sample facial key point map is obtained by extracting key points of the sample facial area in the first sample image; the sample skin area includes the facial skin area in the sample facial area.
[0207] The first sample image may be an image including a human face. The skin area in the first sample image is used as the sample skin area, and the face area in the first sample image is used as the sample face area.
[0208] In some implementations, a large number of initial sample images may be generated through an image generation model, and then images meeting a screening condition may be screened from the initial sample images as first sample images.
[0209] The image generation model may be a Stable Diffusion XL (SDXL) series model. In some embodiments, a prompt text may be input into the image generation model, and the image generation model generates an image according to the prompt text and outputs an initial sample image. The prompt text is used to guide the image generation model to generate an image.
[0210] For example, the prompt text may be text that satisfies the text template "gender, age, number of people in the image, quality of people in the image, clothing of the people, requirements for the face of the people, requirements for the skin of the people, image resolution, image quality, and requirements for image authenticity". Among them, gender can be male or female, the optional range of age is {5,10,20,30,40,50,60,70}, the quality of the person in the image is used to indicate the clarity and size of the person in the image (the higher the quality of the person in the image, the higher the clarity of the person and the larger the person; conversely, the lower the quality of the person in the image, the lower the clarity of the person and the smaller the person), the face requirements can include whether the face is symmetrical, the face shape, and the fatness of the face, etc., the skin requirements can include fine skin and rough skin, etc., the image quality refers to the quality of the image itself, which can include low quality and high quality, etc., and the image quality can be indicated by the aesthetic quality score below; the image authenticity requirement is used to indicate the authenticity of the image. The higher the image authenticity requirement, the more similar the image is to the real image obtained by shooting a real person, and the lower the image authenticity requirement, the more similar the image is to an automatically generated virtual image. Under this text template, by changing the input gender and age, a large number of initial sample images of multiple age groups and different genders can be generated.
[0211] In some embodiments, the screening condition may include screening the initial sample images whose image scores are higher than the score threshold as the first sample image. The image score may be the aesthetic quality score of the scoring standard used by the Laion dataset, which is used to evaluate the quality of the image. The aesthetic quality score ranges from [0,10]. The higher the aesthetic quality score, the better the quality of the image. In this embodiment, the score threshold may be 6.5.
[0212] In some embodiments, the screening condition may also be to screen the initial sample image whose facial area ratio reaches a ratio threshold as the first sample image; wherein the ratio threshold may be 1 / 16. The facial area detection model (e.g., RetinaFace Model, high-resolution face recognition model) may be used to detect the facial area of the initial sample image to obtain the position information of the facial area in the initial sample image. Then, based on the position information of the identified facial area, the facial area in the initial sample image is determined, and then the ratio of the facial area is calculated. The ratio may be an area ratio, i.e., the ratio of the pixel area of the facial area to the pixel area of the initial sample image, or a pixel number ratio, i.e., the ratio of the number of pixels in the facial area to the total number of pixels in the initial sample image.
[0213] The position information of the face region in the initial sample image may be the position information of the detection frame surrounding the face region. Fig.14 As shown in 801 in FIG. , the detection frame is, for example, Fig.14 For the rectangular box 802 in the image, the output position information may be (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the rectangular box, and w and h are the width and height of the rectangular box, respectively.
[0214] Based on this screening condition, the facial area in the first sample image is larger, so that the first sample image includes more facial information, which improves the effectiveness and accuracy of the first sample image, and further makes the model recognition ability of the first neural network and the second neural network trained according to the first sample image stronger.
[0215] At the same time, the smaller the proportion of the face in the image, the greater the possibility of face distortion and blur, and the larger the proportion of the face in the image, the smaller the possibility of face distortion and blur. Therefore, the initial sample image whose face area proportion reaches the proportion threshold is selected as the first sample image. The larger the proportion of the face area in the first sample image, the smaller the possibility of face distortion and blur in the face area of the first sample image, so as to effectively ensure the subsequent training effect.
[0216] In some other embodiments, the screening condition may include screening initial sample images whose image scores are higher than a score threshold and whose facial area ratio reaches a ratio threshold as the first sample image.
[0217] In some embodiments, before S510, the method further includes: performing skin region segmentation on the first sample image to determine a sample skin region in the first sample image; performing mask processing on the sample skin region to obtain a sample skin mask; performing occlusion processing on the sample skin region in the first sample image based on the sample skin mask to obtain a sample skin occlusion image; processing the first sample image based on the sample skin mask to obtain a sample skin image; performing key point extraction on the sample face region in the first sample image to obtain a sample face key point map. The specific process of determining the sample skin mask, the sample skin region, and the sample face key point map here refers to the specific process of determining the skin mask, the skin region, and the face key point map described above, and will not be repeated here.
[0218] Among them, performing skin area segmentation on the sample image and determining the sample skin area in the sample image includes: performing spatial detail feature extraction on the first sample image through a first branch network to obtain a sample spatial feature map; performing semantic feature extraction on the first sample image through a second branch network to obtain a sample semantic feature map; performing downsampling processing on the sample spatial feature map to obtain a downsampled spatial feature map; performing upsampling processing on the sample semantic feature map to obtain an upsampled semantic feature map; aggregating the sample semantic feature map with the downsampled spatial feature map to obtain a first sample aggregated feature map; aggregating the sample spatial feature map with the upsampled semantic feature map to obtain a second sample aggregated feature map; fusing the first sample aggregated feature map and the second sample aggregated feature map to obtain a sample feature map; and determining the sample skin area in the first sample image based on the sample feature map.
[0219] The process of determining the sample skin area in the sample image by using the skin segmentation model here refers to the aforementioned process of determining the skin area of the target image, and will not be described in detail.
[0220] Among them, the sample skin area in the sample image is occluded based on the sample skin mask to obtain the sample skin occluded image, including: normalizing the first sample image to a first pixel value range to obtain a first intermediate sample image; normalizing the sample skin mask to a second pixel value range to obtain a first sample skin mask, and the second pixel value range is within the first pixel value range; performing a dot product of the first intermediate sample image and the first sample skin mask to obtain a second intermediate image; and performing a denormalization process on the second intermediate image to obtain a sample skin occluded image.
[0221] Among them, segmenting the sample skin area from the sample image based on the sample skin mask to obtain the sample skin image includes: normalizing the sample skin mask to a second pixel value range to obtain a second sample skin mask; and performing dot multiplication on the second sample skin mask and the first sample image to obtain the sample skin image.
[0222] The specific process of determining the sample skin mask, the sample skin area and the sample facial key point map here refers to the specific process of determining the skin mask, the skin area and the facial key point map mentioned above, which will not be repeated here.
[0223] You can follow Fig.15 The process shown determines other training data for training the skin replacement model based on the first sample image, specifically including: a sample facial key point map, a sample skin occlusion image, a sample object description text, a sample skin mask, and a sample skin image. Among them, the facial key points of the first sample image can be extracted to obtain a sample facial key point map, the skin area of the first sample image can be segmented to obtain a sample skin area, and a sample skin mask is determined according to the sample skin area. Then, the sample skin mask and the first sample image are used to determine the sample skin image and the sample skin occlusion image. At the same time, a sample object description text for the first sample image can be generated by a text generation model: a long-haired Asian woman carrying a backpack.
[0224] S520, respectively encode the sample skin image, the sample skin occlusion image, and the sample facial key point map to obtain a first sample feature map of the sample skin image, a second sample feature map of the sample skin occlusion image, and a third sample feature map of the sample facial key point map.
[0225] The encoder in the variational autoencoder can be used to encode the sample skin image, the sample skin occlusion image and the sample facial key point map respectively to obtain a first sample feature map of the sample skin image, a second sample feature map of the sample skin occlusion image and a third sample feature map of the sample facial key point map.
[0226] S530, extract features from the first sample feature map through the first neural network to obtain a sample feature extraction result; use the second sample feature map, the third sample feature map and the sample skin mask as diffusion conditions, perform reverse diffusion processing on the sample noise image based on the sample feature extraction result through the second neural network to obtain a sample skin quality fusion feature map.
[0227] The specific implementation process of S530 is similar to the process of determining the copy fusion feature map above, and will not be repeated here. Similarly, the sample object description text of the first sample image can also be used as a diffusion condition, wherein the sample object description text of the first sample image can be automatically generated by the above-mentioned trained text generation model.
[0228] S540: Decode the sample skin quality fusion feature map to obtain a sample skin quality fusion image.
[0229] After obtaining the sample skin quality fusion feature map, the sample skin quality fusion feature map can be decoded by a decoder to generate an image, and the generated image is used as the sample skin quality fusion image.
[0230] S550: Based on the difference between the sample skin quality fusion image and the first sample image, adjust the parameters of the first neural network and the second neural network until the first training end condition is met.
[0231] In the present application, the difference between the sample skin quality fusion image and the sample image can be determined as the fusion loss through the cross entropy loss function or the mean square error loss function, and the parameters of the first neural network and the second neural network can be adjusted through the fusion loss.
[0232] Among them, the first training end condition may include that the fusion loss is less than the third threshold, and / or the number of iterations reaches the third iteration number threshold, etc. This application does not limit the third threshold and the third iteration number threshold. When the first training end condition is reached, the first neural network and the second neural network after the last parameter adjustment can be obtained, which are used to extract the feature extraction result of the first feature map and perform inverse diffusion processing to obtain the skin quality fusion feature map.
[0233] In this embodiment, the training process of the skin-changing model is as follows: Fig.16 As shown, the sample skin image 1501 can be encoded by an encoder to obtain a first sample feature map, the sample skin occlusion image 1502 can be encoded by an encoder to obtain a second sample feature map, the sample facial key point map 1503 can be encoded by an encoder to obtain a third sample feature map, the first sample image 1504 can be encoded by an encoder to obtain a sample encoding feature map, and random noise is added to the sample encoding feature map to obtain a sample noise feature map corresponding to the sample noise image; then, the first sample feature map is subjected to feature extraction by a first neural network to obtain a sample feature extraction result, and the sample text encoding result after the sample object description text is encoded, the second sample feature map, the third sample feature map and the sample skin mask 1505 are used as diffusion conditions, and the sample noise feature map sample is subjected to reverse diffusion processing based on the sample feature extraction result by a second neural network to obtain a sample skin quality fusion feature map, and then the sample skin quality fusion feature map is decoded to obtain a sample skin quality fusion image.
[0234] Among them, in each round of denoising process of the reverse diffusion process, the sample feature extraction result extracted by the first neural network includes the first sample self-attention feature output by the first autonomous attention layer: the first sample key vector K1 and the first sample value vector V1, and the second sample self-attention feature output by the second autonomous attention layer of the second neural network includes the second sample key vector K2, the second sample value vector V2 and the target sample query vector Q0. The attention calculation is performed through the first sample key vector K1, the first sample value vector V1, the second sample key vector K2, the second sample value vector K2 and the target sample query vector Q0 to realize the fusion of skin quality and obtain the output sample feature map of this round of denoising. The output sample feature map obtained by the last round of denoising is obtained as the sample skin quality fusion feature map.
[0235] After obtaining the sample skin quality fusion image according to the sample skin quality fusion feature map, the parameters of the first neural network and the second neural network can be adjusted according to the difference between the sample skin quality fusion image and the first sample image until the first training end condition is reached, and the first neural network and the second neural network obtained after the training are obtained as the skin replacement model.
[0236] In this embodiment, the first neural network and the second neural network are trained so that the trained first neural network can accurately extract a skin quality feature map indicating skin quality, and the trained second neural network can accurately perform back diffusion processing based on the feature extraction results extracted by the first neural network to obtain a skin quality fusion feature map, thereby improving the accuracy of the skin quality fusion image generated according to the skin quality fusion feature map.
[0237] At the same time, the sample object description text is extracted through the trained text generation model, and the sample object description text of the second sample image does not need to be manually edited, which improves the generation efficiency of the sample object description text and further improves the training efficiency of the first neural network and the second neural network.
[0238] The first neural network trained through this embodiment can be used to extract features of target skin images of different skin types, so that a skin fusion image integrating different skin types can be generated based on the first neural network and the second neural network. There is no need to train a first neural network for each skin type. The skin resurfacing model can be applicable to different skin types, thereby improving the adaptability of the skin resurfacing model.
[0239] The first sample image used to train the first neural network and the second neural network is generated by an image generation model, and in the training process, the sample skin occlusion image, sample object description text, sample skin mask and sample skin image involved can be automatically generated according to the first sample image. There is no need to manually obtain the first sample image and label the first sample image, thus realizing zero-shot training (zero-shot training also means training with zero manually labeled samples), which greatly improves the training efficiency of the first neural network and the second neural network.
[0240] Furthermore, the features determined by the self-attention layers in the first neural network and the second neural network of the present application are fused, and there is no need to fuse the features determined by all network layers in the first neural network and the second neural network, thereby greatly reducing the parameters used to fuse the features in the first neural network and the second neural network, reducing the parameters that need to be adjusted during the training of the first neural network and the second neural network, and improving the training efficiency of the first neural network and the second neural network.
[0241] See also Fig.17 , Fig.17 A block diagram of an image processing device proposed in one embodiment of the present application is shown. The image processing device 1000 includes:
[0242] The occlusion module 1010 is used to perform occlusion processing on the skin area in the target image to obtain a skin occlusion image;
[0243] The extraction module 1020 is used to extract key points from the face area in the target image to obtain a face key point map; the skin area includes a facial skin area in the face area;
[0244] The mask module 1030 is used to perform mask processing on the skin area in the target image to obtain a skin mask;
[0245] The skin generation module 1040 is used to perform skin generation on the skin-occluded image according to the target skin image, the facial key point map and the skin mask, so as to obtain a skin-quality fused image that incorporates the skin quality presented by the target skin image.
[0246] Optionally, the skin generation module 1040 is also used to encode the target skin image, the skin occlusion image and the facial key point map respectively to obtain a first feature map of the target skin image, a second feature map of the skin occlusion image and a third feature map of the facial key point map; based on the first feature map, the second feature map, the third feature map and the skin mask, perform back diffusion processing on the noise image to obtain a skin quality fusion feature map; decode the skin quality fusion feature map to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image.
[0247] Optionally, the skin generation module 1040 is also used to perform feature extraction on the first feature map through the first neural network to obtain a feature extraction result; using the second feature map, the third feature map and the skin mask as diffusion conditions, through the second neural network, perform back diffusion processing on the noise image based on the feature extraction result to obtain a skin quality fusion feature map.
[0248] Optionally, the first neural network includes K layers of first self-attention layers, and the second neural network includes K layers of second self-attention layers; the feature extraction result includes the first self-attention features input by each first self-attention layer; K is an integer greater than 1; the skin generation module 1040 is also used to use the second feature map, the third feature map and the skin mask as diffusion conditions, through the second neural network, based on the feature extraction results, to sequentially perform M rounds of denoising on the noisy image, and use the output feature map obtained by the Mth round of denoising as the skin quality fusion feature map; M is a positive integer greater than 1; wherein the i-th round of denoising is performed according to the following process: the input feature map of the i-th round of denoising is obtained, wherein the input feature map of the first round of denoising is The second feature map, the third feature map, the skin mask and the noise feature map corresponding to the noise image are fused; when 1<i≤M, the input feature map of the i-th round of denoising is the output feature map of the i-1th round of denoising; in the j-th second self-attention layer, the attention calculation is performed based on the first self-attention feature input by the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising, and the fused feature map output by the j-th second self-attention layer for the i-th round of denoising is obtained; 1≤j≤K, j is a positive integer; the second self-attention feature of the first second self-attention layer for the i-th round of denoising is determined based on the input feature map of the i-th round of denoising; The fused feature map output by the j-th second self-attention layer is denoised to obtain the j-th intermediate denoised feature map for the i-th round of denoising; if j is less than K, based on the j-th intermediate denoised feature map for the i-th round of denoising, the second self-attention feature of the j+1-th second self-attention layer for the i-th round of denoising is determined, j is accumulated and added by 1, and the step of performing attention calculation in the j-th second self-attention layer based on the first self-attention feature input by the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising is returned to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising; if j=K, the j-th intermediate denoised feature map for the i-th round of denoising is used as the output feature map of the i-th round of denoising.
[0249] Optionally, the first self-attention feature includes a first key vector and a first value vector; the second self-attention feature includes a second key vector, a second value vector and a target query vector; the skin generation module 1040 is also used to perform the following processing in the j-th second self-attention layer: fuse the first key vector and the second key vector of the j-th second self-attention layer for the i-th round of denoising to obtain the target key vector of the j-th second self-attention layer for the i-th round of denoising; fuse the first value vector and the second value vector of the j-th second self-attention layer for the i-th round of denoising to obtain the target value vector of the j-th second self-attention layer for the i-th round of denoising; determine the attention score of the j-th second self-attention layer for the i-th round of denoising based on the target query vector and target key vector of the j-th second self-attention layer for the i-th round of denoising; weight the corresponding target value vector according to the attention score of the j-th second self-attention layer for the i-th round of denoising to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising.
[0250] Optionally, the image processing device 1000 also includes a training module for obtaining a sample skin image, a sample skin occlusion image, a sample skin mask, and a sample facial key point map; the sample skin image is determined based on a sample skin area in a first sample image, the sample skin occlusion image is obtained by occluding the sample skin area in the first sample image, the sample skin mask is a mask of the sample skin area, and the sample facial key point map is obtained by extracting key points from the sample facial area in the first sample image; the sample skin area includes a facial skin area in the sample facial area; the sample skin image, the sample skin occlusion image, and the sample facial key point map are encoded and processed respectively to obtain a first sample skin image. A sample feature map, a second sample feature map of a sample skin occluded image, and a third sample feature map of a sample facial key point map; extracting features from the first sample feature map through a first neural network to obtain a sample feature extraction result; using the second sample feature map, the third sample feature map, and the sample skin mask as diffusion conditions, performing reverse diffusion processing on the sample noise image based on the sample feature extraction result through a second neural network to obtain a sample skin texture fusion feature map; decoding the sample skin texture fusion feature map to obtain a sample skin texture fusion image; adjusting parameters of the first neural network and the second neural network based on the difference between the sample skin texture fusion image and the first sample image until the first training end condition is reached.
[0251] Optionally, the training module is also used to perform skin area segmentation on the first sample image to determine the sample skin area in the first sample image; perform mask processing on the sample skin area to obtain a sample skin mask; based on the sample skin mask, perform occlusion processing on the sample skin area in the first sample image to obtain a sample skin occlusion image; based on the sample skin mask, process the first sample image to obtain a sample skin image; perform key point extraction on the sample face area in the first sample image to obtain a sample face key point map.
[0252] Optionally, the training module is also used to normalize the first sample image to a first pixel value range to obtain a first intermediate sample image; normalize the sample skin mask to a second pixel value range to obtain a first sample skin mask, and the second pixel value range is within the first pixel value range; perform a dot product on the first intermediate sample image and the first sample skin mask to obtain a second intermediate image; and perform denormalization on the second intermediate image to obtain a sample skin occlusion image.
[0253] Optionally, the training module is further used to normalize the sample skin mask to a second pixel value range to obtain a second sample skin mask; and perform a dot multiplication on the second sample skin mask and the first sample image to obtain a sample skin image.
[0254] Optionally, the training module is also used to extract spatial detail features of the first sample image through the first branch network to obtain a sample spatial feature map; extract semantic features of the first sample image through the second branch network to obtain a sample semantic feature map; downsample the sample spatial feature map to obtain a downsampled spatial feature map; upsample the sample semantic feature map to obtain an upsampled semantic feature map; aggregate the sample semantic feature map with the downsampled spatial feature map to obtain a first sample aggregated feature map; aggregate the sample spatial feature map with the upsampled semantic feature map to obtain a second sample aggregated feature map; fuse the first sample aggregated feature map and the second sample aggregated feature map to obtain a sample feature map; and determine the sample skin area in the first sample image based on the sample feature map.
[0255] Optionally, the diffusion condition also includes an object description text generated by a text generation model based on the target image; the training module is also used to extract features of the second sample image through a visual encoder to obtain a first sample feature; extract features of the sample description text corresponding to the second sample image through a text encoder to obtain a second sample feature; perform content consistency judgment on the first sample feature and the sample description text through a visual text encoder to obtain a sample judgment result; generate a predicted sample description text based on the first sample encoding result and the sample description text through the text generation model; determine the text generation model loss based on the difference between the predicted sample description text and the sample description text, the difference between the first sample feature and the second sample feature, and the sample judgment result; adjust the parameters of the visual encoder, text encoder, visual text encoder and text generation model through the text generation model loss until the second training end condition is reached.
[0256] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.
[0257] See also Fig.18 , Fig.18 A structural block diagram of an electronic device for executing an image processing method according to an embodiment of the present application is shown. The electronic device may be Figure 1 The terminal device 400 or the server 200 in the embodiment of the present invention should be noted that: Fig.18 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0258] like Fig.18 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, ROM 1202 and RAM 1203 are connected to each other through a bus 1204. The input / output (I / O) interface 1205 is also connected to the bus 1204.
[0259] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.
[0260] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1209, and / or installed from a removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.
[0261] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0262] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer-readable instructions.
[0263] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0264] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.
[0265] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including computer-readable instructions, the computer-readable instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer-readable instructions from the computer-readable storage medium, and the processor executes the computer-readable instructions, so that the electronic device executes the method in any of the above embodiments.
[0266] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit of the module or unit function.
[0267] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0268] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.
[0269] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. It should be understood that the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
[0270] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An image processing method, It is characterized in that The method comprises: Performing occlusion processing on the skin area in the target image to obtain a skin occlusion image; Extracting key points from the face area in the target image to obtain a face key point map; the skin area includes a facial skin area in the face area; Masking the skin area in the target image to obtain a skin mask; the pixel values of the pixels in the non-skin area in the skin mask are different from the pixel values of the pixels in the skin area; the pixel values of different pixels in the non-skin area in the skin mask are the same, and the pixel values of different pixels in the skin area in the skin mask are the same; According to the target skin image, the facial key point map and the skin mask, skin generation is performed on the skin occlusion image to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image; in the process of skin generation, the facial key point map is used to constrain the structure of the facial area, the skin mask is used to specify the area in the skin occlusion image where skin generation needs to be performed, and the target skin image is used to specify the skin quality of the skin to be generated.
2. The method according to claim 1, It is characterized in that The step of generating skin from the skin-blocked image according to the target skin image, the facial key point map and the skin mask to obtain a skin-quality fused image that fuses the skin quality presented by the target skin image comprises: Encoding the target skin image, the skin occlusion image, and the facial key point map respectively to obtain a first feature map of the target skin image, a second feature map of the skin occlusion image, and a third feature map of the facial key point map; Based on the first feature map, the second feature map, the third feature map and the skin mask, the noise image is subjected to reverse diffusion processing to obtain a skin quality fusion feature map; The skin quality fusion feature map is decoded to obtain a skin quality fusion image that is fused with the skin quality presented by the target skin image.
3. The method according to claim 2, It is characterized in that The step of performing reverse diffusion processing on the noise image based on the first feature map, the second feature map, the third feature map and the skin mask to obtain a skin quality fusion feature map includes: Performing feature extraction on the first feature map by using a first neural network to obtain a feature extraction result; The second feature map, the third feature map and the skin mask are used as diffusion conditions, and the noise image is subjected to back diffusion processing based on the feature extraction result through a second neural network to obtain the skin quality fusion feature map.
4. The method according to claim 3, It is characterized in that The first neural network includes K first self-attention layers, and the second neural network includes K second self-attention layers; the feature extraction result includes the first self-attention features input by each of the first self-attention layers; K is an integer greater than 1; The method uses the second feature map, the third feature map and the skin mask as diffusion conditions, performs reverse diffusion processing on the noise image based on the feature extraction result through a second neural network to obtain the skin quality fusion feature map, including: The second feature map, the third feature map and the skin mask are used as diffusion conditions, and the noise image is subjected to M rounds of denoising in sequence based on the feature extraction result through a second neural network, and the output feature map obtained by the M-th round of denoising is used as the skin quality fusion feature map; M is a positive integer greater than 1; Among them, the i-th round of denoising is performed according to the following process: Obtaining an input feature map of the i-th round of denoising, wherein the input feature map of the first round of denoising is obtained by fusing the second feature map, the third feature map, the skin mask, and the noise feature map corresponding to the noise image; when 1<i≤M, the input feature map of the i-th round of denoising is the output feature map of the i-1-th round of denoising; In the j-th second self-attention layer, attention calculation is performed based on the first self-attention feature input from the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising processing, to obtain a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing; 1≤j≤K, j is a positive integer; the second self-attention feature of the first second self-attention layer for the i-th round of denoising processing is determined based on the input feature map of the i-th round of denoising processing; Performing denoising on the fused feature map output by the second self-attention layer of the j-th layer to obtain the j-th intermediate denoised feature map for the i-th round of denoising; If j is less than K, based on the j-th intermediate denoising feature map for the i-th round of denoising, determine the second self-attention feature of the j+1-th second self-attention layer for the i-th round of denoising, add 1 to j cumulatively, and return to execute the step of performing attention calculation in the j-th second self-attention layer based on the first self-attention feature input from the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising, to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising; If j=K, the jth intermediate denoising feature map for the i-th round of denoising is used as the output feature map of the i-th round of denoising.
5. The method according to claim 4, It is characterized in that The first self-attention feature includes a first key vector and a first value vector; the second self-attention feature includes a second key vector, a second value vector and a target query vector; In the j-th second self-attention layer, attention calculation is performed based on the first self-attention feature input by the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising processing, to obtain a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing, including: The following processing is performed in the second self-attention layer of the jth layer: The first key vector and the second key vector of the j-th second self-attention layer for the i-th round of denoising are merged to obtain a target key vector of the j-th second self-attention layer for the i-th round of denoising; The first value vector and the second value vector of the j-th second self-attention layer for the i-th round of denoising are merged to obtain a target value vector of the j-th second self-attention layer for the i-th round of denoising; Determine an attention score of the j-th second self-attention layer for the i-th round of denoising according to the target query vector of the j-th second self-attention layer for the i-th round of denoising and the target key vector; The corresponding target value vector is weighted according to the attention score of the j-th second self-attention layer for the i-th round of denoising processing to obtain a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing.
6. The method according to claim 3, It is characterized in that The diffusion condition also includes object description text for the target image.
7. The method according to any one of claims 3 to 6, It is characterized in that The method further comprises: Acquire a sample skin image, a sample skin occlusion image, a sample skin mask, and a sample facial key point map; the sample skin image is determined based on a sample skin region in a first sample image, the sample skin occlusion image is obtained by occluding the sample skin region in the first sample image, the sample skin mask is a mask of the sample skin region, and the sample facial key point map is obtained by extracting key points from the sample facial region in the first sample image; the sample skin region includes a facial skin region in the sample facial region; Encoding the sample skin image, the sample skin occlusion image, and the sample facial key point map respectively to obtain a first sample feature map of the sample skin image, a second sample feature map of the sample skin occlusion image, and a third sample feature map of the sample facial key point map; Performing feature extraction on the first sample feature map through a first neural network to obtain a sample feature extraction result; Using the second sample feature map, the third sample feature map and the sample skin mask as diffusion conditions, performing reverse diffusion processing on the sample noise image based on the sample feature extraction result through a second neural network to obtain a sample skin quality fusion feature map; Decoding the sample skin quality fusion feature map to obtain a sample skin quality fusion image; Based on the difference between the sample skin quality fusion image and the first sample image, parameters of the first neural network and the second neural network are adjusted until a first training end condition is reached.
8. The method according to claim 7, It is characterized in that The obtaining of the sample skin image, the sample skin occlusion image, the sample skin mask and the sample facial key point map includes: Performing skin region segmentation on the first sample image to determine a sample skin region in the first sample image; Performing mask processing on the sample skin area to obtain the sample skin mask; Based on the sample skin mask, a sample skin area in the first sample image is subjected to occlusion processing to obtain a sample skin occlusion image; Based on the sample skin mask, processing the first sample image to obtain the sample skin image; Key points are extracted from the sample face region in the first sample image to obtain the sample face key point map.
9. The method according to claim 8, It is characterized in that The step of performing occlusion processing on the sample skin area in the first sample image based on the sample skin mask to obtain a sample skin occlusion image includes: Normalizing the first sample image to a first pixel value range to obtain a first intermediate sample image; Normalizing the sample skin mask to a second pixel value range to obtain a first sample skin mask, wherein the second pixel value range is within the first pixel value range; Performing a dot multiplication on the first intermediate sample image and the first sample skin mask to obtain a second intermediate image; The second intermediate image is subjected to inverse normalization processing to obtain the sample skin occlusion image.
10. The method according to claim 8, It is characterized in that The step of processing the first sample image based on the sample skin mask to obtain the sample skin image includes: Normalizing the sample skin mask to a second pixel value range to obtain a second sample skin mask; The second sample skin mask and the first sample image are dot-multiplied to obtain the sample skin image.
11. The method according to claim 8, It is characterized in that The step of performing skin region segmentation on the first sample image to determine a sample skin region in the first sample image includes: Extracting spatial detail features from the first sample image through a first branch network to obtain a sample spatial feature map; Extracting semantic features from the first sample image through a second branch network to obtain a sample semantic feature map; Performing downsampling processing on the sample space feature map to obtain a downsampled space feature map; Performing upsampling processing on the sample semantic feature map to obtain an upsampled semantic feature map; Aggregating the sample semantic feature map with the downsampled spatial feature map to obtain a first sample aggregated feature map; Aggregating the sample spatial feature map with the upsampled semantic feature map to obtain a second sample aggregated feature map; Fusing the first sample aggregate feature map and the second sample aggregate feature map to obtain a sample feature map; Based on the sample feature map, a sample skin area in the first sample image is determined.
12. The method according to any one of claims 3 to 6, It is characterized in that The diffusion condition also includes object description text generated by a text generation model based on the target image; The method further comprises: Extract features of the second sample image by using a visual encoder to obtain features of the first sample; Extracting features of the sample description text corresponding to the second sample image by a text encoder to obtain second sample features; Performing content consistency determination on the first sample feature and the sample description text by using a visual text encoder to obtain a sample determination result; Generate a predicted sample description text according to the first sample encoding result and the sample description text through a text generation model; Determine a text generation model loss based on a difference between the predicted sample description text and the sample description text, a difference between the first sample feature and the second sample feature, and the sample determination result; The parameters of the visual encoder, the text encoder, the visual text encoder and the text generation model are adjusted through the text generation model loss until a second training end condition is reached.
13. An image processing device, It is characterized in that The image processing device comprises: An occlusion module is used to perform occlusion processing on the skin area in the target image to obtain a skin occlusion image; An extraction module, configured to extract key points from a face region in the target image to obtain a face key point map; the skin region includes a facial skin region in the face region; A mask module, used for performing mask processing on the skin area in the target image to obtain a skin mask; the pixel values of the pixels in the non-skin area in the skin mask are different from the pixel values of the pixels in the skin area; the pixel values of different pixels in the non-skin area in the skin mask are the same, and the pixel values of different pixels in the skin area in the skin mask are the same; A skin generation module is used to perform skin generation on the skin occlusion image according to the target skin image, the facial key point map and the skin mask, so as to obtain a skin quality fusion image that integrates the skin quality presented by the target skin image; during the skin generation process, the facial key point map is used to constrain the structure of the facial area, the skin mask is used to specify the area in the skin occlusion image where skin generation is required, and the target skin image is used to specify the skin quality of the skin to be generated.
14. The device according to claim 13, It is characterized in that The skin generation module is further used for: Encoding the target skin image, the skin occlusion image, and the facial key point map respectively to obtain a first feature map of the target skin image, a second feature map of the skin occlusion image, and a third feature map of the facial key point map; Based on the first feature map, the second feature map, the third feature map and the skin mask, the noise image is subjected to reverse diffusion processing to obtain a skin quality fusion feature map; The skin quality fusion feature map is decoded to obtain a skin quality fusion image that is fused with the skin quality presented by the target skin image.
15. The device according to claim 14, It is characterized in that The skin generation module is further used for: Performing feature extraction on the first feature map by using a first neural network to obtain a feature extraction result; The second feature map, the third feature map and the skin mask are used as diffusion conditions, and the noise image is back-diffused based on the feature extraction result through a second neural network to obtain a skin quality fusion feature map.
16. The device according to claim 15, It is characterized in that The first neural network includes K first self-attention layers, and the second neural network includes K second self-attention layers; the feature extraction result includes the first self-attention features input by each first self-attention layer; K is an integer greater than 1; The skin generation module is further used for: The second feature map, the third feature map and the skin mask are used as diffusion conditions, and the noise image is subjected to M rounds of denoising in sequence based on the feature extraction result through a second neural network, and the output feature map obtained by the M-th round of denoising is used as the skin quality fusion feature map; M is a positive integer greater than 1; Among them, the i-th round of denoising is performed according to the following process: Obtaining an input feature map of the i-th round of denoising, wherein the input feature map of the first round of denoising is obtained by fusing the second feature map, the third feature map, the skin mask, and the noise feature map corresponding to the noise image; when 1<i≤M, the input feature map of the i-th round of denoising is the output feature map of the i-1-th round of denoising; In the j-th second self-attention layer, attention calculation is performed based on the first self-attention feature input from the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising processing, to obtain a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing; 1≤j≤K, j is a positive integer; the second self-attention feature of the first second self-attention layer for the i-th round of denoising processing is determined based on the input feature map of the i-th round of denoising processing; Performing denoising on the fused feature map output by the second self-attention layer of the j-th layer to obtain the j-th intermediate denoised feature map for the i-th round of denoising; If j is less than K, based on the j-th intermediate denoising feature map for the i-th round of denoising, determine the second self-attention feature of the j+1-th second self-attention layer for the i-th round of denoising, add 1 to j, and return to execute the step of performing attention calculation in the j-th second self-attention layer based on the first self-attention feature input from the j-th first self-attention layer and the second self-attention feature of the j-th second self-attention layer for the i-th round of denoising, to obtain the fused feature map output by the j-th second self-attention layer for the i-th round of denoising; If j=K, the jth intermediate denoising feature map for the i-th round of denoising is used as the output feature map of the i-th round of denoising.
17. The device according to claim 16, It is characterized in that The first self-attention feature includes a first key vector and a first value vector; the second self-attention feature includes a second key vector, a second value vector and a target query vector; the skin generation module is further used to: The following processing is performed in the second self-attention layer of the jth layer: The first key vector and the second key vector of the j-th second self-attention layer for the i-th round of denoising are merged to obtain a target key vector of the j-th second self-attention layer for the i-th round of denoising; The first value vector and the second value vector of the j-th second self-attention layer for the i-th round of denoising are merged to obtain a target value vector of the j-th second self-attention layer for the i-th round of denoising; Determine an attention score of the j-th second self-attention layer for the i-th round of denoising according to the target query vector of the j-th second self-attention layer for the i-th round of denoising and the target key vector; The corresponding target value vector is weighted according to the attention score of the j-th second self-attention layer for the i-th round of denoising processing to obtain a fused feature map output by the j-th second self-attention layer for the i-th round of denoising processing.
18. The device according to claim 15, It is characterized in that The diffusion condition also includes object description text for the target image.
19. The device according to any one of claims 15 to 18, It is characterized in that The image processing device further comprises a training module, which is used for: Acquire a sample skin image, a sample skin occlusion image, a sample skin mask, and a sample facial key point map; the sample skin image is determined based on a sample skin region in a first sample image, the sample skin occlusion image is obtained by occluding the sample skin region in the first sample image, the sample skin mask is a mask of the sample skin region, and the sample facial key point map is obtained by extracting key points from the sample facial region in the first sample image; the sample skin region includes a facial skin region in the sample facial region; Encoding the sample skin image, the sample skin occlusion image, and the sample facial key point map respectively to obtain a first sample feature map of the sample skin image, a second sample feature map of the sample skin occlusion image, and a third sample feature map of the sample facial key point map; Performing feature extraction on the first sample feature map through a first neural network to obtain a sample feature extraction result; Using the second sample feature map, the third sample feature map and the sample skin mask as diffusion conditions, performing reverse diffusion processing on the sample noise image based on the sample feature extraction result through a second neural network to obtain a sample skin quality fusion feature map; Decoding the sample skin quality fusion feature map to obtain a sample skin quality fusion image; Based on the difference between the sample skin quality fusion image and the first sample image, parameters of the first neural network and the second neural network are adjusted until a first training end condition is reached.
20. The device according to claim 19, It is characterized in that The training module is also used for: Performing skin region segmentation on the first sample image to determine a sample skin region in the first sample image; Performing mask processing on the sample skin area to obtain the sample skin mask; Based on the sample skin mask, a sample skin area in the first sample image is subjected to occlusion processing to obtain a sample skin occlusion image; Based on the sample skin mask, processing the first sample image to obtain the sample skin image; Key points are extracted from the sample face region in the first sample image to obtain the sample face key point map.
21. The device according to claim 20, It is characterized in that The training module is also used for: Normalizing the first sample image to a first pixel value range to obtain a first intermediate sample image; Normalizing the sample skin mask to a second pixel value range to obtain a first sample skin mask, wherein the second pixel value range is within the first pixel value range; Performing a dot multiplication on the first intermediate sample image and the first sample skin mask to obtain a second intermediate image; The second intermediate image is subjected to inverse normalization processing to obtain the sample skin occlusion image.
22. The device according to claim 20, It is characterized in that The training module is also used for: Normalizing the sample skin mask to a second pixel value range to obtain a second sample skin mask; The second sample skin mask and the first sample image are dot-multiplied to obtain the sample skin image.
23. The device according to claim 20, It is characterized in that The training module is also used for: Extracting spatial detail features from the first sample image through a first branch network to obtain a sample spatial feature map; Extracting semantic features from the first sample image through a second branch network to obtain a sample semantic feature map; Performing downsampling processing on the sample space feature map to obtain a downsampled space feature map; Performing upsampling processing on the sample semantic feature map to obtain an upsampled semantic feature map; Aggregating the sample semantic feature map with the downsampled spatial feature map to obtain a first sample aggregated feature map; Aggregating the sample spatial feature map with the upsampled semantic feature map to obtain a second sample aggregated feature map; Fusing the first sample aggregate feature map and the second sample aggregate feature map to obtain a sample feature map; Based on the sample feature map, a sample skin area in the first sample image is determined.
24. The device according to any one of claims 15 to 18, It is characterized in that The diffusion condition also includes an object description text generated by a text generation model based on the target image; the image processing device also includes a training module, and the training module is used to: Extract features of the second sample image by using a visual encoder to obtain features of the first sample; Extracting features of the sample description text corresponding to the second sample image by a text encoder to obtain second sample features; Performing content consistency determination on the first sample feature and the sample description text by using a visual text encoder to obtain a sample determination result; Generate a predicted sample description text according to the first sample encoding result and the sample description text through a text generation model; Determine a text generation model loss based on a difference between the predicted sample description text and the sample description text, a difference between the first sample feature and the second sample feature, and the sample determination result; The parameters of the visual encoder, the text encoder, the visual text encoder and the text generation model are adjusted through the text generation model loss until a second training end condition is reached.
25. An electronic device, It is characterized in that include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 12 is implemented.
26. A computer-readable storage medium, It is characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.
27. A computer program product, It is characterized in that The method comprises computer-readable instructions, which implement the method according to any one of claims 1 to 12 when executed by a processor.
