A convolutional neural network skin quality processing method and processing device based on human body analysis prior support

Abstract

The present invention discloses a convolutional neural network skin processing method based on human body analysis prior support, comprising the following steps: A. constructing a human skin analysis model to obtain a human body analysis map; B. constructing a convolutional neural network model for portrait skin processing; C. introducing human skin analysis information, constructing a human skin analysis information encoding model to guide the portrait skin processing model to perform different processing on different semantic areas of the portrait. The present invention also discloses a convolutional neural network skin processing device based on human body analysis prior support. The present invention uses a convolutional neural network to model the complex pixel mapping relationship between the original portrait and the portrait processed by the photo retoucher, and introduces human body analysis information as the prior information of the convolutional neural network to guide the model to perform corresponding processing on various parts of the human body, especially to distinguish the processing of facial and body skin, so that different areas have different processing effects, and the processing effect is better.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular, relates to a human skin processing technology for digital SLR camera imaging, which is applied to skin quality processing of portraits in photos generated by mobile phones, digital SLR cameras and other devices, and specifically relates to a convolutional neural network skin quality processing method and processing device based on human body analysis prior support. Background Art

[0002] The processing of portrait skin has become a very common image processing requirement in daily life. In recent years, various skin smoothing software has emerged one after another. However, in the high-demand commercial-level portrait retouching scenes, such as wedding photography, advertising posters, etc., professional retouchers are still needed to use software such as PhotoShop for manual processing. According to our research, it takes an average of about 30 minutes for a retoucher to process a 20-megapixel portrait taken by a professional camera. Such efficiency makes it particularly important to develop computer vision technology that is closer to the effect of human retouching. The current portrait skin processing technologies mainly include:

[0003] 1) Using filtering techniques such as Gaussian blur and bilateral filtering to achieve skin smoothing effects. This method has obvious skin smoothing effects and high execution efficiency, but the images processed by this method often have blurring problems, and the overall texture of the skin will be significantly reduced, appearing flat and without three-dimensional sense. In addition, another problem with this method is that it cannot selectively act on the specified area of ​​the image, which will cause the background other than the portrait to be processed, greatly reducing the overall clarity of the image;

[0004] 2) Use image restoration technology to process dark blemishes such as acne on the face. This method first needs to find the blemish area on the skin, and then use image restoration technology to fill it with pixels around the blemish. Although this method can ensure the clarity of the overall image, the repaired local pixel area still has different degrees of skin texture loss.

[0005] In principle, it is difficult for the above methods to ensure that they can handle various complex skin problems while maintaining a natural and delicate commercial-grade portrait processing effect. In addition, the use of the above two methods also requires the use of corresponding facial features positioning technology or skin recognition technology to avoid misprocessing areas outside the skin. What is even more difficult is that in the process of portrait processing by photo retouchers in real scenes, there are large differences in the processing methods and effects of different areas of the human body. For example, the face pays more attention to the preservation of skin texture, while the body skin pays more attention to the overall cleanliness. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a convolutional neural network skin quality processing method and processing device based on human body analysis prior support, which introduces human body analysis information as prior information of the convolutional neural network to guide the model to perform corresponding processing on different parts of the human body, and the processing results are more realistic and clear.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a convolutional neural network skin quality processing method based on human body analysis prior support, comprising the following steps:

[0008] A. Construct a human skin analysis model and obtain a human body analysis map;

[0009] B. Construct a convolutional neural network model for portrait skin processing;

[0010] C. Introduce human skin analysis information and construct a human skin analysis information encoding model to guide the portrait skin processing model constructed in step B to perform different processing on different semantic areas of the portrait.

[0011] Furthermore, in step A, a human skin analysis model is constructed based on the CHIP human analysis dataset and the DeepLab-V3 structure.

[0012] Furthermore, in step B, a convolutional neural network model for portrait skin processing is constructed based on the UNet structure.

[0013] Furthermore, a multi-scale dense convolution module is embedded between each downsampling and upsampling in step B to increase the receptive field of the corresponding layer, so that larger blemishes on the skin can be processed more easily.

[0014] Furthermore, in the step C, a human skin analysis information encoding model is constructed based on HRNet.

[0015] Furthermore, when human skin analysis information is introduced in step C to guide the model to perform different processing on different portrait areas, human body analysis is introduced as prior information for the constructed convolutional neural network model, guiding the model to at least recognize three areas that need to be distinguished: face, body, and background.

[0016] Further, the following steps are included:

[0017] S1. Build a human skin parsing model based on the CIHP human parsing dataset and DeepLab-V3 structure;

[0018] S2, construct training sample set D = {(x i ,y i , m i )|x i ∈X N ,yi ∈Y N , i = 1, 2, ... N}, x i Original portrait image, y i The corresponding retouched photos are manually processed by professional photo retouchers. i For x i The corresponding human body analysis graph, wherein the human body analysis graph is inferred by the human skin analysis model constructed in 1);

[0019] S3. Construct a convolutional neural network model for portrait skin processing. The input of the model is x i and m i , the model output is the predicted retouched image

[0020] S4. Construct model loss, the formula is shown in formula 1:

[0021]

[0022] Among them, φ(y i )=∑ layer＝(relu2_2，relu3_3，relu4_3) w layer VGG layer (y i ) (2)

[0023] Where φ is as shown in formula (2), which represents the output weights of relu2_2, relu3_3, and relu4_3 layers of the pre-trained VGG-16 model;

[0024] S5. Train the convolutional neural network model.

[0025] Preferably, the Adam optimizer is used in step S5, and the learning rate is set to 1e-4. The present invention also discloses a convolutional neural network skin quality processing device based on human body analysis prior support, comprising:

[0026] The human skin parsing model is built based on the CHIP human parsing dataset and the DeepLab-V3 structure to obtain human skin parsing images;

[0027] Convolutional neural network model, built based on UNet structure, for portrait skin processing;

[0028] The human body parsing information guidance model is built based on HRNet and is used to introduce human skin parsing information. It constructs a human skin parsing information encoding model to guide the portrait skin processing model to perform different processing on different semantic areas of the portrait.

[0029] The present invention uses a convolutional neural network to model all complex skin blemishes. The convolutional neural network can make targeted treatments based on the color and texture of different blemishes through high-order nonlinear mapping, without the need to tediously design corresponding positioning and processing methods based on different blemishes.

[0030] The present invention introduces human skin analysis information to guide the model to perform different targeted processing on different portrait areas. Through experiments, it is found that simply using convolutional neural network technology to process portrait skin will result in 1) the facial skin of the portrait retouched output by the model is affected by the body skin effect, so that some shadows and textures of the face are removed, and the face loses a certain degree of three-dimensional sense; 2) some image background areas with skin color or texture similar to the skin are also processed unpredictably, such as the wood grain of some stools is removed, and the rough pink background wall is smoothed. In order to solve the above problems, the method of the present invention introduces human body analysis as prior information for the constructed convolutional neural network model, guiding the model to better identify the three areas that need to be distinguished: face, body, and background.

[0031] The beneficial effects of the present invention are as follows: the present invention utilizes a convolutional neural network to model the complex pixel mapping relationship between the original portrait and the portrait processed by a photo retoucher, and introduces human body analysis information as prior information of the convolutional neural network, guiding the model to perform corresponding processing on various parts of the human body, especially distinguishing between the processing of the face and body skin, so that different areas have different processing effects, and the processing effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flow chart of the method of the present invention.

[0033] Figure 2 A network diagram of the method of the present invention

[0034] Figure 3 This is a comparison diagram of the effects before and after the treatment by the method of the present invention.

[0035] Figure 4 The human body analysis diagram constructed by the method of the present invention.

[0036] Figure 5-1 This is the original unprocessed photo.

[0037] Figure 5-2 This is the final image obtained after the original image is processed by the filtering and skin smoothing method.

[0038] Figure 5-3 This is the final image obtained after the original image is processed by the stain restoration method.

[0039] Figure 5-4 It is the final image obtained after the original image is processed by the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] A convolutional neural network skin quality processing method based on human body analysis prior support includes the following steps:

[0042] A. Construct a human skin analysis model and obtain a human skin analysis map;

[0043] Specifically, a human skin parsing model is constructed based on the CHIP human parsing dataset and the DeepLab-V3 structure to obtain a human skin parsing map;

[0044] Human body analysis aims to parse the person in the image into multiple semantically consistent regions, such as various parts of the body, the composition of the whole body's clothes, etc.

[0045] The CHIP human body parsing dataset is an important public dataset in human body parsing. It includes 30,000 human body annotated samples and defines the semantic areas of the human body into 20 categories (background, hat, hair, gloves, magic mirror, top, dress, coat, socks, shorts, body, scarf, skirt, face, left / right arm, left / right leg, left / right shoe).

[0046] DeeplabV3 was open-sourced by Google for semantic parsing (Rethinking Atrous Convolution for Semantic Image Segmentation, 2017). Its feature is that it can parse objects at multiple scales, designs serial and parallel atrous convolution modules, and uses a variety of different atrous rates to obtain multi-scale content information. It also proposes the Atrous Spatial Pyramid Pooling (ASPP) module to mine convolutional features of different scales and image layer features that encode global content information to improve parsing results.

[0047] B. Construct a convolutional neural network model for portrait skin processing;

[0048] Specifically, a convolutional neural network model for portrait skin processing is constructed based on the UNet structure (U-Net: Convolutional Networks for Biomedical Image Segmentation);

[0049] For the original basic UNet structure, the following optimizations are performed;

[0050] like Figure 2 As shown in the figure, a multi-scale dense convolution module is embedded between each downsampling and upsampling. This module draws on the high feature effectiveness brought by the dense convolution (Densely Connected Convolutional Networks, 2017) to encourage feature reuse and the advantage of the ASPP module in mining multi-scale features. One module is embedded in the original size, two modules are embedded after the first downsampling, three modules are embedded after the second downsampling, and one module is embedded in the upsampling part. From the experimental results, the injection of this module makes the receptive field of the entire network and the single layer at different scales enlarged, and larger areas on the skin, such as larger moles and blocky scars, are easier to process.

[0051] C. Introduce human skin analysis information, build a human body analysis information encoding model based on HRNet to guide the portrait skin processing model built in step B to perform different processing on different semantic areas of the portrait;

[0052] In the above steps, when the human body analysis information is introduced to guide the model to perform different processing on different portrait areas, the human body analysis is introduced as prior information for the constructed convolutional neural network model to guide the model to at least recognize the three areas that need to be distinguished: face, body, and background;

[0053] The present invention finds that directly merging the semantic parsing graph and the original image and inputting them from the network will make it difficult for the image edge and the parsing graph edge to blend naturally. Therefore, we separately built an information encoding network based on HRNet (Deep High-ResolutionRepresentation Learning for Human Pose Estimation, 2019, which is a model proposed to solve human pose estimation) to eliminate or smooth the "essential isolation" between the parsing graph and the real image. We did not make too many changes to the original HRNet, but we did a lot of experiments to determine how to use it for our application scenarios. Specifically, the information encoded by the network is injected into which layer of the main network constructed in step B for better results. Finally, we chose to inject the first upsampling position after the main network completed all downsampling, such as Figure 2 shown.

[0054] More specifically, a convolutional neural network skin quality processing method based on human body analysis prior support includes the following steps:

[0055] S1. Build a human skin parsing model based on the CIHP human parsing dataset and DeepLab-V3 structure;

[0056] S2. Construct training sample set

[0057] D={(x i ,y i , m i )|x i ∈X N ,y i ∈Y N , i = 1, 2, ... N}, x i Original portrait image, y i The corresponding retouched photos are manually processed by professional photo retouchers. i For x i The corresponding human body analysis graph is obtained by inference of the human skin analysis model constructed in 1);

[0058] like Figure 3 As shown, the human body analysis image consists of three parts: a white face area, a gray body area and a black background area.

[0059] S3. Construct a convolutional neural network model for portrait skin processing. The input of the model is x i and m i , the model output is the predicted retouched image

[0060] S4. Construct model loss, the formula is shown in formula 1:

[0061]

[0062] Among them, φ(y i )=∑ layer＝(relu2_2，relu3_3，relu4_3) w layer VGG layer (y i ) (2)

[0063] Where φ is as shown in formula (2), which represents the output weights of relu2_2, relu3_3, and relu4_3 layers of the pre-trained VGG-16 model;

[0064] S5. Training the convolutional neural network model. The method of the present invention uses the Adam optimizer and the learning rate is set to 1e-4.

[0065] like Figure 3As shown, the left picture is the original input portrait picture, and the right picture is the effect picture generated by the method of the present invention. In the right picture, the face, body and background of the human body are processed separately to obtain different results. The blemishes on the face and body skin of the portrait are beautified, including moles, spots, acne marks, wrinkles, etc., so that the skin texture of the portrait in the photo is natural and delicate. Human body analysis is introduced as prior information to guide the model to better identify the three areas that need to be distinguished, namely the face, body and background. In human body analysis, three colors are involved, white-face, gray-human body, and background-black, so that the model network output can keep the background unchanged and only process the white and gray areas. Compared with not distinguishing between the background and skin, some backgrounds with skin color or texture similar to the skin will be misprocessed.

[0066] Figure 5-1 This is the original unprocessed photo. Figure 5-2 The image is obtained by using the existing filtering and skin resurfacing method. Figure 5-3 The image is obtained by using the existing stain repair method. Figure 5-4 This is an image obtained after being processed by the method of the present invention. It can be clearly seen from the figure that the filtering skin resurfacing method significantly reduces the image clarity, and the spot repair method destroys the skin texture of the repaired area. The present invention can selectively process the designated area of ​​the real photo - the portrait part. The image obtained after processing is guaranteed in terms of clarity and skin texture, and the final processing effect is natural and elegant.

[0067] A convolutional neural network skin quality processing device based on human body analysis prior support, characterized by comprising:

[0068] The human skin parsing model is built based on the CHIP human parsing dataset and the DeepLab-V3 structure to obtain human skin parsing images;

[0069] Convolutional neural network model, built based on UNet structure, for portrait skin processing;

[0070] The human body parsing information guidance model is built based on HRNet. It is used to introduce human skin parsing information and construct a human body parsing information encoding model to guide the portrait skin processing model to perform different processing on different semantic areas of the portrait.

[0071] The above specific implementation modes are used to explain the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A convolutional neural network skin quality processing method based on human body analysis prior support, characterized in that The following steps are involved: A. Construct a human skin analysis model and obtain a human body analysis map; B. Construct a convolutional neural network model for portrait skin processing; C. Introduce human skin analysis information, build a human skin analysis information encoding model, and guide the portrait skin processing model built in step B to perform different processing on different semantic areas of the portrait; specifically, S1. Build a human skin parsing model based on the CIHP human parsing dataset and DeepLab-V3 structure; S2, construct training sample set D = {(x i ,y i , m i )|x i ∈X N ,y i ∈Y N , i = 1, 2, ... N}, x i Original portrait image, y i The corresponding retouched photos are manually processed by professional photo retouchers. i For x i The corresponding human body analysis graph, where the human body analysis graph is inferred by the human skin analysis model constructed in S1; S3. Construct a convolutional neural network model for portrait skin processing. The input of the model is x i and m i , the model output is the predicted retouched image S4. Construct model loss, the formula is shown in formula 1: where, φ(y i ) = ∑ layer＝(relu2_2，relu3_3，relu4_3) w layer VGG layer (y i ) (2) Where φ is as shown in formula (2), which represents the pre-trained VGG-16 model relu2_2, relu33, rel u 43 layers of output weighting; S5. Train the convolutional neural network model.

2. The convolutional neural network skin quality processing method based on human body analysis prior support according to claim 1 is characterized in that: In the step A, a human skin parsing model is constructed based on the CIHP human parsing dataset and the DeepLab-V3 structure.

3. The convolutional neural network skin quality processing method based on human body analysis prior support according to claim 1 or 2, characterized in that: In the step B, a convolutional neural network model for portrait skin processing is constructed based on the UNet structure.

4. The convolutional neural network skin quality processing method based on human body analysis prior support according to claim 3 is characterized in that: In the step B, a multi-scale dense convolution module is embedded between each downsampling and upsampling.

5. The convolutional neural network skin quality processing method based on human body analysis prior support according to claim 3 is characterized in that: In the step C, a human skin analysis information coding model is constructed based on HRNet.

6. The convolutional neural network skin quality processing method based on human body analysis prior support according to claim 1 is characterized in that: When human skin analysis information is introduced in step C to guide the model to perform different processing on different portrait areas, human analysis is introduced as prior information for the constructed convolutional neural network model to guide the model to at least recognize three areas that need to be distinguished: face, body, and background.

7. The convolutional neural network skin quality processing method based on human body analysis prior support according to claim 1 is characterized in that: In step S5, the Adam optimizer is used, and the learning rate is set to 1e-4.

8. A convolutional neural network skin quality processing device based on human body analysis prior support, used to implement the convolutional neural network skin quality processing method based on human body analysis prior support as described in any one of claims 1-7, characterized in that include: The human skin parsing model is built based on the CIHP human parsing dataset and the DeepLab-V3 structure to obtain human skin parsing images; Convolutional neural network model, built based on UNet structure, for portrait skin processing; The human body parsing information guidance model is built based on HRNet and is used to introduce human skin parsing information. It constructs a human skin parsing information encoding model to guide the portrait skin processing model to perform different processing on different semantic areas of the portrait.

Images