Intelligent switching method and system for face component structured mask learning

Abstract

The invention discloses an intelligent switching method and system for face component structured mask learning, and the method comprises the steps: extracting a portrait mask with an edge perception capability, and reconstructing a high-fidelity background region with spatial continuity through combining a background reconstruction network; generating a standardized face image with geometric consistency by adopting a face detection and posture correction network; generating an optical head portrait with vision and texture consistency; performing structured mask learning on the standardized face image and the optical head image by using a semantic segmentation network to realize pixel-level face analysis; obtaining a multi-component region mask through mask calculation based on morphological operation; generating an optical head portrait with a natural transition effect by using an optical head portrait synthesis module based on Poisson fusion; and generating a hair changing portrait with physical authenticity by using a hair changing portrait synthesis module. According to the method, the visual fidelity, the detail consistency and the identity retentivity of the face image after hair changing are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical head image generation, and in particular to an intelligent hair changing method and system based on face component structured mask learning. BACKGROUND

[0002] In recent years, with the breakthrough progress of generative adversarial network (GAN) and diffusion model, the image attribute editing technology based on deep learning has made significant progress. Researchers usually divide it into three categories according to the editing dimension: global editing, local editing and semantic editing. Among them, the attribute modification technology represented by latent space editing can maintain the semantic information of the original image to a large extent, while realizing precise attribute control, providing a new technical path for hair image generation. Advanced generation models such as StyleGAN can map images to high-dimensional latent space, making it possible to realize fine-grained attribute editing through vector operation, which greatly improves the accuracy and realism of hair editing.

[0003] In the field of optical head image generation, existing methods can be mainly divided into three categories: image segmentation-based repair methods, three-dimensional head reconstruction and latent space editing. Latent space editing has attracted widespread attention due to its non-destructive editing characteristics. Its core idea is to suppress hair generation by adjusting the directionality of the latent vector. Wu et al. (Yiqian Wu, Yong-Liang Yang and Xiaogang Jin. Hairmapper: Removing hair from portraits using gans [C] / / 2022IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp 4227-4236.) learned the specific path of hair removal in the StyleGAN latent space by training a fully connected network, avoiding the limitations of linear transformation, while combining perceptual loss, reconstruction loss and identity loss to ensure that the generated results are highly consistent with the original portrait. Poisson blending and mask optimization are used to eliminate residual shadows and smooth the transition area.

[0004] The patent CN114663274A discloses a portrait image hair removal method and device based on a GAN network, which comprises the following steps: randomly sampling hidden codes in a StyleGAN hidden space to obtain a hair hidden code-score dataset; obtaining a hair separation boundary and a gender separation boundary through support vector machine training; obtaining a male bald hidden code through hair separation boundary editing; training a Male HairMapper model for editing a male hair hidden code based on the data of the previous step; obtaining a female bald hidden code through gender separation boundary and Male HairMapper model editing; training a hair removal model for generating high-quality bald portrait images based on the above data; and inputting a portrait image with hair to be removed into the hair removal model, and outputting a portrait image with hair removed after calculation. The gender boundary solves the problem of the absence of semantic information of the female-bald combination in the StyleGAN hidden space. Furthermore, a hair hidden code set model is constructed based on the StyleGAN hidden space, so that the hair can be removed while keeping other features of the human face unchanged, and the hair in the portrait image can be quickly and automatically removed. However, this method pays too much attention to the human face, and the image dataset used is basically the image of the human face region, which has high requirements for the input image, and cannot keep the features of the background and the body part of the human unchanged.

[0005] In summary, the current models in the field of bald image generation have certain ability to keep the features of the human face unchanged, but have the limitation of paying too much attention to the human face part, and have insufficient ability to keep the features of the background and the body part of the human, and the visual authenticity and detail consistency of the generated image are insufficient. SUMMARY

[0006] The purpose of the present application is to provide an intelligent hair changing method and system for face part structured mask learning, which generates each part region mask through face part structured mask learning and calculation, and generates an image using image fusion.

[0007] To solve the above technical problems, the technical scheme adopted by the present application is as follows: an intelligent hair changing method for face part structured mask learning, comprising:

[0008] For an input image containing a complete human face, an image portrait matting network based on an attention mechanism is used to extract a portrait mask with edge perception ability, and a background reconstruction network is used to reconstruct a high-fidelity background region with spatial continuity;

[0009] A face detection and posture correction network is used to generate a standardized human face image with geometric consistency;

[0010] A bald portrait is generated using a bald generation adversarial network and a region suppression module, which has visual and texture consistency;

[0011] The structured mask learning of the standardized face image and the light head image with visual and texture consistency is implemented by using the semantic segmentation network, so as to realize the pixel-level face analysis and obtain the standard face analysis result and the standard light head analysis result;

[0012] The multi-component region mask is obtained through the mask calculation based on the morphological operation;

[0013] The light head portrait synthesis module based on Poisson fusion is used to generate the light head portrait with natural transition effect;

[0014] The hair change portrait synthesis module is used to generate the hair change portrait with physical reality.

[0015] Further, the input image x containing the complete face is standardized and preprocessed, that is, if the long side of the image is greater than 2048, the image is adjusted to have a long side equal to 2048 while keeping the aspect ratio, and if the long side of the image is less than 512, the image is adjusted to have a long side equal to 512 while keeping the aspect ratio; the preprocessed image is input into the portrait matting network S m , to generate the portrait mask m h ; then the preprocessed image and the portrait mask m h are input into the background reconstruction network G p , to generate the high-fidelity background region x bg .

[0016] Further, the preprocessed image is input into the face detection and pose correction network D f , to obtain the face key point information and calculate the affine transformation matrix; then the face alignment is realized through the matrix transformation to correct the face pose, and the standard face image x f with a size of 1024x1024 is generated.

[0017] Further, the standard face image x f is input into the light head generation and region suppression module, in which the image is mapped into the feature latent space to generate the encoding vector of the feature latent space, and then the feature encoding is edited in a directional manner to suppress the generation of the hair and obtain the standard light head image x b . b After the face detail restoration of the standard light head image x b , the repaired light head image x' b is obtained. b .

[0018] Further, the standard face image x f and the light head image x" b are input into the semantic segmentation network S fThe face analysis is performed on the image to generate a standard face analysis result m f and a standard bald head analysis result m bald In the analysis result, the image is divided into 11 semantic labels, including background, face, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, mouth and hair.

[0019] Further, by using the portrait mask m h , the standard face analysis result m f and the standard bald head analysis result m bald , mask calculation is performed to obtain a multi-component region mask, which specifically includes a head edge mask m around , a head center mask m center , a bald head mask m head , a body mask m body and a background mask m bg .

[0020] Further, by using the multi-component region mask, the high-fidelity background region x bg , the input image containing the complete face x and the bald head portrait x b , image fusion is performed to generate a bald head portrait x h .

[0021] Further, the head pose and facial key point detection are performed on the bald head portrait x h , and then the hairstyle replacement network G h is used to realize hairstyle replacement to generate a hairstyle-replaced portrait x o .

[0022] The application also provides an intelligent hairstyle replacement system for face component structured mask learning, which is used to realize the above method, and the system comprises:

[0023] A portrait mask extraction module is used to extract a portrait mask with edge perception ability from an input image containing a complete face by using a portrait matting network based on an attention mechanism, and a high-fidelity background region with spatial continuity is reconstructed by combining a background reconstruction network;

[0024] A standardized face image generation module is used to generate a standardized face image with geometric consistency by using a face detection and pose correction network;

[0025] A bald head generation adversarial network and region suppression module is used to generate a bald head portrait with visual and texture consistency;

[0026] A face analysis module is used to perform structured mask learning on the standardized face image and the bald head image with visual and texture consistency by using a semantic segmentation network to realize pixel-level face analysis, so as to obtain a standard face analysis result and a standard bald head analysis result;

[0027] a mask calculation module, which obtains a multi-component region mask through mask calculation based on morphological operation;

[0028] a bald head portrait synthesis module based on Poisson fusion, which is used for generating a bald head portrait with natural transition effect;

[0029] a hair change portrait synthesis module, which is used for generating a hair change portrait with physical reality.

[0030] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the program.

[0031] Compared with the prior art, the beneficial effects of the present application are as follows:

[0032] (1) The present application proposes component-level mask learning and separate calculation, which can realize face component analysis and reduce the interference of non-target regions;

[0033] (2) Face component-level editing and pixel space fusion generation effectively balance the face biometric feature fidelity and the naturalness of the generated region;

[0034] (3) Significantly improve the visual fidelity, detail consistency and identity preservation of the face image after hair change. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the intelligent hair change method of face component structured mask learning of the present application.

[0036] Figure 2 The sub-flowchart of the bald head generation adversarial and region suppression module used in the present application.

[0037] Figure 3 The sub-flowchart of the multi-component mask calculation module used in the present application.

[0038] Figure 4 The sub-flowchart of the bald head portrait synthesis module used in the present application.

[0039] Figure 5 The sub-flowchart of the hair change portrait synthesis module used in the present application. DETAILED DESCRIPTION

[0040] The present application will be further introduced in combination with specific embodiments.

[0041] The intelligent hair changing method of face component structured mask learning of the embodiment first extracts a portrait mask with edge perception ability from an input image containing a complete face by using a portrait matting network based on an attention mechanism, and reconstructs a high-fidelity background area with spatial continuity by combining a background reconstruction network; a face detection and pose correction network is used to generate a standardized face image with geometric consistency; a light head generative adversarial network and a region suppression module are used to generate a light head portrait with visual and texture consistency; a semantic segmentation network is used to learn a structured mask from the standardized face image and the light head image with visual and texture consistency to achieve pixel-level face analysis, and obtain a standard face analysis result and a standard light head analysis result; a mask calculation based on morphological operation is used to obtain a multi-component region mask; a light head portrait synthesis module based on Poisson fusion is used to generate a light head portrait with a natural transition effect; finally, a hair changing portrait synthesis module is used to generate a physically realistic hair changing portrait. Combined with Figure 1 The method specifically comprises the following steps:

[0042] Step 1, multi-scale standardization preprocessing based on bilinear interpolation is performed on an input image x containing a complete face, that is, when the long side of the image exceeds 2048 pixels, the long and short sides are scaled to a long side of 2048 pixels by resampling; when the long side is less than 512 pixels, the image is adjusted to a long side of 512 pixels by combining edge padding and super-resolution reconstruction to maintain the aspect ratio, and the larger image is reduced to avoid memory overflow and improve the processing efficiency of the subsequent module, and the smaller image is enlarged to improve the clarity of the image and better preserve the features in the image.

[0043] The preprocessed image is input into a portrait matting network S m , a sub-pixel level precision portrait mask m h is generated, the value range is [0, 1], 1 represents the main body of the person, and 0 represents the background, and then the preprocessed image and the portrait mask m h are spliced and input into a background reconstruction network G p , a high-fidelity background area x bg with spatial continuity is generated.

[0044] Step 2, the preprocessed image is input into a face detection and pose correction network D f for multi-angle face detection, in which the position information of the face key points (such as eyes and corners of the mouth) is obtained by detecting the face in the image, and then the affine transformation matrix of the face key points with the standard pose is calculated to align the detected face key points to the target position, and the face alignment is realized by a bilinear sampling network to realize nonlinear face pose correction, and a standardized face image x fThere is a distortion control parameter in the bilinear sampling network, which allows local elastic deformation to adapt to different postures (such as side face, lowered head, etc.).

[0045] Step 3: Transform the standard face image x f After being reduced to 256×256, it is input into the optical head generation adversarial and regional suppression module, combined with Figure 2 In this module, the image is first input into the light head generation network G based on the StyleGAN architecture b In this network, a pre-trained encoder is used to embed the image into the latent space of StyleGAN to generate an encoding vector in the 18×512-dimensional feature latent space, and then the hair region positioning matrix M is designed. hair ∈R (18×1) , the encoding vector is edited in a targeted manner in the latent space to obtain the latent code of the bald head. This part of the operation is to remove the hair area of ​​the image and reveal the semantic content of the areas blocked by the hair, such as the background and ears. Then, by mixing the latent code of the bald head with the encoding vector of the input image, only the features of the hair area are replaced and the original features of the non-hair area are retained, the standard bald head image x is generated. b .

[0046] Then, by combining the standard face analysis results m f , standard face image x f and the standard bald head image x b After restoring facial details, we get the bald head image x' to be repaired b , the bald head image to be repaired x' b After reducing it to 512×512, perform face restoration to enhance facial details, repair artifacts, improve image quality, and generate a bald head portrait with a size of 1024×1024 with visual consistency x" b .

[0047] Step 4: Transform the standard face image x f and the standard bald head image x b Input to the semantic segmentation network S f Perform face analysis in , and generate standard face analysis result m f And the standard optical head analysis result m bald ,In the parsing results, the images are divided into 11 categories of semantic ,labels, including background, face, left eyebrow, right eyebrow, left eye, ,right eye, nose, upper lip, lower lip, mouth and hair.

[0048] Step 5: Combine Figure 3 , by using the portrait mask m h , standard face analysis result m f And the standard optical head analysis result m baldPerform mask calculation to obtain the multi-component area mask and extract the standard optical head parsing result m bald The part with pixel value greater than 0 can generate the bald head mask m head , similarly extract the standard face analysis result m f The head area, and then the portrait mask m h Subtract this head area to get the body mask m body Subtract the bald mask m from the mask with all pixel values ​​1 in the entire image area head and body mask m body The background mask m can be generated bg For the standard face analysis result m f The area with a pixel value of 10 is extracted to generate a hair mask. head Perform mask separation to generate head edge mask m around and the head center mask m center .

[0049] Step 6: Use the body mask body Multiply the input image x containing the complete face and use the background mask m bg Multiply the high-fidelity background area by x bg , using the optical mask m head Multiply the bald image by x" b , these regions are fused to generate the final bald head portrait x h ,like Figure 4 shown.

[0050] Step 7: Detect head posture through network D h and facial key point detection network D g For bald portrait x h Perform head pose and facial key point detection to generate head pose e h and facial key points p f , then the hairstyle image x s , head posture h , facial key points p f and bald head image x h Input to Hairstyle Add Network G h Realize hairstyle replacement to generate hair-changing portrait x o ,like Figure 5 shown.

[0051] The present application realizes face component analysis by component level mask learning and separation calculation, reduces the interference of non-target regions, and effectively balances the face biometric feature fidelity and the naturalness of the generated region by using face component level editing and pixel space fusion generation. The method significantly improves the visual fidelity, detail consistency and identity preservation of the face image after changing the face, and solves the problem that the previous method pays too much attention to retaining the features of the face region.

Claims

1. An intelligent hair replacement method based on structured mask learning of facial parts, characterized in that: include: For input images containing complete faces, an attention-based portrait cutout network is used to extract edge-aware portrait masks, and a background reconstruction network is combined to reconstruct high-fidelity background regions with spatial continuity. A face detection and pose correction network is used to generate standardized face images with geometric consistency; Using the bald head generative adversarial network and the region suppression module to generate bald head portraits with visual and texture consistency; Using semantic segmentation networks, we perform structured mask learning on standardized face images and bald head images with visual and texture consistency to achieve pixel-level face parsing, obtaining standard face parsing results and standard bald head parsing results. The multi-component region mask is obtained by mask calculation based on morphological operations; Use the bald portrait synthesis module based on Poisson fusion to generate bald portraits with natural transition effects; The hair-changing portrait synthesis module is used to generate physically realistic hair-changing portraits.

2. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 1, characterized in that: For input images containing complete faces, we use an attention-based portrait cutout network to extract edge-aware portrait masks, and combine it with a background reconstruction network to reconstruct high-fidelity background regions with spatial continuity. Specifically: (1) The input image x containing the complete face is preprocessed with multi-scale normalization based on bilinear interpolation. That is, when the long side of the image exceeds 2048 pixels, it is resampled to maintain the aspect ratio and scaled to 2048 pixels. When the long side is less than 512 pixels, the image is adjusted to 512 pixels by combining edge filling and super-resolution reconstruction to maintain the aspect ratio. (2) Input the preprocessed image into the portrait cutout network S containing the channel attention module m Generate a sub-pixel accurate portrait mask m h ; (3) Combine the preprocessed image and portrait mask m h After splicing, input to the background reconstruction network G p Generate a high-fidelity background region x with spatial continuity bg .

3. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 2, characterized in that: The face detection and posture correction network is used to generate a standardized face image with geometric consistency, including the following steps: (1) By inputting the preprocessed image into the face detection and posture correction network D f Perform multi-angle face detection, in which the network detects faces in images and obtains the location information of facial key points; (2) The affine transformation matrix between the position information of the facial key points and the standard posture is calculated to align the detected facial key points to the target position, and the nonlinear facial posture correction is implemented through the bilinear sampling network to achieve face alignment and generate a 1024×1024 pixel standard face image x f ; There is a distortion control parameter in the bilinear sampling network, which allows local elastic deformation to adapt to different postures.

4. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 3, characterized in that: Using the bald head generative adversarial network and the region suppression module, we generate bald head portraits with visual and texture consistency, including the following steps: (1) By taking the standard face image x f Input to the bald head generation adversarial and regional suppression module, in which the image is first input to the bald head generation network G based on the StyleGAN architecture b In this network, a pre-trained encoder is used to embed the image into the latent space of StyleGAN to generate an encoding vector in the 18×512-dimensional feature latent space; (2) By designing the hair area positioning matrix M hair ∈R (18×1) , the encoding vector is edited in the latent space to obtain the latent code of the bald head. This part of the operation is to remove the hair area of ​​the image and reveal the area blocked by the hair; then the latent code of the bald head is mixed with the encoding vector of the input image to replace only the features of the hair area and retain the original features of the non-hair area to generate the standard bald head image x b ; (3) By combining the standard face analysis results m f , standard face image x f and the standard bald head image x b After restoring facial details, we get the bald head image x' to be repaired b , the bald head image to be repaired x' b After reducing the size to 512×512, perform face restoration to enhance facial details, repair artifacts, improve image quality, and generate a bald head portrait with a size of 1024×1024 with visual consistency x" b .

5. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 4, characterized in that: The semantic segmentation network is used to perform structured mask learning on the standardized face image and the bald head image with visual and texture consistency to achieve pixel-level face parsing, and obtain the standard face parsing results and the standard bald head parsing results; the standard face image x f and bald head image x″ b Input to the semantic segmentation network S f Perform face analysis in , and generate standard face analysis results m with sub-pixel segmentation accuracy f And the standard optical head analysis result m bald , which contains parsing results of 11 categories: background, face, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, mouth and hair.

6. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 5, characterized in that: The multi-component region mask is obtained by mask calculation based on morphological operation, specifically: by using the portrait mask m h , standard face analysis result m f And the standard optical head analysis result m bald Perform mask calculation to obtain a multi-component area mask, including the head edge mask m around , head center mask m center , bald mask m head , body mask m body and background mask m bg .

7. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 6, characterized in that: The bald portrait synthesis module based on Poisson fusion is used to generate a bald portrait with a natural transition effect. By using multi-component region masks, high-fidelity background region x bg , input image x containing a complete face and bald head image x″ b Perform image fusion to generate bald portrait x h .

8. The intelligent hair replacement method based on structured mask learning of facial parts according to claim 7, characterized in that: Generating a physically realistic hair-changing portrait using the hair-changing portrait synthesis module includes the following steps: (1) For bald portrait x h Perform head posture and facial key point detection to obtain the head posture e h and facial key points p f ; (2) By taking the hairstyle image x s , head posture h , facial key points p f and bald head image x h Input to Hairstyle Add Network G h Realize hairstyle replacement to generate hair-changing portrait x o .

9. An intelligent hair replacement system based on structured mask learning of facial parts, characterized by: For implementing the method according to any one of claims 1 to 8, the system comprises: The portrait mask extraction module is used to extract edge-aware portrait masks from input images containing complete faces using an attention-based portrait cutout network, and then reconstruct high-fidelity background regions with spatial continuity using a background reconstruction network. The standardized face image generation module uses a face detection and posture correction network to generate standardized face images with geometric consistency; Bald head generative adversarial network and region suppression module, used to generate bald head portraits with visual and texture consistency; The face parsing module uses a semantic segmentation network to perform structured mask learning on standardized face images and bald head images with visual and texture consistency to achieve pixel-level face parsing, obtaining standard face parsing results and standard bald head parsing results; The mask calculation module obtains the multi-component region mask through mask calculation based on morphological operations; A bald portrait synthesis module based on Poisson fusion is used to generate bald portraits with natural transition effects; The hair-changing portrait synthesis module is used to generate hair-changing portraits with physical realism.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images