Face image makeup transfer method and system based on image makeup transfer network

Abstract

The application discloses a kind of face image makeup transfer method and system based on image makeup transfer network, first obtain original face image;Then the original face image obtained is input into image makeup transfer network, generates final makeup transfer result;The image makeup transfer network of the application is composed of semantic corresponding subnetwork and image coloring subnetwork;Semantic corresponding subnetwork is used to learn the dense semantic corresponding relationship between unmade-up image and made-up image.The main function of coloring subnetwork is to render the makeup features after semantic alignment to target image, to generate final makeup transfer result.The application defines makeup transfer as image coloring problem, which better avoids the problem of lack of paired data.A kind of alternative attention mechanism is proposed to calculate the semantic corresponding relationship at pixel level at low resolution, and use this corresponding relationship to aggregate patch blocks at high resolution, which greatly reduces the computational complexity without losing makeup details.

Description

Technical Field

[0001] This invention pertains to image processing technology based on Generative Adversarial Networks (GANs), and relates to a method and system for transferring makeup from facial images, specifically a method and system for transferring makeup from facial images based on image coloring and alternative attention. Background Technology

[0002] Against the backdrop of the booming cosmetics market, how to quickly and accurately provide users with personalized beauty products has gradually become a research hotspot in the field of computer vision. As an effective means to solve this problem, makeup transfer technology has attracted much attention due to its wide range of applications and huge market demand. In recent years, makeup transfer has gradually gained widespread attention from scholars both domestically and internationally. By combining decoupled representations with generative adversarial networks, numerous scholars have made significant contributions to promoting the development of makeup transfer technology.

[0003] However, two key challenges remain to be addressed in makeup transfer. One is the significant impact of synthesized pseudo-paired data on the quality of the generated transfer results. Due to the inherent nature of makeup transfer, paired data is practically unavailable for supervised network training. To overcome this, existing methods use histogram matching or facial feature point distortion to synthesize pseudo-paired data for network training; therefore, the quality of the synthesized data severely affects the network's generation performance. The other challenge is the computational cost of semantic correspondence. Existing methods have demonstrated that semantic correspondence learning can effectively improve the quality of makeup transfer. However, semantic correspondence first requires calculating the correlation between each pixel and all pixels, and then aggregating all features based on the correlation magnitude. Its computational complexity is quadratic with the feature map size, severely limiting the practical application of makeup transfer. Summary of the Invention

[0004] To address the shortcomings of the aforementioned background technology, this invention proposes a method and system for facial image makeup transfer based on image coloring and alternative attention.

[0005] The technical solution adopted by the method of the present invention is: a face image makeup transfer method based on an image makeup transfer network, comprising the following steps:

[0006] Step 1: Obtain the original face image;

[0007] Step 2: Input the acquired original face image into the image makeup transfer network to generate the final makeup transfer result;

[0008] The image makeup transfer network consists of a semantic correspondence subnetwork and an image coloring subnetwork.

[0009] The semantic correspondence subnetwork comprises a feature extractor and a substitute attention module; the feature extractor is used to extract rich spatial semantic features for feature matching; and the substitute attention module is used to successfully map the semantic correspondence relationship learned at a low resolution to high-resolution features.

[0010] The image colorization subnetwork comprises an identity encoder, a color distiller and a decoder; the identity encoder is used to extract a person's identity features; the color distiller is used to distill the makeup features of a reference image; and the decoder is used to input the unmodified person's identity features and the makeup features after semantic alignment into the decoder to generate a final makeup transfer result.

[0011] The feature extractor comprises three Convolution blocks with a step size of 2; the identity encoder and the color distiller each comprise two Convolution blocks and two Residual blocks with a step size of 2; the decoder comprises four Residual blocks and two Convolution blocks and performs an upsampling operation through bilinear interpolation; and the substitute attention module comprises a pixel unshuffle operation, a cross-attention and a pixelshuffle.

[0012] The Convolution block comprises a Convolution Layer, an Instantiation Normalization Layer and a ReLU Activation Layer connected in sequence; and the Residual block comprises two Convolution blocks connected in sequence, wherein the last ReLU Activation Layer is removed.

[0013] The technical scheme of the system of the present application is as follows: a face image makeup transfer system based on an image makeup transfer network, comprising an original face image acquisition module and a makeup transfer module:

[0014] The original face image acquisition module is used to acquire an original face image.

[0015] The makeup transfer module is used to input the acquired original face image into the image makeup transfer network to generate a final makeup transfer result.

[0016] The image makeup transfer network comprises a semantic correspondence subnetwork and an image colorization subnetwork.

[0017] The semantic correspondence subnetwork comprises a feature extractor and a substitute attention module; the feature extractor is used to extract rich spatial semantic features for feature matching; the substitute attention module is used to successfully map the semantic correspondence relationship learned at low resolution to high resolution features;

[0018] The image coloring subnetwork comprises an identity encoder, a color distiller and a decoder; the identity encoder is used to extract the identity features of the person; the color distiller is used to distill the makeup features of the reference image; and the decoder is used to input the unmodified identity features of the person and the makeup features after semantic alignment into the decoder to generate the final makeup transfer result.

[0019] The feature extractor is composed of 3 Convolution blocks with a step of 2; the identity encoder and the color distiller are each composed of 2 Convolution blocks and 2 Residual blocks with a step of 2; the decoder is composed of 4 Residual blocks and 2 Convolution blocks, and is operated by bilinear interpolation for upsampling; and the substitute attention module is composed of a pixel unshuffle operation, a cross-attention and a pixelshuffle.

[0020] The Convolution block is composed of a Convolution Layer layer, an Instantiation Normalization Layer layer and a ReLU Activation Layer layer connected in sequence; and the Residual block is composed of two Convolution blocks connected in sequence, wherein the last ReLU Activation Layer layer is removed.

[0021] Compared with the prior art, the beneficial effects of the present application include:

[0022] (1) Without generating pseudo paired data, the complexity of data collection is reduced;

[0023] (2) The generated makeup transfer result has higher makeup similarity than the existing method;

[0024] (3) The proposed substitute attention mechanism greatly reduces the computational cost. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1It is a network overall architecture diagram in the embodiment of the present application, containing a semantic corresponding sub-network and an image coloring sub-network. The semantic corresponding sub-network contains a feature extractor and a proposed alternative attention mechanism, and the image coloring sub-network contains an identity encoder, a color distillation encoder and a decoder.

[0026] Figure 2 It is a makeup transfer schematic diagram from image coloring in the embodiment of the present application. The network takes a grayscale unmade-up image and a colorful made-up image as input, and the target is to transfer the color of the made-up image to the unmade-up image.

[0027] Figure 3 It is an alternative attention visual interpretation diagram in the embodiment of the present application. The proposed alternative attention mechanism calculates a correlation matrix and aggregated features at different resolutions, greatly reducing the calculation cost without losing makeup details.

[0028] Figure 4 It is an alternative attention module specific operation flow in the embodiment of the present application.

[0029] Figure 5 It is a comparison result of the embodiment of the present application and other methods. The first three rows are comparison results under the condition of frontal face, and the last three rows are comparison results under different poses and expressions.

[0030] Figure 6 It is a comparison result method diagram of the embodiment of the present application. In order to better compare the makeup transfer effects of different methods, the results are locally enlarged.

[0031] Figure 7 It is a semantic corresponding effect and transfer result diagram of the embodiment of the present application. In order to verify the generalization ability of the model, the last two rows select two stars for experiments.

[0032] Figure 8 It is a makeup interpolation diagram of the embodiment of the present application. The present application extracts makeup features, and realizes the interpolation of makeup effects through the interpolation of makeup features.

[0033] Figure 9 It is a makeup editing example 1 of the embodiment of the present application. The present application proposes a user interaction mode, and the user can edit the makeup he likes.

[0034] Figure 10 It is a makeup editing example 2 of the embodiment of the present application. The present application proposes a user interaction mode, and the user can edit the makeup he likes. DETAILED DESCRIPTION

[0035] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0036] Makeup transfer aims to transfer the makeup style of a reference image to a source image while preserving the identity of the source image. While existing makeup transfer methods have achieved encouraging results, they still have two drawbacks: one is that the makeup similarity of the generated results is affected by synthetic pseudo-pairing data, and the other is that the learned semantic correspondences require a large amount of computation.

[0037] This embodiment proposes a makeup transfer method based on image colorization and alternative attention. Considering the close relationship between makeup and color changes, makeup transfer is defined as a sample-based colorization problem. Specifically, a color reference image is used as a sample to colorize a grayscale source image, i.e., makeup is simulated through colorization. Simultaneously, a novel alternative attention effectively maps low-resolution feature semantics to high-resolution features, thereby significantly reducing computational costs. Furthermore, this embodiment explores a new, high-demand makeup editing scenario and verifies the effectiveness of the proposed method in this scenario. Extensive experiments demonstrate the effectiveness and rationality of the proposed method.

[0038] This invention provides a face image makeup transfer method based on an image makeup transfer network, comprising the following steps:

[0039] Step 1: Obtain the original face image;

[0040] Step 2: Input the acquired original face image into the image makeup transfer network to generate the final makeup transfer result;

[0041] Please see Figure 1 This is a diagram illustrating the overall architecture of the image makeup transfer network in this embodiment. The image makeup transfer network in this embodiment includes a semantic correspondence subnetwork and an image coloring subnetwork. The semantic correspondence subnetwork is used to learn the dense semantic correspondence between the unmade-up image and the made-up image. Because makeup transfer requires transferring makeup to the corresponding semantic position in the target image, learning semantic correspondence is necessary. The coloring subnetwork's main function is to render the semantically aligned makeup features into the target image, generating the final makeup transfer result. These two subnetworks have clear objectives and each performs its own function, working together to generate a realistic makeup result.

[0042] Please see Figure 2 and Figure 3The semantic correspondence subnetwork includes a feature extractor and an alternative attention module. The feature extractor extracts rich spatial semantic features for feature matching. This embodiment observes that faces have certain structural features; when one point on the face corresponds to another point, there is also a correspondence around those two points. Based on this observation, this embodiment proposes a novel alternative attention module that successfully maps the semantic correspondence learned at low resolution to high-resolution features, significantly reducing computational complexity while aligning the semantic information of high-resolution features. The image color subnetwork includes an identity encoder, a color distiller, and a decoder. The identity encoder extracts the identity features of the person, and the color distiller distills the makeup features of the reference image. The unmodified identity features and the semantically aligned makeup features (completed by the aforementioned semantic correspondence subnetwork) are input into the decoder to generate the final makeup transfer result. The specific network structure consists of two main stacked modules: a convolution block and a residual block. A convolution block consists of a convolution layer, an instantiation normalization layer, and a reLU activation layer. A residual block consists of two convolution blocks, but the final activation layer is removed. The feature extractor consists of three convolution blocks with a stride of 2, the identity encoder and color distiller consist of two convolution blocks with a stride of 2 and two residual blocks, and the decoder consists of four residual blocks and two convolution blocks, with upsampling performed via bilinear interpolation. The alternative attention module consists of a pixel unshuffling operation, a cross-attention operation, and a pixel shuffle operation.

[0043] For the real data (x) gray Let X be the set of unmade-up colored images, and x be the unmade-up colored images, x∈X; let Y be the set of colored images with makeup, and y be the colored images with makeup, y∈X; x gray A grayscale image representing the color image x without makeup;

[0044] Step 2 includes the following sub-steps:

[0045] (1) Parse the faces corresponding to x and y (x parse y parse The two components are connected together and input into the feature extractor E. feature In the process, obtain the feature variables of the unmade-up image x and the made-up image y. and wherein x parse , y parse represents face parsing, which is obtained from a semantic segmentation network BiSeNet;

[0046] (2) L2 regularization is applied to the feature variables along the channel dimension and

[0047] (3) For each local feature variable at a spatial position, a matching score describing the size of the semantic correspondence is calculated in a dot product manner to model the semantic correspondence, and a correlation matrix M is obtained:

[0048]

[0049] wherein p and q are spatial coordinates in the source image and the reference image respectively, represents a local feature variable at the spatial position p of the source image feature variable , represents a local feature variable at the spatial position q of the reference image feature variable , and M(p, q) ∈ [0, 1] represents a matching score of the two spatial positions p and q; the closer the matching score is to 0, the weaker the semantic correspondence is; on the contrary, the closer the matching score is to 1, the stronger the semantic correspondence is;

[0050] (4) In the image colorization subnetwork, the color non-makeup image x is first converted into a grayscale image x gray by grayscale processing, and then input into the identity encoder E identity to extract the identity feature

[0051] (5) Then the color makeup image y is combined with the face parsing to separate the face region from y parse , and the face region is multiplied by y to obtain y face , which is then input into the color distiller E color to obtain the color feature and the background color feature of the non-makeup image x wherein the non-face region is separated from x parse , and the non-face region is multiplied by x to obtain x back ;

[0052] (6) The correlation matrix M calculated and the color feature of the makeup image are input into the proposed substitute attention module; first, the color feature The channel unshuffling operation is performed to match the spatial size with the correlation matrix M, and then the features at different spatial positions are aggregated to obtain the semantically aligned makeup features by using the correlation matrix M

[0053]

[0054] wherein, softmax j indicates performing the softmax operation on the j column, M(i,j) indicates the calculated correlation matrix, ∈ is a scaling factor, which is set to 100, pixel_unshuffle indicates the channel unshuffling operation, and i and j respectively indicate the spatial coordinates;

[0055] (7) The face color features of the distorted makeup image with the same size as the input features are obtained by using the inverse operation pixel_shuffle of the channel shuffling

[0056] (8) Finally, the identity features of the non-makeup image the background color features of the non-makeup image and the face color features of the distorted makeup image are concatenated in the channel dimension and input into the decoder to obtain the final makeup transfer result

[0057] During the training process, the input data includes the color simulation data (y gray , y warp ) and the real data (x gray , y); wherein, y gray and y warp respectively indicate the grayscale image and the image after affine transformation distortion of y, such as Figure 2 ; x gray indicates the grayscale image of the color non-makeup image x, and y indicates the color makeup image.

[0058] The following describes the makeup transfer process by taking the real data as an example. The processing flow of the simulation data is the same, and only the input is different.

[0059] In the semantic correspondence subnetwork, in order to calculate the correspondence, the embodiment adopts the strategy of extracting high-level features and then calculating the correlation in the feature space. First, the x and y corresponding to the face analysis channel dimension are connected and input into the feature extractor, E feature indicates the feature extractor. Then, the dot product operation in the document Deep exemplar-based colorization is used to calculate the semantic correspondence correlation matrix M. The source image x s and the reference image yr Feature variables and The next step is to perform a matching operation that measures the semantic relevance of different spatial locations. To obtain more accurate semantic correspondences, L2 regularization is applied along the channel dimension to the feature variables before feature matching. and In this process, normalized feature variables are obtained. Then, for the local feature variables at each spatial location, a matching score describing the magnitude of the semantic correspondence is calculated using a dot product method. The semantic correspondence is modeled, and the mathematical formula is described as follows:

[0060]

[0061] Where p and q are the spatial coordinates in the source image and the reference image, respectively. Indicates the feature variables located in the source image Local characteristic variables at spatial location p Indicates the feature variables located in the reference image The local feature variable at spatial location q, M(p, q) ∈ [0, 1], represents the matching score between two spatial locations p and q. The closer the matching score is to 0, the weaker the semantic correspondence; conversely, the closer the matching score is to 1, the stronger the semantic correspondence.

[0062] Inspired by decoupling representation, this embodiment constructs an image color sub-network to decompose the input image into content features and makeup features. By exchanging makeup features, a makeup transfer effect is achieved. In the makeup transfer network, firstly, content encoders and makeup encoders with different functions are used to decompose the source image and reference image into content features and makeup features, respectively. Then, using the established semantic correspondence, the makeup features of the reference image are distorted to align semantically with the content features of the source image. Finally, the unmodified content features of the source image and the semantically aligned makeup features of the reference image are fused and input into the decoder to generate the final makeup transfer result. The specific process is as follows: In the image color sub-network, the colored, unmade-up image x is first converted to grayscale to obtain a grayscale image x. gray Then it is input into the identity encoder to extract identity features. Then, the colored makeup image y, combined with face analysis, undergoes simple image processing to obtain y. face Then it is input into a color distiller to obtain color features. In addition, since additional information such as the background of the unmade-up image does not need to be changed, the background color features of the unmade-up image are also extracted.

[0063] In the makeup transfer task, there are two main aspects to learn. One is to establish the semantic correspondence between the source image and the reference image, and the other is to extract the makeup style of the reference image. The semantic correspondence ensures that the reference makeup is accurately rendered to the semantically corresponding position of the source image, and the extraction of the makeup style ensures that the generated result has a similar makeup style to the reference makeup. According to the above analysis, this embodiment designs two networks, one semantic correspondence network for establishing semantic correspondence, and one makeup transfer network for feature decoupling and result generation. However, the computational complexity of the semantic correspondence network is proportional to the square of the spatial size of the image. Therefore, this embodiment designs an alternative attention module to solve this problem, so that high-resolution features can also be used for semantic correspondence operations, greatly improving the makeup similarity.

[0064] After that, the correlation matrix M obtained by calculation and the color features of the makeup image are input into the proposed alternative attention mechanism. As Figure 4 , first, the color features are channel unshuffling operations to match the spatial size with the correlation matrix M. Then, the correlation matrix M is used to aggregate features at different spatial positions to obtain semantically aligned makeup features:

[0065]

[0066] Finally, the twisted features with the same size as the input features are obtained through the channel shuffling operation pixel_shuffle. Figure Two The intuitive explanation of this operation is that the correlation is calculated at a low resolution, and this correlation is mapped to a high-resolution patch for feature aggregation.

[0067] After inputting into the alternative attention module, the semantically aligned features of the non-makeup image are obtained in combination with the correlation matrix M. Finally, the identity features of the non-makeup image , the background color features of the non-makeup image , and the twisted facial color features of the makeup image are concatenated in the channel dimension and input into the decoder to obtain the final makeup transfer result

[0068] The loss function used in the training process includes a semantic loss function, an identity loss function, a local color loss function, and a reconstruction loss function, and a total loss function is obtained by weighting;

[0069] Semantic loss function: the embodiment uses the semantic loss function proposed in the article A symmetric semantic-aware transformer network for makeup transfer and removal, which constrains the semantic correspondence to be established in the same semantic space without supervision.

[0070] Identity loss function: the embodiment uses the identity loss function proposed in the article A symmetric semantic-aware transformer network for makeup transfer and removal, which constrains the gradient consistency of the transfer result and the non-makeup image to maintain the consistency of the identity.

[0071] Local color loss function: the embodiment proposes a differentiable local color histogram loss function:

[0072]

[0073] Where item∈{lip, eye, face} represents that the face region is divided into lips, eyes and face (the face does not include lips and eyes), ⊙ represents point-by-point multiplication, hist represents histogram statistics, Mask item represents the parsing mask corresponding to the face semantic information, and ||·||1 represents the L1 norm. This loss function constrains the generated result in the same local semantic region to be consistent with the reference makeup color distribution.

[0074] Reconstruction loss function: when the input data is (y gray , y warp ), the embodiment uses L1Norm and VGG19-based perceptual loss to constrain the reconstruction error, and constrain the consistency between the transfer result and y gt . In addition, the embodiment also includes the widely used adversarial loss function, in which the least square loss is used instead of the negative log-likelihood loss to stabilize the training.

[0075] The experimental effect of the present application is shown in Figure 7 , and the comparison effect of other methods is shown in Figure 5 .

[0076] See Figure 6 , which is a comparison result method diagram of the embodiment of the present application. In order to better compare the makeup transfer effects of different methods, the results are locally enlarged.

[0077] Comparing the eye shadow makeup transfer effect, the area-level semantic correspondence cannot meet the needs of eye shadow makeup transfer, and the generation result of BeautyGAN cannot effectively transfer the eye shadow makeup of the reference image to the result, such as the second row and the fifth row. For the fifth row of makeup eye shadow, PSGAN also fails to achieve satisfactory makeup transfer effect, and the eye shadow style of the generated result is significantly different from the reference image.

[0078] Comparing the blush makeup transfer effect, BeautyGAN fails to effectively extract the blush style, resulting in no blush makeup transfer effect in the generated result, such as the third row and the sixth row. PSGAN extracts part of the blush information, but the blush color of the generated result is still relatively light, and there is still a large difference from the reference makeup, such as the sixth row and the seventh row. In contrast, whether it is lipstick, eye shadow or blush, the makeup transfer result of the present embodiment is highly similar to the makeup style of the reference picture. In the fifth, sixth and seventh rows, for large-area blush and colorful eye shadow, other makeup transfer algorithms fail to effectively render the makeup of the reference image to the source image, while the makeup transfer algorithm of the present embodiment can still generate a real and accurate makeup transfer result.

[0079] The present application can be further improved:

[0080] (1) Makeup transfer strength control: Because the method extracts makeup information, it can control the strength of makeup transfer by giving a certain weight. For example, Figure 8 .

[0081] (2) Makeup editing: The proposed network structure can perform makeup editing. The meaning is that users can apply their favorite colors to the corresponding semantic positions of the reference image, and the method of the present embodiment can generate an ideal makeup transfer result, thereby achieving the purpose of makeup editing. Other makeup transfer technologies based on generative networks do not have this function. For example, Figure 9 , Figure 10 .

[0082] The present application studies makeup transfer from the perspective of coloring, defines makeup transfer as an image coloring problem, and therefore uses a large number of easily collected color images as self-supervised data for the network, thereby better avoiding the problem of lack of paired data. A large number of experiments have verified the rationality and effectiveness of this makeup transfer strategy in real-world environments. The present application proposes an alternative attention mechanism to calculate pixel-level semantic correspondence at low resolution, and uses this correspondence to aggregate patch blocks at high resolution, thereby greatly reducing the computational load without losing makeup details. The present application explores a new makeup editing scenario and verifies the effectiveness of the method of the present application in this scenario.

[0083] It should be understood that the above description is merely a detailed explanation of the preferred embodiments and is not intended to limit the patent protection scope of the present application. Any modification or alternation made by those skilled in the art without departing from the scope of the present application shall fall within the patent protection scope of the present application. The patent protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for face image makeup transfer based on an image makeup transfer network, characterized in that, Includes the following steps: Step 1: Obtain the original face image; Step 2: Input the acquired original face image into the image makeup transfer network to generate the final makeup transfer result; The image makeup transfer network consists of a semantic correspondence subnetwork and an image coloring subnetwork. The semantic correspondence subnetwork includes a feature extractor and an alternative attention module; The feature extractor is used to extract rich spatial semantic features for feature matching; the alternative attention module is used to successfully map the semantic correspondence learned at low resolution to high resolution features. The image color sub-network includes an identity encoder, a color distiller, and a decoder; the identity encoder is used to extract the identity features of the person; the color distiller is used to distill the makeup features of the reference image. The decoder is used to input the unmodified personal identity features and semantically aligned makeup features into the decoder to generate the final makeup transfer result. The feature extractor consists of three convolution blocks with a stride of 2 and a bilinear interpolation operation; the identity encoder and the color distiller each consist of two convolution blocks with a stride of 2 and two residual blocks; the decoder consists of four residual blocks and two convolution blocks, and performs upsampling through bilinear interpolation; the alternative attention module consists of a pixel unshuffle operation, a cross-attention operation, and a pixel shuffle operation. The Convolution block consists of a Convolution Layer, an Instantiation Normalization Layer, and a ReLUActivation Layer connected in sequence; the Residual block consists of two Convolution blocks connected in sequence, with the last ReLUActivation Layer removed. The image makeup transfer network is trained by jointly training a semantic correspondence subnetwork and an image coloring subnetwork. The loss function used during the training of the semantic correspondence subnetwork includes a local color loss function: in, This indicates that the face area is divided into lips, eyes, and face. This represents point-to-point multiplication. This represents histogram statistics. This represents the parsing mask corresponding to the semantic information of the face. Represents the L1 norm; Representing images respectively x and y The binary masks corresponding to the lip, eye, and face regions are separated by... and get,( )for x and y The corresponding face analysis; This indicates the makeup transfer results obtained during the training process; The loss function used in the training process of the color subnetwork on the image includes the local color loss function: in, This indicates that the face area is divided into lips, eyes, and face. This represents point-to-point multiplication. This represents histogram statistics. This represents the parsing mask corresponding to the semantic information of the face. Represents the L1 norm; These represent the binary masks for the lip, eye, and face regions in image x and y, respectively, separated by... and get; This indicates the results of makeup transfer obtained during the training process.

2. The face image makeup transfer method based on an image makeup transfer network according to claim 1, characterized in that: In step 2, for real data The collection of color photos of people without makeup is as follows: Colored unmade-up images are , The collection of colorful makeup images is as follows: Color makeup images are , ; Represents a color image without makeup. grayscale image; Step 2 includes the following sub-steps: (1) x and y The corresponding face analysis ( The two inputs are connected together and fed into the feature extractor. Get unmade-up photos Feature variables and , , ;in, This indicates that the face is parsed and obtained from the semantic segmentation network BiSeNet; (2) Apply L2 regularization along the channel dimension to the feature variables and In this process, normalized feature variables are obtained; (3) For the local feature variables at each spatial location, the matching score describing the magnitude of the semantic correspondence is calculated by the dot product method, and the semantic correspondence is modeled to obtain the relevance matrix. : in, and These are the spatial coordinates in the source image and the reference image, respectively. Indicates the feature variables located in the source image Spatial location Local characteristic variables at that location, Indicates the feature variables located in the reference image Spatial location Local characteristic variables at that location, Indicates two spatial locations and The closer the matching score is to 0, the weaker the semantic correspondence; conversely, the closer the matching score is to 1, the stronger the semantic correspondence. (4) In the image on-color network, the colored unmade-up image x First, a grayscale image is obtained by converting it to grayscale. Then input into the identity encoder Extracting identity features ; (5) For color makeup images Combining facial analysis, using Separate the face region, multiply it by y, and obtain... Then input into the color distiller In the process of obtaining color features and unmade-up images x Background color features Among them, utilizing Separate the non-face regions and combine them with... x Performing a dot product, we get ; (6) Calculate the obtained correlation matrix Color characteristics of makeup images The input is fed into the alternative attention module; first, the color features are processed. Perform a channel deshuffling operation to adjust the spatial size and correlation matrix. Matching; then using the correlation matrix By aggregating features from different spatial locations, semantically aligned makeup features are obtained. ; ； in, Indicates to j The column performs a softmax operation. This represents the calculated correlation matrix. Scaling factor This indicates a channel deshuffling operation. i , j Represent spatial coordinates respectively; (7) Shuffling operation through the channel Obtain facial color features from a distorted makeup image with the same feature size as the input. ; (8) Finally, identify the characteristics of the unmade-up image. Background color characteristics of unmade-up images and facial color features in distorted makeup images The channel dimensions are concatenated and input into the decoder to obtain the final makeup transfer result. .

3. The face image makeup transfer method based on an image makeup transfer network according to claim 1, characterized in that: The input data during the training of the semantic correspondence subnetwork includes coloring simulation data. and real data ;in Let represent the grayscale image of y and the image after affine transformation distortion, respectively. Represents a color image without makeup. grayscale image, The image represents a colored makeup image; the loss functions used during training include semantic loss function, identity loss function, local color loss function, and reconstruction loss function; a weighted overall loss function is obtained.

4. The face image makeup transfer method based on an image makeup transfer network according to claim 1, characterized in that: The input data during the training of the image coloring subnetwork includes coloring simulation data. and real data ;in, These represent the grayscale image of y and the image after affine transformation distortion, respectively; Represents a color image without makeup. grayscale image, The image represents a colored makeup image; the loss functions used during training include semantic loss function, identity loss function, local color loss function, reconstruction loss function, and adversarial loss function; a weighted overall loss function is obtained; in the adversarial loss function, least squares loss is used instead of negative log-likelihood loss to stabilize training.

5. A face image makeup transfer system based on an image makeup transfer network, used to implement the method described in any one of claims 1-4; characterized in that, Includes a module for acquiring original facial images and a module for applying makeup: The original face image acquisition module is used to acquire the original face image; The makeup transfer module is used to input the acquired original face image into the image makeup transfer network to generate the final makeup transfer result. The image makeup transfer network consists of a semantic correspondence subnetwork and an image coloring subnetwork. The semantic correspondence subnetwork includes a feature extractor and an alternative attention module; The feature extractor is used to extract rich spatial semantic features for feature matching; the alternative attention module is used to successfully map the semantic correspondence learned at low resolution to high resolution features. The image color sub-network includes an identity encoder, a color distiller, and a decoder; the identity encoder is used to extract the identity features of the person; the color distiller is used to distill the makeup features of the reference image. The decoder is used to input the unmodified personal identity features and semantically aligned makeup features into the decoder to generate the final makeup transfer result. The feature extractor consists of three convolution blocks with a stride of 2 and a bilinear interpolation operation; the identity encoder and the color distiller each consist of two convolution blocks with a stride of 2 and two residual blocks; the decoder consists of four residual blocks and two convolution blocks, and performs upsampling through bilinear interpolation; the alternative attention module consists of a pixel unshuffle operation, a cross-attention operation, and a pixel shuffle operation. The Convolution block consists of a Convolution Layer, an Instantiation Normalization Layer, and a ReLUActivation Layer connected in sequence; the Residual block consists of two Convolution blocks connected in sequence, with the last ReLUActivation Layer removed.

Images