A neural network-based intelligent makeup imitation system and makeup imitation generation method

Abstract

The present invention provides an intelligent makeup imitation system based on a neural network. The system comprises: a facial attribute encoder (FAEnc) for extracting feature maps of an original image and a reference image; a makeup transfer module (MTM) for processing the feature map of the reference image; and an application makeup decoder (MADec) for decoding the feature map of the original image and a makeup matrix output by the makeup transfer module (MTM), thereby generating a natural and realistic makeup imitation image. The system retains the facial details of the source face while accurately transferring the makeup details, thus solving the problem of detail loss in makeup imitation generation tasks, which results in only copying the overall makeup style but not accurately transferring the makeup details.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent imitation makeup, and in particular relates to an intelligent imitation makeup system and an imitation makeup generation method based on a neural network. Background Art

[0002] In existing technologies, by treating makeup style as color distribution, spatial information is ignored and high-frequency information about makeup details is lost. Only the overall makeup style can be copied, and the makeup details cannot be accurately transferred. Capturing the correspondence between the source image and the reference image through traditional pixel-level cross-attention requires a lot of computational overhead, which limits the size of the hidden feature map and leads to color leakage problems. During the makeup transfer process, since the image is downsampled to a low-resolution feature map, high-frequency information is lost, resulting in changes in detailed attributes that should remain unchanged during the imitation process. Summary of the Invention

[0003] To address the above problems, the present invention provides an intelligent makeup imitation system based on a neural network to solve the problem in the existing technology that details are lost in the makeup imitation generation task, and only the overall makeup style can be copied but the makeup details cannot be accurately transferred.

[0004] According to a first aspect of an embodiment of the present invention, there is provided an intelligent makeup imitation system based on a neural network, comprising:

[0005] Facial attribute encoder (FAEnc), used to extract feature maps of the original image and reference image;

[0006] Makeup Transfer Module (MTM), which processes the feature map of the reference image;

[0007] A makeup decoder (MADec) is applied to decode the feature map of the original image and the makeup matrix output by the makeup transfer module (MTM), thereby generating an imitation makeup image.

[0008] Preferably, the facial attribute encoder (FAEnc) is used to extract the high-resolution feature map and the low-resolution feature map of the original image and the high-resolution feature map and the low-resolution feature map of the reference image.

[0009] Preferably, the makeup transfer module (MTM) includes a first attention module and a sliding window attention module, wherein:

[0010] The first attention module is used to process the low-resolution feature map of the reference image and align the low-resolution feature map of the reference image with the face of the source image by using pixel-wise cross-attention. Each position on the source image corresponds to the area with the most similar features on the reference image. The low-resolution feature map of the reference image is deformed according to these soft pixel correspondences.

[0011] Sliding window attention module to align the high-resolution feature map of the reference image with the source image face.

[0012] Preferably, the sliding window attention module is used to align the high-resolution feature map of the reference image with the face of the source image, comprising: roughly aligning the high-resolution feature map of the reference image with the face of the source image in space by thin plate spline transformation;

[0013] Utilizing staggered overlapping panes to segment the high-resolution feature map of the reference image and applying the crisscross attention multiple times;

[0014] The smoothed high-resolution makeup matrix is ​​output by weighted summation based on position weights.

[0015] Preferably, the applied makeup decoder is further configured to perform element-wise multiplication of the deformed high-resolution makeup matrix with the high-resolution feature map of the source image;

[0016] The applied makeup decoder is used to perform element-wise multiplication of the deformed low-resolution makeup matrix and the low-resolution feature map of the source image.

[0017] Preferably, the applied makeup decoder is further used to fuse the result of element-wise multiplication of the deformed high-resolution makeup matrix and the high-resolution feature map of the source image with the result of element-wise multiplication of the deformed low-resolution makeup matrix and the low-resolution feature map of the source image, and generate a final imitation makeup image.

[0018] Preferably, the applied makeup decoder is further configured to manipulate the deformed makeup matrix through linear interpolation to achieve makeup transfer in a specified area with controllable concentration.

[0019] Preferably, the pseudo-real value: the applied makeup decoder is further used to generate a pseudo-real value for transferring the makeup of the reference image to the face of the source image in a coarse-to-fine manner, including two stages of color matching and detail matching, based on thin plate spline transform, histogram matching and linear annealing;

[0020] The pseudo-true value is used to supervise the neural network generator to learn the required mapping, providing additional supervision signals for the generation of makeup details.

[0021] Preferably, the color matching includes: dividing the facial area into skin, lips, and eye area, performing histogram matching on the corresponding areas of the source image and the makeup-transferred image generated by the neural network generator, respectively, to make their color distribution consistent with the color distribution of the corresponding areas of the reference image, and mixing the two matching results with a specific weight;

[0022] The detail matching includes: dividing the facial area into skin, lips and eye area, warping the corresponding area in the reference image to align with the corresponding area in the source image based on the detected facial key points, mixing the aligned corresponding area of ​​the reference image with the pseudo-true value obtained in the color matching stage with specific weights, and smoothing the edges.

[0023] Preferably, the linear annealing method includes: in the color matching stage, gradually reducing the mixing weight to generate a pseudo-real value with a more natural color; in the detail matching stage, gradually increasing the mixing weight in the early stage of training to increase the target difficulty, allowing the neural network generator to first learn the global color distribution and then learn the local makeup details; in the later stage of training, reducing the mixing weight to avoid the neural network generator from learning artificial traces.

[0024] According to a second aspect of an embodiment of the present invention, a method for generating imitation makeup is provided, comprising:

[0025] Extract feature maps of the original image and the reference image;

[0026] Processing the feature map of the reference image;

[0027] The feature map of the original image and the processed feature map of the reference image are decoded to generate an imitation makeup image.

[0028] Preferably, the feature maps of the original image and the reference image are extracted as follows:

[0029] Extract the high-resolution feature map and low-resolution feature map of the original image and the high-resolution feature map and low-resolution feature map of the reference image.

[0030] Preferably, the processing of the feature map of the reference image is specifically:

[0031] Processing the low-resolution feature map of the reference image and aligning it with the face of the source image through pixel-wise cross-attention, where each location on the source image corresponds to the region with the most similar features on the reference image, and the low-resolution feature map of the reference image is deformed according to these soft pixel correspondences;

[0032] The high-resolution feature map of the reference image is pane-processed, specifically: the high-resolution feature map of the reference image is roughly spatially aligned with the source image face through thin plate spline transformation; the high-resolution feature map of the reference image is segmented using staggered overlapping panes, and the cross-attention is applied multiple times; and a smoothed high-resolution makeup matrix is ​​output through weighted summation based on position weights.

[0033] The present invention has at least the following effects:

[0034] The problem of being able to more accurately transfer makeup details in the makeup imitation generation task.

[0035] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is a schematic diagram of an embodiment of the makeup imitation system of the present invention;

[0038] Figure 2 This is a flow chart provided by an embodiment of the makeup imitation system of the present invention;

[0039] Figure 3 Schematic diagram of the sliding window attention module of the makeup imitation system of the present invention. DETAILED DESCRIPTION

[0040] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0041] Figure 1 This is a schematic diagram provided by an embodiment of the imitation makeup system of the present invention.

[0042] like Figure 1 As shown, the present embodiment is a neural network-based intelligent makeup imitation system, comprising:

[0043] Facial attribute encoder (FAEnc), used to extract feature maps of the original image and reference image;

[0044] Makeup Transfer Module (MTM), which processes the feature map of the reference image;

[0045] A makeup decoder (MADec) is applied to decode the feature map of the original image and the makeup matrix output by the makeup transfer module (MTM) to generate an imitation makeup image.

[0046] In specific practice, the facial attribute encoder (FAEnc) is used to extract the high-resolution feature map and the low-resolution feature map of the original image and the high-resolution feature map and the low-resolution feature map of the reference image.

[0047] Reference Figure 2 , which is a flowchart provided for an embodiment of the makeup imitation system of the present invention. In specific practice, the makeup transfer module (MTM) includes a first attention module and a sliding window attention module, wherein:

[0048] The first attention module is used to process the low-resolution feature map of the reference image and align the low-resolution feature map of the reference image with the face of the source image by using pixel-wise cross-attention. Each position on the source image corresponds to the area with the most similar features on the reference image. The low-resolution feature map of the reference image is deformed according to these soft pixel correspondences.

[0049] Sliding window attention module to align the high-resolution feature map of the reference image with the source image face.

[0050] In specific practice, the sliding window attention module is used to align the high-resolution feature map of the reference image with the face of the source image, including: roughly aligning the high-resolution feature map of the reference image with the face of the source image in space through thin plate spline transformation;

[0051] Utilizing staggered overlapping panes to segment the high-resolution feature map of the reference image and applying the crisscross attention multiple times;

[0052] The smoothed high-resolution makeup matrix is ​​output by weighted summation based on position weights.

[0053] In practice, the makeup decoder is applied to element-wise multiply the deformed high-resolution makeup matrix with the high-resolution feature map of the source image.

[0054] The makeup decoder is applied to element-wise multiply the deformed low-resolution makeup matrix with the low-resolution feature map of the source image.

[0055] In specific practice, the result of element-wise multiplication of the deformed high-resolution makeup matrix and the high-resolution feature map of the source image is fused with the result of element-wise multiplication of the deformed low-resolution makeup matrix and the low-resolution feature map of the source image to generate the final imitation makeup image.

[0056] In specific practice, the high-resolution makeup matrix contains high-frequency information;

[0057] The low-resolution makeup matrix contains low-frequency information;

[0058] The high-resolution makeup matrix and the low-resolution makeup matrix are respectively composed of high-resolution features and low-resolution features of the reference image.

[0059] In practice, the makeup matrix is ​​deformed by the attention matrix and matched with the face in the source image, including:

[0060] An attention matrix containing the pixel-level soft alignment relationship between the makeup matrix and the source image face is calculated by pixel-level cross-attention, and the makeup matrix is ​​deformed and adapted to the source image according to the soft alignment relationship.

[0061] In specific practice, the deformed makeup matrix is ​​manipulated by linear interpolation to achieve makeup transfer in a specified area with controllable concentration.

[0062] In practice, the attention module, because the source image and the reference image have differences in expression and posture, the feature map Y obtained from the reference image H and Y L It cannot be directly applied to the source image. This paper aligns the feature map with the source face through attention. Attention can be interpreted as each position on the source image corresponds to the area with the most similar features on the reference image. The feature map is deformed according to these soft pixel correspondences. Formally, given a pair of feature maps extracted from the source image and the reference image, Where C, H and W are the number of channels, height and width of the feature map, and the attention matrix The calculation method is

[0063]

[0064] in is a learnable parameter, The feature map is spliced ​​with the position embedding. Here, the position embedding adopts the key point embedding method to introduce spatial features into attention. Given N facial key points For the i-th pixel x on the image i ,Position Embedding By x i The vector is calculated relative to these key points.

[0065]

[0066] Where x(·) and y(·) represent the x-coordinate and y-coordinate of a point respectively, and then p iis normalized to the two-norm to ensure independence from the size and is related to the pixel x i The visual feature vectors at are concatenated.

[0067] Makeup features From the reference feature map Y through the learnable weight The linear projection is extracted, and the makeup matrix and Y keep the spatial correspondence.

[0068] The attention matrix A is applied to the makeup feature Γ0 to deform it and align it with the spatial distribution of X,

[0069] Γ=A(YV) (3)

[0070] Afterwards, the deformed makeup matrix Γ becomes the input of the makeup decoder and is then element-wise multiplied with the feature map X of the source image.

[0071]

[0072] Because the deformed makeup matrix Γ corresponds to the source image in space, before inputting the makeup decoder, the operation on Γ can be used to achieve the generation of controllable makeup imitation with controllable shade for specific areas.

[0073] In specific practice, the sliding window attention module is used to: spatially align the feature map of the reference image with the face of the source image through thin plate spline transformation;

[0074] Segmenting the feature map of the reference image using staggered overlapping panes and applying the crisscross attention multiple times;

[0075] The smoothed makeup matrix is ​​output by weighted summation based on position weights.

[0076] In practice, traditional pixel-level cross-attention requires a large amount of computational overhead, which is proportional to the square of the number of pixels. The limited cost limits the size of the input feature map. Therefore, during the downsampling process, a large amount of high-frequency information is lost, making it difficult to retain detailed features, such as wrinkles in the source image and makeup details to be transferred, such as eye shadow in the reference image. The present invention designs a sliding window attention module suitable for higher-resolution feature maps, which can process makeup details and support precise editing of the makeup matrix along the spatial dimension. The sliding window attention module includes:

[0077] The reference feature map Y is roughly aligned with the source image face in space through the thin plate spline transformation. The thin plate spline transformation is determined by N control points. The coordinates of these control points in the original space and the target space are expressed as and Where C is set as the coordinates of N coordinate points on the source surface, C′ is set as the coordinates of the reference surface, and then the thin plate spline transformation converts Y into Y′ through parameterized window sampling to adapt C to C′. The two-dimensional thin plate spline transformation in the preset formula is differentiable with respect to the input, and gradient descent can be applied to optimize the neural network model.

[0078] Reference Figure 3 , a schematic diagram of the sliding window attention module of the makeup imitation system of the present invention. In sliding window attention, because the source image feature map X and the transformed reference image feature map Y′ are roughly aligned, local attention is sufficient to allow a point on X to acquire detailed makeup from the corresponding neighboring region on Y. However, segmenting the feature map into non-overlapping pane units for local attention presents discontinuities, resulting in obvious artifacts at the pane edges in the output image. The present invention utilizes staggered overlapping panes to segment the reference image feature map and applies the cross-attention multiple times. Formally, the feature map segmentation is defined by an S×S pane, where the hyperparameter S is the pane size. Each 3S / 2×3S / 2 region generates four staggered overlapping segmented panes, with a sliding distance of S / 2 between panes. The attention module is then applied to each corresponding pane of the source image feature map and the reference image feature map. Sliding window attention can also be understood as: the feature map is segmented into S / 2×S / 2 patches, and each patch on the source image feature map X participates in four attention calculations, with attention paid to the reference image feature map Y′ in each attention calculation.

[0079] Smoothing To ensure that the output makeup matrix is ​​continuous along the spatial dimension, the attention outputs (local makeup matrices) generated by the multiple panes are aggregated and summed based on the position weights to output the smoothed makeup matrix. Let w be a pane unit, x i ∈w is a pixel in the cell, position x i The attention value output in this pane According to the weight W(x i ,w) is scaled, where W(x i , w) is composed of pixels x i The relative position of the pane unit w is determined

[0080]

[0081] where c w is the center of the pane w, x(·) and y(·) represent the x-coordinate and y-coordinate of a point respectively, and the position x in the complete makeup matrix is i The final value is

[0082]

[0083] W(x i, w) meets the following two requirements: W(x i , w) is continuous in space; for each pixel x i , Therefore W(x i , w) ensures the continuity of the output attention matrix.

[0084] In practice, the loss function for training the sliding window attention autoencoder is the adversarial loss and Cycle consistency loss Perceptual loss and makeup loss The weighted sum of .

[0085] The total loss is expressed as

[0086] The makeup loss is the Euclidean distance between the image generated by the model and the pseudo-true value

[0087]

[0088] In practice, the pseudo-truth value is generated by transferring the makeup of the reference image to the face of the source image through a coarse-to-fine approach, including color matching and detail matching, based on thin plate spline transform, histogram matching and linear annealing.

[0089] The pseudo-true value is used to supervise the neural network generator to learn the required mapping, providing additional supervision signals for the generation of makeup details.

[0090] In specific practice, the color matching includes: dividing the facial area into skin, lips and eye area, using histogram matching for the corresponding areas of the source image and the makeup-transferred image generated by the neural network generator, so that their color distribution is consistent with the color distribution of the corresponding areas of the reference image, and mixing the two matching results with specific weights.

[0091] In specific practice, the detail matching includes: dividing the facial area into skin, lips and eye area, warping the corresponding area in the reference image to align with the corresponding area in the source image based on the detected facial key points, mixing the aligned corresponding area of ​​the reference image with the pseudo-true value obtained in the color matching stage with specific weights, and smoothing the edges.

[0092] In specific practice, the linear annealing method includes: in the color matching stage, gradually reducing the mixing weight to generate a pseudo-real value with a more natural color; in the detail matching stage, gradually increasing the mixing weight in the early stage of training to increase the target difficulty, allowing the neural network generator to first learn the global color distribution and then learn the local makeup details; in the later stage of training, reducing the mixing weight to avoid the neural network generator from learning artificial traces.

[0093] The embodiment of the present application also discloses a method for generating imitation makeup, which specifically includes the following steps: extracting feature maps of the original image and the reference image;

[0094] Processing the feature map of the reference image;

[0095] The feature map of the original image and the processed feature map of the reference image are decoded to generate an imitation makeup image.

[0096] In specific practice, the feature maps of the original image and the reference image are extracted as follows:

[0097] Extract the high-resolution feature map and low-resolution feature map of the original image and the high-resolution feature map and low-resolution feature map of the reference image.

[0098] In specific practice, the processing of the feature map of the reference image is specifically as follows:

[0099] Processing the low-resolution feature map of the reference image and aligning it with the face of the source image through pixel-wise cross-attention, where each location on the source image corresponds to the region with the most similar features on the reference image, and the low-resolution feature map of the reference image is deformed according to these soft pixel correspondences;

[0100] The high-resolution feature map of the reference image is pane-processed, specifically: the high-resolution feature map of the reference image is roughly spatially aligned with the source image face through thin plate spline transformation; the high-resolution feature map of the reference image is segmented using staggered overlapping panes, and the cross-attention is applied multiple times; and a smoothed high-resolution makeup matrix is ​​output through weighted summation based on position weights.

[0101] It can be understood that the technical solution provided in this embodiment is an intelligent makeup imitation system based on a neural network, which includes: a facial attribute encoder (FAEnc) for extracting feature maps of the original image and the reference image; a makeup transfer module (MTM) for processing the feature map of the reference image; and an application makeup decoder (MADec) for decoding the feature map of the original image and the makeup matrix output by the makeup transfer module (MTM), thereby generating a natural and realistic makeup imitation image, while retaining the facial details of the source face while accurately transferring the makeup details, solving the problem of details being lost in the makeup imitation generation task, and only being able to copy the overall makeup style but not accurately transferring the makeup details.

[0102] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0103] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is at least two.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0105] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0106] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0107] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0108] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0109] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0110] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An intelligent makeup imitation system based on neural network, characterized in that: The imitation makeup system comprises: Facial attribute encoder (FAEnc), used to extract feature maps of the original image and reference image; Makeup Transfer Module (MTM), which processes the feature map of the reference image; Applying a makeup decoder (MADec) to decode the feature map of the original image and the makeup matrix output by the makeup transfer module (MTM) to generate an imitation makeup image; The makeup transfer module (MTM) includes a first attention module and a sliding window attention module, wherein: The first attention module is used to process the low-resolution feature map of the reference image and align the low-resolution feature map of the reference image with the face of the source image by using pixel cross attention. Each position on the source image corresponds to the area with the most similar features on the reference image. The low-resolution feature map of the reference image is deformed according to the soft pixel correspondence. Specifically: Given a pair of feature maps extracted from a source image and a reference image, , where C, H and W are the number of channels, height and width of the feature map, and the attention matrix The calculation method is: (1) in, is a learnable parameter, X̃,Ỹ is the concatenation of the feature map and the position embedding; wherein, the position embedding adopts the key point embedding method, specifically: Introducing spatial features into attention, given N facial key points , for the i-th pixel on the image ,Position Embedding is The vector is calculated relative to these key points. （2) Where x(·) and y(·) represent the x-coordinate and y-coordinate of a point respectively, then Normalized to the 2-norm to ensure independence from size and to the pixel The visual feature vectors at are spliced; Makeup features From the reference feature map Y through the learnable weight The linear projection of is extracted, and the makeup matrix and Y maintain spatial correspondence; The attention matrix A is applied to the makeup features , to deform it and align it with the spatial distribution of X to generate the deformed makeup matrix , as shown below: (3) Deformed makeup matrix becomes the input of the makeup decoder and is then multiplied element-wise with the feature map X of the source image as shown below: (4) Because the deformed makeup matrix corresponds to the source image spatially, so before applying the makeup decoder to the input, The operation can realize the generation of imitation makeup with controllable shade for specific areas; Sliding window attention module to align the high-resolution feature map of the reference image with the source image face.

2. The makeup imitation system according to claim 1, characterized in that: The facial attribute encoder (FAEnc) is used to extract the high-resolution feature map and the low-resolution feature map of the original image and the high-resolution feature map and the low-resolution feature map of the reference image.

3. The makeup imitation system according to claim 1, characterized in that: The sliding window attention module aligns the high-resolution feature map of the reference image with the face of the source image, including: roughly aligning the high-resolution feature map of the reference image with the face of the source image in space through thin plate spline transformation; Utilizing staggered overlapping panes to segment the high-resolution feature map of the reference image and applying the crisscross attention multiple times; The smoothed high-resolution makeup matrix is ​​output by weighted summation based on position weights.

4. The imitation makeup system according to claim 1 or 3, characterized in that: The applied makeup decoder is further configured to perform element-wise multiplication of the deformed high-resolution makeup matrix with the high-resolution feature map of the source image; The applied makeup decoder is further configured to perform element-wise multiplication of the deformed low-resolution makeup matrix with the low-resolution feature map of the source image.

5. The makeup imitation system according to claim 4, characterized in that: The applied makeup decoder is further used to fuse the result of element-wise multiplication of the deformed high-resolution makeup matrix and the high-resolution feature map of the source image with the result of element-wise multiplication of the deformed low-resolution makeup matrix and the low-resolution feature map of the source image, and generate a final imitation makeup image.

6. The makeup imitation system according to claim 1, characterized in that: The applied makeup decoder is further used to manipulate the deformed makeup matrix through linear interpolation to achieve makeup transfer in a specified area with controllable concentration.

7. The makeup imitation system according to claim 1, characterized in that: Also includes: The applied makeup decoder is further used to generate a pseudo-truth value for transferring the makeup of the reference image to the face of the source image in a coarse-to-fine manner, including two stages of color matching and detail matching, based on thin plate spline transform, histogram matching and linear annealing method; The pseudo-true value is used to supervise the neural network generator to learn the required mapping, providing additional supervision signals for the generation of makeup details.

8. The makeup imitation system according to claim 7, characterized in that: The color matching includes dividing the facial area into skin, lips, and eye area, performing histogram matching on the corresponding areas of the source image and the makeup-transferred image generated by the neural network generator, respectively, to make their color distribution consistent with the color distribution of the corresponding areas of the reference image, and mixing the two matching results with a specific weight; The detail matching includes: dividing the facial area into skin, lips and eye area, warping the corresponding area in the reference image to align with the corresponding area in the source image based on the detected facial key points, mixing the aligned corresponding area of ​​the reference image with the pseudo-true value obtained in the color matching stage with specific weights, and smoothing the edges.

9. The imitation makeup system according to claim 7, characterized in that: The linear annealing method includes: in the color matching stage, gradually reducing the mixing weight to generate a pseudo-real value with a more natural color; in the detail matching stage, in the early stage of training, gradually increasing the mixing weight to increase the target difficulty, allowing the neural network generator to first learn the global color distribution and then learn the local makeup details; in the later stage of training, reducing the mixing weight to prevent the neural network generator from learning artificial traces.

10. A method for generating imitation makeup, characterized in that: The method comprises: Extract feature maps of the original image and the reference image; Processing the feature map of the reference image; Decoding the feature map of the original image and the processed feature map of the reference image to generate an imitation makeup image; The processing of the feature map of the reference image is specifically as follows: The low-resolution feature map of the reference image is processed and aligned with the face of the source image by pixel cross attention. Each position on the source image corresponds to the area with the most similar features on the reference image. The low-resolution feature map of the reference image is deformed according to these soft pixel correspondences, specifically: Given a pair of feature maps extracted from a source image and a reference image, , where C, H and W are the number of channels, height and width of the feature map, and the attention matrix The calculation method is: (1) in, is a learnable parameter, X̃,Ỹ is the concatenation of the feature map and the position embedding; wherein, the position embedding adopts the key point embedding method, specifically: Introducing spatial features into attention, given N facial key points , for the i-th pixel on the image ,Position Embedding is The vector is calculated relative to these key points. （2) Where x(·) and y(·) represent the x-coordinate and y-coordinate of a point respectively, then Normalized to the 2-norm to ensure independence from size and to the pixel The visual feature vectors at are spliced; Makeup features From the reference feature map Y through the learnable weight The linear projection of is extracted, and the makeup matrix and Y maintain spatial correspondence; The attention matrix A is applied to the makeup features , to deform it and align it with the spatial distribution of X to generate the deformed makeup matrix , as shown below: (3) Deformed makeup matrix becomes the input of the makeup decoder and is then multiplied element-wise with the feature map X of the source image as shown below: (4) Because the deformed makeup matrix corresponds to the source image spatially, so before applying the makeup decoder to the input, The operation can realize the generation of imitation makeup with controllable shade for specific areas; The high-resolution feature map of the reference image is pane-processed, specifically: the high-resolution feature map of the reference image is roughly spatially aligned with the source image face through thin plate spline transformation; the high-resolution feature map of the reference image is segmented using staggered overlapping panes, and the cross-attention is applied multiple times; and a smoothed high-resolution makeup matrix is ​​output through weighted summation based on position weights.

11. The makeup imitation generation method according to claim 10, characterized in that: The feature maps of the original image and the reference image are specifically extracted as follows: Extract the high-resolution feature map and low-resolution feature map of the original image and the high-resolution feature map and low-resolution feature map of the reference image.

Images