A model training, image processing method, device, equipment and medium

Abstract

The present invention discloses a model training, image processing method, device, equipment and medium. In the embodiment of the present invention, any sample image pair in a sample set is obtained, and the parameters of the generator of the lip makeup injection model are adjusted to finally obtain a trained lip makeup injection model. Feature matching is performed based on the original image and the pre-saved image, and after determining the target image matching the original image, the pre-trained lip makeup injection model is used to finally determine the completed image. This can improve the makeup texture of the generated image after makeup and improve the user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a model training, image processing method, device, equipment and medium. Background Art

[0002] With the rapid development of computer vision technology, lip makeup is widely used in content production and entertainment videos. The main process is to transfer the lip makeup information in the target image to the original image based on the original image and the target image, thereby generating the image after makeup.

[0003] However, since each person's skin color is different, it is difficult to provide and recommend the most suitable makeup to each person based on their skin color. In other words, it is difficult to ensure the beauty of the makeup and the naturalness of the generated image after makeup.

[0004] In the prior art, lip makeup can only provide a few sets of fixed makeup looks with few options, resulting in poor makeup texture of the images generated based on the fixed makeup looks, affecting the user experience. Summary of the invention

[0005] The present invention provides a model training, image processing method, device, equipment and medium, which are used to solve the problem in the prior art that the makeup texture of the makeup-applied images generated based on fixed makeup is poor, which affects the user experience.

[0006] In a first aspect, an embodiment of the present invention provides a model training method, the method comprising:

[0007] Obtain any sample image pair in the sample set, wherein the sample image pair includes: a sample original image and a sample target image;

[0008] Inputting the sample original image and the sample target image into the generator of the lip makeup injection model to obtain a predicted image after the makeup is completed;

[0009] Inputting the sample original image and the predicted image after makeup application into a pre-trained contrast feature extraction network to determine a first loss value;

[0010] Inputting the predicted image after makeup application into a pre-trained discriminator to determine a second loss value;

[0011] Inputting the predicted image after makeup application and the sample target image into a pre-trained discriminator to determine a third loss value;

[0012] According to the first loss value, the second loss value, and the third loss value and value, the parameters of the generator of the lip makeup injection model are adjusted.

[0013] In a second aspect, an embodiment of the present invention provides an image processing method, the method comprising:

[0014] Perform feature matching between the original image and a pre-saved image to determine a target image in the pre-saved image that matches the original image;

[0015] The pre-trained lip makeup injection model is used to determine a completed makeup image according to the input target image and the original image.

[0016] In a third aspect, an embodiment of the present invention provides a model training device, the device comprising:

[0017] An acquisition module is used to acquire any sample image pair in the sample set, wherein the sample image pair includes: a sample original image and a sample target image; the sample original image and the sample target image are input into a generator of a lip makeup injection model to obtain a predicted image after makeup is completed;

[0018] A first determination module is used to input the sample original image and the predicted image after makeup application into a pre-trained contrast feature extraction network to determine a first loss value; input the predicted image after makeup application into a pre-trained discriminator to determine a second loss value; input the predicted image after makeup application and the sample target image into a pre-trained discriminator to determine a third loss value;

[0019] An adjustment module is used to adjust the parameters of the generator of the lip makeup injection model according to the first loss value, the second loss value and the third loss value.

[0020] In a fourth aspect, an embodiment of the present invention provides an image processing device, the device comprising:

[0021] A matching module, used for performing feature matching between an original image and a pre-saved image, and determining a target image in the pre-saved image that matches the original image;

[0022] The second determination module is used to determine the completed makeup image according to the input target image and the original image through the pre-trained lip makeup injection model.

[0023] In a fifth aspect, an embodiment of the present invention provides an electronic device, which comprises at least a processor and a memory, wherein the processor is used to execute the steps of any of the above-mentioned model training methods or the steps of any of the above-mentioned image processing methods when executing a computer program stored in the memory.

[0024] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the above-mentioned model training methods or the steps of any of the above-mentioned image processing methods.

[0025] In an embodiment of the present invention, any sample image pair in a sample set is obtained, wherein the sample image pair includes: a sample original image and a sample target image; the sample original image and the sample target image are input into a generator of a lip makeup injection model to obtain a predicted image after makeup is completed; the sample original image and the predicted image after makeup is completed are input into a pre-trained contrast feature extraction network to determine a first loss value; the predicted image after makeup is completed is input into a pre-trained discriminator to determine a second loss value; the predicted image after makeup is completed and the sample target image are input into a pre-trained discriminator to determine a third loss value; and the parameters of the generator of the lip makeup injection model are adjusted according to the first loss value, the second loss value, and the third loss value. In the embodiment of the present invention, any sample image pair in the sample set is obtained, the parameters of the generator of the lip makeup injection model are adjusted, and finally a trained lip makeup injection model is obtained. Feature matching is performed based on the original image and the pre-saved image, and after the target image matching the original image is determined, the completed image is finally determined based on the pre-trained lip makeup injection model. This can improve the makeup texture of the generated image after makeup and enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a model training process provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of an image processing process provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a detailed process of feature fusion based on a residual module with adaptive normalization provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of a lip makeup process provided by an embodiment of the present invention;

[0031] Figure 5A schematic diagram of a process for obtaining a makeup-completed image based on a lip makeup injection model provided by an embodiment of the present invention;

[0032] Figure 6 A schematic diagram of the structure of a model training device provided by an embodiment of the present invention;

[0033] Figure 7 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention;

[0034] Figure 8 An electronic device is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0036] In order to improve the makeup texture of a generated image after makeup application and enhance user experience, an embodiment of the present invention provides an image processing method, device, electronic device and storage medium.

[0037] Embodiment 1:

[0038] Figure 1 A schematic diagram of a model training process provided by an embodiment of the present invention, the process includes the following steps:

[0039] S101: Acquire any sample image pair in a sample set, wherein the sample image pair includes: a sample original image and a sample target image.

[0040] In an embodiment of the present invention, before inputting the original image and the target image into the lip makeup injection model to obtain the completed image, the lip makeup injection model is trained in advance. For the sake of distinction, the original image of the training process is referred to as a sample original image, and the target image obtained by feature matching the sample original image with a pre-saved image during the training process is referred to as a sample target image. The completed image obtained by inputting the sample original image and the sample target image into the lip makeup injection model during the training process is referred to as a predicted image of completed makeup.

[0041] When training the lip makeup injection model, a sample set for training is predetermined. In an embodiment of the present invention, the sample set includes many sample image pairs, and the sample image pairs include: a sample image and a sample target image determined based on feature matching between the sample image and a pre-saved image.

[0042] S102: Input the sample original image and the sample target image into a generator of a lip makeup injection model to obtain a predicted image of a completed makeup application.

[0043] In an embodiment of the present invention, in order to generate a predicted image of a completed makeup, the lip makeup injection model includes a generator, and the sample image pair, that is, the sample original image and the sample target image, are input into the generator of the lip makeup injection model to obtain a predicted image of a completed makeup.

[0044] S103: Inputting the sample original image and the predicted image after makeup application into a pre-trained contrast feature extraction network to determine a first loss value.

[0045] In order to continuously optimize the lip makeup injection model in training, it is necessary to update the parameters of the generator according to the loss function in the lip makeup injection model. In an embodiment of the present invention, in order to make the generated predicted image after makeup maintain the characteristics of the original image, in an embodiment of the present invention, a first loss value can be determined according to the sample original image and the predicted image after makeup, wherein the first loss value is a feature loss value. Specifically, the sample original image and the predicted image after makeup are input into a pre-trained contrast feature extraction network, which can be a super-resolution test sequence network (Visual Geometry Group Network, VGG). Specifically, the first loss value is determined according to vgg_loss=vgg(gen_img)-vgg(src_img)||;

[0046] Among them, the vgg_loss is the first loss value, the vgg(gen_img) represents the feature value obtained after the predicted sample image with makeup is input into the pre-trained contrast feature extraction network, and the vgg(scr_img) represents the feature value obtained after the original sample image is input into the pre-trained contrast feature extraction network.

[0047] S104: Inputting the predicted image after makeup application into a pre-trained discriminator to determine a second loss value.

[0048] In order to make the generated makeup-completed image more natural, in an embodiment of the present invention, a second loss value may be determined according to a predicted sample image after makeup and a pre-trained discriminator, that is, the predicted sample image after makeup is input into the pre-trained discriminator to determine the second loss value, wherein the second loss value is an adversarial loss value, wherein the discriminator may be a discriminator network (dis), specifically, according to gan_loss=E[log(1-dis(gen_img))], the second loss value is determined;

[0049] Among them, gan_loss is the second loss value, and dis(gen_img) is the feature value obtained by inputting the predicted sample image with makeup into the pre-trained discriminator.

[0050] That is to say, in the process of determining the second loss value, the feature value obtained by inputting the predicted sample image with makeup into the pre-trained discriminator is determined, and then the difference between the preset value and the feature value is determined, and the difference is determined as the first value, wherein the preset value is 1, and the first value is logarithmically operated to obtain the second value, and the second value is taken as the expected value to obtain the second loss value.

[0051] S105: Inputting the predicted image after makeup application and the sample target image into a pre-trained discriminator to determine a third loss value.

[0052] In order to make the style of the generated image with makeup closer to the style of the target image and make the lip makeup more textured, in an embodiment of the present invention, the predicted image with makeup and the sample target image are input into a discriminator based on pre-training to determine a third loss value, wherein the third loss value is a feature matching loss value. The discriminator is a multi-scale discriminator, and the discriminator can also be dis.

[0053] Specifically, according to determining the third loss value;

[0054] Among them, ganFeat_loss is the third loss value, dis(gen_img) is the feature value obtained after the predicted image with makeup is passed through the discriminator, and dis(scr_img) is the feature value obtained after the sample target image is passed through the discriminator.

[0055] S106: Adjust the parameters of the generator of the lip makeup injection model according to the first loss value, the second loss value and the third loss value.

[0056] In the embodiment of the present invention, after obtaining the first loss value, the second loss value and the third loss value, that is, after obtaining the feature loss value, the adversarial loss value and the feature matching loss value, the parameters of the generator of the lip makeup injection model are adjusted according to the first loss value, the second loss value and the third loss value. Specifically, the loss function is determined according to total_loss=vgg_loss+gan_loss+ganFeat_loss;

[0057] Among them, total_loss is the loss function, vgg_loss is the first loss value, gan_loss is the second loss value, and ganFeat_loss is the third loss value.

[0058] After determining the loss function according to the first loss value, the second loss value and the third loss value, determine the independent variable of the loss function, wherein the independent variable of the loss function is also the parameter of the generator of the lip makeup injection model. In order to adjust the parameters of the generator of the lip makeup injection model, thereby realizing the training of the lip makeup injection model, in an embodiment of the present invention, the independent variable of the loss function is differentiated, the value of the independent variable corresponding to the minimum value of the loss function is determined, and the value is determined as the adjustment parameter corresponding to the generator and adjusted, wherein the smaller the value of the loss function, the better the lip makeup effect of the lip makeup injection model, and the larger the value of the loss function, the worse the lip makeup effect of the lip makeup injection model.

[0059] Embodiment 2:

[0060] In order to obtain a discriminator for training the lip makeup injection model, based on the above embodiment, in an embodiment of the present invention, the process of training the discriminator includes:

[0061] Inputting the predicted image after makeup application and the sample original image into the discriminator to determine the fourth loss value;

[0062] The parameters of the discriminator are adjusted according to the fourth loss value.

[0063] In an embodiment of the present invention, when training the discriminator in the lip makeup injection model, the predicted image of the completed makeup and the original sample image are input into the discriminator to obtain a fourth loss value, wherein the fourth loss value is an adversarial loss value, and the adversarial loss value is the loss function corresponding to the discriminator.

[0064] Specifically, according to dis_loss=E[logdis(gen_img)]+E[log(1-dis(scr_img))], the fourth loss value is determined;

[0065] Among them, dis_loss is the fourth loss value, dis(gen_img) is the feature value obtained by inputting the predicted image with makeup into the pre-trained discriminator, and dis(scr_img) is the feature value obtained by inputting the original image of the sample into the pre-trained discriminator.

[0066] That is to say, in the process of determining the fourth loss value, the first eigenvalue obtained by inputting the predicted image with makeup into the pre-trained discriminator is determined, and then a logarithmic operation is performed on the first feature to obtain a third value, the third value is taken as an expected value, and the fourth value is determined, the second eigenvalue obtained by inputting the original image of the sample into the pre-trained discriminator is determined, the difference between the preset value and the second eigenvalue is determined, and the difference is determined as the fifth value, wherein the preset value is 1, the fifth value is logarithmically operated to obtain a sixth value, the sixth value is taken as an expected value to obtain a seventh value, the sum of the seventh value and the fourth value is determined, and the sum is determined as the fourth loss value.

[0067] In order to adjust the parameters of the discriminator of the lip makeup injection model, in an embodiment of the present invention, the independent variable of the loss function is differentiated, the value of the independent variable corresponding to the minimum value of the fourth loss value is determined, and the value is determined as the adjustment parameter corresponding to the discriminator and adjusted, wherein the smaller the value of the fourth loss value, the better the lip makeup effect of the lip makeup injection model, and the larger the value of the fourth loss value, the worse the lip makeup effect of the lip makeup injection model.

[0068] Embodiment 3:

[0069] Figure 2 A schematic diagram of an image processing process provided by an embodiment of the present invention, the process includes the following steps:

[0070] S201: Perform feature matching between an original image and a pre-saved image to determine a target image in the pre-saved image that matches the original image.

[0071] The image processing method provided by the embodiment of the present invention is applied to an electronic device, wherein the electronic device may be an intelligent terminal device such as a mobile phone, a tablet computer, a PC, or a server.

[0072] Since each user has a different skin color, the lip makeup that suits each person is different. In order to select a suitable makeup, in an embodiment of the present invention, multiple images are pre-saved in the electronic device for feature matching with the original image to be lip-made up. The pre-saved image may be a pre-saved model image with lip makeup, and the skin color and / or lip makeup of the model in each image is different. After the original image is feature matched with the pre-saved image, a target image in the pre-saved image that matches the original image is determined based on the matching result of the pre-saved image with the original image.

[0073] S202: Input the target image and the original image into a pre-trained lip makeup injection model to determine a completed makeup image.

[0074] In an embodiment of the present invention, in order to accurately determine the image after makeup is applied, a lip makeup injection model is pre-trained. After obtaining the target image, the target image and the original image are input into the pre-trained lip makeup injection model to determine the image after makeup is applied.

[0075] Since in the embodiment of the present invention, feature matching is performed based on the original image and the pre-saved image to determine the target image matching the original image, and then the completed image is finally determined based on the pre-trained lip makeup injection model, the makeup texture of the generated image after makeup can be improved, thereby improving the user experience.

[0076] Embodiment 4:

[0077] In order to improve the makeup texture of the image after makeup is applied and enhance the user experience, based on the above embodiment, in an embodiment of the present invention, the lip makeup injection model completed by pre-training determines the image after makeup is applied according to the input target image and the original image, including:

[0078] Obtain a target makeup feature map corresponding to the target image and the original image by injecting a first sub-model for makeup feature extraction into a pre-trained lip makeup model;

[0079] A second sub-model for extracting shape attribute features is implemented by injecting the pre-trained lip makeup into the model to obtain a shape attribute feature map of the original image;

[0080] The shape attribute feature map and the target makeup feature map corresponding to the target image and the original image are input into a pre-trained lip makeup injection model to implement a third sub-model of lip makeup injection, and determine a completed makeup image.

[0081] In an embodiment of the present invention, in order to extract makeup features in a target image, the pre-trained lip makeup injection model has a first sub-model capable of extracting makeup features. In order to extract shape attribute features of the original image, the pre-trained lip makeup injection model has a second sub-model capable of extracting shape attribute features. In order to realize lip makeup injection, the pre-trained lip makeup injection model has a third sub-model capable of lip makeup injection.

[0082] Since makeup is only applied to the lips of the person in the original image when obtaining the image after makeup based on the original image and the target image, it is necessary to ensure that other features except the lips in the original image remain unchanged, wherein the other features except the lips may be tooth features, skin features, etc., so it is necessary to obtain the makeup features of the target image and the makeup features of the original image respectively. In order to obtain the target makeup feature map and improve the accuracy of makeup feature extraction, in an embodiment of the present invention, the target image and the original image are processed by the first sub-model of makeup feature extraction implemented in the pre-trained lip makeup injection model, that is, the target image and the original image are input into the first sub-model of makeup feature extraction implemented in the pre-trained lip makeup injection model, and the target makeup feature map corresponding to the target image and the original image is obtained.

[0083] In an embodiment of the present invention, the original image is processed by the second sub-model for extracting shape attribute features in the pre-trained lip makeup injection model, that is, the original image is input into the second sub-model for extracting shape attribute features in the pre-trained lip makeup injection model to obtain a shape attribute feature map of the original image, wherein the shape attribute feature map includes: upper and lower lip shape attribute feature maps, teeth shape attribute feature maps, and face shape attribute feature maps including skin parts, etc.

[0084] In an embodiment of the present invention, in order to accurately determine the image after makeup is completed, after obtaining the target makeup feature map and the shape attribute feature map, the target makeup feature map and the shape attribute feature map are input into the pre-trained lip makeup injection model to implement the third sub-model of lip makeup injection to determine the image after makeup is completed.

[0085] Since the lip makeup injection model has been pre-trained, the lip makeup injection model includes a first sub-model for realizing makeup feature extraction, a second sub-model for realizing shape attribute feature extraction, and a third sub-model for realizing lip makeup injection. Therefore, based on the target image, the original image and the pre-trained lip makeup injection model, a completed image can be obtained, thereby improving the makeup texture of the completed image, helping users determine the makeup that suits them, and improving the user experience.

[0086] Embodiment 5:

[0087] In order to obtain the target makeup feature map, retain the lip features of the target image and the features other than the lip features in the original image, thereby ensuring the makeup texture of the generated image after makeup, on the basis of the above embodiments, in an embodiment of the present invention, the first sub-model for makeup feature extraction is injected into the pre-trained lip makeup model, and before obtaining the target makeup feature map corresponding to the target image and the original image, the method further includes:

[0088] Based on the fourth sub-model for image segmentation in the pre-trained lip makeup injection model, each sub-target image in the target image and each sub-original image in the original image are obtained, and the first sub-target image including the upper and lower lips in the sub-target image and each second sub-original image except the lips in the sub-original image are obtained.

[0089] Since makeup is only applied to the lips of the face in the original image when obtaining the image after makeup based on the original image and the target image, it is necessary to obtain the lip features of the target image and other features in the original image except the lips. In an embodiment of the present invention, in order to obtain the lip features of the target image for the target image and other features in the original image except the lips for the original image, the pre-trained lip makeup injection model has a fourth sub-model that can realize image segmentation, and the first sub-model for makeup feature extraction is implemented in the pre-trained lip makeup injection model. Before obtaining the target makeup feature map corresponding to the target image and the original image, each sub-target image in the target image and each sub-original image of the original image are obtained based on the fourth sub-model, wherein each sub-image in the target image includes: a sub-target image containing the upper lip, a sub-target image containing the lower lip, a sub-target image containing the skin, and a sub-target image containing the teeth; each sub-original image of the original image includes: a sub-original image containing the upper lip, a sub-original image containing the lower lip, a sub-original image containing the skin, and a sub-original image containing the teeth.

[0090] After obtaining each sub-target image in the target image and each sub-original image of the original image, the sub-target image including the upper lip and the sub-target image including the lower lip included in each sub-target image in the target image are determined as first sub-target images, and each sub-original image except the sub-original image including the lips in each sub-original image of the original image is determined as each second sub-original image, that is, the sub-original image including the skin and the sub-original image including the teeth are determined as each second sub-original image.

[0091] In order to determine the target makeup feature map corresponding to the target image and the original image, based on the above embodiments, in an embodiment of the present invention, the first sub-model for makeup feature extraction is injected into the pre-trained lip makeup model, and obtaining the target makeup feature map corresponding to the target image and the original image includes:

[0092] Based on the convolution layer of the first sub-model for makeup feature extraction in the pre-trained lip makeup injection model, the first sub-target image, and each of the second sub-original images, a first feature map corresponding to the first sub-target image and each of the second sub-original images is obtained;

[0093] Based on the aggregate output layer of the first sub-model, each first feature map is fused into a second feature map, and the second feature map is determined as a target makeup feature map corresponding to the target image and the original image.

[0094] In order to obtain the target makeup feature map, the first sub-target image and the second sub-original image are input into the pre-trained lip makeup injection model to implement the first sub-model for makeup feature extraction. The first sub-model includes a convolution layer for convolving the sub-target image and the sub-original image, and an aggregation output layer for fusing the feature map.

[0095] Specifically, after the first sub-target image and the second sub-original image are input into the pre-trained lip makeup injection model, in the convolution layer of the first sub-model, a convolution operation is performed on each first sub-target image and each second sub-original image to obtain a first feature map corresponding to each sub-image, wherein the first feature map can be represented as F HWD , where H is the height of the first feature map, w is the width of the first feature map, and D is the number of output channels, that is, the total number of first sub-target images and second sub-original images. The first sub-target image contains: a first sub-target image containing the upper lip and a first sub-target image containing the lower lip, that is, the number of first sub-target images is 2, and the second sub-original image contains: a second sub-original image containing the skin and a second sub-original image containing the teeth, then the number of the second sub-original images is also 2, so D=4.

[0096] In order to make the target makeup feature map more accurate, based on the aggregation output layer in the first sub-model of the pre-trained lip makeup injection model, each first feature map is feature aggregated and aggregated into a second feature map, and the second feature map is determined as the target makeup feature map corresponding to the target image and the original image. The target makeup features corresponding to the target image can be determined based on the target makeup feature map.

[0097] Since the first sub-model for realizing makeup feature extraction in the pre-trained lip makeup injection model includes a convolutional layer and an aggregation output layer, a target makeup feature map is determined based on the target image, the original image and the pre-trained lip makeup injection model. The target makeup feature map can retain both the lip features of the target image and the features other than the lip features in the original image, thereby ensuring the makeup texture of the generated image with makeup applied and improving the user experience.

[0098] Embodiment 6:

[0099] In order to obtain the shape attribute feature map of the original image, based on the above embodiments, in an embodiment of the present invention, the second sub-model for extracting shape attribute features is injected into the pre-trained lip makeup model, and obtaining the shape attribute feature map of the original image includes:

[0100] Based on the second sub-model for extracting shape attribute features in the pre-trained lip makeup injection model and each sub-original image, each third feature map is obtained, and each third feature map is determined as the shape attribute feature map of the original image.

[0101] In order to obtain the shape attribute feature map of each part in the original image, wherein the shape attribute features include the attribute features of the upper lip shape, the attribute features of the lower lip shape, the shape attribute features of the face including the skin, and the attribute features of the teeth shape. The second sub-model for shape attribute feature extraction is implemented by injecting the pre-trained lip makeup into the model, and each sub-original image is input into the second sub-model. The second sub-model processes each sub-original image through convolution, down-samples, and then up-samples to obtain the third feature map of each sub-original image, and each third feature map is determined as the shape attribute feature map of the original image. The shape attribute features of the original image can be determined based on the shape attribute feature map of the original image, wherein the shape attribute features of the original image can be the shape attribute features of the upper lip, the shape attribute features of the lower lip, the shape attribute features of the face including the skin, and the shape attribute features of the teeth of the original image.

[0102] Embodiment 7:

[0103] In order to determine the image after makeup is applied and improve the makeup texture of the generated image after makeup is applied, based on the above embodiments, in an embodiment of the present invention, the third sub-model for implementing lip makeup injection in the pre-trained lip makeup injection model determines the image after makeup is applied according to the shape attribute feature map and the target makeup feature map corresponding to the target image and the original image, including:

[0104] Based on the first convolutional layer of the third sub-model for implementing lip makeup injection in the pre-trained lip makeup injection model, up-sampling each of the shape attribute feature maps to obtain each fourth feature map;

[0105] Obtaining a fifth feature map based on the first residual layer of the third sub-model, each of the fourth feature maps, and the target makeup feature map;

[0106] Based on the activation function layer of the third sub-model, the fifth feature map and the preset activation function, obtain a sixth feature map;

[0107] Determine a seventh feature map based on the second convolutional layer of the third sub-model and the sixth feature map;

[0108] Determine an eighth feature map based on the second residual layer of the third sub-model and the seventh feature map;

[0109] Based on the third residual layer of the third sub-model, the target makeup feature map and the eighth feature map, an image with completed makeup is determined.

[0110] In order to ensure that the generated finished image can maintain the shape of the original image and have the lip makeup information of the target image, it is necessary to inject the target makeup features of the target image into the shape attribute features of the original image, that is, to implement the third sub-model of lip makeup injection based on the pre-trained lip makeup injection model, and determine the finished image according to the target makeup feature map and shape attribute feature map corresponding to the target image and the original image.

[0111] In an embodiment of the present invention, the third sub-model for implementing lip makeup injection in the pre-trained lip makeup injection model includes a convolution layer for convolving the feature map, an activation function layer for activating the feature map, and a residual layer for performing feature fusion on the feature map.

[0112] Specifically, the shape attribute feature map is input into the first convolutional layer in the third sub-model, the shape attribute feature map is upsampled to obtain each fourth feature map, the fourth feature map is input into the first residual layer of the third sub-model, and the fifth feature map is obtained according to each fourth feature map and the target makeup feature map, wherein the residual block in the first residual layer can be an adaptive normalized residual module.

[0113] Among them, the normalized residual module of the adaptive embodiment can depict the original image in the form of mean and variance, and implement style injection by changing the mean and variance of the original image, that is, injecting the target makeup features of the target image into the shape attribute features of the original image. Specifically, the corresponding formula in the adaptive residual module is:

[0114] Among them, μ_y1 is the mean value corresponding to the target makeup feature map to be injected, Should is the value obtained after convolution operation on the target makeup feature map. is the value obtained after convolution operation on the shape attribute feature map, α μ are pre-set parameters.

[0115] Among them, σ_y1 is the variance corresponding to the target makeup feature map to be injected, Should is the value obtained after convolution operation on the target makeup feature map. is the value obtained after convolution operation on the shape attribute feature map, α σ are pre-set parameters.

[0116] Figure 3A detailed process diagram of feature fusion based on a residual module with adaptive normalization provided by an embodiment of the present invention is now provided for Figure 3 Conduct analysis.

[0117] Y is the target image and the original image, Aij is the target makeup feature map obtained based on the first sub-model of the pre-trained lip makeup injection model, that is, based on the first sub-model of the pre-trained lip makeup injection model, the target image and the original image are segmented to obtain the first sub-target image containing the upper and lower lips and each second sub-original image in the sub-original image except the lips, and the target makeup feature map is obtained after feature aggregation of the first sub-target image and the second sub-original image.

[0118] Based on the second sub-model of the pre-trained lip makeup injection model, a shape attribute feature map of the original image is obtained, and the shape attribute feature map is up-sampled to obtain an up-sampled feature map, that is, a fourth feature map. The target makeup attribute feature map corresponding to the original image and the target image and the fourth feature map are input into the residual layer of the third sub-model of the pre-trained lip makeup injection model to realize makeup feature injection (Style Map). In the residual layer, the target makeup feature map is convolved to obtain β s And obtain, β s Convolution operation is performed on the shape attribute feature map to obtain a o and γ o , β s Multiply by a σ Get Beta s a σ , γ s Multiply by a μ Get γ s a μ , a o Multiply by 1-a σ Get a o (1-a σ ), and γ o Multiply by 1-a μ Get γ o (1-a μ ), determine β s a σ with a o (1-a σ ), that is, β s a σ +a o (1-a σ ), determine γ s a μ With γ o (1-a μ ), that is, γ sa μ +γ o (1-a μ ), thereby determining the mean and variance of the target makeup feature map to be injected, where the mean of the target makeup feature map to be injected is β s a σ +a o (1-a σ ), the variance of the target makeup feature map to be injected is γ s a μ +γ o (1-a μ ), determine the mean u(x) and variance σ(x) of the shape attribute eigenvalue, where u(x) is where x is the shape attribute eigenvalue, then determine (Batch Norm), and finally obtained

[0119] After obtaining the fifth feature map, the fifth feature map is input into the activation function layer of the third sub-model, and the sixth feature map is obtained according to the fifth feature map and a preset activation function, wherein the activation function may be a rectified linear unit (ReLU).

[0120] The sixth feature map is input into the second convolutional layer of the third sub-model, a convolution operation is performed on the sixth feature map to obtain a seventh feature map, the seventh feature map is input into the second residual layer of the third sub-model, and an eighth feature map is determined according to the seventh feature map. The second residual layer has the same structure as the first residual layer, so the residual block in the second residual layer is also an adaptive normalized residual module.

[0121] Specifically, the formula corresponding to the adaptive residual module in the second residual layer is:

[0122] Among them, μ_y2 is the mean value corresponding to the target makeup feature map to be injected, Should is the value obtained after convolution operation on the target makeup feature map. is the value obtained after the convolution operation on the seventh feature map, α μ are pre-set parameters, where σ_y2 is the variance of the target makeup feature map to be injected, Should is the value obtained after convolution of the target makeup feature map. is the value obtained after the convolution operation on the seventh feature map, α σare pre-set parameters.

[0123] After obtaining the eighth feature map, based on the third residual layer of the third sub-model, according to the target makeup feature map and the eighth feature map, an image with makeup completed is determined.

[0124] Figure 4 A schematic diagram of a lip makeup process provided by an embodiment of the present invention, Figure 5 A schematic diagram of a process for obtaining a makeup-completed image based on a lip makeup injection model provided in an embodiment of the present invention is now directed to Figure 4 and Figure 5 Provide explanation.

[0125] Feature matching is performed on the original image and the pre-saved image to obtain a target image, which is an image that matches the pre-saved image with the original image. The target image is input into the first sub-model of the pre-trained lip makeup injection model, which includes convolution layers (conv-Layers), transposed convolution layers (Tconv-Layers) and aggregation layers (rejion-vise-average pooling). According to the original image and the target image, and based on the first sub-model, a target makeup feature map corresponding to the original image and the target image is obtained, including an upper lip target makeup feature map A11, a lower lip target makeup feature map A21, a tooth target makeup feature map A31 and a skin target makeup feature map A41. Based on the second sub-model of the pre-trained lip makeup injection model, a shape attribute feature map of the original image is obtained, and the shape attribute feature map includes an upper lip shape attribute feature B11, a lower lip shape attribute feature B21, a tooth shape attribute feature B31 and a skin shape attribute feature B41.

[0126] After obtaining A11, A21, A31, A41, B11, B21, B31 and B41, A11 and B11, A21 and B21, A31 and B31, A41 and B41 are respectively input into the third sub-model of the pre-trained lip makeup injection model to output the completed makeup image, wherein the lip makeup injection model includes an upsampling layer (Upsample), a residual layer (ResBlk) and a convolutional layer (Conv).

[0127] Specifically, the target makeup attribute feature map and the shape attribute feature map are input into the third sub-model of the pre-trained lip makeup injection model. In the process of determining the image after makeup is completed, after obtaining the source user's lip shape feature map, that is, after obtaining the shape attribute feature map of the original image, that is, after obtaining the third feature map, the shape attribute feature map is convolved (Conv) to obtain a fourth feature map. According to the fourth feature map and the target image lip makeup feature map, that is, the target makeup feature map, the variance σ_y1 corresponding to the target makeup feature map to be injected and the mean μ_y1 corresponding to the target makeup feature map to be injected are determined, and the fourth feature map and the target makeup feature map are input into To the residual layer, a fifth feature map is obtained, and the fifth feature map is input into the activation function layer. According to the fifth feature map and the preset activation function, a sixth feature map is obtained, and the sixth feature map is convolved to determine the seventh feature map, and according to the target makeup feature map and the seventh feature map, the variance σ_y2 and the mean μ_y2 are determined, wherein the method for determining σ_y2 and μ_y2 is the same as the method for determining σ_y1 and μ_y1, which will not be repeated here, and the seventh feature map and the target makeup feature map are input into the second residual layer to obtain an eighth feature map, and the eighth feature map and the shape attribute feature map are input into the third residual layer to determine the image after makeup is completed.

[0128] Embodiment 8:

[0129] In order to determine a target image in a pre-saved image that matches an original image, and further determine a suitable makeup, based on the above embodiment, in an embodiment of the present invention, the original image and the pre-saved image are subjected to feature matching to determine a target image in the image that matches the original image, including:

[0130] For each pre-saved image, a first difference in the number of pixel points corresponding to each pixel value is determined based on the number of pixel points corresponding to each pixel value in the skin color histogram of the original image and the number of pixel points corresponding to each pixel value in the skin color histogram of the image; a second difference in the L channel pixel mean of the Lab channel of the original image and the L channel pixel mean of the Lab channel of the image is determined; and based on the second difference and the first difference in the number of pixel points corresponding to each pixel value, it is determined whether the image is a target image that matches the original image.

[0131] In an embodiment of the present invention, for each pre-saved image, a skin color histogram of the image and a skin color histogram of the original image are determined, wherein the skin color histogram records the number of pixel points of each pixel value, wherein the value range of the pixel value is [0,255], and the horizontal axis of the skin color histogram is each pixel value, and the vertical axis is the number of pixel points corresponding to each pixel value. Based on the number of pixel points corresponding to each pixel value in the skin color histogram of the original image and the number of pixel points corresponding to each pixel value in the skin color histogram of the image, a first difference in the number of pixel points corresponding to each pixel value can be determined. For example, if the number of pixel points with a pixel value of 0 in the skin color histogram of the original image is 20, and the number of pixel points with a pixel value of 0 in the skin color histogram of the image is 10, then the first difference in the number of pixel points corresponding to the pixel value of 0 is 10.

[0132] Among them, for each pre-saved image, the number of pixel points corresponding to each pixel value in the skin color histogram of the original image is determined. The number of pixel points corresponding to each pixel value in the skin color histogram of the image is the existing technology and will not be described here.

[0133] In an embodiment of the present invention, a second difference between the L channel pixel mean of the Lab channel of the original image and the L channel pixel mean of the Lab channel of the image can be determined. In order to determine a target image that matches the original image in a pre-saved image, the target image that matches the original image can be determined based on the second difference and the first difference in the number of pixel points corresponding to each pixel value.

[0134] Among them, determining the L channel pixel mean of the Lab channel of the original image and the L channel pixel mean of the Lab channel of the image is a prior art and will not be described in detail here.

[0135] In order to determine a target image that matches an original image in a pre-saved image, based on the above embodiments, in an embodiment of the present invention, determining whether the image is a target image that matches the original image according to the second difference and the first difference in the number of pixels corresponding to each pixel value includes:

[0136] Determine a second sum value of the original image and the image according to the second difference value and a first difference value of the number of pixel points corresponding to each pixel value;

[0137] If the second sum value is the minimum value corresponding to each pre-saved image, the image is determined as a target image matching the original image.

[0138] In an embodiment of the present invention, in order to determine a target image that matches an original image in a pre-saved image, for any pre-saved image, after determining a first difference between a second difference value and the number of pixel points corresponding to each pixel value, a second sum value of the original image and the image is determined according to the second difference value and the first difference between the number of pixel points corresponding to each pixel value. Specifically, according to determining the second sum;

[0139] Among them, dst' is the second sum value, i is the pixel value, hist scri -hist dsti is the first difference, hist scri is the number of pixels with pixel value i in the skin color histogram of the original image, hist dsti is the number of pixels with pixel value i in the skin color histogram of the image, L scr -L dst is the second difference, L scr is the L channel pixel mean of the Lab channel of the original image, L dst is the mean value of the L channel pixels of the Lab channel of the image, and α and β are preset parameters.

[0140] In order to determine the target image that matches the original image in the pre-saved images, after determining the second sum value of the original image and each image, the images are sorted in order from small to large according to the size of the second sum value, and the image ranked first is used as the target image that matches the original image. That is, for each pre-saved image, if the second sum value corresponding to the image is the minimum value corresponding to each pre-saved image, the image is determined as the target image that matches the original image. In other words, the smaller the second sum value, the higher the matching degree of the image with the original image, and the larger the second sum value, the lower the matching degree of the image with the original image.

[0141] Embodiment 9:

[0142] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present invention, the device comprising:

[0143] The acquisition module 601 is used to acquire any sample image pair in the sample set, wherein the sample image pair includes: a sample original image and a sample target image; the sample original image and the sample target image are input into the generator of the lip makeup injection model to obtain a predicted image after makeup is completed;

[0144] The first determination module 602 is used to input the sample original image and the predicted image after makeup into a pre-trained contrast feature extraction network to determine a first loss value; input the predicted image after makeup into a pre-trained discriminator to determine a second loss value; input the predicted image after makeup and the sample target image into a pre-trained discriminator to determine a third loss value;

[0145] The adjustment module 603 is used to adjust the parameters of the generator of the lip makeup injection model according to the first loss value, the second loss value and the third loss value.

[0146] A device provided in an embodiment of the present invention can be used to execute the model training method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0147] Embodiment 10:

[0148] Figure 7 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention, the device comprising:

[0149] A matching module 701 is used to perform feature matching between an original image and a pre-saved image, and determine a target image in the pre-saved image that matches the original image;

[0150] The second determination module 702 is used to determine the makeup-completed image according to the input target image and the original image through the pre-trained lip makeup injection model.

[0151] An apparatus provided by an embodiment of the present invention can be used to execute the image processing method provided by any of the above embodiments, and has corresponding functions and beneficial effects.

[0152] Embodiment 11:

[0153] Based on the above embodiments, an embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes: a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804;

[0154] The memory 803 stores a computer program, and when the program is executed by the processor 801, the processor 801 performs the following steps: first, any sample image pair in the sample set is obtained, wherein the sample image pair includes: a sample original image and a sample target image, the sample original image and the sample target image are input into the generator of the lip makeup injection model to obtain a predicted image after makeup is completed, and then the sample original image and the predicted image after makeup are input into a pre-trained contrast feature extraction network to determine a first loss value, the predicted image after makeup is completed is input into a pre-trained discriminator to determine a second loss value; the predicted image after makeup is completed and the sample target image are input into a pre-trained discriminator to determine a third loss value; and the parameters of the generator of the lip makeup injection model are adjusted according to the first loss value, the second loss value, and the third loss value.

[0155] Or the processor 801 executes the following steps: first, feature matching is performed between the original image and a pre-saved image to determine a target image in the pre-saved image that matches the original image, and then, through a pre-trained lip makeup injection model, a completed makeup image is determined based on the input target image and the original image.

[0156] An electronic device provided by an embodiment of the present invention can be used to execute the model training method or image processing method provided by any of the above embodiments, and has corresponding functions and beneficial effects.

[0157] Embodiment 12:

[0158] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device, and when the program is executed on the electronic device, the electronic device implements the following steps when executing: first, obtain any sample image pair in the sample set, wherein the sample image pair includes: a sample original image and a sample target image, input the sample original image and the sample target image into the generator of the lip makeup injection model to obtain a predicted image after makeup is completed, and then input the sample original image and the predicted image after makeup is completed into a pre-trained contrast feature extraction network to determine a first loss value, input the predicted image after makeup is completed into a pre-trained discriminator to determine a second loss value; input the predicted image after makeup is completed and the sample target image into the pre-trained discriminator to determine a third loss value; and adjust the parameters of the generator of the lip makeup injection model according to the first loss value, the second loss value, and the third loss value.

[0159] Alternatively, the electronic device is configured to implement the following steps when executed: first, feature matching is performed between the original image and a pre-saved image to determine a target image in the pre-saved image that matches the original image. Then, a pre-trained lip makeup injection model is used to determine a makeup-completed image based on the input target image and the original image.

[0160] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model training method, characterized in that: The method comprises: Obtain any sample image pair in the sample set, wherein the sample image pair includes: a sample original image and a sample target image, wherein the sample target image is a target image obtained by performing feature matching between the sample original image and each pre-saved image; each of the pre-saved images is a human body image after lip makeup, and the skin color and / or lip makeup of the human body in different images are different; Inputting the sample original image and the sample target image into the generator of the lip makeup injection model to obtain a predicted image after the makeup is completed; Input the sample original image and the predicted image after makeup application into a pre-trained contrast feature extraction network to determine a first loss value; the first loss value is a feature loss value; Inputting the predicted image after makeup application into a pre-trained discriminator to determine a second loss value; the second loss value is an adversarial loss value; Input the predicted image after makeup application and the sample target image into a pre-trained discriminator to determine a third loss value; the third loss value is a feature matching loss value; According to the first loss value, the second loss value, and the third loss value and value, adjusting the parameters of the generator of the lip makeup injection model; The process of training the discriminator includes: Inputting the predicted image after makeup application and the sample original image into the discriminator to determine a fourth loss value; the fourth loss value is an adversarial loss value; The parameters of the discriminator are adjusted according to the fourth loss value.

2. An image processing method based on the model training method according to claim 1, characterized in that: The method comprises: Perform feature matching between the original image and a pre-saved image to determine a target image in the pre-saved image that matches the original image; The target image and the original image are input into a pre-trained lip makeup injection model to determine a completed makeup image.

3. The method according to claim 2, characterized in that By using the pre-trained lip makeup injection model, according to the input target image and the original image, determining the image after makeup is completed includes: Obtain a target makeup feature map corresponding to the target image and the original image by injecting a first sub-model for makeup feature extraction into a pre-trained lip makeup model; A second sub-model for extracting shape attribute features is implemented by injecting the pre-trained lip makeup into the model to obtain a shape attribute feature map of the original image; The shape attribute feature map, the target image and the target makeup feature map corresponding to the original image are input into a third sub-model for implementing lip makeup injection of a pre-trained lip makeup injection model to determine a completed makeup image.

4. The method according to claim 3, characterized in that Before the first sub-model for makeup feature extraction is injected into the pre-trained lip makeup model, and the target makeup feature map corresponding to the target image and the original image is obtained, the method further includes: Based on the fourth sub-model for image segmentation in the pre-trained lip makeup injection model, each sub-target image in the target image and each sub-original image in the original image are obtained, and the first sub-target image including the upper and lower lips in the sub-target image and each second sub-original image except the lips in the sub-original image are obtained.

5. The method according to claim 4, characterized in that The first sub-model for extracting makeup features by injecting the pre-trained lip makeup into the model, and obtaining the target makeup feature map corresponding to the target image and the original image comprises: Based on the convolution layer of the first sub-model for makeup feature extraction in the pre-trained lip makeup injection model, the first sub-target image and each of the second sub-original images, a first feature map corresponding to the first sub-target image and each of the second sub-original images is obtained; Based on the aggregate output layer of the first sub-model, each first feature map is fused into a second feature map, and the second feature map is determined as a target makeup feature map corresponding to the target image and the original image.

6. The method according to claim 4, characterized in that The second sub-model for extracting shape attribute features by injecting the pre-trained lip makeup into the model, and obtaining the shape attribute feature map of the original image comprises: Based on the second sub-model for extracting shape attribute features in the pre-trained lip makeup injection model and each sub-original image, each third feature map is obtained, and each third feature map is determined as the shape attribute feature map of the original image.

7. The method according to claim 6, characterized in that The third sub-model for implementing lip makeup injection in the pre-trained lip makeup injection model determines the image with makeup completed according to the shape attribute feature map and the target makeup feature map corresponding to the target image and the original image, including: Based on the first convolutional layer of the third sub-model for implementing lip makeup injection in the pre-trained lip makeup injection model, up-sampling each of the shape attribute feature maps to obtain each fourth feature map; Obtaining a fifth feature map based on the first residual layer of the third sub-model, each of the fourth feature maps, and the target makeup feature map; Based on the activation function layer of the third sub-model, the fifth feature map and the preset activation function, obtain a sixth feature map; Determine a seventh feature map based on the second convolutional layer of the third sub-model and the sixth feature map; Determine an eighth feature map based on the second residual layer of the third sub-model and the seventh feature map; Based on the third residual layer of the third sub-model, the target makeup feature map and the eighth feature map, an image with completed makeup is determined.

8. The method according to claim 2, characterized in that: The performing feature matching between the original image and the pre-saved image to determine a target image in the pre-saved image that matches the original image comprises: For each pre-saved image, a first difference in the number of pixel points corresponding to each pixel value is determined based on the number of pixel points corresponding to each pixel value in the skin color histogram of the original image and the number of pixel points corresponding to each pixel value in the skin color histogram of the image; a second difference in the L channel pixel mean of the Lab channel of the original image and the L channel pixel mean of the Lab channel of the image is determined; and based on the second difference and the first difference in the number of pixel points corresponding to each pixel value, it is determined whether the image is a target image that matches the original image.

9. The method according to claim 8, characterized in that The determining, based on the second difference and the first difference in the number of pixels corresponding to each pixel value, whether the image is a target image matching the original image comprises: Determine a second sum value of the original image and the image according to the second difference value and a first difference value of the number of pixel points corresponding to each pixel value; If the second sum value is the minimum value corresponding to each pre-saved image, the image is determined as a target image matching the original image.

10. A model training device, characterized in that: The device comprises: An acquisition module is used to acquire any sample image pair in the sample set, wherein the sample image pair includes: a sample original image and a sample target image, wherein the sample target image is a target image obtained by feature matching between the sample original image and a pre-saved image; the sample original image and the sample target image are input into a generator of a lip makeup injection model to obtain a predicted image after makeup is completed; wherein the pre-saved images are images of a human body after lip makeup, and the skin color and / or lip makeup of the human body in different images are different; A first determination module is used to input the sample original image and the predicted image after makeup into a pre-trained contrast feature extraction network to determine a first loss value; the first loss value is a feature loss value; the predicted image after makeup is input into a pre-trained discriminator to determine a second loss value; the second loss value is an adversarial loss value; the predicted image after makeup and the sample target image are input into a pre-trained discriminator to determine a third loss value; the third loss value is a feature matching loss value; an adjustment module, configured to adjust parameters of a generator of the lip makeup injection model according to the first loss value, the second loss value, and the third loss value; The first determination module is specifically used to input the predicted image after makeup application and the sample original image into the discriminator to determine a fourth loss value; the fourth loss value is an adversarial loss value; and the parameters of the discriminator are adjusted according to the fourth loss value.

11. An image processing device, characterized in that: The device comprises: A matching module, used for performing feature matching between an original image and a pre-saved image, and determining a target image in the pre-saved image that matches the original image; The second determination module is used to input the target image and the original image into a lip makeup injection model pre-trained based on the model training method according to claim 1 to determine the image after makeup is applied.

12. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory stores a computer program; When the program is executed by the processor, the processor is caused to execute the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: It stores a computer program executable by an electronic device; When the program is executed on the electronic device, the electronic device is enabled to execute the steps of any one of the methods of claims 1-9.

Images