A method, apparatus and device for lip augmentation based on an image encoder

Abstract

The application discloses a method and device for enriching lips based on an image encoder, equipment and a storage medium, which comprises the following steps: inputting an image to be processed, pre-processing the image to be processed to obtain a lip image, and the image to be processed comprising a face region; inputting the lip image into an encoding network to obtain a vector; inputting the vector into a generation network to obtain an intermediate result of a feature image and a lip generation image, and simultaneously inputting the feature image into a segmentation network to obtain a lip mask image; and performing fusion processing on the lip region of the lip generation image and the lip image through the lip mask image to obtain a target result image. The application can realize natural enhancement of the three-dimensional degree of lips to achieve a remodeling type lip enrichment beautifying effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a lip augmentation method, apparatus, and device based on an image encoder. Background Technology

[0002] The lips, located in the center of the face, are the only organ on the entire face with a bright color. Ancient Chinese sayings often used "red lips and white teeth" to describe a beautiful woman, and today lipstick is an essential accessory for every woman who loves beauty, demonstrating the importance of lips to appearance. While the ancients traditionally considered small, cherry-like lips beautiful, the clash between Eastern and Western aesthetic standards has led to a surge in the popularity of bold and sexy images among Western female celebrities. Full, rounded, and thick red lips have thus become a new aesthetic element. Full, beautiful lips mainly possess the following characteristics:

[0003] (1) The cupid's bow and valence are distinct, tending towards an "M" lip shape;

[0004] (2) Full and three-dimensional, with a large volume;

[0005] (3) The lower lip is slightly thicker than the upper lip, with appropriate proportions and thickness;

[0006] (4) The lip edges have smooth and beautiful curves;

[0007] (5) The corners of the mouth are symmetrical and slightly upturned;

[0008] To cater to the aforementioned aesthetic, some lip augmentation algorithms have emerged for images, but they are not perfect. They only cause pixel-level deformation and do not improve the three-dimensionality and fullness of the lips. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to propose a lip augmentation method, apparatus and device based on an image encoder, which aims to solve the problem that existing lip augmentation methods cannot improve the three-dimensionality and fullness of the lips.

[0010] To achieve the above objectives, the present invention provides a lip augmentation method based on an image encoder, the method comprising:

[0011] Input the image to be processed, preprocess the image to be processed to obtain a lip image, the image to be processed includes a face region;

[0012] The lip image is input into an encoding network to obtain a vector;

[0013] The vector is input into the generation network to obtain the intermediate result of the feature image and the lip generation image. At the same time, the feature image is input into the segmentation network to obtain the lip mask image.

[0014] The lip image and the generated lip image are fused together using the lip mask image to obtain the target result image.

[0015] Preferably, the segmentation network structure includes an upsampling module and a convolutional layer; the step of inputting the feature image into the segmentation network to obtain the lip mask image includes:

[0016] The feature images are input into the upsampling module layer by layer in order of increasing resolution. The current result obtained by the upsampling module of the current layer is connected to the feature image input to the next layer. After passing through the upsampling module of 6 layers, the final result is passed through the convolutional layer of 2 layers to obtain the lip mask image.

[0017] Preferably, the training process of the generative network includes:

[0018] A training image set is obtained, and the training image set is preprocessed to obtain lip training images and lip mask training images. The training image set includes a set of face images with full lips.

[0019] The generator network is obtained by training based on the lip training image, the first result image output by the generator network, and a preset first loss function.

[0020] Preferably, the training process of the segmentation network includes:

[0021] The segmentation network is obtained by training based on the lip mask training image, the second result image output by the segmentation network, and a preset second loss function.

[0022] Preferably, the training process of the encoding network includes:

[0023] The lip training image is input into the encoding network to obtain the first vector;

[0024] The lip training image is input into the pre-trained decoupled network to obtain the second vector;

[0025] The encoding network is obtained by training based on the first vector, the second vector, and a preset third loss function.

[0026] Preferably, the first loss function is L pic =λ1L2+λ2L perc +λ3L w ,in,

[0027]

[0028]

[0029]

[0030] H represents the image height, W represents the image width, λ1, λ2, λ3 represent the weights of the corresponding loss function, and φ j This represents the feature map output by the last convolutional layer of the j-th module of the VGG16 network. Represents the target image of the lips. Let w represent the first result graph, w' represent the first vector, and w' represent the second vector.

[0031] Preferably, the second loss function is in, Represents the target mask image. This represents the second result graph.

[0032] Preferably, the third loss function is L total =L pic +L mask .

[0033] To achieve the above objectives, the present invention also provides a lip augmentation device based on an image encoder, the device comprising:

[0034] A preprocessing unit is used to input an image to be processed, preprocess the image to be processed, and obtain a lip image, wherein the image to be processed includes a face region;

[0035] The first processing unit is used to input the lip image into the encoding network to obtain a vector;

[0036] The second processing unit is used to input the vector into the generation network to obtain the intermediate result of the feature image and the lip generation image, and simultaneously input the feature image into the segmentation network to obtain the lip mask image.

[0037] The fusion processing unit is used to fuse the generated lip image, the lip mask image, and the lip image in the lip region to obtain the target result image.

[0038] To achieve the above objectives, the present invention also proposes an apparatus comprising a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of an image encoder-based lip augmentation method as described in the above embodiments.

[0039] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of an image encoder-based lip augmentation method as described in the above embodiments.

[0040] Beneficial effects:

[0041] The above solution, through the proposed image encoder-based lip augmentation method, can intelligently thicken the upper and lower lips, giving them a full and plump shape, with smooth lip edge lines.

[0042] The above scheme preprocesses the input image to obtain a lip image; inputs the lip image into an encoding network to obtain a vector; inputs the vector into a generator network to obtain an intermediate result of the feature image and a generated lip image; simultaneously, inputs the feature image into a segmentation network to obtain a lip mask image; and fuses the generated lip image, the lip mask image, and the lip image to obtain the target result image. This approach can achieve a natural enhancement of lip three-dimensionality, thus achieving a reshaping and plumping lip enhancement effect.

[0043] The above scheme, through the designed segmentation network, uses the feature image output by the generator network as the input of the segmentation network, which enables feature reuse and reduces the amount of computation. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a lip augmentation method based on an image encoder, as provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic flowchart of an image encoder-based lip augmentation method provided in an embodiment of the present invention.

[0047] Figure 3 This is a comparison diagram showing the effect of input image processing before and after processing according to an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the connection method of a segmentation network provided in an embodiment of the present invention.

[0049] Figure 5 This is a schematic diagram of the connection method of the upsampling module provided in an embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram of the connection method of the encoding network provided in an embodiment of the present invention.

[0051] Figure 7This is a schematic diagram of the connection method of the generated network provided in an embodiment of the present invention.

[0052] Figure 8 This is a schematic diagram illustrating the specific training process of each network provided in an embodiment of the present invention.

[0053] Figure 9 This is a schematic diagram of the structure of a lip augmentation device based on an image encoder, provided in an embodiment of the present invention.

[0054] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0057] The present invention will be described in detail below with reference to the embodiments.

[0058] Reference Figure 1 The diagram shown is a flowchart of an image encoder-based lip augmentation method according to an embodiment of the present invention.

[0059] In this embodiment, the method includes:

[0060] S11, Input the image to be processed, preprocess the image to be processed to obtain a lip image, the image to be processed includes a face region.

[0061] S12, input the lip image into the encoding network to obtain a vector.

[0062] S13, the vector is input into the generation network to obtain the intermediate result of the feature image and the lip generation image. At the same time, the feature image is input into the segmentation network to obtain the lip mask image.

[0063] S14, the generated lip image and the lip image are fused using the lip mask image to obtain the target result image.

[0064] The process of preprocessing the image to be processed to obtain a lip image includes:

[0065] S11-1, Calculate the face points of the image to be processed using face detection and face alignment algorithms;

[0066] S11-2, Calculate the cropping rectangle for the lip region based on the face points;

[0067] S11-3, the image to be processed is cropped using the cropping rectangle to obtain the straightened lip image.

[0068] See Figure 2 As shown, in this embodiment, the input face image to be processed is fed into a face detection and face alignment network to obtain face points. Based on the obtained face points, the bounding rectangle of the mouth points is calculated, and this rectangle is extended outwards through the obtained lip region. The rotation angle of the face is obtained through the lip region's cropping rectangle. The image is then cropped from the original input image to obtain a aligned lip image, which is then processed to a size of 512*384. The obtained lip image is input into an encoding network to obtain a vector with a length (dimension) of 512. This vector is simultaneously input into a generation network, and the intermediate result from the generation network is then input into a segmentation network to obtain a beautified lip generated image and a lip mask image. Using the obtained lip mask image, the lip image and the lip generated image are fused to obtain the plumped lip result. Compared to traditional lip plumping methods, the lip plumping method of this embodiment does not involve pixel-level stretching, but rather a natural enhancement of lip three-dimensionality, resulting in a reshaping lip plumping effect. See also... Figure 3 The image shows a comparison of the before and after processing (the left image is the image to be processed, and the right image is the result after lip augmentation).

[0069] Furthermore, the segmentation network structure includes an upsampling module and convolutional layers; the step of inputting the feature image into the segmentation network to obtain the lip mask image includes:

[0070] The feature images are input into the upsampling module layer by layer in order of increasing resolution. The current result obtained by the upsampling module of the current layer is connected to the feature image input to the next layer. After passing through the upsampling module of 6 layers, the final result is passed through the convolutional layer of 2 layers to obtain the lip mask image.

[0071] In this embodiment, the feature layers from the generative network process are used as input to the segmentation network to achieve feature reuse and significantly reduce computational load. Specifically, in the generative network process, the tRGB image starts from the smallest resolution of 8*6 and is fed into the upsampling module layer by layer. The result of this layer (the feature map of the next layer is twice the length and width of this layer, and after the upsampling module, the length and width of the feature map are consistent) is channel-concatenated with the tRGB image input to the next layer. Finally, after 6 upsampling modules, it is enlarged to a size of 512*384 and directly channel-concatenated with the largest resolution tRGB (512*384) image, resulting in a size of 21*512*384. Finally, after two convolutional layers, a lip mask image of size 1*512*384 is obtained. See also Figure 4 The diagram shows the connection method of the segmented network, and as shown in the figure. Figure 5 The diagram shows the connection method of the upsampling module.

[0072] The encoding network consists of 5 downsampled residual blocks and 2 fully connected layers, which reduces the input lip image from a resolution of 3*512*384 to 512*16*12, and finally to a vector of length 512, as shown below. Figure 6 As shown.

[0073] The generator network consists of 7 upsampled residual blocks, transforming the initial 4*3*512 constant matrix into a 512*384*3 color image. The connection method is as follows: Figure 7 As shown, random noise is added to enhance image details, such as lip lines and other lip details. Style parameters are generated through a decoupled network. Furthermore, the w vector is copied 18 times, and each w vector is mapped to the s space through an affine transformation to form the style s, which is then fused into the generator network. The style s is multiplied by the weight w* matrix in the generator network and finally normalized. The processing follows the formula:

[0074] (1)w' ijk =s i ·w ijk

[0075] (2)

[0076] Furthermore, the training process of the generative network includes:

[0077] A training image set is obtained, and the training image set is preprocessed to obtain lip training images and lip mask training images. The training image set includes a set of face images with full lips.

[0078] The generator network is obtained by training based on the lip training image, the first result image output by the generator network, and a preset first loss function.

[0079] Furthermore, the training process of the segmentation network includes:

[0080] The segmentation network is obtained by training based on the lip mask training image, the second result image output by the segmentation network, and a preset second loss function.

[0081] Furthermore, the training process of the encoding network includes:

[0082] The lip training image is input into the encoding network to obtain the first vector;

[0083] The lip training image is input into the pre-trained decoupled network to obtain the second vector;

[0084] The encoding network is obtained by training based on the first vector, the second vector, and a preset third loss function.

[0085] In this embodiment, the preprocessing of the image set data includes:

[0086] a) Download facial images of people with full lips that meet the requirements as dataset I;

[0087] b) Obtain the face point set FP in dataset I using currently mature CNN-based face detection and face alignment methods;

[0088] c) Obtain facial feature mask images from dataset I using currently mature CNN-based facial feature segmentation methods, and select the lip channel as the lip mask image, denoted as M.

[0089] d) Calculate the bounding rectangle of the mouth points using the FP obtained in b), and extend it outward to obtain the cropping rectangle of the lip region. Obtain the rotation angle of the face using the lip cropping rectangle. Crop the lip points from the original image and the lip mask image based on the original image to obtain the corrected lip training image and lip mask training image, respectively. Process both to a size of 512*384 to form datasets I* and M*. At the same time, transform the face point set FP to the coordinates of the lip image and only take the lip points, denoted as FP*.

[0090] In this embodiment, the loss functions of each network are as follows:

[0091] (i) The loss function of the generative network is consistent with that of StyleGAN2. The loss function of the generative network consists of three parts. Specifically, an initial L2 loss is chosen, and to ensure the restoration of facial details, a perceptual loss based on VGG16 is introduced. The L2 loss is expressed as follows:

[0092]

[0093] Perceptual_loss can be expressed as:

[0094]

[0095] Where, φ j This represents the feature map output by the last convolutional layer of the j-th module of the VGG16 network. Represents the target image of the lips. This represents the output diagram of the generator network.

[0096] To ensure that the vector w obtained from the lip image after the encoder has the same distribution as the vector w' obtained from the decoupled network of the stylegan2 pre-trained model, a vector distance loss is introduced, as follows:

[0097]

[0098] The total loss function of the generator network is as follows:

[0099] L pic =λ1L2+λ2L perc +λ3L w ; where λ1, λ2, λ3 represent the weights of the corresponding loss function.

[0100] (ii) The loss function for the segmentation network is chosen as L1_loss, and the formula is as follows:

[0101]

[0102] in, Represents the target mask image. This is a diagram showing the output of the segmentation network.

[0103] (III) Since the encoding network and the segmentation network are trained simultaneously, we have:

[0104] L total =L pic +L mask

[0105] like Figure 8The specific training process for each network is shown below. The network model training is divided into two main modules. First, the generator network is trained. In this embodiment, the StyleGAN2 generator network is selected. To achieve good results with a small number of samples and improve training speed, a pre-trained StyleGAN2 face model is loaded for transfer training. Because the pre-trained face model is adapted to a resolution of 512*512 (aspect ratio 1:1), the weights are compressed to 512*384 (aspect ratio 4:3). This ensures good results even with a small number of lip enhancement samples. Second, the encoding network and the segmentation network are trained simultaneously, while the frozen generator network is not trained.

[0106] Reference Figure 9 The diagram shown is a schematic representation of a lip augmentation device based on an image encoder according to an embodiment of the present invention.

[0107] In this embodiment, the device 90 includes:

[0108] Preprocessing unit 91 is used to input the image to be processed, preprocess the image to be processed, and obtain a lip image, wherein the image to be processed includes a face region;

[0109] The first processing unit 92 is used to input the lip image into the encoding network to obtain a vector;

[0110] The second processing unit 93 is used to input the vector into the generation network to obtain the intermediate result of the feature image and the lip generation image, and at the same time input the feature image into the segmentation network to obtain the lip mask image.

[0111] The fusion processing unit 94 is used to perform lip region fusion processing on the generated lip image, the lip mask image, and the lip image to obtain the target result image.

[0112] Furthermore, the network structure of the segmentation network includes an upsampling module and a convolutional layer; the second processing unit 93 is used for:

[0113] The feature images are input into the upsampling module layer by layer in order of increasing resolution. The current result obtained by the upsampling module of the current layer is connected to the feature image input to the next layer. After passing through the upsampling module of 6 layers, the final result is passed through the convolutional layer of 2 layers to obtain the lip mask image.

[0114] Furthermore, the training process of the generative network includes:

[0115] A training image set is obtained, and the training image set is preprocessed to obtain lip training images and lip mask training images. The training image set includes a set of face images with full lips.

[0116] The generator network is obtained by training based on the lip training image, the first result image output by the generator network, and a preset first loss function.

[0117] Furthermore, the training process of the segmentation network includes:

[0118] The segmentation network is obtained by training based on the lip mask training image, the second result image output by the segmentation network, and a preset second loss function.

[0119] Furthermore, the training process of the encoding network includes:

[0120] The lip training image is input into the encoding network to obtain the first vector;

[0121] The lip training image is input into the pre-trained decoupled network to obtain the second vector;

[0122] The encoding network is obtained by training based on the first vector, the second vector, and a preset third loss function.

[0123] Furthermore, the first loss function is L pic =λ1L2+λ2L perc +λ3L w ,in,

[0124]

[0125]

[0126]

[0127] H represents the image height, W represents the image width, λ1, λ2, λ3 represent the weights of the corresponding loss function, and φ j This represents the feature map output by the last convolutional layer of the j-th module of the VGG16 network. Represents the target image of the lips. Let w represent the first result graph, w' represent the first vector, and w' represent the second vector.

[0128] Furthermore, the second loss function is in, Represents the target mask image. This represents the second result graph.

[0129] Furthermore, the third loss function is L total =L pic +L mask .

[0130] Each unit module of the device 90 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0131] This invention also provides a device comprising the image encoder-based lip augmentation apparatus described above, wherein the image encoder-based lip augmentation apparatus can employ... Figure 9 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0132] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0133] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as image playback function), etc.; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0134] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0135] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0136] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The illustrated lip augmentation method is based on an image encoder. The computer-readable storage medium may be a read-only memory, a hard disk, or an optical disk, etc.

[0137] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0138] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method of lip augmentation based on an image encoder, characterized in that, The method comprises: inputting a to-be-processed image, pre-processing the to-be-processed image to obtain a lip image, and the to-be-processed image comprising a face region; inputting the lip image into an encoding network to obtain a vector; inputting the vector into a generation network to obtain an intermediate result of a feature image and a lip generated image, and inputting the feature image into a segmentation network to obtain a lip mask image; wherein the training process of the generation network comprises: obtaining a training image set, pre-processing the training image set to obtain a lip training image and a lip mask training image, and the training image set comprising a set of full-lip face images; training based on the lip training image, a first result image output by the generation network and a preset first loss function to obtain the generation network; The first loss function is wherein, ， ， ， H represents image height, W represents image width, represents a weight of a corresponding loss function, represents a feature map output by the last convolutional layer of the VGG16 network, , w i represents a first vector, represents a second vector;​ wherein the training process of the segmentation network comprises: training based on the lip mask training image, a second result image obtained by the segmentation network and a preset second loss function to obtain the segmentation network; The second loss function is wherein, denotes the second result map; fusing the lip generated image and the lip image in a lip region through the lip mask image to obtain a target result image.

2. The method of claim 1, wherein the image encoder is based on, The network structure of the segmentation network comprises an up-sampling module and a convolution layer; and the inputting of the feature image into the segmentation network to obtain the lip mask image comprises: inputting the feature image into the up-sampling module layer by layer in an order from small to large resolution, and performing channel connection between a current result obtained by the up-sampling module of a current layer and the feature image input by a next layer until the up-sampling module is used for 6 layers, and then the obtained final result is input into the convolution layer for 2 layers to obtain the lip mask image.

3. The method of claim 1, wherein the image encoder is based on, The training process of the encoding network comprises: inputting the lip training image into the encoding network to obtain a first vector; inputting the lip training image into a pre-trained decoupling network to obtain a second vector; training based on the first vector, the second vector and a preset third loss function to obtain the encoding network.

4. The method of claim 3, wherein the image encoder is based on, The third loss function is .

5. A lip augmentation device based on an image encoder, characterized by, The device comprises: a pre-processing unit configured to input a to-be-processed image, pre-process the to-be-processed image to obtain a lip image, and the to-be-processed image comprising a face region; a first processing unit configured to input the lip image into an encoding network to obtain a vector; a second processing unit configured to input the vector into a generation network to obtain an intermediate result of a feature image and a lip generated image, and input the feature image into a segmentation network to obtain a lip mask image; wherein the training process of the generation network comprises: obtaining a training image set, pre-processing the training image set to obtain a lip training image and a lip mask training image, and the training image set comprising a set of full-lip face images; training based on the lip training image, a first result image output by the generation network and a preset first loss function to obtain the generation network; The first loss function is wherein, ， ， ， H represents image height, W represents image width, represents a weight of a corresponding loss function, represents a feature map output by the last convolutional layer of the VGG16 network, , w i represents a first vector, represents a second vector;​ wherein the training process of the segmentation network comprises: training the segmentation network based on the lip mask training image, a second result image output by the segmentation network, and a preset second loss function, to obtain the segmentation network; The second loss function is wherein, denotes the second result map; a fusion processing unit configured to perform fusion processing on a lip region of the lip generation image, the lip mask image, and the lip image, to obtain a target result image.

6. An apparatus, comprising: A computer program product comprising a processor, a memory, and a computer program stored in the memory, the computer program being executable by the processor to implement the steps of the method of any one of claims 1 to 4.

Images