A pursed-lip smile editing method, device, equipment and readable storage medium

Abstract

The present invention provides a pursed-lip smile editing method, device, equipment and readable storage medium, which relates to the technical field of computer vision and pattern recognition, and includes obtaining a target pursed-lip smile face image to be edited; inputting the target pursed-lip smile face image into a pre-trained face editing model to output a new pursed-lip smile image; based on a convolutional neural network and a loss function, extracting and training the pursed-lip smile image to obtain a final pursed-lip smile image. The beneficial effects of the present invention are that it can cover most user scenarios and the algorithm is stable; by using an attention module, the network learning ability is improved, and the pursed-lip smile effect is greatly improved; and the authenticity of nasolabial folds and mouth shapes of the network output image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and pattern recognition. Specifically, it relates to a pursed-lip smile editing method, device, equipment and readable storage medium. Background Art

[0002] Portrait pursed-lip smile editing (with the mouth closed, obvious nasolabial folds beside the apple muscles, and an obvious smile) has important applications in image beauty. For example, in a beauty app, users can upload various expression pictures of themselves, and then use the pursed-lip editing algorithm to change the portrait expression in the picture to a pursed-lip smile for special effects gameplay, greatly improving the user experience.

[0003] Currently in the industry, the pursed-lip smile editing effect is not good enough, and there are the following problems: through traditional image algorithms, warping the image to be edited according to facial landmarks to the pursed-lip smile effect has problems such as unnatural facial texture and unrealistic smile after warping; and most of the current mainstream pursed-lip smile editing algorithms are based on generative networks (GANs), and the generated portrait features by the generative network will change, resulting in not looking like the person himself and not being realistic enough. Summary of the Invention

[0004] The purpose of the present invention is to provide a pursed-lip smile editing method, device, equipment and readable storage medium to improve the above problems. To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0005] In the first aspect, the present application provides a pursed-lip smile editing method, including:

[0006] Obtain a target pursed-lip smile face image to be edited;

[0007] Input the target pursed-lip smile face image into a pre-trained face editing model to output a new pursed-lip smile image;

[0008] Based on a convolutional neural network and a loss function, extract and train the pursed-lip smile image to obtain a final pursed-lip smile image.

[0009] Preferably, before obtaining the target pursed-lip smile face image to be edited, it includes:

[0010] Collect face data images;

[0011] Based on the key point detection method, detect and crop the collected face data images to obtain a target face image of 512*512;

[0012] Retouch the target face image to obtain the target pursed-lip smile face image to be edited.

[0013] Preferably, input the pursed - lip smile target face image into a pre - trained face editing model to output a new pursed - lip smile image, which includes:

[0014] Input the target face image into a face semantic segmentation network for synthesis to obtain a mask image of the face region. Use the mask image as prior information, where the prior information includes the nasolabial fold part and the mouth part;

[0015] Build a face editing model;

[0016] Input the prior information and the pursed - lip smile target face image to be edited into the face editing model to obtain a new pursed - lip smile image.

[0017] Preferably, based on a convolutional neural network and a loss function, extract and train the pursed - lip smile image to obtain a final pursed - lip smile image, which includes:

[0018] Build an attention module, and use a convolutional layer to extract features in the attention module to obtain a first network branch and a second network branch;

[0019] Perform a softmax operation on the first network branch to obtain a first result;

[0020] Calculate the first result according to the second network branch and preset learning parameters to obtain a second result;

[0021] Optimize the second result according to the loss function to obtain a final pursed - lip smile image.

[0022] In a second aspect, the present application also provides a pursed - lip smile editing device, including:

[0023] An acquisition module: used to acquire a pursed - lip smile target face image to be edited;

[0024] An input module: used to input the pursed - lip smile target face image into a pre - trained face editing model to output a new pursed - lip smile image;

[0025] A training module: used to extract and train the pursed - lip smile image based on a convolutional neural network and a loss function to obtain a final pursed - lip smile image.

[0026] Preferably, before the acquisition module, it includes:

[0027] A collection unit: used to collect face data images;

[0028] A detection unit: used to detect and crop the collected face data image based on the key - point detection method to obtain a 512 * 512 target face image;

[0029] Obtaining unit: used to retouch the target face image to obtain a pursed - lip smiling target face image to be edited.

[0030] Preferably, the input module, which includes:

[0031] Synthesis unit: used to input the target face image into a face semantic segmentation network for synthesis to obtain a mask image of the face region, and use the mask image as prior information, where the prior information includes the nasolabial fold part and the mouth part;

[0032] Establishment unit: used to establish a face editing model;

[0033] Input unit: used to input the prior information and the pursed - lip smiling target face image to be edited into the face editing model to obtain a new pursed - lip smiling image.

[0034] Preferably, the training module, which includes:

[0035] Extraction unit: used to establish an attention module, and use a convolutional layer to extract features in the attention module to obtain a first network branch and a second network branch;

[0036] Operation unit: used to perform a softmax operation on the first network branch to obtain a first result;

[0037] Calculation unit: used to calculate the first result according to the second network branch and a preset learning parameter to obtain a second result;

[0038] Optimization unit: used to optimize the second result according to a loss function to obtain a final pursed - lip smiling image.

[0039] In a third aspect, the present application further provides a pursed - lip smile editing device, including:

[0040] A memory, used to store a computer program;

[0041] A processor, used to implement the steps of the pursed - lip smile editing method when executing the computer program.

[0042] In a fourth aspect, the present application further provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above - mentioned pursed - lip smile editing method.

[0043] The beneficial effects of the present invention are as follows: The present invention proposes a new pursed - lip smile editing scheme, which is trained based on the CNN network structure. Without the need for complicated parameter adjustment, it can convert the expressions of various expression portrait images into pursed - lip smiles with one key, covering most user scenarios and having a stable algorithm. Based on the CNN deep - learning method, a feature fusion module for portrait image quality restoration is proposed, which uses an attention module to enhance the network's learning ability and greatly improve the pursed - lip smile effect. At the same time, in terms of the loss function, in addition to the image - to - image loss, a mask - based loss function is also proposed, which can improve the authenticity of nasolabial folds and mouth shapes in the network output image.

[0044] Other features and advantages of the present invention will be described in the following specification. And, in part, they will become obvious from the specification or can be understood by implementing the embodiments of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the structures specifically pointed out in the written specification, claims, and drawings. Brief Description of the Drawings

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, so they should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of the pursed - lip smile editing method described in the embodiments of the present invention;

[0047] Figure 2 It is a schematic structural diagram of the pursed - lip smile editing device described in the embodiments of the present invention;

[0048] Figure 3 It is a schematic structural diagram of the pursed - lip smile editing device described in the embodiments of the present invention.

[0049] In the figure: 701, acquisition module; 7011, acquisition unit; 7012, detection unit; 7013, obtaining unit; 702, input module; 7021, synthesis unit; 7022, establishment unit; 7023, input unit; 703, training module; 7031, extraction unit; 7032, operation unit; 7033, calculation unit; 7034, optimization unit; 800, pursed - lip smile editing device; 801, processor; 802, memory; 803, multimedia component; 804, I / O interface; 805, communication component. Detailed Embodiments

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are used only to distinguish the description and should not be understood as indicating or implying relative importance.

[0052] Example 1:

[0053] This embodiment provides a method for editing a pursed smile.

[0054] See also Figure 1 , the figure shows that the method includes step S100, step S200 and step S300.

[0055] S100: Obtain a target face image of a person with a pursed smile to be edited.

[0056] It is understood that before this step, the following steps are included:

[0057] Collect facial data graph;

[0058] Based on the key point detection method, the collected face data image is detected and cropped to obtain a 512*512 target face image;

[0059] The target face image is retouched to obtain a target face image of the person showing a pursed smile to be edited.

[0060] It's understandable that this step is a preprocessing step. First, a large number of facial data images are collected as the original images. Second, the face in each image is detected using a facial landmark detection algorithm and cropped to a 512x512 face image x. Third, each original image x is manually retouched (using Meitu Xiu Xiu) to obtain the target face image y with a pursed smile.

[0061] S200: Input the target face image of the pursed smile into a pre-trained face editing model, and output a new pursed smile image.

[0062] It is understandable that this step includes:

[0063] Input the target face image into a face semantic segmentation network for synthesis to obtain a mask image of the face region, and use the mask image as prior information, where the prior information includes the nasolabial fold area and the mouth area;

[0064] It should be noted that first, the original image x is input into the face semantic segmentation network, and then the nasolabial fold area and the mouth area are combined into a single mask as prior information mask_x (which only has two values, 0 and 1, with the nasolabial fold area and the mouth area being 1 and the rest being 0).

[0065] Build a face editing model;

[0066] Input the prior information and the target face image of the pursed - lip smile to be edited into the face editing model to obtain a new pursed - lip smile image.

[0067] It is understandable that then mask_x and the face image to be edited are input into the face editing network G, and finally a generated pursed - lip smile image is output

[0068] S300. Based on a convolutional neural network and a loss function, extract and train the pursed - lip smile image to obtain the final pursed - lip smile image.

[0069] It is understandable that this step includes:

[0070] Build an attention module, and use a convolutional layer to extract features in the attention module to obtain a first network branch and a second network branch;

[0071] Perform a softmax operation on the first network branch to obtain a first result;

[0072] Calculate the first result according to the second network branch and preset learning parameters to obtain a second result;

[0073] It should be noted that for the newly proposed attention module, the attention module has two branches fm x0 and fm x1 , which means that the two branches of the network respectively use convolutional layers to extract features from the same input, and then perform attention learning to further enhance the authenticity of the pursed - lip smile. (The input network features are respectively extracted by two convolutional layers to obtain fm x0 and fm x1 , then perform a softmax operation on fm x0 and multiply it by fm x1 , and finally multiply by α, then add β)

[0074]

[0075] In the formula, α is a learnable parameter, and β is also a learnable parameter. fm x0 and fm x1 represent that two branches of the network extract features from the same input using convolutional layers respectively.

[0076] Optimize the second result according to the loss function to obtain the final pursed-lip smile image.

[0077] It should be noted that in this step, it can be understood as the training process, and the specific process is as follows:

[0078] a) Overall process: Input the original face image x and mask_x into the G network, and output the pursed-lip smile image Then Calculate the loss with the target image y to optimize the model.

[0079] b) Loss function loss

[0080] The loss function includes: L2 loss, Perceptual loss, and mask loss.

[0081] Calculate the L2Loss between the network output image G(x) and the target image y:

[0082] L2_loss = ||G(x) - y||2

[0083] In the formula, G(x) represents the output of the input image x through the generation network G y represents the target image, and L2 loss represents the squared loss. Perceptual Loss is calculated using the vgg-19 network to enhance the texture quality of the generated image.

[0084]

[0085] In the formula, Perceptual loss represents the perceptual loss function, L represents multiple levels of features, and i represents the i-th level of features.

[0086] For maskLoss, the mouth and nasolabial fold masks are extracted from the target image and the network output image through the semantic segmentation model, and then mask supervision is performed, which can enhance the realism of the nasolabial folds and mouth shape in the network output image.

[0087]

[0088] In the formula, represents: extracting the mouth and nasolabial fold masks from the network output effect through the semantic segmentation model, mask yTaking the mouth and nasolabial folds masks of the target image through the semantic segmentation model.

[0089] Embodiment 2:

[0090] As Figure 2 shown, this embodiment provides a pursed-lip smile editing device. Refer to Figure 2 The device includes an acquisition module 701, an input module 702, and a training module 703, where:

[0091] The acquisition module 701: is used to acquire the target face image of the pursed-lip smile to be edited;

[0092] The input module 702: is used to input the target face image of the pursed-lip smile into a pre-trained face editing model and output a new pursed-lip smile image;

[0093] The training module 703: is used to extract and train the pursed-lip smile image based on a convolutional neural network and a loss function to obtain the final pursed-lip smile image.

[0094] Specifically, before the acquisition module 701, there are a collection unit 7011, a detection unit 7012, and an obtaining unit 7013, where:

[0095] The collection unit 7011: is used to collect face data images;

[0096] The detection unit 7012: is used to detect and crop the collected face data image based on the key point detection method to obtain a target face image of 512*512;

[0097] The obtaining unit 7013: is used to retouch the target face image to obtain the target face image of the pursed-lip smile to be edited.

[0098] Specifically, in the input module 702, there are a synthesis unit 7021, a building unit 7022, and an input unit 7023, where:

[0099] The synthesis unit 7021: is used to input the target face image into a face semantic segmentation network for synthesis to obtain a mask image of the face area, and use the mask image as prior information, where the prior information includes the nasolabial fold part and the mouth part;

[0100] The building unit 7022: is used to build a face editing model;

[0101] The input unit 7023: is used to input the prior information and the target face image of the pursed-lip smile to be edited into the face editing model to obtain a new pursed-lip smile image.

[0102] Specifically, the training module 703 includes an extraction unit 7031, an operation unit 7032, a calculation unit 7033, and an optimization unit 7034, where:

[0103] The extraction unit 7031: is used to establish an attention module, and uses a convolutional layer to extract features in the attention module, obtaining a first network branch and a second network branch;

[0104] The operation unit 7032: is used to perform a softmax operation on the first network branch to obtain a first result;

[0105] The calculation unit 7033: is used to calculate the first result according to the second network branch and a preset learning parameter to obtain a second result;

[0106] The optimization unit 7034: is used to optimize the second result according to a loss function to obtain a final pursed-lip smile image.

[0107] It should be noted that regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0108] Embodiment 3:

[0109] Corresponding to the above method embodiments, a pursed-lip smile editing device is also provided in this embodiment. The pursed-lip smile editing device described below can be correspondingly referred to the pursed-lip smile editing method described above.

[0110] Figure 3 It is a block diagram of a pursed-lip smile editing device 800 shown according to an exemplary embodiment. As Figure 3 shown, the pursed-lip smile editing device 800 may include: a processor 801, a memory 802. The pursed-lip smile editing device 800 may further include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0111] Among them, the processor 801 is used to control the overall operation of the pursed-lip smile editing device 800 to complete all or part of the steps in the above-mentioned pursed-lip smile editing method. The memory 802 is used to store various types of data to support the operation of the pursed-lip smile editing device 800. These data may include, for example, instructions for any application or method operating on the pursed-lip smile editing device 800, as well as application-related data, such as contact data, received and sent messages, pictures, audio, video, and so on. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 803 may include a screen and an audio component. Among them, the screen may be a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, and the microphone is used to receive external audio signals. The received audio signals may be further stored in the memory 802 or sent through the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the pursed-lip smile editing device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G or 4G, or a combination of one or more of them. Therefore, the corresponding communication component 805 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0112] In an exemplary embodiment, the pursed-lip smile editing device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above-mentioned pursed-lip smile editing method.

[0113] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-mentioned pursed-lip smile editing method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions may be executed by the processor 801 of the pursed-lip smile editing device 800 to complete the above-mentioned pursed-lip smile editing method.

[0114] Embodiment 4:

[0115] Corresponding to the above method embodiment, a readable storage medium is further provided in this embodiment. A readable storage medium described below can be mutually referred to with a pursed-lip smile editing method described above.

[0116] A readable storage medium has a computer program stored thereon. When the computer program is executed by a processor, the steps of the pursed-lip smile editing method in the above method embodiment are implemented.

[0117] Specifically, the readable storage medium may be various readable storage media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0118] In summary, the present invention addresses the problems of non-realistic pursed-lip smiles and insufficiently realistic skin textures that currently exist. Based on the CNN deep learning method, a feature fusion module for portrait image quality restoration is proposed. By using an attention module, the network learning ability is improved, and the pursed-lip smile effect is greatly enhanced. At the same time, in terms of the loss function, in addition to the image-to-image loss, a mask-based loss function is also proposed, which can improve the realism of nasolabial folds and mouth shapes in the network output image.

[0119] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0120] As mentioned above, the above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or replacements, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A pursed-lip smile editing method, characterized in that, Including: Obtain the pursed - lip smile target face image to be edited; Input the pursed - lip smile target face image into a pre - trained face editing model, and output a new pursed - lip smile image, which includes: Obtain the target face image, input the target face image into a face semantic segmentation network for synthesis to obtain a mask image of the face region, and use the mask image as prior information, where the prior information includes the nasolabial fold area and the mouth area; Establish a face editing model; Input the prior information and the pursed - lip smile target face image to be edited into the face editing model to obtain a new pursed - lip smile image; Based on a convolutional neural network and a loss function, extract and train the pursed - lip smile image to obtain the final pursed - lip smile image, which includes: Establish an attention module, use a convolutional layer to extract features in the attention module to obtain a first network branch and a second network branch; Perform a softmax operation on the first network branch to obtain a first result; Calculate the first result according to the second network branch and preset learning parameters to obtain a second result; Optimize the second result according to the loss function to obtain the final pursed - lip smile image.

2. The method for pursed-lip smile editing according to claim 1, characterized in that , Before obtaining the pursed - lip smile target face image to be edited, it includes: Collect face data images; Based on the key - point detection method, detect and crop the collected face data image to obtain a 512*512 target face image; Retouch the target face image to obtain the pursed - lip smile target face image to be edited.

3. A pursed-lip smile editing device, characterized in that, Including: Acquisition module: used to obtain the pursed - lip smile target face image to be edited; Input module: used to input the pursed - lip smile target face image into a pre - trained face editing model and output a new pursed - lip smile image, which includes: Synthesis unit: used to obtain the target face image, input the target face image into a face semantic segmentation network for synthesis to obtain a mask image of the face region, and use the mask image as prior information, where the prior information includes the nasolabial fold area and the mouth area; Establishment unit: used to establish a face editing model; Input unit: used to input the prior information and the pursed - lip smile target face image to be edited into the face editing model to obtain a new pursed - lip smile image; Training module: used to extract and train the pursed - lip smile image based on a convolutional neural network and a loss function to obtain the final pursed - lip smile image, which includes: Extraction unit: used to establish an attention module, use a convolutional layer to extract features in the attention module to obtain a first network branch and a second network branch; Operation unit: used to perform a softmax operation on the first network branch to obtain a first result; Calculation unit: used to calculate the first result according to the second network branch and preset learning parameters to obtain a second result; Optimization unit: used to optimize the second result according to the loss function to obtain the final pursed - lip smile image.

4. The pursed-lip smile editing device according to claim 3, wherein Before the acquisition module, it includes: Collection unit: used to collect face data images; Detection unit: used to detect and crop the collected face data image based on the key - point detection method to obtain a 512*512 target face image; Obtaining unit: configured to retouch the target face image to obtain a pursed-lip smiling target face image to be edited.

5. A pursed-lip smile editing device, characterized in that, Comprising: A memory, configured to store a computer program; A processor, configured to implement the steps of the pursed-lip smile editing method according to any one of claims 1 to 2 when executing the computer program.

6. A readable storage medium, characterized in that: A computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the pursed-lip smile editing method according to any one of claims 1 to 2 are implemented.

Images