Facial expression editing method and device, computer device and storage medium

Abstract

The present application discloses a face expression editing method and device, computer equipment and storage medium, relates to the artificial intelligence technical field and the financial technology field, and collects face image data sets containing various expressions; optical flow extraction is performed on the original expression and the target expression to obtain first optical flow information and second optical flow information; the original expression combined with the first optical flow information and the original expression combined with the first optical flow information and the second optical flow information are used as input, and a neutral expression image and a target expression image are predicted through a generator; the neutral expression image and the target expression image are compared with a neutral expression true value image and a target expression true value image to obtain a first loss function value and a second loss function value; the generator is trained to generate a face expression editing model; and based on the face expression editing model, the expression of an original face image is edited. The present application can realize high-accuracy and natural face expression editing and conversion while maintaining the identity features of the face.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and financial technology, specifically to a method, apparatus, computer device, and computer-readable storage medium for editing facial expressions. Background Technology

[0002] Currently, facial editing technology, as an important branch of image processing, involves operations such as denoising, enhancement, deformation, and synthesis of facial images to modify their appearance and expressions. With the development of digital image processing technology, facial editing technology has shown broad application prospects in various fintech-related fields such as facial beautification, facial recognition, and facial animation, supporting functions such as social networking, shopping, interactive games, and resource transfer. However, due to the high dimensionality, high nonlinearity, and high sensitivity of facial images, ensuring that the edited image retains the original image's identity information while achieving the desired editing effect has always been a key focus and challenge in this field.

[0003] Facial expressions are a crucial means of human emotional communication, conveying complex feelings and emotions. In daily interactions, facial expressions act as a window, allowing us to understand the inner world of others. With the rise of deep learning technology, facial expression recognition and editing have become a research hotspot in the field of computer vision. Facial expression editing technology aims to transform the expression in a given facial image into a target expression while preserving the facial's identity features. This technology has enormous application potential in fields such as photography, filmmaking, and the entertainment industry. As research deepens, facial expression editing technology is receiving increasing attention from both academia and industry.

[0004] Currently, although some methods based on Generative Adversarial Networks (GANs) have made significant progress in facial expression editing tasks, existing technologies still face several challenges. In particular, when there is a large difference between the input facial expression and the target expression, the generated image may exhibit overlap artifacts, especially in areas with dense facial muscle changes such as the eyes, mouth, and nose. Furthermore, existing GAN-based facial expression editing methods typically employ a one-step transformation approach, and coupled with network capacity limitations, this makes it difficult to avoid image distortion when performing large-scale expression editing.

[0005] In summary, how to provide a method, apparatus, computer device, and non-volatile computer-readable storage medium for editing facial expressions that can achieve high accuracy and naturalness in editing and converting facial expressions while maintaining facial identity features is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method, apparatus, computer device and computer-readable storage medium for editing facial expressions, aiming to solve the problem of how to achieve high accuracy and naturalness in editing and converting facial expressions while maintaining facial identity features.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for editing facial expressions, comprising:

[0009] Collect a dataset of facial images containing various expressions;

[0010] Based on the optical flow extraction network, optical flow is extracted from the acquired original expression and the target expression respectively to obtain the corresponding first optical flow information and second optical flow information;

[0011] Using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the neutral expression image and the target expression image corresponding to the original expression and the target expression, respectively.

[0012] The neutral expression image and the target expression image are compared with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and the corresponding first loss function value and second loss function value are calculated.

[0013] The generator is trained based on the first loss function value and the second loss function value to generate a facial expression editing model;

[0014] Based on the aforementioned facial expression editing model, facial expression editing of the original facial image is achieved.

[0015] Secondly, the present invention provides a facial expression editing device, comprising:

[0016] The collection module is used to collect a dataset of facial images containing various expressions;

[0017] The extraction module is used to extract optical flow from the acquired original expression and the target expression based on the optical flow extraction network, so as to obtain the corresponding first optical flow information and second optical flow information.

[0018] The prediction module is used to use the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and to predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively through the generator;

[0019] The comparison module is used to compare the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value.

[0020] The training module is used to train the generator based on the first loss function value and the second loss function value to generate a facial expression editing model;

[0021] The editing module is used to edit the expressions of the original face image based on the facial expression editing model.

[0022] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the facial expression editing method described above.

[0023] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the facial expression editing method described above.

[0024] Compared to existing technologies, this invention provides a method, apparatus, computer device, and computer-readable storage medium for editing facial expressions. The method involves collecting a dataset of facial images containing various expressions; using an optical flow extraction network, extracting optical flow from the acquired original expression and target expression to obtain corresponding first and second optical flow information; using the original expression combined with the first optical flow information, and the original expression combined with the first and second optical flow information as input, a generator predicts neutral expression images and target expression images corresponding to the original expression and the target expression, respectively; comparing the neutral expression images and target expression images with the neutral expression ground truth images and target expression ground truth images in the facial image dataset, respectively, and calculating corresponding first and second loss function values; training the generator based on the first and second loss function values ​​to generate a facial expression editing model; and using the facial expression editing model to edit expressions from original facial images. Thus, by generating a facial expression editing model, this invention can edit facial expressions on original facial images with high accuracy and naturalness while maintaining facial identity features. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating the application environment of a facial expression editing method provided in an embodiment of the present invention.

[0027] Figure 2 This is a flowchart illustrating a facial expression editing method according to an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the program modules of a facial expression editing device provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

[0030] Figure 5 This is another structural schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0033] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0035] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0037] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0038] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0039] An embodiment of the present invention provides a method for editing facial expressions, which can be applied to, for example... Figure 1In the application environment shown, the client communicates with the server via a network. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0040] Please see Figure 2 This invention provides a method for editing facial expressions, wherein the method includes the following steps:

[0041] S100: Collect a dataset of facial images containing various expressions;

[0042] S200. Based on the optical flow extraction network, optical flow is extracted from the acquired original expression and the target expression respectively to obtain the corresponding first optical flow information and second optical flow information.

[0043] S300: Using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the neutral expression image and the target expression image corresponding to the original expression and the target expression, respectively.

[0044] S400: Compare the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value;

[0045] S500. Based on the first loss function value and the second loss function value, the generator is trained to generate a facial expression editing model;

[0046] S600. Based on the facial expression editing model, the facial expression of the original facial image is edited.

[0047] In this specific implementation, a dataset of facial images containing various expressions is collected. Then, based on an optical flow extraction network, optical flow is extracted from the acquired original and target expressions to obtain corresponding first and second optical flow information. The original expression combined with the first and second optical flow information, and vice versa, are used as inputs to generate neutral and target expression images corresponding to the original and target expressions, respectively. The neutral and target expression images are then compared with the ground truth images of neutral and target expressions in the facial image dataset, respectively, and corresponding first and second loss function values ​​are calculated. Based on the first and second loss function values, the generator is trained to generate a facial expression editing model. Finally, based on the facial expression editing model, expression editing of the original facial image is achieved. Thus, this embodiment, through the generated facial expression editing model, can achieve high accuracy and naturalness in facial expression editing and conversion while maintaining facial identity features.

[0048] Furthermore, in the aforementioned facial expression editing method, step S100, collecting a dataset of facial images containing multiple expressions, specifically includes the following steps:

[0049] Collect facial images containing a variety of expressions;

[0050] The face image is preprocessed;

[0051] The preprocessed face images are labeled and organized to obtain a face image dataset.

[0052] In this specific implementation, the following steps are taken: collecting facial images containing multiple expressions; preprocessing the facial images; labeling the preprocessed facial images; and organizing them to obtain a facial image dataset. The specific implementation process of each of the above steps is roughly as follows:

[0053] 1. Collect facial images containing various expressions.

[0054] Data sources: Facial images were collected through publicly available facial expression datasets, social media, volunteer recruitment, or professional photography.

[0055] Diversity: Ensure that the collected facial images include faces of different ages, genders, ethnicities, and lighting conditions, as well as a variety of natural expressions such as happiness, sadness, and anger;

[0056] 2. Preprocess the face image

[0057] Image quality: Image processing techniques are used to remove noise, adjust brightness and contrast to improve image quality;

[0058] Image size: Adjust all facial images to a uniform resolution and aspect ratio to facilitate subsequent processing;

[0059] Face detection and alignment: Use face detection algorithms to detect faces in face images and align them to ensure that key feature points such as eyes, nose and mouth are in the same position;

[0060] 3. Annotate the preprocessed face images.

[0061] Facial expression annotation: Label each facial image with an expression category, such as "happy," "sad," etc.; this can be done by expert annotation or by using annotation tools.

[0062] Key point annotation: Annotate the key points of the face in the face image, such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc. This is necessary for some advanced expression editing techniques;

[0063] Data cleaning: Check the accuracy of the annotations, and correct or delete incorrectly annotated face images if necessary;

[0064] 4. The resulting face image dataset was compiled.

[0065] Data organization: The labeled face images are classified and organized according to standards such as expression category and image quality to obtain a face image dataset;

[0066] Database construction: Store the face image dataset in a database for easy retrieval and use;

[0067] Through the above process, a high-quality, diverse, and accurately labeled facial image dataset can be constructed, providing a solid foundation for subsequent expression editing or other related tasks.

[0068] Furthermore, in the facial expression editing method, step S200, based on an optical flow extraction network, involves extracting optical flow from the acquired original expression and the target expression to obtain corresponding first optical flow information and second optical flow information. This specifically includes the following steps:

[0069] Predetermine the optical flow extraction network;

[0070] The acquired original facial expression is input into the optical flow extraction network for optical flow extraction to obtain the first optical flow information from the original facial expression to the neutral facial expression;

[0071] The acquired target expression is input into the optical flow extraction network for optical flow extraction to obtain the second optical flow information from the target expression to the neutral expression.

[0072] In this specific implementation, an optical flow extraction network is predetermined; the acquired original facial expression is input into the optical flow extraction network for optical flow extraction to obtain first optical flow information from the original facial expression to a neutral facial expression; the acquired target facial expression is input into the optical flow extraction network for optical flow extraction to obtain second optical flow information from the target facial expression to the neutral facial expression; the specific implementation process of each of the above steps is roughly as follows:

[0073] 1. Determine (select or construct) the optical flow extraction network

[0074] Network selection: Select an existing optical flow extraction network that has been proven effective in optical flow estimation tasks;

[0075] Network construction: If the existing optical flow extraction network does not meet the requirements, a new optical flow extraction network can be constructed; this usually involves defining the network architecture, selecting an appropriate loss function, and optimizing the algorithm.

[0076] 2. Extract the first optical flow information

[0077] Input network: The acquired raw facial expressions are input into the optical flow extraction network;

[0078] Optical flow calculation: The optical flow extraction network analyzes the original facial expression, calculates and outputs the first optical flow information from the original facial expression to the neutral facial expression;

[0079] 3. Extract the second optical flow information

[0080] Input network: The acquired target facial expression is input into the optical flow extraction network;

[0081] Optical flow calculation: The optical flow extraction network analyzes the target facial expression, calculates and outputs the second optical flow information from the target facial expression to the neutral facial expression;

[0082] 4. Post-processing optical flow information

[0083] Optical flow optimization: Post-processing of the first and second optical flow information output by the optical flow extraction network, such as smoothing and removing outliers, to improve the quality of the first and second optical flow information;

[0084] Through the above process, the corresponding optical flow information can be effectively extracted from the original and target facial expression images, providing a solid foundation for subsequent facial expression editing or other related tasks.

[0085] Furthermore, in the facial expression editing method, step S300, using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and using a generator to predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively, specifically includes the following steps:

[0086] Pre-build generators;

[0087] Using the original expression combined with the first optical flow information as input, the generator predicts the neutral expression image corresponding to the original expression.

[0088] Using the original facial expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the target facial expression image corresponding to the target facial expression.

[0089] In this specific implementation, a generator is pre-constructed; using the original expression combined with the first optical flow information as input, the generator predicts the neutral expression image corresponding to the original expression; using the original expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the target expression image corresponding to the target expression; the specific implementation process of each of the above steps is roughly as follows:

[0090] 1. Build the generator:

[0091] Design a network: Design a deep learning network, i.e., a generator, that can take the original facial expression and optical flow information as input and output a new facial expression image;

[0092] Initialize parameters: Initialize the network parameters of the generator, which usually involves random weight allocation or loading parameters from a pre-trained model;

[0093] 2. Prepare input data

[0094] Get raw emojis: Select or collect raw emojis, which will be used as one of the inputs to the generator;

[0095] Optical flow information extraction: Use an optical flow extraction network to obtain first optical flow information (optical flow from original expression to neutral expression) and second optical flow information (optical flow from original expression to target expression);

[0096] 3. Generate neutral expression images

[0097] Data fusion: Combining the original facial expression image with the first optical flow information as input to the generator;

[0098] Model inference: Forward propagation is performed through a generator, which predicts neutral expression images based on the input data;

[0099] 4. Generate the target facial expression image

[0100] Secondary data fusion: The original facial expression image, the first optical flow information, and the second optical flow information are combined and used as input to the generator again;

[0101] Model inference again: Another forward pass is performed through the generator, which predicts the target facial expression image;

[0102] 5. Post-processing facial images

[0103] Facial expression image optimization: Post-processing of the generated neutral facial expression image and the target facial expression image, such as color correction and detail enhancement, to improve image quality;

[0104] Through the above process, the generator can effectively predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively, providing a solid foundation for subsequent expression editing or other related tasks.

[0105] Further, in the facial expression editing method, step S400, comparing the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the facial image dataset, respectively, and calculating the corresponding first loss function value and second loss function value, specifically includes the following steps:

[0106] Select the neutral expression ground truth image and the target expression ground truth image corresponding to the neutral expression image and the target expression image from the face image dataset;

[0107] The neutral expression image is compared with the neutral expression ground truth image, and the first loss function value between the two is calculated;

[0108] The target facial expression image is compared with the target facial expression ground truth image, and the second loss function value between the two is calculated.

[0109] In specific implementation, in this embodiment, a neutral expression ground truth image and a target expression ground truth image corresponding to the neutral expression image and the target expression image are selected from the face image dataset; the neutral expression image and the neutral expression ground truth image are compared, and a first loss function value between them is calculated; the target expression image and the target expression ground truth image are compared, and a second loss function value between them is calculated; the specific implementation process of the above steps is roughly as follows:

[0110] 1. Selection of the truth image

[0111] Select the neutral expression ground truth image and the target expression ground truth image corresponding to the neutral expression image and the target expression image from the face image dataset;

[0112] Ensure that the selected neutral expression ground truth image and target expression ground truth image are consistent with their corresponding neutral expression image and target expression image in terms of resolution, size, and alignment for accurate comparison;

[0113] 2. Image Feature Extraction

[0114] Feature representation: Using predefined feature extraction methods (such as pre-trained deep learning models) to extract feature representations from images, which helps with subsequent comparisons and loss calculations;

[0115] 3. Calculate the value of the first loss function:

[0116] Comparing neutral expression images: The generated neutral expression images are compared with the corresponding neutral expression ground truth images at the feature level;

[0117] Loss calculation: Based on a predetermined loss function (such as mean squared error, structural similarity index SSIM, perceptual loss, etc.), the difference between the two is calculated to obtain the first loss function value;

[0118] 4. Calculate the value of the second loss function:

[0119] Compare the target expression image: Compare the generated target expression image with the corresponding ground truth image of the target expression at the feature level;

[0120] Loss calculation: The difference between the two is calculated using the same loss function to obtain the second loss function value;

[0121] Through the above process, the first loss function value and the second loss function value can be calculated, providing a solid foundation for subsequent expression editing or other related tasks.

[0122] Furthermore, in the aforementioned facial expression editing method, step S500, training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model, specifically includes the following steps:

[0123] Based on the first loss function value and the second loss function value, the generator is iteratively trained using the backpropagation algorithm;

[0124] When the generator training reaches the preset requirements, a facial expression editing model is generated.

[0125] In this specific implementation, based on the first loss function value and the second loss function value, the generator is iteratively trained using a backpropagation algorithm; when the generator training reaches a preset requirement, a facial expression editing model is generated; the specific implementation process of each of the above steps is roughly as follows:

[0126] 1. Loss Function Optimization

[0127] Backpropagation: Backpropagation is performed using the calculated first loss function value and the second loss function value to calculate the gradient of the generator network parameters;

[0128] Parameter update: Update the generator's parameters using gradient descent or other optimization algorithms to reduce loss and improve image generation quality;

[0129] 2. Generator Evaluation and Iteration

[0130] Performance evaluation: Evaluate the generator's performance in generating neutral and target expression images to ensure that the generated images retain the original identity information while achieving the expected expression editing effect;

[0131] Iterative training: The generator is trained iteratively multiple times based on the loss function value and the quality of the generated image to continuously optimize the generator's performance;

[0132] 3. Evaluate the pre-set requirements

[0133] Performance criteria: Set preset performance requirements, such as the lower limit of the loss function value, the upper limit of accuracy, or performance on the validation set;

[0134] Model validation: The generator is tested periodically on the validation set to ensure that the generated images meet the requirements and that the model is not overfitting;

[0135] 4. Generate facial expression editing models

[0136] The training is considered complete when the generator training reaches the preset performance requirements, such as a sufficiently low loss function value and stable performance on the validation set.

[0137] Model saving: Save the generator's parameters to form the final facial expression editing model for later use or further fine-tuning;

[0138] 5. Subsequent Applications

[0139] Model Deployment: Deploying the trained facial expression editing model to real-world applications, such as mobile apps, online services, or offline software;

[0140] User interaction: In practical applications, users can use this facial expression editing model to edit the expressions of the original facial images in order to achieve expression conversion and generation;

[0141] Through the above process, it can be ensured that the generator is gradually optimized during training, and finally a facial expression editing model that can accurately edit facial expressions can be generated.

[0142] Furthermore, in the aforementioned facial expression editing method, step S600, which involves editing the expression of the original facial image based on the facial expression editing model, specifically includes the following steps:

[0143] Obtain the original face image to be edited;

[0144] The original face image is input into the face expression editing model to generate the target face image after expression editing;

[0145] Output or display the target face image.

[0146] In specific implementation, in this embodiment, the original face image to be edited is obtained; the original face image is input into the face expression editing model to generate the target face image after expression editing; the target face image is output or displayed; wherein, the specific implementation process of each of the above steps is roughly as follows:

[0147] 1. Obtain the original face image

[0148] Image acquisition: Obtaining raw facial images for expression editing from user uploads, databases, or other sources;

[0149] Image preprocessing: Perform necessary preprocessing on the original face image, including cropping, scaling, etc., to match the input size of the face expression editing model;

[0150] 2. Face detection and alignment

[0151] Face detection: Use face detection algorithms to locate faces in the original face image;

[0152] Aligning faces: Aligning detected faces to a standard pose, which may include adjusting the positions of the eyes, nose, and mouth to ensure consistency with the alignment of the face images used during training;

[0153] 3. Input facial expression editing model

[0154] Load Model: Load the trained facial expression editing model;

[0155] Model inference: The preprocessed and aligned original face image is input into the face expression editing model, which generates the target face image after expression editing based on the learned expression transformation mapping;

[0156] 4. Post-processing of face images

[0157] Color correction: Perform color correction on the generated target face image to ensure that its color distribution is consistent with the input original face image;

[0158] Detail enhancement: Enhances the details of the generated target face image, improves image quality, and makes it look more natural;

[0159] 5. Output or display the target face image

[0160] Image conversion: Converting the generated target face image into a format suitable for output or display, such as JPEG, PNG, etc.

[0161] Display Image: Display the target face image after facial expression editing on the user interface, or save it for later use;

[0162] Output image: Of course, the target face image after expression editing can also be output to a specified location as needed;

[0163] 6. User Feedback

[0164] Provide a feedback mechanism: Allow users to evaluate the generated target face image and provide feedback;

[0165] Model iteration: Based on user feedback and editing results, the facial expression editing model is fine-tuned and optimized to improve the accuracy and naturalness of expression editing and conversion;

[0166] Through the above process, a complete workflow can be achieved from acquiring the original face image to outputting or displaying the target face image after expression editing, providing users with a high-quality expression editing and conversion experience.

[0167] As can be seen from the above method embodiments, the facial expression editing method provided by the present invention includes: collecting a facial image dataset containing multiple expressions; performing optical flow extraction on the acquired original expression and target expression respectively based on an optical flow extraction network to obtain corresponding first optical flow information and second optical flow information; using the original expression combined with the first optical flow information and the original expression combined with the first optical flow information and the second optical flow information as input, and using a generator to predict the neutral expression image and target expression image corresponding to the original expression and the target expression respectively; comparing the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the facial image dataset respectively, and calculating the corresponding first loss function value and second loss function value; training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model; and realizing expression editing of the original facial image based on the facial expression editing model. Thus, the method of the present invention, through the generated facial expression editing model, can achieve high accuracy and naturalness in facial expression editing and conversion of the original facial image while maintaining facial identity features.

[0168] Understandably, the facial expression editing method provided in this embodiment of the invention can be applied to facial expression editing scenarios related to the fintech field, specifically:

[0169] 1. Integration into applications: The facial expression editing function can be integrated into the corresponding applications of financial institutions to support relevant functions such as social networking, shopping, interactive games, and resource transfer;

[0170] 2. Real-time editing: In scenarios requiring real-time facial expression editing, such as video or live streaming, ensure that the facial expression editing model can respond quickly and perform real-time facial expression editing.

[0171] It should be understood that although this application provides the method operation steps as described in the embodiments or flowcharts, conventional or non-inventive labor may include more or fewer operation steps, and these operation steps are not necessarily executed sequentially according to the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is merely one way of executing many steps and does not represent the only execution order. It should be noted that there is no necessary sequential order between the above steps. Those skilled in the art can understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in exchange, etc. Moreover, at least some steps in the embodiments or flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn, alternately, or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0172] Based on the above method embodiments, please refer to Figure 3 Another embodiment of the present invention also provides a facial expression editing device, wherein the device includes:

[0173] Collection module 11 is used to collect a dataset of facial images containing various expressions;

[0174] Extraction module 12 is used to extract optical flow from the acquired original expression and target expression based on the optical flow extraction network to obtain the corresponding first optical flow information and second optical flow information.

[0175] Prediction module 13 is used to use the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and to predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively through the generator;

[0176] The comparison module 14 is used to compare the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value.

[0177] Training module 15 is used to train the generator based on the first loss function value and the second loss function value to generate a facial expression editing model;

[0178] The editing module 16 is used to edit the expressions of the original face image based on the face expression editing model.

[0179] Furthermore, in the aforementioned facial expression editing device, the collection module 11 specifically includes:

[0180] The collection unit is used to collect facial images containing a variety of expressions;

[0181] A preprocessing unit is used to preprocess the face image;

[0182] The annotation unit is used to annotate the preprocessed face images and organize them to obtain a face image dataset.

[0183] Furthermore, in the aforementioned facial expression editing device, the extraction module 12 specifically includes:

[0184] The determination unit is used to predetermine the optical flow extraction network;

[0185] The first extraction unit is used to input the acquired original expression into the optical flow extraction network for optical flow extraction to obtain the first optical flow information from the original expression to the neutral expression;

[0186] The second extraction unit is used to input the acquired target expression into the optical flow extraction network for optical flow extraction, and obtain the second optical flow information from the target expression to the neutral expression.

[0187] Furthermore, in the aforementioned facial expression editing device, the prediction module 13 specifically includes:

[0188] Building blocks are used to pre-build generators;

[0189] The first prediction unit is used to use the original expression combined with the first optical flow information as input to predict the neutral expression image corresponding to the original expression through the generator.

[0190] The second prediction unit is used to use the original expression combined with the first optical flow information and the second optical flow information as input to predict the target expression image corresponding to the target expression through the generator.

[0191] Furthermore, in the aforementioned facial expression editing device, the comparison module 14 specifically includes:

[0192] The selection unit is used to select, from the face image dataset, the neutral expression ground truth image and the target expression ground truth image corresponding to the neutral expression image and the target expression image;

[0193] The first comparison unit is used to compare the neutral expression image with the neutral expression ground truth image and calculate the first loss function value between the two.

[0194] The second comparison unit is used to compare the target expression image with the target expression ground truth image and calculate the second loss function value between the two.

[0195] Furthermore, in the aforementioned facial expression editing device, the training module 15 specifically includes:

[0196] An iterative training unit is used to iteratively train the generator based on the first loss function value and the second loss function value using a backpropagation algorithm.

[0197] The model generation unit is used to generate a facial expression editing model when the generator training reaches the preset requirements.

[0198] Furthermore, in the aforementioned facial expression editing device, the editing module 16 specifically includes:

[0199] The acquisition unit is used to acquire the original face image to be edited.

[0200] The input unit is used to input the original face image into the face expression editing model to generate the target face image after expression editing;

[0201] An output or display unit is used to output or display the target face image.

[0202] It should be noted that, in the device embodiments of the present invention, the information interaction and execution process between the above modules are based on the same concept as in the method embodiments of the present invention. For details on their specific functions and the resulting technical effects, please refer to the aforementioned method embodiments section, which will not be repeated here.

[0203] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the facial expression editing method server-side as described in any of the above method embodiments.

[0204] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a client, and its internal structure diagram can be as follows. Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements the functions or steps of the facial expression editing method on the client side as described in any of the above method embodiments.

[0205] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown is only a schematic diagram of a part of the structure related to the present invention and does not constitute a limitation on the computer device on which the present invention is applied. The specific computer device may include more components than shown in the figure, or combine certain components, or have different component arrangements.

[0206] The processor referred to herein can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0207] The memory includes readable storage media, internal memory, etc., where internal memory can be the RAM of a computer device. Internal memory provides an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal storage units and external storage devices of the computer device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0208] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the facial expression editing method as described in any of the above method embodiments. The computer-readable storage medium may be non-volatile or volatile.

[0209] It should be noted that the functions or steps that can be achieved by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0210] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0211] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0212] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0213] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for editing facial expressions, characterized in that, include: Collect a dataset of facial images containing various expressions; Based on the optical flow extraction network, optical flow is extracted from the acquired original expression and the target expression respectively to obtain the corresponding first optical flow information and second optical flow information; Using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the neutral expression image and the target expression image corresponding to the original expression and the target expression, respectively. The neutral expression image and the target expression image are compared with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and the corresponding first loss function value and second loss function value are calculated. The generator is trained based on the first loss function value and the second loss function value to generate a facial expression editing model; Based on the aforementioned facial expression editing model, facial expression editing of the original facial image is realized; The optical flow extraction network extracts optical flow from both the acquired original facial expression and the target facial expression to obtain corresponding first and second optical flow information, including: Predetermine the optical flow extraction network; The acquired original facial expression is input into the optical flow extraction network for optical flow extraction to obtain the first optical flow information from the original facial expression to the neutral facial expression; The acquired target expression is input into the optical flow extraction network for optical flow extraction to obtain the second optical flow information from the target expression to the neutral expression. The step of using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and using a generator to predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively, includes: Pre-build generators; Using the original expression combined with the first optical flow information as input, the generator predicts the neutral expression image corresponding to the original expression. Using the original facial expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the target facial expression image corresponding to the target facial expression.

2. The facial expression editing method according to claim 1, characterized in that, The collected facial image dataset, containing various facial expressions, includes: Collect facial images containing a variety of expressions; The face image is preprocessed; The preprocessed face images are labeled and organized to obtain a face image dataset.

3. The facial expression editing method according to claim 1, characterized in that, The step of comparing the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and calculating the corresponding first loss function value and second loss function value, includes: Select the neutral expression ground truth image and the target expression ground truth image corresponding to the neutral expression image and the target expression image from the face image dataset; The neutral expression image is compared with the neutral expression ground truth image, and the first loss function value between the two is calculated; The target facial expression image is compared with the target facial expression ground truth image, and the second loss function value between the two is calculated.

4. The facial expression editing method according to claim 1, characterized in that, The step of training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model includes: Based on the first loss function value and the second loss function value, the generator is iteratively trained using the backpropagation algorithm; When the generator training reaches the preset requirements, a facial expression editing model is generated.

5. The facial expression editing method according to claim 1, characterized in that, The process of editing facial expressions based on the facial expression editing model includes: Obtain the original face image to be edited; The original face image is input into the face expression editing model to generate the target face image after expression editing; Output or display the target face image.

6. A facial expression editing device, characterized in that, include: The collection module is used to collect a dataset of facial images containing various expressions; The extraction module is used to extract optical flow from the acquired original expression and the target expression based on the optical flow extraction network, so as to obtain the corresponding first optical flow information and second optical flow information. The prediction module is used to use the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and to predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively through the generator; The comparison module is used to compare the neutral expression image and the target expression image with the neutral expression ground truth image and the target expression ground truth image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value. The training module is used to train the generator based on the first loss function value and the second loss function value to generate a facial expression editing model; The editing module is used to edit the expressions of the original face image based on the facial expression editing model. The optical flow extraction network extracts optical flow from both the acquired original facial expression and the target facial expression to obtain corresponding first and second optical flow information, including: Predetermine the optical flow extraction network; The acquired original facial expression is input into the optical flow extraction network for optical flow extraction to obtain the first optical flow information from the original facial expression to the neutral facial expression; The acquired target expression is input into the optical flow extraction network for optical flow extraction to obtain the second optical flow information from the target expression to the neutral expression. The step of using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and using a generator to predict the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively, includes: Pre-build generators; Using the original expression combined with the first optical flow information as input, the generator predicts the neutral expression image corresponding to the original expression. Using the original facial expression combined with the first optical flow information and the second optical flow information as input, the generator predicts the target facial expression image corresponding to the target facial expression.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the facial expression editing method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the facial expression editing method as described in any one of claims 1-5.

Images