Facial expression editing method and device, computer equipment and storage medium

Abstract

The invention discloses a facial expression editing method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. Performing optical flow extraction on the original expression and the target expression to obtain first optical flow information and second optical flow information; using the combination of the original expression and the first optical flow information, and the combination of the original expression and the first optical flow information and the second optical flow information as input, and predicting a neutral expression image and a target expression image through a generator; comparing the neutral expression image and the target expression image with the neutral expression truth value image and the target expression truth value image to obtain a first loss function value and a second loss function value; training the generator to generate a facial expression editing model; and realizing expression editing of the original face image based on the face expression editing model. According to the method, the face identity features can be kept, and meanwhile, high-accuracy and high-naturalness face expression editing and conversion can be achieved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a facial expression editing method, device, computer equipment and computer-readable storage medium. Background Art

[0002] At present, face editing technology is an important branch in the field of image processing. It involves operations such as denoising, enhancement, deformation and synthesis of face images to modify the appearance and expression of the face. With the development of digital image processing technology, face editing technology has shown broad application prospects in many financial technology-related fields such as face beautification, face recognition, and face animation, and can support corresponding social, shopping, interactive games, resource transfer and other functions. However, due to the characteristics of high dimensionality, high nonlinearity and high sensitivity of face images, how to ensure that the edited image can not only maintain the identity information of the original image, but also achieve the expected editing effect has always been the focus and difficulty of research in this field.

[0003] Facial expressions are an important way for humans to communicate emotions, and they can convey complex emotions and moods. In daily human communication, facial expressions are like a window that allows us to gain insight into each other's inner world. With the rise of deep learning technology, facial expression recognition and editing have become a research hotspot in the field of computer vision. Facial expression editing technology aims to convert the expression in a given facial image into a target expression while retaining the identity features of the face. This technology has great application potential in photography, filmmaking, entertainment industry and other fields. With the deepening of research, facial expression editing technology has attracted more and more attention from academia and industry.

[0004] Although some methods based on generative adversarial networks (GANs) have made significant progress in facial expression editing tasks, existing technologies still face some challenges. In particular, when there is a large difference between the input facial expression and the target expression, the generated image may have overlapping artifacts, especially in areas where facial muscles change intensively, such as the eyes, mouth, and nose. In addition, existing GAN-based facial expression editing methods usually adopt a one-step conversion method, and the limitation of network capacity makes it difficult to avoid image distortion when performing large-span expression editing.

[0005] To sum up, how to provide a facial expression editing method, device, computer equipment and non-volatile computer-readable storage medium that can maintain facial identity features while achieving high-accuracy and natural facial expression editing and conversion is a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a facial expression editing method, apparatus, computer equipment and computer-readable storage medium, aiming to solve the problem of how to achieve high-accuracy and natural facial expression editing and conversion while maintaining facial identity features.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for editing facial expressions, comprising:

[0009] Collect a dataset of facial images containing a variety of expressions;

[0010] Based on the optical flow extraction network, optical flow extraction is performed on the obtained original expression and the target expression respectively to obtain corresponding first optical flow information and second optical flow information;

[0011] Using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, respectively predicting a neutral expression image and a target expression image corresponding to the original expression and the target expression through a generator;

[0012] Compare the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value;

[0013] Based on the first loss function value and the second loss function value, training the generator to generate a facial expression editing model;

[0014] Based on the facial expression editing model, expression editing of the original facial image is achieved.

[0015] In a second aspect, the present invention provides a facial expression editing device, comprising:

[0016] A collection module is used to collect a dataset of facial images containing multiple expressions;

[0017] An extraction module, used to perform optical flow extraction on the acquired original expression and the target expression respectively based on an optical flow extraction network to obtain corresponding first optical flow information and second optical flow information;

[0018] A prediction module, configured to use the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and respectively predict a neutral expression image and a target expression image corresponding to the original expression and the target expression through a generator;

[0019] A comparison module, used to compare the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value;

[0020] A training module, used for training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model;

[0021] The editing module is used to implement expression editing of the original face image based on the face expression editing model.

[0022] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the facial expression editing method as described above when executing the computer program.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the facial expression editing method as described above.

[0024] Compared with the prior art, the present invention provides a facial expression editing method, apparatus, computer equipment and computer-readable storage medium, wherein a facial image data set containing multiple expressions is collected; based on an optical flow extraction network, optical flow extraction is performed on the obtained original expression and target expression respectively to obtain corresponding first optical flow information and second optical flow information; using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, a generator is used to predict a neutral expression image and a target expression image corresponding to the original expression and the target expression respectively; the neutral expression image and the target expression image are compared with the neutral expression true value image and the target expression true value image in the facial image data set, and the corresponding first loss function value and second loss function value are calculated; based on the first loss function value and the second loss function value, the generator is trained to generate a facial expression editing model; based on the facial expression editing model, expression editing of the original facial image is realized. In this way, the present invention can edit the expression of the original face image through the generated face expression editing model, while maintaining the face identity features, and can also achieve high-accuracy and natural face expression editing and conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of an application environment of a facial expression editing method provided by an embodiment of the present invention.

[0027] Figure 2 A schematic flow chart of a facial expression editing method provided by one embodiment of the present invention.

[0028] Figure 3 A schematic diagram of program modules of a facial expression editing device provided by one embodiment of the present invention.

[0029] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention.

[0030] Figure 5 Another structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0035] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" etc. described in the specification of the present invention mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0038] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0039] A facial expression editing method provided by an embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment shown, the client communicates with the server through the network. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPC), netbooks, cloud computing devices, personal digital assistants (PDA) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0040] See also Figure 2 The embodiment of the present invention provides a method for editing facial expressions, wherein the method comprises the following steps:

[0041] S100, collect a dataset of facial images containing multiple expressions;

[0042] S200, based on the optical flow extraction network, respectively performing optical flow extraction on the acquired original expression and the target expression to obtain corresponding first optical flow information and second optical flow information;

[0043] S300, using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and respectively predicting a neutral expression image and a target expression image corresponding to the original expression and the target expression through a generator;

[0044] S400, comparing the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image dataset, respectively, and calculating corresponding first loss function values ​​and second loss function values;

[0045] S500, training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model;

[0046] S600: Based on the facial expression editing model, realize expression editing for the original facial image.

[0047] In the specific implementation, in this embodiment, by collecting a facial image dataset containing multiple expressions; then, based on the optical flow extraction network, optical flow extraction is performed on the obtained original expression and target expression respectively to obtain the corresponding first optical flow information and second optical flow information; then, the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information are used as input, and the neutral expression image and the target expression image corresponding to the original expression and the target expression are predicted by the generator respectively; then, the neutral expression image and the target expression image are compared with the neutral expression true value image and the target expression true value image in the facial image dataset respectively, and the corresponding first loss function value and second loss function value are calculated; then, based on the first loss function value and the second loss function value, the generator is trained to generate a facial expression editing model; finally, based on the facial expression editing model, the expression editing of the original facial image is realized. In this way, the facial expression editing model generated by this embodiment can edit the expression of the original facial image while maintaining the facial identity features, and can also realize high-accuracy and natural facial expression editing and conversion.

[0048] Furthermore, the facial expression editing method, wherein the step S100, collecting a facial image data set containing multiple expressions, specifically comprises the steps of:

[0049] Collect facial images containing a variety of expressions;

[0050] Preprocessing the facial image;

[0051] The preprocessed face images are labeled and sorted to obtain a face image dataset.

[0052] In the specific implementation, in this embodiment, face images containing multiple expressions are collected; the face images are preprocessed; the preprocessed face images are annotated to obtain a face image data set; wherein the specific implementation process of each of the above steps is roughly as follows:

[0053] 1. Collect facial images with multiple expressions

[0054] Data sources: Collect facial images through public facial expression datasets, social media, volunteer recruitment, or professional photography;

[0055] Diversity: Ensure that the collected face images contain faces of different ages, genders, races, and lighting conditions, as well as a variety of natural expressions such as happiness, sadness, anger, etc.

[0056] 2. Preprocess the face image

[0057] Image quality: Use image processing techniques to remove noise, adjust brightness and contrast to improve image quality;

[0058] Image size: adjust all face images to a uniform resolution and aspect ratio for easy subsequent processing;

[0059] Face detection and alignment: Use the face detection algorithm to detect the face in the face image and align it to ensure that the key feature points such as eyes, nose and mouth are in the same position;

[0060] 3. Label the preprocessed face images

[0061] Expression annotation: Label each face image with an expression category, such as “happy”, “sad”, etc. This can be done by expert annotation or using annotation tools;

[0062] Key point annotation: Annotate the key points of the face in the face image, such as the corners of the eyes, the corners of the mouth, the tip of the nose, etc. This is necessary for some advanced expression editing techniques;

[0063] Data cleaning: Check the accuracy of annotations and make corrections or delete incorrectly annotated face images if necessary;

[0064] 4. Organize and obtain face image dataset

[0065] Data organization: Classify and organize the labeled face images according to expression categories, image quality and other standards to obtain a face image dataset;

[0066] Database construction: Store the face image dataset in a database for easy retrieval and use;

[0067] Through the above process, a high-quality, diverse and accurately annotated facial image dataset can be constructed, providing a solid foundation for subsequent expression editing or other related tasks.

[0068] Furthermore, the facial expression editing method, wherein the step S200, based on the optical flow extraction network, performs optical flow extraction on the acquired original expression and the target expression respectively to obtain corresponding first optical flow information and second optical flow information, specifically comprises the steps of:

[0069] Predetermine the optical flow extraction network;

[0070] Inputting the acquired original expression into the optical flow extraction network to perform optical flow extraction, and obtaining first optical flow information from the original expression to the neutral expression;

[0071] The acquired target expression is input into the optical flow extraction network for optical flow extraction to obtain second optical flow information from the target expression to the neutral expression.

[0072] In the specific implementation, in this embodiment, an optical flow extraction network is predetermined; the original expression obtained is input into the optical flow extraction network for optical flow extraction, and the first optical flow information from the original expression to the neutral expression is obtained; the target expression obtained is input into the optical flow extraction network for optical flow extraction, and the second optical flow information from the target expression to the neutral expression is obtained; wherein, the specific implementation process of each of the above steps is roughly as follows:

[0073] 1. Determine (select or build) the optical flow extraction network

[0074] Network selection: Choose an off-the-shelf optical flow extraction network that has proven effective in optical flow estimation tasks;

[0075] Network construction: If the existing optical flow extraction network does not meet the requirements, a new optical flow extraction network can be built; this usually involves defining the network architecture, selecting an appropriate loss function and optimization algorithm;

[0076] 2. Extract the first optical flow information

[0077] Input network: input the acquired original expression into the optical flow extraction network;

[0078] Optical flow calculation: The optical flow extraction network analyzes the original expression, calculates and outputs the first optical flow information from the original expression to the neutral expression;

[0079] 3. Extract the second optical flow information

[0080] Input network: Input the acquired target expression into the optical flow extraction network;

[0081] Optical flow calculation: The optical flow extraction network analyzes the target expression, calculates and outputs the second optical flow information from the target expression to the neutral expression;

[0082] 4. Post-processing optical flow information

[0083] Optical flow optimization: post-processing the first optical flow information and the second optical flow information output by the optical flow extraction network, such as smoothing and removing outliers, to improve the quality of the first optical flow information and the second optical flow information;

[0084] Through the above process, the corresponding optical flow information can be effectively extracted from the original expression and target expression images, providing a solid foundation for subsequent expression editing or other related tasks.

[0085] Further, the facial expression editing method, wherein the step S300, using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, respectively predicting the neutral expression image and the target expression image corresponding to the original expression and the target expression through a generator, specifically comprises the steps of:

[0086] Pre-built generators;

[0087] Using the original expression combined with the first optical flow information as input, predicting a neutral expression image corresponding to the original expression through the generator;

[0088] The original expression combined with the first optical flow information and the second optical flow information are used as input, and a target expression image corresponding to the target expression is predicted by the generator.

[0089] In the specific implementation, in this embodiment, a generator is pre-built; the original expression is combined with the first optical flow information as input, and the neutral expression image corresponding to the original expression is predicted by the generator; the original expression is combined with the first optical flow information and the second optical flow information as input, and the target expression image corresponding to the target expression is predicted by the generator; wherein the specific implementation process of each of the above steps is roughly as follows:

[0090] 1. Build the generator:

[0091] Design network: Design a deep learning network, the generator, which can accept raw expressions and optical flow information as input and output new expression images;

[0092] Initialization parameters: Initialize the network parameters of the generator, which usually involves assigning random weights or loading the parameters of a pre-trained model;

[0093] 2. Prepare input data

[0094] Get original expressions: Select or collect original expressions, which will be used as one of the inputs of the generator;

[0095] Extract optical flow information: Use the optical flow extraction network to obtain the first optical flow information (optical flow from original expression to neutral expression) and the second optical flow information (optical flow from original expression to target expression);

[0096] 3. Generate neutral expression images

[0097] Data fusion: Combine the original expression image and the first optical flow information as the input of the generator;

[0098] Model reasoning: forward propagation is performed through the generator, which predicts a neutral expression image based on the input data;

[0099] 4. Generate target expression image

[0100] Second data fusion: combine the original expression image, the first optical flow information and the second optical flow information and use them as the input of the generator again;

[0101] Model reasoning again: another forward propagation through the generator, the generator predicts the target expression image;

[0102] 5. Post-processing expression images

[0103] Expression image optimization: Post-processing of the generated neutral expression image and target expression image, such as color correction and detail enhancement, is performed to improve image quality;

[0104] Through the above process, the neutral expression image and the target expression image corresponding to the original expression and the target expression can be effectively predicted by the generator, respectively, providing a solid foundation for subsequent expression editing or other related tasks.

[0105] Further, the facial expression editing method, wherein the step S400, comparing the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the facial image data set, respectively, and calculating the corresponding first loss function value and second loss function value, specifically comprises the steps of:

[0106] Selecting a neutral expression true value image and a target expression true value image corresponding to the neutral expression image and the target expression image from the face image dataset;

[0107] Comparing the neutral expression image with the neutral expression true value image, and calculating a first loss function value between the two;

[0108] The target expression image is compared with the target expression true value image, and a second loss function value between the two is calculated.

[0109] In the specific implementation, in this embodiment, a neutral expression true value image and a target expression true value image corresponding to the neutral expression image and the target expression image are selected from the face image data set; the neutral expression image is compared with the neutral expression true value image, and a first loss function value between the two is calculated; the target expression image is compared with the target expression true value image, and a second loss function value between the two is calculated; wherein, the specific implementation process of each of the above steps is roughly as follows:

[0110] 1. True value image selection

[0111] Selecting a neutral expression true value image and a target expression true value image corresponding to the neutral expression image and the target expression image from a face image dataset;

[0112] Ensure that the selected neutral expression ground truth images and target expression ground truth images are consistent with their corresponding neutral expression images and target expression images in resolution, size, and alignment to enable accurate comparison;

[0113] 2. Image feature extraction

[0114] Feature representation: Use a predefined feature extraction method (such as a pre-trained deep learning model) to extract feature representations from the image, which helps in subsequent comparison and loss calculation;

[0115] 3. Calculate the first loss function value:

[0116] Comparing neutral expression images: Compare the generated neutral expression images with the corresponding neutral expression ground truth images at the feature level;

[0117] Loss calculation: According to the predetermined loss function (such as mean square error, structural similarity index SSIM, perceptual loss, etc.), the difference between the two is calculated to obtain the first loss function value;

[0118] 4. Calculate the second loss function value:

[0119] Comparing the target expression image: comparing the generated target expression image with the corresponding target expression ground truth image at the feature level;

[0120] Loss calculation: Also use the loss function to calculate the difference between the two and get the second loss function value;

[0121] Through the above process, the first loss function value and the second loss function value can be calculated, providing a solid foundation for subsequent expression editing or other related tasks.

[0122] Furthermore, the facial expression editing method, wherein the step S500, based on the first loss function value and the second loss function value, training the generator to generate a facial expression editing model, specifically comprises the steps of:

[0123] Iteratively training the generator through a back propagation algorithm based on the first loss function value and the second loss function value;

[0124] When the generator training reaches the preset requirements, a facial expression editing model is generated.

[0125] In the specific implementation, in this embodiment, based on the first loss function value and the second loss function value, the generator is iteratively trained through the back propagation algorithm; when the generator training meets the preset requirements, a facial expression editing model is generated; wherein, the specific implementation process of each of the above steps is roughly as follows:

[0126] 1. Loss function optimization

[0127] Back propagation: Use the calculated first loss function value and the second loss function value to perform back propagation and calculate the gradient of the generator network parameters;

[0128] Parameter update: Update the parameters of the generator using gradient descent or other optimization algorithms to reduce the loss and improve the quality of image generation;

[0129] 2. Generator Evaluation and Iteration

[0130] Performance evaluation: Evaluate the performance of the generator in generating neutral expression images and target expression images to ensure that the generated images not only maintain the original identity information but also achieve the expected expression editing effect;

[0131] Iterative training: According to the loss function value and the quality of the generated image, the generator is trained multiple times to continuously optimize the performance of the generator;

[0132] 3. Evaluate the preset requirements

[0133] Performance criteria: set preset performance requirements, such as a lower limit on the loss function value, an upper limit on the accuracy, or performance on the validation set;

[0134] Model validation: Regularly test the generator on the validation set to ensure that the generated image quality meets the requirements and the model is not overfitting;

[0135] 4. Generate facial expression editing model

[0136] Satisfaction conditions: When the training of the generator reaches the preset performance requirements, such as when the loss function value is low enough and the performance on the validation set is stable, the training is considered complete;

[0137] Model saving: Save the parameters of the generator to form the final facial expression editing model for subsequent use or further fine-tuning;

[0138] 5. Subsequent Applications

[0139] Model deployment: deploy the trained facial expression editing model to actual applications, such as mobile applications, online services, or offline software;

[0140] User interaction: In actual applications, users can use the facial expression editing model to edit the original facial image to achieve expression conversion and generation;

[0141] Through the above process, it can be ensured that the generator is gradually optimized during the training process, and finally a facial expression editing model that can accurately edit facial expressions is generated.

[0142] Further, the facial expression editing method, wherein the step S600, based on the facial expression editing model, realizes expression editing for the original facial image, specifically comprises the steps of:

[0143] Obtain the original face image to be edited;

[0144] Inputting the original face image into the face expression editing model to generate a target face image after expression editing;

[0145] The target face image is output or displayed.

[0146] In the specific implementation, in this embodiment, an original face image to be edited is obtained; the original face image is input into the face expression editing model to generate a target face image after expression editing; the target face image is output or displayed; wherein the specific implementation process of each of the above steps is roughly as follows:

[0147] 1. Get the original face image

[0148] Image acquisition: obtaining the original face image to be edited from user uploads, databases or other sources;

[0149] Image preprocessing: Perform necessary preprocessing on the original face image, including cropping, scaling, etc. to match the input size of the facial expression editing model;

[0150] 2. Face detection and alignment

[0151] Detect faces: Use face detection algorithms to locate faces in original face images;

[0152] Align faces: Align detected faces to a standard pose, which may include adjusting the position of the eyes, nose, and mouth to ensure consistent alignment with the face images used during training;

[0153] 3. Input facial expression editing model

[0154] Load model: load the trained facial expression editing model;

[0155] Model inference: The preprocessed and aligned original face image is input into the facial expression editing model, and the facial expression editing model generates the target face image after expression editing based on the learned expression conversion mapping;

[0156] 4. Post-processing face images

[0157] Color correction: Perform color correction on the generated target face image to ensure that its color distribution is consistent with the input original face image;

[0158] Detail enhancement: Enhance the details of the generated target face image to improve the image quality and make it look more natural;

[0159] 5. Output or display the target face image

[0160] Image conversion: convert the generated target face image into a format suitable for output or display, such as JPEG, PNG, etc.

[0161] Display image: Display the target face image after expression editing on the user interface, or save it for subsequent use;

[0162] Output image: Of course, you can also output the target face image after expression editing to a specified location as needed;

[0163] 6. User Feedback

[0164] Provide feedback mechanism: Allow users to evaluate the generated target face image and provide feedback;

[0165] Model iteration: Fine-tune and optimize the facial expression editing model based on user feedback and editing results to improve the accuracy and naturalness of expression editing and conversion;

[0166] Through the above process, the complete flow from acquiring the original face image to outputting or displaying the target face image after expression editing can be realized, which can provide users with high-quality expression editing and conversion experience.

[0167] It can be seen from the above method embodiments that the facial expression editing method provided by the present invention includes: collecting a facial image data set containing multiple expressions; based on an optical flow extraction network, performing optical flow extraction on the acquired original expression and target expression respectively to obtain corresponding first optical flow information and second optical flow information; using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, respectively predicting the neutral expression image and the target expression image corresponding to the original expression and the target expression through a generator; comparing the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the facial image data set, and calculating the corresponding first loss function value and second loss function value; based on the first loss function value and the second loss function value, training the generator to generate a facial expression editing model; based on the facial expression editing model, realizing expression editing of the original facial image. In this way, the method of the present invention can achieve high-accuracy and natural facial expression editing and conversion while maintaining facial identity features through the generated facial expression editing model for the expression editing of the original facial image.

[0168] It can be understood that the facial expression editing method provided in the embodiment of the present invention can be applied to facial expression editing scenarios related to the financial technology field, specifically:

[0169] 1. Integration into applications: The facial expression editing function can be integrated into the corresponding applications of financial institutions to support corresponding social, shopping, interactive games, resource transfer and other functions;

[0170] 2. Real-time editing: In scenarios where real-time expression editing is required, such as video or live broadcast, ensure that the facial expression editing model can respond quickly and perform real-time expression editing.

[0171] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0172] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a facial expression editing device, wherein the device comprises:

[0173] A collection module 11 is used to collect a facial image dataset containing multiple expressions;

[0174] An extraction module 12 is used to perform optical flow extraction on the acquired original expression and the target expression respectively based on an optical flow extraction network to obtain corresponding first optical flow information and second optical flow information;

[0175] A prediction module 13, configured to use the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and respectively predict a neutral expression image and a target expression image corresponding to the original expression and the target expression through a generator;

[0176] A comparison module 14 is used to compare the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image data set, respectively, and calculate the corresponding first loss function value and second loss function value;

[0177] A training module 15, configured to train the generator based on the first loss function value and the second loss function value to generate a facial expression editing model;

[0178] The editing module 16 is used to implement expression editing of the original facial image based on the facial expression editing model.

[0179] Furthermore, in the facial expression editing device, the collection module 11 specifically comprises:

[0180] A collection unit, used for collecting facial images containing multiple expressions;

[0181] A preprocessing unit, used for preprocessing the face image;

[0182] The labeling unit is used to label the preprocessed face images and obtain a face image data set.

[0183] Furthermore, in the facial expression editing device, the extraction module 12 specifically comprises:

[0184] A determination unit, used for predetermining an optical flow extraction network;

[0185] A first extraction unit, configured to input the acquired original expression into the optical flow extraction network for optical flow extraction, and obtain first optical flow information from the original expression to the neutral expression;

[0186] The second extraction unit is used to input the acquired target expression into the optical flow extraction network to perform optical flow extraction, and obtain second optical flow information from the target expression to the neutral expression.

[0187] Furthermore, in the facial expression editing device, the prediction module 13 specifically comprises:

[0188] Build units, used to pre-build generators;

[0189] A first prediction unit, configured to use the original expression in combination with the first optical flow information as input, and predict a neutral expression image corresponding to the original expression through the generator;

[0190] The second prediction unit is used to use the original expression combined with the first optical flow information and the second optical flow information as input to predict the target expression image corresponding to the target expression through the generator.

[0191] Furthermore, in the facial expression editing device, the comparison module 14 specifically comprises:

[0192] A selection unit, used for selecting a neutral expression true value image and a target expression true value image corresponding to the neutral expression image and the target expression image from the face image data set;

[0193] A first comparison unit, used for comparing the neutral expression image with the neutral expression true value image, and calculating a first loss function value between the two;

[0194] The second comparison unit is used to compare the target expression image with the target expression true value image, and calculate a second loss function value between the two.

[0195] Furthermore, in the facial expression editing device, the training module 15 specifically comprises:

[0196] An iterative training unit, configured to iteratively train the generator through a back propagation algorithm based on the first loss function value and the second loss function value;

[0197] The model generation unit is used to generate a facial expression editing model when the generator training reaches a preset requirement.

[0198] Furthermore, in the facial expression editing device, the editing module 16 specifically comprises:

[0199] An acquisition unit, used for acquiring an original face image to be edited;

[0200] An input unit, used for inputting the original face image into the face expression editing model to generate a target face image after expression editing;

[0201] An output or display unit is used to output or display the target face image.

[0202] It should be noted that in the embodiment of the device of the present invention, the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the aforementioned method embodiment part and will not be repeated here.

[0203] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of the facial expression editing method in any of the above method embodiments are implemented.

[0204] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the facial expression editing method in any of the above method embodiments are implemented.

[0205] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0206] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0207] Among them, the memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of a computer device, and in other embodiments, it can also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory can also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory can also be used to temporarily store data that has been output or is to be output.

[0208] Based on the above method embodiments, another embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the facial expression editing method in any of the above method embodiments is implemented. The computer-readable storage medium may be non-volatile or volatile.

[0209] It should be noted that the above-mentioned functions or steps that can be implemented by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be found in the relevant descriptions in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.

[0210] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and / or any other suitable types of memory.

[0211] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0212] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0213] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A facial expression editing method, characterized in that: include: Collect a dataset of facial images containing a variety of expressions; Based on the optical flow extraction network, optical flow extraction is performed on the obtained original expression and the target expression respectively to obtain corresponding first optical flow information and second optical flow information; Using the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, respectively predicting a neutral expression image and a target expression image corresponding to the original expression and the target expression through a generator; Compare the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value; Based on the first loss function value and the second loss function value, training the generator to generate a facial expression editing model; Based on the facial expression editing model, expression editing of the original facial image is achieved.

2. The facial expression editing method according to claim 1, characterized in that: The collection contains a dataset of facial images with various expressions, including: Collect facial images containing a variety of expressions; Preprocessing the facial image; The preprocessed face images are labeled and sorted to obtain a face image dataset.

3. The facial expression editing method according to claim 1, characterized in that: The optical flow extraction network is based on which the optical flow extraction is performed on the original expression and the target expression to obtain the corresponding first optical flow information and second optical flow information, including: Predetermine the optical flow extraction network; Inputting the acquired original expression into the optical flow extraction network to perform optical flow extraction, and obtaining first optical flow information from the original expression to the neutral expression; The acquired target expression is input into the optical flow extraction network for optical flow extraction to obtain second optical flow information from the target expression to the neutral expression.

4. The facial expression editing method according to claim 1, characterized in that: The method uses the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and predicts the neutral expression image and the target expression image corresponding to the original expression and the target expression respectively through a generator, including: Pre-built generators; Using the original expression combined with the first optical flow information as input, predicting a neutral expression image corresponding to the original expression through the generator; The original expression combined with the first optical flow information and the second optical flow information are used as input, and a target expression image corresponding to the target expression is predicted by the generator.

5. The facial expression editing method according to claim 1, characterized in that: The step of comparing the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image dataset, respectively, and calculating the corresponding first loss function value and second loss function value, comprises: Selecting a neutral expression true value image and a target expression true value image corresponding to the neutral expression image and the target expression image from the face image dataset; Comparing the neutral expression image with the neutral expression true value image, and calculating a first loss function value between the two; The target expression image is compared with the target expression true value image, and a second loss function value between the two is calculated.

6. The facial expression editing method according to claim 1, characterized in that: The step of training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model includes: Based on the first loss function value and the second loss function value, iteratively training the generator through a back propagation algorithm; When the generator training reaches the preset requirements, a facial expression editing model is generated.

7. The facial expression editing method according to claim 1, characterized in that: The method of implementing expression editing for the original face image based on the face expression editing model includes: Obtain the original face image to be edited; Inputting the original face image into the face expression editing model to generate a target face image after expression editing; The target face image is output or displayed.

8. A facial expression editing device, characterized in that: include: A collection module is used to collect a dataset of facial images containing multiple expressions; An extraction module, used to perform optical flow extraction on the acquired original expression and the target expression respectively based on an optical flow extraction network to obtain corresponding first optical flow information and second optical flow information; A prediction module, configured to use the original expression combined with the first optical flow information, and the original expression combined with the first optical flow information and the second optical flow information as input, and respectively predict a neutral expression image and a target expression image corresponding to the original expression and the target expression through a generator; A comparison module, used to compare the neutral expression image and the target expression image with the neutral expression true value image and the target expression true value image in the face image dataset, respectively, and calculate the corresponding first loss function value and second loss function value; A training module, used for training the generator based on the first loss function value and the second loss function value to generate a facial expression editing model; The editing module is used to implement expression editing of the original face image based on the face expression editing model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the facial expression editing method as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the facial expression editing method as described in any one of claims 1 to 7 is implemented.

Images