Facial expression editing method, electronic equipment and storage medium

By using a multi-stage training expression editing model in the face expression editing technology, the problem of insufficient expression intensity control and local details in the existing technology is solved, and high-quality and natural expression editing image generation is achieved.

CN118918207APending Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411060563.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing facial expression editing techniques lack attention to expression intensity control and local details, resulting in low quality and naturalness of generated images, and possible artifacts and blur.

Method used

The expression editing model trained by editing ability is adopted, including image feature extraction network, expression control network and image generation network. Through the collaborative work of image feature extraction, expression control and generation network, high-quality expression editing images are generated. The model ensures the accuracy and nature of expression editing through multi-stage training, including training of expression control network, image feature extraction network and image generation network.

Benefits of technology

The generated expression editing images are of higher quality, and the expression intensity and detail control are more refined, avoiding artifacts and blurring, and improving the naturalness and authenticity of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918207A_ABST
    Figure CN118918207A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a facial expression editing method, electronic equipment and a storage medium. According to the facial expression editing method provided by the embodiment of the invention, a target editing expression type, an expression intensity editing instruction and an initial expression image of a to-be-edited expression need to be acquired firstly, and further, an expression editing model subjected to editing ability training is acquired; wherein the expression editing model comprises an image feature extraction network, an expression control network and an image generation network. Further, feature extraction is carried out on the initial expression image through an image feature extraction network, and image feature information is obtained; and in the expression control network, according to the target editing expression type and the expression intensity editing instruction, generating expression editing information, and through the image generation network, according to the image feature information and the expression editing information, generating a target expression image after expression editing. Therefore, the expression editing image with higher quality can be generated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a facial expression editing method, electronic device, and storage medium. Background Art

[0002] Facial expression editing technology is a technology that adjusts the expression in a facial image to match the expected target expression without changing the identity of the face. Facial expression editing technology has significant application potential in the medical and financial fields. In the medical field, facial expression editing can be used for psychotherapy and subject emotion analysis. In the financial field, facial expression editing technology can be applied to customer service and risk management.

[0003] In the related technologies of facial expression editing, structures such as conditional generative adversarial networks (GAN) or variational autoencoders (VAE) can be used to take the input face image and the target expression type label as conditions to generate an output face image with the target expression. However, this method does not pay enough attention to the local details of the expression, which may cause artifacts and blurring during the expression conversion process, which affects the quality and naturalness of the generated image. Therefore, how to generate higher quality expression editing images has become a problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a facial expression editing method, an electronic device, and a storage medium, which can generate a higher quality expression editing image.

[0005] According to the first aspect of the present application, the facial expression editing method includes:

[0006] Obtaining the target editing expression type, expression intensity editing instructions, and the initial expression image of the expression to be edited;

[0007] Acquire an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network;

[0008] Extracting features from the initial expression image through the image feature extraction network to obtain image feature information;

[0009] In the expression control network, expression editing information is generated according to the target editing expression type and the expression intensity editing instruction;

[0010] The target expression image after the expression is edited is generated through the image generation network according to the image feature information and the expression editing information.

[0011] According to some embodiments of the present application, the step of obtaining an expression editing model that has undergone editing capability training includes:

[0012] Determine an original expression editing model; wherein the original expression editing model includes the original image feature extraction network, the original expression control network and the original image generation network;

[0013] Acquire multiple expression sample images, expression category labels corresponding to each of the expression sample images, prior expression information corresponding to each of the expression sample images, and prior feature information corresponding to each of the expression sample images;

[0014] Based on the plurality of expression sample images, the expression type label corresponding to each expression sample image, and the prior expression information corresponding to each expression type label, the original expression control network is trained in the first stage;

[0015] After the first stage of training, based on the plurality of expression sample images and the prior feature information corresponding to each of the expression sample images, the original image feature extraction network is trained in the second stage;

[0016] After the second stage of training, based on the plurality of expression sample images and the expression type label corresponding to each of the expression sample images, the original image generation network is trained in the third stage;

[0017] The original expression editing model that has undergone the first stage training, the second stage training, and the third stage training is determined as the expression editing model that has undergone editing capability training.

[0018] According to some embodiments of the present application, the first stage training of the original expression control network based on the plurality of expression sample images, the expression category label corresponding to each expression sample image, and the prior expression information corresponding to each expression category label includes:

[0019] Inputting the expression type label into the original expression control network, so that the expression control network performs expression control training on the expression type label to generate first expression training data;

[0020] Comparing the prior expression information corresponding to the expression type label with the first expression training data to obtain first deviation data;

[0021] The model parameters of the expression control network are updated based on the first deviation data, and the expression type label is input into the original expression control network again until the expression control network meets the preset first stage termination condition, thereby terminating the first stage training.

[0022] According to some embodiments of the present application, the step of inputting the expression type label into the original expression control network so that the expression control network performs expression control training on the expression type label to generate first expression training data includes:

[0023] Inputting the expression type label into the original expression control network, and introducing an intensity control training signal into the expression control network;

[0024] In the expression control network, expression control training is performed on the expression type label based on the intensity control training signal to generate the first expression training data.

[0025] According to some embodiments of the present application, the second stage training of the original image feature extraction network based on the plurality of expression sample images and the prior feature information corresponding to each expression sample image includes:

[0026] Inputting the expression sample image into the original image feature extraction network so that the image feature extraction network performs image feature extraction training on the expression sample image to obtain first image feature training data;

[0027] Comparing the prior feature information corresponding to the expression sample image with the first image feature training data to obtain second deviation data;

[0028] The model parameters of the image feature extraction network are updated based on the second deviation data, and the execution returns to inputting the expression sample image into the image feature extraction network until the image feature extraction network meets the preset second-stage termination condition, thereby terminating the second-stage training.

[0029] According to some embodiments of the present application, the third stage training of the original image generation network based on the plurality of expression sample images and the expression type label corresponding to each expression sample image includes:

[0030] Inputting the expression type label into the expression control network trained in the first stage, so that the expression control network performs expression control training on the expression type label to obtain second expression training data;

[0031] Inputting the expression sample image into the image feature extraction network after the second stage training, so that the image feature extraction network performs image feature extraction training on the expression sample image to obtain second image feature training data;

[0032] Inputting the second expression training data and the second image feature training data into the original image generation network, so that the image generation network generates expression editing training data according to the second expression training data and the second image feature training data;

[0033] Performing a true value comparison between the expression sample image and the corresponding expression editing training data to obtain third deviation data;

[0034] The model parameters of the image generation network are updated based on the third deviation data, and the expression type label is input into the expression control network after the first stage training, and the image generation network meets the preset third stage termination condition, thereby terminating the third stage training.

[0035] According to some embodiments of the present application, the step of obtaining a plurality of expression sample images, an expression type label corresponding to each of the expression sample images, a priori expression information corresponding to each of the expression sample images, and a priori feature information corresponding to each of the expression sample images includes:

[0036] Acquire a plurality of expression sample images and an expression type label corresponding to each of the expression sample images;

[0037] Inputting each of the expression sample images into the pre-trained expression parsing model for expression parsing processing to obtain the prior expression information corresponding to each of the expression sample images;

[0038] Each of the expression sample images is input into the pre-trained feature extraction model for feature extraction processing to obtain the prior feature information corresponding to each of the expression sample images.

[0039] According to the second aspect of the present application, the facial expression editing device includes:

[0040] A data acquisition unit, used to acquire a target edited expression type, an expression intensity edit instruction, and an initial expression image of the expression to be edited;

[0041] A model acquisition unit, used to acquire an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network;

[0042] The image feature extraction network is used to extract features from the initial expression image to obtain image feature information;

[0043] The expression control network is used to generate expression editing information according to the target editing expression type and the expression intensity editing instruction;

[0044] The image generation network is used to generate a target expression image after the expression is edited according to the image feature information and the expression editing information.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the facial expression editing method as described in any one of the embodiments of the first aspect of the present application.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement a facial expression editing method as described in any one of the embodiments of the first aspect of the present application.

[0047] The facial expression editing method, electronic device, and storage medium according to the embodiments of the present application have at least the following beneficial effects:

[0048] According to the facial expression editing method of the embodiment of the present application, it is necessary to first obtain the target editing expression type, expression intensity editing instructions and the initial expression image of the expression to be edited, and further, obtain an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network and an image generation network. Further, feature extraction is performed on the initial expression image through the image feature extraction network to obtain image feature information. In the expression control network, expression editing information is generated according to the target editing expression type and the expression intensity editing instructions, and through the image generation network, a target expression image after the edited expression is generated according to the image feature information and the expression editing information. In this way, a higher quality expression editing image can be generated.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0051] Figure 1 A schematic diagram of a process for editing facial expressions provided in an embodiment of the present application;

[0052] Figure 2 for Figure 1The flowchart of step S102 in the embodiment of the present application is shown;

[0053] Figure 3 for Figure 2 The flowchart of step S202 in the embodiment of the present application is shown;

[0054] Figure 4 for Figure 2 The flowchart of step S203 in the embodiment of the present application is shown;

[0055] Figure 5 for Figure 4 The flowchart of step S401 in the embodiment of the present application is shown;

[0056] Figure 6 for Figure 2 The flowchart of step S204 in the embodiment of the present application is shown;

[0057] Figure 7 for Figure 2 The flowchart of step S205 in the embodiment of the present application is shown;

[0058] Figure 8 is a structural schematic diagram of a facial expression editing device provided in an embodiment of the present application;

[0059] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0061] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0062] In the description of the present application, it should be understood that descriptions involving orientation, such as up, down, left, right, front, back, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0063] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0064] In the description of this application, it should be noted that, unless otherwise clearly defined, the terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application in combination with the specific content of the technical solution. In addition, the identification of specific steps below does not represent a limitation on the order of steps and execution logic. The execution order and execution logic between each step should be understood and inferred with reference to the contents described in the embodiment.

[0065] Facial expression editing is a technique that adjusts the expression in a facial image to match the expected target expression without changing the identity of the face. This technology has important application value in many fields, including human-computer interaction, entertainment, education, and psychotherapy. Despite its wide range of potential applications, the facial expression editing task itself is full of challenges. It requires that when generating images, not only the authenticity and naturalness of the image must be guaranteed, but also the type and intensity of the expression must be precisely controlled.

[0066] Facial expression editing technology has significant application potential in the medical and financial fields.

[0067] In the medical field, this technology can be used for psychotherapy and emotional analysis of subjects. For example, psychologists can use facial expression editing technology to simulate different emotional reactions to help subjects identify and express their emotional states. In addition, by analyzing the changes in the subject's expression, doctors can better understand the subject's psychological state and provide more personalized treatment plans.

[0068] In the financial sector, facial expression editing technology can be applied to customer service and risk management. Financial institutions can use this technology to improve customer service experience, for example, by generating virtual customer service representatives with positive expressions to enhance the friendliness and affinity of customer interactions. At the same time, in terms of risk management, banks and financial institutions can use facial expression analysis to evaluate customers' emotional reactions during business processes and identify possible risky behaviors or abnormal patterns. In addition, in financial promotional materials, by accurately controlling the intensity of character expressions, promotional information can be conveyed more effectively, attracting the attention of potential customers and improving the persuasiveness of promotional materials.

[0069] Overall, facial expression editing technology can bring innovative services and analysis tools to the medical and financial fields by providing more natural and controllable expression changes, helping to improve service efficiency, enhance customer experience, and play an important role in risk management.

[0070] Generative Adversarial Networks (GAN) is a deep learning model consisting of two parts: the generator and the discriminator. These two parts compete with each other during the training process, forming an adversarial training mechanism.

[0071] The goal of the generator is to produce realistic data, such as images, audio, or text, which should be statistically indistinguishable from real data. It usually uses a deep neural network to learn the distribution of data and generate new samples. The generator receives random noise as input and gradually improves the quality of the data it generates by learning the distribution of real data.

[0072] The role of the discriminator is to distinguish between the fake data generated by the generator and the real data in the real data set. It is also a deep neural network that makes judgments by analyzing the characteristics of the input data. The training goal of the discriminator is to improve its discrimination ability so that it can accurately identify fake data.

[0073] During the training process of the Generative Adversarial Network, the generator and the discriminator engage in a "game". The generator tries to test the discriminator so that the data it generates is mistakenly identified as real data; while the discriminator strives to improve its ability to distinguish between real and fake data. This adversarial training drives the performance of both networks, and ultimately the generator is able to produce high-quality samples that are difficult to distinguish from real data.

[0074] In recent years, generative adversarial networks have made significant progress in the field of image generation, which provides new possibilities for facial expression editing methods based on generative adversarial networks. Generative adversarial networks consist of generators and discriminators, which learn the distribution of real data through adversarial training to generate high-quality images.

[0075] In the related technologies of facial expression editing, a conditional generative adversarial network or variational autoencoder (VAE) can be used to take the input facial image and the target expression type label as conditions to generate an output facial image with the target expression. However, this method has some problems that need to be solved:

[0076] First, the relevant technology has shortcomings in controlling the intensity of expression. It can often only generate expressions corresponding to the labels but cannot adjust the intensity of the expression.

[0077] Secondly, related technologies do not pay enough attention to the local details of expressions, which may lead to artifacts and blurs during the expression conversion process, which affects the quality and naturalness of the generated images.

[0078] Thirdly, related technologies ignore the retention of facial identity information when converting expression information, which may cause the generated image to have a change in identity characteristics, the so-called "face-changing" phenomenon.

[0079] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a facial expression editing method, an electronic device, and a storage medium, which can generate a higher quality expression editing image.

[0080] Reference Figure 1 The facial expression editing method according to the embodiment of the present application may include but is not limited to:

[0081] Step S101, obtaining a target edit expression type, an expression intensity edit instruction, and an initial expression image of the expression to be edited;

[0082] Step S102, obtaining an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network;

[0083] Step S103, extracting features from the initial expression image through an image feature extraction network to obtain image feature information;

[0084] Step S104, in the expression control network, generating expression editing information according to the target editing expression type and the expression intensity editing instruction;

[0085] Step S105, generating a target expression image after the expression is edited according to the image feature information and the expression editing information through an image generation network.

[0086] According to the facial expression editing method of the embodiment of the present application, it is necessary to first obtain the target editing expression type, expression intensity editing instructions and the initial expression image of the expression to be edited, and further, obtain an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network and an image generation network. Further, feature extraction is performed on the initial expression image through the image feature extraction network to obtain image feature information. In the expression control network, expression editing information is generated according to the target editing expression type and the expression intensity editing instructions, and through the image generation network, a target expression image after the edited expression is generated according to the image feature information and the expression editing information. In this way, a higher quality expression editing image can be generated.

[0087] In step S101 of some embodiments, the target editing expression type, expression intensity editing instructions and the initial expression image of the expression to be edited are obtained. It should be noted that step S101 is intended to collect the input information necessary for facial expression editing. Specifically, the information to be collected includes the type of target editing expression and the editing instructions of expression intensity. The type of target editing expression in the embodiment of the present application refers to the type of expression that the user hopes to eventually generate, such as basic emotional expressions such as happiness, sadness, and surprise. These types define the direction and expected results of the editing to ensure that the editing process has a clear goal. The editing instructions of the expression intensity of the embodiment of the present application involve the intensity of the expression that the user wants to show, which can be a continuous range from mild to significant. By accurately controlling the intensity of the expression, more delicate and situation-specific expression images can be generated to meet different application requirements. The introduction of editing instructions makes expression editing not only a type conversion, but also includes the depth and richness of emotional expression. The initial expression image to be edited in the embodiment of the present application is the object of the actual editing operation, which is the original image that the user wants to modify the expression.

[0088] In step S102 of some embodiments, an expression editing model that has been trained in editing capabilities is obtained; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network. It should be noted that the core link of the facial expression editing method is to obtain an expression editing model that has been specially trained and has expression editing capabilities. The editing capability training of this expression editing model is intended to train the model to accurately edit the input initial expression image to generate an image that meets the expression type and intensity specified by the user. The expression editing model consists of three key components: an image feature extraction network, an expression control network, and an image generation network, each of which plays a different but complementary role.

[0089] First, the image feature extraction network is tasked with extracting key visual features from the initial expression image. These features include not only basic facial structure information, such as the location of the eyes, nose, and mouth, but also subtle changes associated with expression, such as wrinkles at the corners of the eyes or the curvature of the corners of the mouth. By extracting these features, the model is able to capture the details in the image and provide the necessary information for subsequent expression editing.

[0090] Secondly, the expression control network generates corresponding expression editing information based on the target editing expression type and expression intensity editing instructions provided by the user. It should be pointed out that the expression control network can use the learned knowledge and rules to convert instructions into specific parameters that can affect image generation, such as adjusting the upward angle of the corners of the eyes or the curvature of the corners of the mouth to achieve different expression effects.

[0091] Thirdly, the image generation network combines the feature information obtained by the image feature extraction network and the expression editing information generated by the expression control network to generate the final edited target expression image. This network is responsible for converting abstract features and editing instructions into visual image output, ensuring that the generated image meets the user's requirements while maintaining a high degree of authenticity and naturalness.

[0092] The collaborative work of the image feature extraction network, the expression control network, and the image generation network is the key to achieving accurate expression editing. The image feature extraction network provides the necessary visual information, the expression control network performs precise control according to the user's instructions, and the image generation network converts this information and control parameters into specific image output. Through this division of labor and cooperation, the entire expression editing model can flexibly respond to different editing needs and generate high-quality facial expression images.

[0093] Reference Figure 2 According to some embodiments of the present application, step S102 acquires an expression editing model that has been trained in editing capabilities, which may include but is not limited to:

[0094] Step S201, determining an original expression editing model; wherein the original expression editing model includes an original image feature extraction network, an original expression control network and an original image generation network;

[0095] Step S202, obtaining a plurality of expression sample images, an expression type label corresponding to each expression sample image, prior expression information corresponding to each expression sample image, and prior feature information corresponding to each expression sample image;

[0096] Step S203, performing first-stage training on the original expression control network based on the plurality of expression sample images, the expression category label corresponding to each expression sample image, and the prior expression information corresponding to each expression category label;

[0097] Step S204, after the first stage training, based on the plurality of expression sample images and the prior feature information corresponding to each expression sample image, the original image feature extraction network is subjected to the second stage training;

[0098] Step S205, after the second stage training, based on the plurality of expression sample images and the expression type label corresponding to each expression sample image, the original image generation network is subjected to the third stage training;

[0099] Step S206, determining the original expression editing model that has undergone the first stage training, the second stage training, and the third stage training as the expression editing model that has undergone editing capability training.

[0100] In step S201 of some embodiments, the original expression editing model is determined. The original expression editing model includes the original image feature extraction network, the original expression control network and the original image generation network. It should be noted that determining the original expression editing model is the first step in building an expression editing model with advanced expression editing capabilities. The original expression editing model consists of three basic networks: an image feature extraction network, an expression control network and an image generation network. Each basic network plays a unique and important role in the model, working together to achieve complex expression editing tasks.

[0101] In step S202 of some embodiments, a plurality of expression sample images, an expression type label corresponding to each expression sample image, prior expression information corresponding to each expression sample image, and prior feature information corresponding to each expression sample image are obtained. It should be noted that a plurality of expression sample images are obtained, and these expression sample images are the core data in the training process. Each expression sample image is equipped with a corresponding expression type label, and these expression type labels clearly indicate the type of expression in the expression sample image, such as happiness, sadness, or surprise. At the same time, each expression sample image is also accompanied by prior expression information and prior feature information. The prior expression information provides the original expression editing model with an initial understanding of the expression features, and the prior feature information provides the original expression editing model with an initial understanding of the facial features.

[0102] Reference Figure 3 According to some embodiments of the present application, step S202 obtains multiple expression sample images, expression category labels corresponding to each expression sample image, prior expression information corresponding to each expression sample image, and prior feature information corresponding to each expression sample image, which may include but are not limited to:

[0103] Step S301, obtaining a plurality of expression sample images and an expression type label corresponding to each expression sample image;

[0104] Step S302, inputting each expression sample image into a pre-trained expression analysis model for expression analysis processing to obtain prior expression information corresponding to each expression sample image;

[0105] Step S303: input each expression sample image into a pre-trained feature extraction model for feature extraction processing to obtain prior feature information corresponding to each expression sample image.

[0106] According to some embodiments of the present application, the goal of step S202 is to collect and process key data in order to lay a solid foundation for subsequent expression editing model training. This stage includes obtaining multiple expression sample images and various types of information related to these images.

[0107] In step S301 of some embodiments, a plurality of expression sample images are collected, and these expression sample images show different emotions and expression states. Each expression sample image needs to be equipped with a corresponding expression category label, and these expression category labels serve as the basis for the classification of expressions, and clearly indicate the basic emotional categories of the expressions in the expression sample images, such as happiness, sadness, anger, etc.

[0108] In step S302 of some embodiments, each expression sample image is input into a pre-trained expression parsing model. The expression parsing model is responsible for in-depth analysis of the input expression sample images to identify and parse the expression features in each expression sample image, thereby obtaining the prior expression information corresponding to each sample image. This prior expression information is a high-level description of the expression features, which captures the key visual elements of the expression and provides important information for understanding the semantic content of the expression.

[0109] In step S303 of some embodiments, each expression sample image is further input into a pre-trained feature extraction model. This pre-trained feature extraction model is intended to be used to extract detailed facial features from the expression sample images, including but not limited to the position and shape of the eyes, nose, and mouth. Through this process, the prior feature information corresponding to each expression sample image is obtained. These prior feature information are indispensable in the subsequent expression editing process because they provide the expression editing model with rich facial details and help generate highly realistic expression images.

[0110] Through the embodiment of the present application shown in step S301 to step S303, not only the expression sample images and their expression category labels are obtained, but also the prior expression information and prior feature information of each image are obtained. These information together constitute the comprehensive data set required for training the expression editing model, provide the necessary input for the original expression editing model to be trained at various stages, and ensure that the model can accurately learn and simulate different expression features.

[0111] In step S203 of some embodiments, the original expression control network is trained in the first stage based on a plurality of expression sample images, expression category labels corresponding to each expression sample image, and prior expression information corresponding to each expression category label. It should be noted that the original expression control network can be trained in the first stage using expression sample images, expression category labels, and prior expression information. The purpose of this stage is to allow the expression control network to learn how to generate corresponding expression control information according to the expression category labels, so as to control the generation of subsequent expressions.

[0112] Reference Figure 4 According to some embodiments of the present application, step S203 performs a first-stage training on the original expression control network based on a plurality of expression sample images, an expression category label corresponding to each expression sample image, and a priori expression information corresponding to each expression category label, which may include but is not limited to:

[0113] Step S401, inputting the expression type label into the original expression control network, so that the expression control network performs expression control training on the expression type label to generate first expression training data;

[0114] Step S402, comparing the prior expression information corresponding to the expression type label with the first expression training data to obtain first deviation data;

[0115] Step S403, updating the model parameters of the expression control network based on the first deviation data, returning to execute inputting the expression type label into the original expression control network until the expression control network meets the preset first stage termination condition, and ending the first stage training.

[0116] In step S401 of some embodiments, the expression type label is first input into the original expression control network. The purpose of this step is to allow the expression control network to learn and adapt to different expression type labels and generate first expression training data. The first expression training data is the basic response of the expression control network to the expression type label and is the starting point for subsequent training and model optimization.

[0117] In step S402 of some embodiments, the prior expression information corresponding to the expression type label is compared with the first expression training data. Through this comparison, first deviation data can be obtained. The first deviation data reflects the difference between the expression information generated by the current expression control network and the actual expression characteristics, which is key information for evaluating the performance of the expression control network and guiding model adjustment.

[0118] In step S403 of some embodiments, the model parameters of the expression control network are updated based on the first deviation data. This parameter update process is to reduce the deviation and improve the accuracy of the expression control network in generating expressions. The updated expression control network is trained with the expression type label again, generates new first expression training data, and compares it with the prior expression information, and repeats this cycle until the performance of the expression control network meets the preset first stage termination condition. The end of this stage indicates that the expression control network can generate the corresponding expression control information according to the expression type label more accurately.

[0119] In the embodiment of the present application shown by step S404 to step S403, the first stage training not only improves the responsiveness of the expression control network to the expression category label, but also ensures the accuracy and reliability of the expression control information generated by the expression control network through continuous parameter updating and optimization. This lays the foundation for the subsequent first stage training and third stage training, so that the entire expression editing model can complete the expression editing task more efficiently and accurately after training.

[0120] Reference Figure 5 According to some embodiments of the present application, step S401 inputs the expression type label into the original expression control network so that the expression control network performs expression control training on the expression type label to generate first expression training data, including:

[0121] Step S501, inputting the expression type label into the original expression control network, and introducing the intensity control training signal into the expression control network;

[0122] Step S502: In the expression control network, expression control training is performed on the expression type label based on the intensity control training signal to generate first expression training data.

[0123] In step S501 of some embodiments, expression type labels are input into the original expression control network. On this basis, in order to increase the depth and precision of training, the embodiments of the present application can also introduce intensity control training signals into the expression control network. It should be understood that the role of the intensity control training signal is to guide the expression control network not only to recognize the types of expressions during the training process, but also to learn to control the intensity of expressions. In this way, the expression control network can gradually understand and learn the subtle differences in expressions during continuous training, so that it can be used to generate more accurate and natural expressions.

[0124] In step S502 of some embodiments, the expression control network uses the received expression type label and intensity control training signal to perform expression control training. In this training process, the expression control network of the embodiment of the present application converts the expression type label into a specific expression feature based on the intensity control training signal, that is, generates the first expression training data. It should be understood that the first expression training data is the product of the expression control network in the expression control training process, and they reflect the network's responsiveness to different expression types and intensity requirements.

[0125] The embodiment of the present application shown by step S501 to step S502 can not only enable the expression control network to recognize and generate basic expression types, but also enable the expression control network to master the control of expression intensity, providing a basis for generating expressions with subtle differences. It should be pointed out that this training method improves the expressiveness and adaptability of the expression control network.

[0126] In step S204 of some embodiments, after the first stage of training, the original image feature extraction network is trained in the second stage based on multiple expression sample images and the prior feature information corresponding to each expression sample image. It should be noted that after the first stage of training is completed, the image feature extraction network enters the second stage of training, which focuses on the image feature extraction network. The image feature extraction network is trained using the expression sample images and the corresponding prior feature information. This enables it to extract facial features more accurately and provide the necessary basic data for expression generation.

[0127] Reference Figure 6 According to some embodiments of the present application, step S204 performs a second-stage training on the original image feature extraction network based on a plurality of expression sample images and prior feature information corresponding to each expression sample image, which may include but is not limited to:

[0128] Step S601, inputting the expression sample image into the original image feature extraction network, so that the image feature extraction network performs image feature extraction training on the expression sample image to obtain first image feature training data;

[0129] Step S602, comparing the prior feature information corresponding to the expression sample image with the first image feature training data to obtain second deviation data;

[0130] Step S603, based on the second deviation data, the model parameters of the image feature extraction network are updated, and the expression sample image is input into the image feature extraction network again until the image feature extraction network meets the preset second stage termination condition, thereby terminating the second stage training.

[0131] In step S601 of some embodiments, the expression sample image is input into the original image feature extraction network. The purpose of this process is to let the image feature extraction network learn how to extract useful image features from the expression sample image, including but not limited to the position and shape of the eyes, nose, and mouth. Through this training, the image feature extraction network generates first image feature training data, which are the preliminary results of the image feature extraction network extracting features from the sample image.

[0132] In step S602 of some embodiments, the prior feature information corresponding to the expression sample image is carefully compared with the first image feature training data generated by the image feature extraction network. This comparison process reveals the difference between the first image feature training data extracted by the image feature extraction network and the prior feature information, and obtains the second deviation data. Among them, the second deviation data is an important basis for evaluating the performance of the image feature extraction network and guiding the further optimization of the expression editing model.

[0133] In step S603 of some embodiments, the model parameters of the image feature extraction network are updated based on the second deviation data. This parameter update process is intended to reduce the second deviation data between the first image feature training data and the prior feature information, thereby improving the accuracy of feature extraction by the image feature extraction network. The updated image feature extraction network performs feature extraction training on the expression sample image again, generates new first image feature training data, and compares it with the prior feature information. This process is repeated until the performance of the image feature extraction network meets the preset second-stage termination conditions and meets the expected requirements. At this point, the second-stage training is completed, and the image feature extraction network can more accurately extract facial features in the expression sample image.

[0134] In the embodiment of the present application shown in step S601 to step S603, the second stage training strengthens the ability of the image feature extraction network, enabling it to provide more accurate and comprehensive feature information for subsequent expression editing tasks. This not only improves the quality and naturalness of expression editing, but also is a key link in the training process of the entire expression editing model.

[0135] In step S205 of some embodiments, after the second stage of training, the original image generation network is trained in the third stage based on a plurality of expression sample images and expression type labels corresponding to each expression sample image. It should be noted that, based on the second stage of training, the expression editing model is then trained in the third stage. Among them, the third stage of training is aimed at the image generation network, which aims to use the expression sample images and expression type labels to train the image generation network to generate new expression images that meet the label requirements, ensuring that the generated images meet the expression requirements and have high quality.

[0136] Reference Figure 7 According to some embodiments of the present application, step S205 performs a third-stage training on the original image generation network based on a plurality of expression sample images and an expression type label corresponding to each expression sample image, which may include but is not limited to:

[0137] Step S701, inputting the expression type label into the expression control network trained in the first stage, so that the expression control network performs expression control training on the expression type label, and obtains second expression training data;

[0138] Step S702, inputting the expression sample image into the image feature extraction network after the second stage training, so that the image feature extraction network performs image feature extraction training on the expression sample image to obtain second image feature training data;

[0139] Step S703, inputting the second expression training data and the second image feature training data into the original image generation network, so that the image generation network generates expression editing training data according to the second expression training data and the second image feature training data;

[0140] Step S704, performing a truth value comparison between the expression sample image and the corresponding expression editing training data to obtain third deviation data;

[0141] Step S705, based on the third deviation data, the model parameters of the image generation network are updated, and the expression type label is input into the expression control network after the first stage training, until the image generation network meets the preset third stage termination condition, and the third stage training is ended.

[0142] In step S701 of some embodiments, the expression type label is input into the expression control network that has been trained in the first stage. This step enables the expression control network to continue to conduct in-depth training on the expression type label and generate second expression training data. After the expression control network has undergone the first stage of training, these second expression training data are the expression control network's further understanding and response to the expression type label, and the second expression training data provides a basis for generating specific expression features.

[0143] In step S702 of some embodiments, the expression sample image is input into the image feature extraction network that has been trained in the second stage. The purpose of this step is to allow the image feature extraction network to perform feature extraction training on the expression sample image to generate second image feature training data. These data contain the key features of the face and provide the necessary visual information for generating a specific expression image.

[0144] In step S703 of some embodiments, the second expression training data and the second image feature training data are input into the original image generation network together. The goal of this step is to allow the image generation network to generate expression editing training data based on these training data. Among them, the second expression training data and the second image feature training data are direct inputs for the image generation network to generate new expression images, reflecting the comprehensive understanding of the expression features and facial features by the image feature extraction network.

[0145] In step S704 of some embodiments, the expression sample image is compared with the corresponding expression editing training data to obtain third deviation data. This step obtains the third deviation data by comparing the difference between the generated expression editing training data and the actual expression sample image. It should be understood that the third deviation data is used to evaluate the accuracy of the network generated image and provide a basis for further optimization.

[0146] In step S705 of some embodiments, the model parameters of the image generation network are updated based on the third deviation data. This parameter update process is intended to reduce deviations and improve the quality and authenticity of network-generated images. The updated network-generated images are trained again to generate new expression editing training data, which are compared with the corresponding expression sample images. This process is repeated until the performance of the image generation network meets the preset third-stage termination conditions and then meets the expected requirements. At this point, the third-stage training is completed, and the image generation network can generate high-quality expression images that meet user expectations.

[0147] Through steps S701 to S705 of the embodiment of the present application, not only the generation capability of the image generation network is improved, but also the visual naturalness and authenticity of the generated expression images are ensured, which contributes a key link to the training process of the entire expression editing model.

[0148] In step S206 of some embodiments, the original expression editing model that has undergone the first stage training, the second stage training, and the third stage training is determined as an expression editing model that has undergone editing capability training. It should be noted that after the continuous training of the above three stages, the original expression editing model is optimized and adjusted to form an expression editing model that has undergone editing capability training. It should be understood that the expression editing model that has undergone editing capability training can accurately edit facial expressions according to the type and intensity of expression specified by the user, and generate image output that meets expectations. The entire training process ensures the high performance and efficiency of the model in expression editing tasks, so that it can meet the needs of different application scenarios.

[0149] In step S103 of some embodiments, feature extraction is performed on the initial expression image through an image feature extraction network to obtain image feature information. It should be noted that the main task of the image feature extraction network is to capture the key visual information in the image and convert this information into feature representations that can be used for further processing. Through the image feature extraction network, the basic structural features of the face in the initial expression image can be extracted, such as the position and shape of the eyes, nose and mouth, which are the core elements that constitute the basic recognition information of the face. At the same time, the image feature extraction network can also capture subtle details related to the expression, such as the degree of opening and closing of the eyes, wrinkles at the corners of the eyes, and the ups and downs of the corners of the mouth. These detailed features are crucial for understanding and reproducing facial expressions because they directly affect the communication and perception of expressions. It should be clear that the image feature information includes a series of descriptive features, which include not only static facial features, but also dynamic expression changes. These features provide a rich source of information for subsequent expression control and image generation. For example, by analyzing the features of the eyes and mouth, it can be inferred whether a person is smiling or frowning. This in-depth feature extraction capability enables the editing model to understand and reproduce facial expressions more accurately, thereby maintaining a high degree of authenticity and naturalness when generating new expression images.

[0150] In addition, the design and training of the feature extraction network in some embodiments also take into account the differences and complexity between different expressions, ensuring that a wide range of facial expression changes can be covered when extracting features. This comprehensive feature extraction not only provides a solid foundation for subsequent facial expression editing, but also provides possibility for realizing high-quality facial expression editing. Therefore, the feature extraction in step S103 is an indispensable link in the whole facial expression editing method, which ensures the accuracy and effectiveness of the editing process.

[0151] In step S104 of some embodiments, in the expression control network, expression editing information is generated according to the target editing expression type and expression intensity editing instructions. It should be noted that the expression control network allows the user to adjust the generated expression type and intensity according to specific editing needs. Based on this, the deficiencies in the related technology in controlling the intensity of expression can be solved. The embodiments of the present application are not limited to generating expressions corresponding to labels, but can also adjust the expression type and expression intensity. It should be pointed out that the expression control network receives the target editing expression type, where the target editing expression type refers to a basic emotion category, such as happiness, sadness or surprise. The expression control network needs to be able to recognize and understand these emotion categories and convert them into corresponding visual features.

[0152] The editing instructions for the intensity of expression in the embodiment of the present application are also an important factor that the expression control network needs to consider. In some cases, the expression that the user wants to generate may be slight, moderate, or strong, so the expression control network needs to be able to adjust the prominence of the expression features according to the expression intensity editing instructions. For example, a slight sad expression may only require a slight drooping of the eyebrows and a downward bend of the corners of the mouth, while a strong sad expression may require more obvious changes in facial features.

[0153] In this way, the expression control network can generate a series of expression editing information, which is then passed to the image generation network. This editing information includes not only basic expression features, but also more detailed expression adjustments, such as the degree of eye opening and closing, the bending angle of the mouth corners, etc. This fine control enables the generated image to more accurately reflect the user's editing intentions in terms of both the type and intensity of expression.

[0154] It should be understood that the expression editing information output by the expression control network is a set of detailed instructions that guide the image generation network on how to adjust the features of the initial expression image to generate the final expression image. This design not only improves the flexibility of the editing process, but also enhances the naturalness and authenticity of the generated image, ensuring that users can obtain the expression editing results that meet their expectations.

[0155] In step S105 of some embodiments, a target expression image after editing the expression is generated through an image generation network according to the image feature information and the expression editing information. It should be noted that the image generation network needs to receive image feature information from the image feature extraction network, and these image feature information contain the basic structure of the face in the initial expression image and the details related to the expression. In addition, the image generation network also receives the expression editing information from the expression control network, and the expression editing information guides the image generation network on how to adjust the features of the expression according to the user's instructions. The image generation network fuses the two parts of information, the image feature information and the expression editing information, to generate a target expression image that is both natural and in line with expectations.

[0156] To achieve this goal, the image generation network needs to be highly flexible and creative. It should not only accurately reproduce the basic structure of the face, but also adjust facial features such as the shape of the eyes and the curvature of the mouth corners according to the expression editing information. This adjustment needs to be very subtle to ensure that the generated image is both visually natural and has the desired expression effect.

[0157] In addition, the image generation network also takes into account the natural features of the original image, such as skin texture, lighting effects, etc., to ensure that the generated target expression image is visually consistent with the original image. This involves not only the adjustment of facial features, but also the coordination and realism of the overall image. In this way, the generated image can not only convey the expression expected by the user, but also maintain the natural beauty and realism of the original image.

[0158] The target expression image is an edited facial expression image, which not only meets the editing needs of the user, but also maintains the natural features of the original image. Based on this, the facial expression editing method of the embodiment of the present application provides users with an efficient, flexible and high-quality expression editing solution.

[0159] Reference Figure 8 According to the embodiment of the present application, the facial expression editing device 800 may include but is not limited to:

[0160] The data acquisition unit 801 is used to acquire the target editing expression type, the expression intensity editing instruction and the initial expression image of the expression to be edited;

[0161] The model acquisition unit 802 is used to acquire an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network;

[0162] An image feature extraction network 803 is used to extract features from the initial expression image to obtain image feature information;

[0163] The expression control network 804 is used to generate expression editing information according to the target editing expression type and the expression intensity editing instruction;

[0164] The image generation network 805 is used to generate a target expression image after the expression is edited according to the image feature information and the expression editing information.

[0165] It can be seen that the contents in the above-mentioned facial expression editing method embodiment are all applicable to the embodiment of my facial expression editing device. The functions specifically implemented by the embodiment of my facial expression editing device are the same as those in the above-mentioned facial expression editing method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned facial expression editing method embodiment.

[0166] Reference Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0167] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0168] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the facial expression editing method of the embodiment of this application;

[0169] Input / output interface 903, used to implement information input and output;

[0170] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0171] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0172] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0173] The embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the above-mentioned facial expression editing method.

[0174] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0175] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, which may include but is not limited to any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0176] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0177] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, which may include but is not limited to a number of instructions for a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0181] It should also be understood that the various implementation methods provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0182] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A facial expression editing method, characterized in that: include: Obtaining the target editing expression type, expression intensity editing instructions, and the initial expression image of the expression to be edited; Acquire an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network; Extracting features from the initial expression image through the image feature extraction network to obtain image feature information; In the expression control network, expression editing information is generated according to the target editing expression type and the expression intensity editing instruction; The target expression image after the expression is edited is generated through the image generation network according to the image feature information and the expression editing information.

2. The method according to claim 1, characterized in that The step of obtaining the expression editing model trained in editing capability includes: Determine an original expression editing model; wherein the original expression editing model includes the original image feature extraction network, the original expression control network and the original image generation network; Acquire multiple expression sample images, expression category labels corresponding to each of the expression sample images, prior expression information corresponding to each of the expression sample images, and prior feature information corresponding to each of the expression sample images; Based on the plurality of expression sample images, the expression type label corresponding to each expression sample image, and the prior expression information corresponding to each expression type label, the original expression control network is trained in the first stage; After the first stage of training, based on the plurality of expression sample images and the prior feature information corresponding to each of the expression sample images, the original image feature extraction network is trained in the second stage; After the second stage of training, based on the plurality of expression sample images and the expression type label corresponding to each of the expression sample images, the original image generation network is subjected to a third stage of training; The original expression editing model that has undergone the first stage training, the second stage training, and the third stage training is determined as the expression editing model that has undergone editing capability training.

3. The method according to claim 2, characterized in that The first stage training of the original expression control network based on the plurality of expression sample images, the expression type label corresponding to each expression sample image, and the prior expression information corresponding to each expression type label includes: Inputting the expression type label into the original expression control network, so that the expression control network performs expression control training on the expression type label to generate first expression training data; Comparing the prior expression information corresponding to the expression type label with the first expression training data to obtain first deviation data; The model parameters of the expression control network are updated based on the first deviation data, and the expression type label is input into the original expression control network again until the expression control network meets the preset first stage termination condition, thereby terminating the first stage training.

4. The method according to claim 3, characterized in that The step of inputting the expression type label into the original expression control network so that the expression control network performs expression control training on the expression type label to generate first expression training data comprises: Inputting the expression type label into the original expression control network, and introducing an intensity control training signal into the expression control network; In the expression control network, expression control training is performed on the expression type label based on the intensity control training signal to generate the first expression training data.

5. The method according to claim 2, characterized in that: The second stage training of the original image feature extraction network based on the plurality of expression sample images and the prior feature information corresponding to each expression sample image includes: Inputting the expression sample image into the original image feature extraction network so that the image feature extraction network performs image feature extraction training on the expression sample image to obtain first image feature training data; Comparing the prior feature information corresponding to the expression sample image with the first image feature training data to obtain second deviation data; The model parameters of the image feature extraction network are updated based on the second deviation data, and the execution returns to inputting the expression sample image into the image feature extraction network until the image feature extraction network meets the preset second-stage termination condition, thereby terminating the second-stage training.

6. The method according to claim 2, characterized in that The third stage training of the original image generation network based on the plurality of expression sample images and the expression type label corresponding to each expression sample image includes: Inputting the expression type label into the expression control network trained in the first stage, so that the expression control network performs expression control training on the expression type label to obtain second expression training data; Inputting the expression sample image into the image feature extraction network after the second stage training, so that the image feature extraction network performs image feature extraction training on the expression sample image to obtain second image feature training data; Inputting the second expression training data and the second image feature training data into the original image generation network, so that the image generation network generates expression editing training data according to the second expression training data and the second image feature training data; Performing a true value comparison between the expression sample image and the corresponding expression editing training data to obtain third deviation data; The model parameters of the image generation network are updated based on the third deviation data, and the expression type label is input into the expression control network after the first stage training, and the image generation network meets the preset third stage termination condition, thereby terminating the third stage training.

7. The method according to claim 2, characterized in that The step of acquiring a plurality of expression sample images, an expression type label corresponding to each of the expression sample images, prior expression information corresponding to each of the expression sample images, and prior feature information corresponding to each of the expression sample images includes: Acquire a plurality of expression sample images and an expression type label corresponding to each of the expression sample images; Inputting each of the expression sample images into the pre-trained expression parsing model for expression parsing processing to obtain the prior expression information corresponding to each of the expression sample images; Each of the expression sample images is input into the pre-trained feature extraction model for feature extraction processing to obtain the prior feature information corresponding to each of the expression sample images.

8. A facial expression editing device, characterized in that: include: A data acquisition unit, used to acquire a target edited expression type, an expression intensity edit instruction, and an initial expression image of the expression to be edited; A model acquisition unit, used to acquire an expression editing model that has been trained in editing capabilities; wherein the expression editing model includes an image feature extraction network, an expression control network, and an image generation network; The image feature extraction network is used to extract features from the initial expression image to obtain image feature information; The expression control network is used to generate expression editing information according to the target editing expression type and the expression intensity editing instruction; The image generation network is used to generate a target expression image after the expression is edited according to the image feature information and the expression editing information.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor implements the facial expression editing method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the facial expression editing method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Expression editing method based on facial component, electronic equipment and readable storage medium

    CN119152082A