Facial expression transfer method and expression transfer device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a face expression migration method and an expression migration device, an electronic device and a storage medium, and belong to the technical field of artificial intelligence. The method comprises: obtaining an original face image of a target object and a target expression category; determining a first training set according to the target object, the first training set comprising a plurality of first face sample images; determining a second training set according to the target expression category, the second training set comprising a plurality of second face sample images; inputting the second face sample images into a preset initial recurrent probability model for pre-training to obtain a pre-training model; inputting the first face sample images and the second face sample images into the pre-training model to obtain a first face processing image, and then obtaining a face expression migration model; and inputting the original face image into the face expression migration model to convert the original face image into a target face image matching the target expression category. The embodiments of the present application can efficiently and accurately realize migration of a face expression in an image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a facial expression transfer method and device, an electronic device, and a storage medium. Background Art

[0002] Facial expression transfer involves modifying a person's facial features without changing the original facial features. While existing facial expression transfer methods can address this problem, they all suffer from drawbacks. For example, methods based on deep learning generative adversarial networks (GANs) suffer from significant domain differences between images with different facial expressions, making the adversarial structure less stable and leading to lower accuracy in expression transfer. Therefore, efficiently and accurately completing facial expression transfer has become a pressing technical challenge. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a facial expression migration method and expression migration device, electronic device and storage medium, which can efficiently and accurately realize the migration of facial expressions in images.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for transferring facial expressions, the method comprising:

[0005] Obtain the original face image and target expression category of the target object;

[0006] Determining a first training set according to the target object, wherein the first training set includes a plurality of first face sample images, and the original face images and the first face sample images are images of the same target object;

[0007] Determining a second training set according to the target expression category, wherein the second training set includes a plurality of second face sample images, and the expression category corresponding to the second face sample images is the target expression category;

[0008] Inputting each of the second face sample images into a preset initial cyclic probability model for pre-training to obtain a pre-trained model;

[0009] Inputting the first face sample image and the second face sample image into the pre-trained model to obtain a first face processed image;

[0010] Calculating a model loss value according to the first face processing image, and adjusting model parameters of the pre-trained model according to the model loss value to obtain a facial expression transfer model;

[0011] The original facial image is input into the facial expression transfer model, so as to convert the original facial image into a target facial image matching the target expression category through the facial expression transfer model.

[0012] In some embodiments, the pre-trained model includes a first diffusion denoising sub-model and a second diffusion denoising sub-model; inputting the first face sample image and the second face sample image into the pre-trained model to obtain a first face processed image includes:

[0013] Inputting the first face sample image into the first diffusion denoising sub-model, performing forward image processing on the first face sample image using the first diffusion denoising sub-model to obtain a second face processed image;

[0014] performing inverse image processing on the second face-processed image using the second diffusion denoising sub-model to obtain a third face-processed image;

[0015] Performing facial feature extraction on the third processed face image to obtain a facial feature image;

[0016] inputting the second face sample image and the face feature image into the first diffusion denoising sub-model, and performing forward image processing on the second face sample image using the first diffusion denoising sub-model to obtain a fourth face processed image;

[0017] Perform inverse image processing on the fourth face-processed image using the second diffusion denoising sub-model to obtain a first face-processed image.

[0018] In some embodiments, calculating a model loss value based on the first face processing image, and adjusting model parameters of the pre-trained model based on the model loss value to obtain a facial expression transfer model, includes:

[0019] determining a circulation loss value according to the first face sample image, the second face sample image, the third face processed image, and the first face processed image;

[0020] Determining a spatial similarity loss value based on the first face sample image and the second face sample image;

[0021] determining an expression structure loss value according to the second processed face image and the fourth processed face image;

[0022] Performing weighted calculation on the cycle loss value, the spatial similarity loss value, and the expression structure loss value to obtain a model loss value;

[0023] The model parameters of the pre-trained model are adjusted according to the model loss value, and the adjusted model is continued to be trained based on the first training set and the second training set until the model loss value meets the preset training end condition to obtain a facial expression transfer model.

[0024] In some embodiments, determining the cycle loss value based on the first face sample image, the second face sample image, the third face processed image, and the first face processed image includes:

[0025] performing distance loss calculation on the first face sample image and the third processed face image to obtain a first distance loss value;

[0026] performing distance loss calculation on the second face sample image and the first processed face image to obtain a second distance loss value;

[0027] A weighted calculation is performed on the first distance loss value and the second distance loss value to obtain the cycle loss value.

[0028] In some embodiments, determining the expression structure loss value based on the second processed face image and the fourth processed face image includes:

[0029] Get the preset rectangular mask area;

[0030] performing mask processing on the first face sample image according to the rectangular mask area to obtain a first mask image;

[0031] performing mask processing on the second face sample image according to the rectangular mask area to obtain a second mask image;

[0032] performing mask processing on the second processed face image according to the rectangular mask area to obtain a third mask image;

[0033] performing mask processing on the fourth face-processed image according to the rectangular mask area to obtain a fourth mask image;

[0034] An expression structure loss value is determined according to the first mask image, the second mask image, the third mask image, and the fourth mask image.

[0035] In some embodiments, determining the expression structure loss value based on the first mask image, the second mask image, the third mask image, and the fourth mask image includes:

[0036] performing structural similarity calculation on the first mask image and the fourth mask image to obtain a first structural similarity loss value;

[0037] performing structural similarity calculation on the second mask image and the third mask image to obtain a second structural similarity loss value;

[0038] A weighted calculation is performed on the first structural similarity loss value and the second structural similarity loss value to obtain the expression structure loss value.

[0039] In some embodiments, inputting each second face sample image into a preset initial cyclic probability model for pre-training to obtain a pre-trained model includes:

[0040] Extracting facial features from the original facial image to obtain a target feature image;

[0041] Inputting each second face sample image into a preset initial cyclic probability model, and processing the second face sample image using the initial cyclic probability model to obtain a sample feature image;

[0042] Determining an image loss value according to the sample feature image and the target feature image;

[0043] The model parameters of the initial cyclic probability model are adjusted according to the image loss value, and the adjusted model is continuously trained based on the second training set until the image loss value meets a preset training end condition to obtain a pre-trained model.

[0044] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides a facial expression transfer device, the device comprising:

[0045] Image and category acquisition module, used to obtain the original face image and target expression category of the target object;

[0046] A first training set determination module is configured to determine a first training set based on the target object, wherein the first training set includes a plurality of first face sample images, and the original face images and the first face sample images are images of the same target object;

[0047] A second training set determination module is configured to determine a second training set according to the target expression category, wherein the second training set includes a plurality of second face sample images, and the expression category corresponding to the second face sample images is the target expression category;

[0048] A first model training module, configured to input each of the first face sample images into a preset initial cyclic probability model for pre-training to obtain a pre-trained model;

[0049] A model input module, configured to input the first face sample image and the second face sample image into the pre-trained model to obtain a first face processed image;

[0050] a second model training module, configured to calculate a model loss value based on the first processed face image, and adjust model parameters of the pre-trained model based on the model loss value to obtain a facial expression transfer model;

[0051] The expression transfer module is used to input the original facial image into the facial expression transfer model to convert the original facial image into a target facial image matching the target expression category through the facial expression transfer model.

[0052] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0053] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method described in the first aspect above.

[0054] The facial expression transfer method, expression transfer device, electronic device, and storage medium proposed in the embodiments of the present application obtain an original facial image of a target object and a target expression category, and then determine a first training set based on the target object, wherein the first training set includes multiple first facial sample images, and the original facial image and the first facial sample images belong to the same target object. Then, a second training set is determined based on the target expression category, wherein the second training set includes multiple second facial sample images, and the expression category corresponding to the second facial sample images is the target expression category. Each second facial sample image is input into a preset initial cyclic probability model for pre-training to obtain a pre-trained model. Thereafter, the first facial sample image and the second facial sample image are input into the pre-trained model to obtain a first facial processed image, and a model loss value is calculated based on the first facial processed image. The model parameters of the pre-trained model are adjusted based on the model loss value to obtain a facial expression transfer model. Finally, the original facial image is input into the facial expression transfer model, and the facial expression transfer model converts the original facial image into a target facial image matching the target expression category. The embodiments of the present application can efficiently and accurately transfer facial expressions in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of the facial expression transfer method provided in an embodiment of the present application;

[0056] Figure 2 yes Figure 1 Flowchart of step S140 in FIG.

[0057] Figure 3 yes Figure 1 Flowchart of step S150 in FIG.

[0058] Figure 4 yes Figure 1 Flowchart of step S160 in FIG.

[0059] Figure 5 yes Figure 4 Flowchart of step S410 in FIG.

[0060] Figure 6 yes Figure 4 Flowchart of step S430 in FIG.

[0061] Figure 7 yes Figure 6 Flowchart of step S660 in FIG.

[0062] Figure 8 Schematic diagram of the structure of the facial expression transfer device provided in an embodiment of the present application;

[0063] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0067] First, let’s analyze some of the terms used in this application:

[0068] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0069] Denoising Diffusion Probabilistic Model (DDPM): mainly calculates the mean of the Gaussian distribution through the results of the noise estimation model.

[0070] Normalization processing: refers to scaling data features so that they fall into a small specific range, removing the unit restrictions of data features and converting them into dimensionless pure values, which facilitates comparison and weighting using indicators of different units or magnitudes.

[0071] L1 loss function: It is used to minimize the error, which is the sum of all absolute differences between the true value and the predicted value.

[0072] Structural Similarity (SSIM): is a metric that measures the similarity between two images.

[0073] Stochastic Gradient Descent Algorithm (SGD): It is mostly used for learning linear classifiers under convex loss functions such as support vector machines and logistic regression.

[0074] With the continuous development of computer vision technology, facial expression transfer technology has attracted the attention and research of numerous scholars. Facial expression transfer refers to modifying the facial features of a person, such as whether they are smiling, sad, or with their eyes closed, without changing the original facial features. This transfer technology can effectively improve the reuse rate of existing facial expression animations and the efficiency of synthesizing new animations, reducing the tedious work of modeling new expressions for animation practitioners. It can provide a new synthesis method and approach for highly realistic human expressions, and has high practical significance.

[0075] At present, many research methods have emerged in the field of facial expression transfer. For example, faces can be reconstructed through multi-view geometry. This method requires collecting facial images from different angles to provide sufficient facial feature information for modeling, thereby further realizing style transfer; facial expression features can also be reconstructed through a stereo color (Red-Green-Blue Depth map, RGBD) camera or a red-green-blue (RGB) camera combined with a three-dimensional morphable face model (3D Morphable FaceModel, 3DMM) method; face models can also be modeled through deep learning methods, and the model can be used to solve facial parameters to achieve expression transfer.

[0076] However, although these methods can solve the problem of facial expression transfer, they all have some drawbacks. For example, methods based on multi-view geometry to reconstruct faces require relatively high equipment accuracy, making facial expression transfer relatively expensive. Methods based on RGBD cameras or RGB cameras have significant limitations in model structure and are difficult to generate model details. For example, wrinkles on the original face disappear after expression transfer. Facial expression transfer methods based on deep learning generative adversarial networks can grasp facial details, but the training stability of generative adversarial networks is very challenging. At the same time, due to the significant domain differences between different expression images, the adversarial structure has low stability and it is difficult to efficiently learn the potential relationship between the original expression and the new expression, resulting in low accuracy of expression transfer. Therefore, based on the shortcomings of current methods and the actual needs of the task, how to efficiently and accurately complete the task of facial expression transfer has become a technical problem that needs to be solved urgently.

[0077] Based on this, the embodiments of the present application provide a facial expression migration method and expression migration device, electronic device and storage medium, which can fully mine the feature information of the original expression and target expression of the task to accurately realize the migration of facial expressions in the image.

[0078] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0079] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0080] The facial expression migration method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The facial expression migration method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application that implements the facial expression migration method, etc., but is not limited to the above forms.

[0081] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0082] The facial expression transfer method and expression transfer device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, a facial expression transfer method in the embodiments of the present application is described.

[0083] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0084] See also Figure 1 , Figure 1 This is an optional flowchart of the facial expression transfer method provided in the embodiment of the present application. Figure 1 The method may specifically include but is not limited to steps S110 to S170. Figure 1 These seven steps are introduced in detail.

[0085] Step S110, obtaining the original face image and target expression category of the target object;

[0086] Step S120, determining a first training set according to the target object, wherein the first training set includes a plurality of first face sample images, and the original face images and the first face sample images are images of the same target object;

[0087] Step S130, determining a second training set according to the target expression category, wherein the second training set includes a plurality of second face sample images, and the expression category corresponding to the second face sample images is the target expression category;

[0088] Step S140: input each second face sample image into a preset initial cyclic probability model for pre-training to obtain a pre-trained model;

[0089] Step S150, inputting the first face sample image and the second face sample image into a pre-trained model to obtain a first face processed image;

[0090] Step S160, calculating a model loss value based on the first face processed image, and adjusting model parameters of the pre-trained model based on the model loss value to obtain a facial expression transfer model;

[0091] Step S170: inputting the original facial image into the facial expression transfer model, so as to convert the original facial image into a target facial image matching the target expression category through the facial expression transfer model.

[0092] It should be noted that in some embodiments, in steps S110 to S170, in order to fully explore the feature information of the original expression and the target expression of the human object, and further explore the potential relationship between the two in different spaces, so as to complete the facial expression transfer task while stabilizing the training. First, the original facial image and the target expression category of the target object are obtained, and a first training set is determined based on the target object, and a second training set is determined based on the target expression category, wherein the first training set includes a plurality of first facial sample images, the original facial image and the first facial sample image belong to the same target object, and the second training set includes a plurality of second facial sample images, and the expression category corresponding to the second facial sample image is the target expression category. Then, each first facial sample image is input into a preset initial cyclic probability model for pre-training to obtain a pre-trained model. Thereafter, the first facial sample image and the second facial sample image are input into the pre-training model to obtain a first facial processed image, and a model loss value is calculated based on the first facial processed image, and the model parameters of the pre-training model are adjusted based on the model loss value to obtain a facial expression transfer model. Finally, the original facial image is input into the facial expression transfer model to convert the original facial image into a target facial image matching the target expression category. The embodiment of the present application can efficiently and accurately transfer facial expressions in an image.

[0093] In step S110 of some embodiments, an original facial image to be subjected to facial expression transfer and a target expression category are obtained. The original facial image is an image containing a target object, and the target expression category represents the expression category to which the target object belongs after the transfer. Expression categories may include happy, sad, angry, and the like. The original expression category corresponding to the original facial image may be any expression category.

[0094] It should be noted that, in actual applications, the facial expression migration method provided in the embodiment of the present application can be applied in an expression migration system. In an online environment, the expression migration system can include a terminal device and a server device, and the facial expression migration method can be deployed on the server side, and the server device and the terminal device can communicate.

[0095] It should be noted that the original facial image captured by the terminal device can be a device with a camera, an original facial image uploaded by the user through the terminal device, or an image of the target object captured from a captured video. The terminal device uploads the captured original person image to the server device.

[0096] In step S120 of some embodiments, in order to ensure that the obtained target facial image can maintain the facial features of the target object, a training set needs to be prepared before training the expression transfer model. In this case, a first training set is determined based on the target object, wherein the first training set D0 includes multiple first facial sample images, and the original facial images and the first facial sample images belong to the same target object. It is understood that the training set used in this application is derived from a real person or a virtual object, that is, the target object can be a real person or a virtual object, wherein the virtual object includes but is not limited to comic characters, game characters, and animation characters.

[0097] It is understood that the first facial sample images in the first training set D0 are multiple different images of the target subject with the same expression category. For example, the first facial sample images may be images of subject A smiling with the face tilted to the left, tilted to the right, or straight on, taken at three different times: morning, noon, and afternoon. It should be noted that the first training set D0 may also be obtained from an existing facial expression database, such as the Real-world Affective Faces Database or the Japanese Female Facial Expression (JAFFE) database.

[0098] In step S130 of some embodiments, in order to convert the expression category of the original facial image into the target expression category to achieve facial expression migration, a second training set D0' is determined according to the target expression category, wherein the second training set D0' includes multiple second facial sample images, and the expression category corresponding to the second facial sample images is the target expression category.

[0099] It is understood that the expression categories of the second facial sample images in the second training set D0' are all target expression categories. That is, the second training set D0' can be a dataset of images of different human subjects making the same expression. The human subjects can be real people or virtual objects. The second training set D0' is obtained in the same manner as the first training set, and will not be further described here.

[0100] It should be noted that in order to overcome the difficulty in obtaining paired matching data images in real life and research, the first training set D0 and the second training set D0' used in training are not limited to paired image data, that is, each first face sample image is not necessarily paired with a second sample image of the same person object. It is only necessary that the images collected in the training set meet the corresponding training set requirements.

[0101] In step S140 of some embodiments, in order to learn the expression features corresponding to the target expression category after facial expression transfer, it is necessary to first learn the expression features of the target expression category, and the expression category corresponding to the second facial sample image is the target expression category. Specifically, each second facial sample image is first input into a preset initial cyclic probability model for pre-training to obtain a pre-trained model, wherein the model parameters of the pre-trained model include expression feature parameters corresponding to the target expression category.

[0102] See also Figure 2 , Figure 2 This is a flowchart of the specific method of step S140 provided in the embodiment of the present application. In some embodiments of the present application, step S140 specifically includes but is not limited to steps S210 to S240. Figure 2 These four steps are introduced in detail.

[0103] Step S210, performing expression feature extraction on the original face image to obtain a target feature image;

[0104] Step S220: input each second face sample image into a preset initial cyclic probability model, and process the second face sample image using the initial cyclic probability model to obtain a sample feature image;

[0105] Step S230, determining an image loss value based on the sample feature image and the target feature image;

[0106] In step S240 , the model parameters of the initial cyclic probability model are adjusted according to the image loss value, and the adjusted model is continuously trained based on the second training set until the image loss value meets the preset training end condition to obtain a pre-trained model.

[0107] Specifically, in order to obtain the common features of the same type of expression category, for example, when the expression category is happy, the corners of the face object's mouth are upward, and when the expression category is sad, the corners of the face object's mouth are downward, etc. First, the expression features of the original face image are extracted to obtain a target feature image. Then, the second face sample image is input into the initial cyclic probability model for processing. The initial cyclic probability model is a cyclic model structure constructed using two DDPMs. The image loss value is determined based on the sample feature image and the target feature image. For example, the image loss value can be solved using a cross-entropy loss function, and the model parameters of the initial cyclic probability model are adjusted according to the image loss value. The adjusted model is continued to be trained based on the second training set D0' until the image loss value meets the preset training end condition to obtain a pre-trained model. The embodiment of the present application can pre-train the model based on the second face sample image in the second training set D0', so that the pre-trained model can first learn the expression features of the target expression category, so that the model parameters of the pre-trained model include the expression feature parameters corresponding to the target expression category, thereby improving the accuracy of expression recognition by facial expression transfer.

[0108] It should be noted that the preset training end condition may be when the image loss value is less than a preset first loss value threshold, or when the recognition accuracy of the initial cyclic probability model is greater than or equal to a preset first accuracy threshold.

[0109] It should be noted that the target feature image can be obtained by first extracting facial key points from the second facial sample image, for example, by using a cascaded deep neural network (DAN) to extract facial key points, or by using a multi-task cascaded convolutional neural network (MTCNN) to extract facial key points, or by using an active shape model (ASM) to extract facial key points. Other methods can also be used, which are not limited here. The target feature image includes the facial image after key point extraction, and the key point locations include but are not limited to eyebrows, eyes, eyeballs, nose, cheeks, and mouth.

[0110] It should be noted that the target feature image can also be composed of line segments, which are connected based on the extracted key points to form line segments. The target feature image includes the facial image after the line segments are generated. In addition, the target feature image can also be reflected as a facial image in the form of points, a facial image in the form of lines, a facial image in the form of curves, etc. This application uses the expression feature image in the form of line segments as an example, but this should not be understood as a limitation of this application.

[0111] See also Figure 3 , Figure 3 This is a flowchart of the specific method of step S150 provided in the embodiment of the present application. In some embodiments of the present application, the pre-trained model includes a first diffusion denoising sub-model and a second diffusion denoising sub-model, and step S150 specifically includes but is not limited to steps S310 to S350. Figure 3 These five steps are introduced in detail.

[0112] Step S310: Input the first face sample image into the first diffusion denoising sub-model, perform forward image processing on the first face sample image through the first diffusion denoising sub-model, and obtain a second face processed image;

[0113] Step S320, performing inverse image processing on the second face-processed image using a second diffusion denoising sub-model to obtain a third face-processed image;

[0114] Step S330, performing facial feature extraction on the third processed face image to obtain a facial feature image;

[0115] Step S340: Input the second face sample image and the face feature image into the first diffusion denoising sub-model, and perform forward image processing on the second face sample image using the first diffusion denoising sub-model to obtain a fourth face processed image;

[0116] Step S350 , performing inverse image processing on the fourth face-processed image using the second diffusion denoising sub-model to obtain the first face-processed image.

[0117] In step S310 of some embodiments, in order to fully mine the facial feature information and expression feature information of the two images and further explore the potential relationship between the corresponding spaces of the two, expression migration of images that are not in the database but have similar features is achieved. The present application applies the idea of ​​the cyclic model to DDPM to obtain a pre-training model, and in order to ensure the data accuracy of the image during the processing, the pre-training model is constructed using two DDPMs, including a first diffusion denoising sub-model and a second diffusion denoising sub-model, to fully mine the feature information between different images. Among them, the first diffusion denoising sub-model and the second diffusion denoising sub-model are both constructed based on the DDPM structure to learn the internal loops and connections between different images. In order to better learn the facial features of the target object, first, the first face sample image is input into the first diffusion denoising sub-model, and the first face sample image is forward processed by the first diffusion denoising sub-model to obtain a second face processed image. Among them, the forward image processing includes a first diffusion processing q and a first denoising processing p. For example, the input first face sample image is recorded as x0, and the first face sample image x0 is subjected to the first diffusion processing q to obtain an image x 0T , and then the image x 0TAfter the first denoising process p, a second face processed image is obtained, and the second face image is recorded as x0'.

[0118] It should be noted that the diffusion process refers to the process of adding noise to the input image. The noise of each addition follows a normal distribution, and each noise addition process has Markov properties, that is, if the distribution of the current step is known, the distribution of future steps is only related to the current state and has nothing to do with the previous ones. The first diffusion process q can be represented by the first face sample image x0 being input through the diffusion process q(x t |x t-1 ) Add noise to the image step by step, t is the step size and t=1,2,K,T, is a hyperparameter, T can be 1000, and is not limited here. After adding T times of noise, we get x T , the x T It obeys a Gaussian distribution with a mean of 0 and a variance of the unit matrix. The specific definition of the diffusion process is shown in formula (1).

[0119]

[0120] Among them, β t is a constant that can take a geometric progression between 0 and 1. N represents normal distribution, I is the unit matrix, and when noise is added for T steps, the noise result is data that obeys the standard Gaussian distribution.

[0121] It should be noted that the denoising process refers to the use of p(x t-1 |x t ) Denoise the image obtained by Gaussian distribution sampling, and after T steps of Markov denoising process, it is expected to obtain a second face processing image corresponding to the first face sample image. The second face processing image can be expressed as pq(x0), that is, there may be errors between the second face processing image and the first face sample image.

[0122] In step S320 of some embodiments, the inverse image processing includes a second diffusion process q' and a second denoising process p', and the second face processing image obtained is subjected to the second diffusion process q' to obtain an image x T0 , and then the image x T0 After the second denoising process p', the third processed face image x0' is obtained. x0' can be expressed as p'q'pq(x0), where p'q'pq(x0) represents the result of the image undergoing two diffusion denoising processes. This application performs model training based on the DDPM model framework, which can avoid the instability of the adversarial model framework and improve the stability of model training.

[0123] In some embodiments, in steps S330 to S350, facial features are extracted from the third face processing image to obtain a facial feature image. After learning the facial features of the target object, the second face sample image and the facial feature image can be spliced ​​to obtain an image to be processed x1, and the image to be processed x1 is input to the first diffusion denoising sub-model for forward image processing. Specifically, the image to be processed x1 is subjected to the first diffusion processing q to obtain an image x1. 1T , and then the image x 1T After the first denoising process p, the fourth face processing image is obtained, and the fourth face processing image pq(x1) is obtained. Then, the fourth face image is subjected to the second diffusion process q' to obtain the image x T1 , and then the image x T1 After the second denoising process p', the required first face processed image x1' is obtained. x1' can also be expressed as p'q'pq(x0). p'q'pq(x0) also represents the result of the image after two diffusion denoising processes.

[0124] It should be noted that the facial feature extraction of the third face processing image can adopt the method of facial key point positioning, for example, using ASM to extract facial key points, or using Active Appearance Models (AAM) to extract facial features to obtain a facial feature image, which is used to accurately locate the facial features of the target object.

[0125] See also Figure 4 , Figure 4 This is a flowchart of the specific method of step S160 provided in the embodiment of the present application. In some embodiments of the present application, step S160 specifically includes but is not limited to steps S410 to S450. Figure 4 These five steps are introduced in detail.

[0126] Step S410, determining a cycle loss value based on the first face sample image, the second face sample image, the third face processed image, and the first face processed image;

[0127] Step S420, determining a spatial similarity loss value based on the first face sample image and the second face sample image;

[0128] Step S430, determining an expression structure loss value based on the second processed face image and the fourth processed face image;

[0129] Step S440, performing weighted calculation on the cycle loss value, the spatial similarity loss value, and the expression structure loss value to obtain a model loss value;

[0130] Step S450, adjust the model parameters of the pre-trained model according to the model loss value, and continue to train the adjusted model based on the first training set and the second training set until the model loss value meets the preset training end condition to obtain a facial expression transfer model.

[0131] It is understandable that in order to deeply explore the feature information of different images in their respective spaces and the potential connections between different spaces, so as to improve the model's ability to transfer facial expressions, the objective function corresponding to the loss value of the model in this application is composed of cycle consistency loss, spatial similarity loss and expression structure. Specifically, based on the first face sample image, the second face sample image, the third face processing image and the first face processing image, the cycle loss value Loss is determined. cycle ; Determine the spatial similarity loss value Loss based on the first face sample image and the second face sample image sim ; Determine the expression structure loss value Loss based on the second face processing image and the fourth face processing image face After that, the cycle loss value Loss cycle , spatial similarity loss value Loss sim And expression structure loss value Loss face Perform weighted calculation to obtain the model loss value Loss sum , according to the model loss value Loss sum The model parameters of the pre-trained model are adjusted, and the adjusted model is continued to be trained based on the first training set and the second training set until the model loss value meets the preset training end condition to obtain a facial expression transfer model.

[0132] To avoid overfitting during model training, stochastic gradient descent can be used to adjust model parameters. At the same time, early stopping can be used to control model progress. Specifically, a portion of the training set's sample data is used as a validation set, and test results are obtained on the validation set. As the number of epochs increases, if the test error on the validation set increases—for example, if the model's validation error increases continuously for more than five iterations—training is stopped, and the weights after the stop are used as the final parameters of the network. The setting condition for the validation error serves as the preset training end condition.

[0133] It should be noted that the preset training end condition can be when the model loss value Loss sum Less than a preset second loss value threshold, or when the recognition accuracy of the pre-trained model is greater than or equal to a preset second accuracy threshold.

[0134] See also Figure 5 , Figure 5This is a flowchart of the specific method of step S410 provided in the embodiment of the present application. In some embodiments of the present application, step S410 specifically includes but is not limited to steps S510 to S530. Figure 5 These three steps are introduced in detail.

[0135] Step S510, performing distance loss calculation on the first face sample image and the first face processed image to obtain a first distance loss value;

[0136] Step S520, performing distance loss calculation on the second face sample image and the third face processed image to obtain a second distance loss value;

[0137] Step S530: Perform weighted calculation on the first distance loss value and the second distance loss value to obtain a cycle loss value.

[0138] Specifically, the cycle loss value includes the loss values ​​of the first face sample image and the second face sample image after two DDPMs respectively, that is, the distance loss is calculated for the first face sample image and the first face processing image to obtain the first distance loss value Loss cycle1 . Calculate the distance loss of the second face sample image and the third face processing image to obtain the second distance loss value Loss cycle2 , and the first distance loss value Loss cycle1 and the second distance loss value Loss cycle2 Perform weighted calculation to obtain the cycle loss value Loss cycle Among them, the weights of the first distance loss value and the second distance loss value can be adjusted according to the situation, as shown in formula (2), which is the cycle loss function corresponding to the cycle loss value. Assuming that the first distance loss value Loss cycle1 and the second distance loss value Loss cycle2 The weights are the same, and when the L1 norm is used to calculate the distance loss, the cycle loss value Loss is obtained cycle .

[0139]

[0140] Here, ||·|| represents the L1 norm.

[0141] In step S430 of some embodiments, the spatial loss function corresponding to the spatial similarity loss value is used to represent the gap between the noise distribution and the standard normal distribution, as shown in formula (3), and the first spatial loss value Loss is determined according to the first face sample image. sim1 , determine the second spatial loss value Loss according to the second face sample image sim2 , determine the spatial similarity loss value Loss sim .

[0142]

[0143] Among them, ε is the result of sampling from the standard normal distribution, ε θ is the model corresponding to the noise processing process, ε θ1 is the parameter selected in the noise model corresponding to the noise addition process of image x0, ε θ2 are the parameters selected in the noise model corresponding to the noise addition process of image x0', Indicates that the loss value at any sampling time is obtained based on the fixed parameters preset by the DDPM model.

[0144] See also Figure 6 , Figure 6 This is a flowchart of the specific method of step S430 provided in the embodiment of the present application. In some embodiments of the present application, step S330 specifically includes but is not limited to steps S610 to S660. Figure 6 These six steps are introduced in detail.

[0145] Step S610, obtaining a preset rectangular mask area;

[0146] Step S620, performing mask processing on the first face sample image according to the rectangular mask area to obtain a first mask image;

[0147] Step S630, performing mask processing on the second face sample image according to the rectangular mask area to obtain a second mask image;

[0148] Step S640, performing mask processing on the second face processing image according to the rectangular mask area to obtain a third mask image;

[0149] Step S650, performing mask processing on the fourth face processing image according to the rectangular mask area to obtain a fourth mask image;

[0150] Step S660 : determining an expression structure loss value according to the first mask image, the second mask image, the third mask image, and the fourth mask image.

[0151] It can be understood that in order to enable the obtained facial expression transfer model to capture a complete image of the target object, the expression structure loss is obtained by setting a constraint on a specific area to limit the facial position of the character object. Specifically, a preset rectangular mask area is obtained, which is represented by mask(). The first face sample image is masked according to the rectangular mask area, that is, the image of the mask area in all the first face sample images of the first training set D0 is extracted, and the average value of each pixel of all the extracted mask images is calculated. The first mask image is obtained based on the obtained multiple average similarities, which can be recorded as mean(mask(D0)). The first mask image is used to represent the representative features of the first face sample image in the rectangular mask area. Then, the second face sample image is masked according to the rectangular mask area to obtain a second mask image, that is, the image of the mask area in all the second face sample images of the second training set D0' is extracted, and the average value of each pixel of all the extracted mask images is calculated. Based on the multiple average similarities obtained, a second mask image is obtained, which can be recorded as mean(mask(D0')). The second mask image is used to represent the representative features of the image in the rectangular mask area. Afterwards, the second face processing image pq(x0) is masked according to the rectangular mask area to obtain a third mask image mask(pq(x0)), and the fourth face processing image p'q'(x0') is masked according to the rectangular mask area to obtain a fourth mask image mask(p'q'(x0')). Finally, the expression structure loss value Loss is determined based on the first mask image, the second mask image, the third mask image and the fourth mask image. face .

[0152] It should be noted that the rectangular mask area can be defined in the image using a rectangular positioning rule so that the face of the human expression image is covered in the rectangular mask area, thereby improving the model's processing of details of the face image.

[0153] See also Figure 7 , Figure 7 This is a flowchart of the specific method of step S660 provided in the embodiment of the present application. In some embodiments of the present application, step S660 specifically includes but is not limited to steps S710 to S730. Figure 7 These three steps are introduced in detail.

[0154] Step S710, performing structural similarity calculation on the first mask image and the fourth mask image to obtain a first structural similarity loss value;

[0155] Step S720, performing structural similarity calculation on the second mask image and the third mask image to obtain a second structural similarity loss value;

[0156] Step S730 , performing weighted calculation on the first structural similarity loss value and the second structural similarity loss value to obtain an expression structural loss value.

[0157] Specifically, the SSIM index is used to measure the structural similarity loss between the two mask images, in the hope that the adjusted noise distribution can eventually approach the normal distribution or even overlap. As shown in formula (4), the structural similarity of the first mask image mean(mask(D0)) and the fourth mask image mask(p'q'(x0')) is calculated to obtain the first structural similarity loss value. The structural similarity of the second mask image mean(mask(D0')) and the third mask image mask(pq(x0)) is calculated to obtain the second structural similarity loss value. The first structural similarity loss value and the second structural similarity loss value are weighted to obtain the expression structure loss value Loss face .

[0158]

[0159] It should be noted that the specific calculation of the structural similarity calculation SSIM is shown in formula (5).

[0160]

[0161] Among them, X and X' represent the two images that need to be calculated, μ X represents the mean of X, μ X' represents the mean of X', represents the variance of X, represents the variance of X', σ X represents the covariance of X, σ X' represents the covariance of X', and C1 and C2 represent two parameters that maintain stability.

[0162] In step S170 of some embodiments, the original facial image is input into a trained facial expression transfer model, so that the original facial image is converted into a target facial image matching the target expression category through the facial expression transfer model, thereby realizing the transfer of the original facial image from the original expression category to the target expression category.

[0163] See also Figure 8 , Figure 8 It is a structural diagram of the facial expression migration device provided in an embodiment of the present application. The device can implement the facial expression migration method of the above embodiment. The device includes an image and category acquisition module 810, a first training set determination module 820, a second training set determination module 830, a first model training module 840, a model input module 850, a second model training module 860 and an expression migration module 870.

[0164] Image and category acquisition module 810, for acquiring the original face image and target expression category of the target object;

[0165] A first training set determination module 820 is configured to determine a first training set according to a target object, wherein the first training set includes a plurality of first face sample images, and the original face images and the first face sample images are images of the same target object;

[0166] A second training set determination module 830 is configured to determine a second training set according to a target expression category, wherein the second training set includes a plurality of second face sample images, and the expression category corresponding to the second face sample images is the target expression category;

[0167] A first model training module 840 is configured to input each first face sample image into a preset initial cyclic probability model for pre-training to obtain a pre-trained model;

[0168] A model input module 850 is configured to input the first face sample image and the second face sample image into a pre-trained model to obtain a first face processed image;

[0169] a second model training module 860 for calculating a model loss value based on the first processed face image, and adjusting model parameters of the pre-trained model based on the model loss value to obtain a facial expression transfer model;

[0170] The expression transfer module 870 is used to input the original facial image into the facial expression transfer model so as to convert the original facial image into a target facial image matching the target expression category through the facial expression transfer model.

[0171] It should be noted that the facial expression migration device of the embodiment of the present application is used to implement the facial expression migration method of the above-mentioned embodiment. The facial expression migration device of the embodiment of the present application corresponds to the aforementioned facial expression migration method. Please refer to the aforementioned facial expression migration method for the specific processing process, which will not be repeated here.

[0172] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements any of the facial expression transfer methods described in the above embodiments. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0173] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0174] The processor 910 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0175] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called by the processor 910 to execute the facial expression transfer method of the embodiments of this application.

[0176] Input / output interface 930, used to implement information input and output;

[0177] Communication interface 940, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0178] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0179] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0180] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the facial expression transfer method in the above embodiment.

[0181] The embodiment of the present application provides a facial expression migration method and expression migration device, electronic device and storage medium. First, the original facial image and target expression category of the target object are obtained, and a first training set is determined based on the target object, and a second training set is determined based on the target expression category. The first training set includes a plurality of first facial sample images, and the original facial image and the first facial sample image belong to the same target object, and the second training set includes a plurality of second facial sample images, and the expression category corresponding to the second facial sample image is the target expression category. Then, each second facial sample image is input into a preset initial cyclic probability model for pre-training to obtain a pre-trained model. Afterwards, the first facial sample image and the second facial sample image are input into the pre-trained model to obtain a first facial processed image. A cyclic loss value is determined based on the first face sample image, the second face sample image, the third face processed image, and the first face processed image; a spatial similarity loss value is determined based on the first face sample image and the second face sample image; and an expression structure loss value is determined based on the second face processed image and the fourth face processed image. The cyclic loss value, the spatial similarity loss value, and the expression structure loss value are weighted to obtain a model loss value. The model parameters of the pre-trained model are adjusted based on the model loss value, and the adjusted model is continuously trained based on the first training set and the second training set until the model loss value meets a preset training end condition, thereby obtaining a facial expression transfer model. Finally, the original facial image is input into the facial expression transfer model, so that the facial expression transfer model converts the original facial image into a target facial image matching the target expression category. The facial expression transfer model proposed in the embodiment of the present application does not rely on paired sample data, but directly models the sample space of the image sample sets corresponding to the two expressions, while focusing on and mining the feature information of each space and the potential relationship between the spaces to achieve the transfer of the target expression category. The embodiment of the present application performs training and prediction based on the cyclic DDPM model framework, avoiding the instability of the adversarial model framework, improving the stability of the model, and at the same time strictly requiring the migration direction and migration quality of the model to achieve better migration effects. The embodiment of the present application constructs the loss function of the model by adopting cyclic loss value, spatial similarity loss value and expression structure loss value. Through multi-faceted loss function constraints, although the model does not rely on paired images, it can realize paired comparison in supervised tasks and realize facial expression migration tasks more accurately and efficiently. The embodiment of the present application can realize the migration of facial expressions in images efficiently and accurately.

[0182] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0184] It will be understood by those skilled in the art that Figures 1 to 7 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0185] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0186] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0187] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0188] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0190] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0193] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A facial expression transfer method, characterized in that: The method comprises: Obtain the original face image and target expression category of the target object; Determining a first training set according to the target object, wherein the first training set includes a plurality of first face sample images, and the original face images and the first face sample images are images of the same target object; Determining a second training set according to the target expression category, wherein the second training set includes a plurality of second face sample images, and the expression category corresponding to the second face sample images is the target expression category; Performing expression feature extraction on the original facial image to obtain a target feature image; inputting each second facial sample image into a preset initial cyclic probability model, processing the second facial sample image through the initial cyclic probability model to obtain a sample feature image; determining an image loss value based on the sample feature image and the target feature image; adjusting model parameters of the initial cyclic probability model based on the image loss value, and continuing to train the adjusted model based on the second training set until the image loss value meets a preset training end condition, thereby obtaining a pre-trained model, wherein the pre-trained model includes a first diffusion denoising sub-model and a second diffusion denoising sub-model; Inputting the first face sample image into the first diffusion denoising sub-model, performing forward image processing on the first face sample image through the first diffusion denoising sub-model to obtain a second face processed image; performing inverse image processing on the second face processed image through the second diffusion denoising sub-model to obtain a third face processed image; performing facial feature extraction on the third face processed image to obtain a facial feature image; inputting the second face sample image and the facial feature image into the first diffusion denoising sub-model, performing forward image processing on the second face sample image through the first diffusion denoising sub-model to obtain a fourth face processed image; performing inverse image processing on the fourth face processed image through the second diffusion denoising sub-model to obtain the first face processed image; Calculating a model loss value according to the first face processing image, and adjusting model parameters of the pre-trained model according to the model loss value to obtain a facial expression transfer model; The original facial image is input into the facial expression transfer model, so as to convert the original facial image into a target facial image matching the target expression category through the facial expression transfer model.

2. The method according to claim 1, characterized in that Calculating a model loss value based on the first face processing image, and adjusting model parameters of the pre-trained model based on the model loss value to obtain a facial expression transfer model, includes: determining a circulation loss value according to the first face sample image, the second face sample image, the third face processed image, and the first face processed image; Determining a spatial similarity loss value based on the first face sample image and the second face sample image; determining an expression structure loss value according to the second processed face image and the fourth processed face image; Performing weighted calculation on the cycle loss value, the spatial similarity loss value, and the expression structure loss value to obtain a model loss value; The model parameters of the pre-trained model are adjusted according to the model loss value, and the adjusted model is continued to be trained based on the first training set and the second training set until the model loss value meets the preset training end condition to obtain a facial expression transfer model.

3. The method according to claim 2, characterized in that The determining of the cycle loss value according to the first face sample image, the second face sample image, the third face processed image, and the first face processed image includes: performing distance loss calculation on the first face sample image and the third processed face image to obtain a first distance loss value; performing distance loss calculation on the second face sample image and the first processed face image to obtain a second distance loss value; A weighted calculation is performed on the first distance loss value and the second distance loss value to obtain the cycle loss value.

4. The method according to claim 2, characterized in that The determining of the expression structure loss value according to the second processed face image and the fourth processed face image includes: Get the preset rectangular mask area; performing mask processing on the first face sample image according to the rectangular mask area to obtain a first mask image; performing mask processing on the second face sample image according to the rectangular mask area to obtain a second mask image; performing mask processing on the second processed face image according to the rectangular mask area to obtain a third mask image; performing mask processing on the fourth face-processed image according to the rectangular mask area to obtain a fourth mask image; An expression structure loss value is determined according to the first mask image, the second mask image, the third mask image, and the fourth mask image.

5. The method according to claim 4, characterized in that The determining of the expression structure loss value according to the first mask image, the second mask image, the third mask image, and the fourth mask image includes: performing structural similarity calculation on the first mask image and the fourth mask image to obtain a first structural similarity loss value; performing structural similarity calculation on the second mask image and the third mask image to obtain a second structural similarity loss value; A weighted calculation is performed on the first structural similarity loss value and the second structural similarity loss value to obtain the expression structure loss value.

6. A facial expression transfer device, characterized in that: The device comprises: Image and category acquisition module, used to obtain the original face image and target expression category of the target object; A first training set determination module is configured to determine a first training set based on the target object, wherein the first training set includes a plurality of first face sample images, and the original face images and the first face sample images are images of the same target object; A second training set determination module is configured to determine a second training set according to the target expression category, wherein the second training set includes a plurality of second face sample images, and the expression category corresponding to the second face sample images is the target expression category; A first model training module is configured to extract expression features from the original facial image to obtain a target feature image; input each second facial sample image into a preset initial cyclic probability model, process the second facial sample image through the initial cyclic probability model to obtain a sample feature image; determine an image loss value based on the sample feature image and the target feature image; adjust model parameters of the initial cyclic probability model based on the image loss value, and continue training the adjusted model based on the second training set until the image loss value meets a preset training end condition, so as to obtain a pre-trained model, wherein the pre-trained model includes a first diffusion denoising sub-model and a second diffusion denoising sub-model; A model input module is configured to input the first face sample image into the first diffusion denoising sub-model, perform forward image processing on the first face sample image through the first diffusion denoising sub-model to obtain a second face-processed image; perform inverse image processing on the second face-processed image through the second diffusion denoising sub-model to obtain a third face-processed image; perform facial feature extraction on the third face-processed image to obtain a facial feature image; input the second face sample image and the facial feature image into the first diffusion denoising sub-model, perform forward image processing on the second face sample image through the first diffusion denoising sub-model to obtain a fourth face-processed image; and perform inverse image processing on the fourth face-processed image through the second diffusion denoising sub-model to obtain the first face-processed image; a second model training module, configured to calculate a model loss value based on the first processed face image, and adjust model parameters of the pre-trained model based on the model loss value to obtain a facial expression transfer model; The expression transfer module is used to input the original facial image into the facial expression transfer model to convert the original facial image into a target facial image matching the target expression category through the facial expression transfer model.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following is achieved: The method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is used to cause a computer to execute: The method according to any one of claims 1 to 5.

Images