Image synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image synthesis method and device, electronic equipment and a storage medium, belongs to the technical field of image processing, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring a reference selected expression image and a target initial expression image; obtaining a target original expression image for the target initial expression image according to the reference selected expression image; obtaining reference selected expression features according to the reference selected expression image; obtaining target original expression features, target original face features and target original head features according to the target original expression image; obtaining a target aligned expression feature for the reference selected expression feature according to the target original expression feature; obtaining target alignment three-dimensional key point features according to the target original head features and the target alignment expression features; and obtaining a target selected expression image for the target object according to the target original face features and the target aligned three-dimensional key point features. According to the embodiment of the invention, a more natural and vivid image can be generated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, is applicable to the field of financial technology, and in particular to an image synthesis method and device, an electronic device and a storage medium. Background Art

[0002] Expression transfer refers to extracting the expression of a specific object and applying it to a specific target object, so that the target object can change its expression while keeping its facial features and head posture unchanged. For example, in the promotion of insurance products, expression transfer technology can be used to generate multiple introduction videos of the same product by different speakers, so as to push recommended videos explained by speakers suitable for the user to the user based on the user's characteristics. However, existing expression transfer technology is usually applicable to the expression changes of the same person, and when transferring expressions across people, the generated images often lack a sense of naturalness. Therefore, how to generate more natural and realistic images in the expression transfer process has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide an image synthesis method and device, an electronic device and a storage medium, aiming to generate more natural and realistic images.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides an image synthesis method, the method comprising:

[0005] Acquire a reference selected expression image of a reference object, and acquire a target initial expression image of a target object; wherein the reference selected expression image has the expression of the reference object, the target initial expression image has the expression of the target object, and the reference object and the target object are different;

[0006] Perform facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image, to obtain a target original expression image;

[0007] Extracting expression features from the reference selected expression image based on a preset target expression transfer model to obtain reference selected expression features;

[0008] Based on the target expression transfer model, the target original expression image is subjected to transfer feature extraction to obtain target original expression features, target original face features and target original head features;

[0009] Performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain a target aligned expression feature;

[0010] Based on the target expression transfer model, three-dimensional key points are generated for the target original head features and the target aligned expression features to obtain target aligned three-dimensional key point features;

[0011] The target object is subjected to image synthesis according to the target original facial features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object.

[0012] In some embodiments, the method further includes pre-training the target expression transfer model, specifically including:

[0013] Obtain a reference original expression image of a reference object;

[0014] Performing expression migration on the reference original expression image based on a preset expression migration model and the reference selected expression image to obtain a predicted selected expression image;

[0015] Calculate the loss value according to the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain a training loss value;

[0016] The preset expression transfer model is optimized in parameters according to the training loss value to obtain the target expression transfer model.

[0017] In some embodiments, performing expression migration on the reference original expression image based on a preset expression migration model and the reference selected expression image to obtain a predicted selected expression image includes:

[0018] Based on the preset expression migration model, migration feature extraction is performed on the reference selected expression image to obtain reference selected expression features, reference selected face features, reference selected head features and reference selected three-dimensional key point features;

[0019] Based on the preset expression migration model, the reference original expression image is subjected to migration feature extraction to obtain reference original expression features, reference original head features and reference original three-dimensional key point features;

[0020] Based on the preset expression transfer model, mask generation is performed on the reference selected expression image and the reference original expression image to obtain a face motion mask matrix;

[0021] Perform motion field calculation based on the reference selected expression features, the reference selected head features, the reference selected three-dimensional key point features, the reference original expression features, the reference original head features and the reference original three-dimensional key point features to obtain a three-dimensional motion field matrix;

[0022] Performing aggregation calculation according to the three-dimensional motion field matrix and the face motion mask matrix to obtain transformed face features;

[0023] The transformed facial features and the reference selected facial features are feature decoded based on the preset expression transfer model to obtain the predicted selected expression image.

[0024] In some embodiments, the calculating of the loss value according to the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain the training loss value includes:

[0025] Performing perceptual loss calculation on the predicted selected expression image and the reference selected expression image to obtain a perceptual loss value;

[0026] Performing adversarial loss calculation on the predicted selected expression image and the reference selected expression image to obtain an adversarial loss value;

[0027] Calculating the mouth consistency loss of the predicted selected expression image and the reference selected expression image to obtain the mouth consistency loss;

[0028] Calculating the affine transformation loss of the predicted selected expression image and the reference selected expression image to obtain an affine loss value;

[0029] Performing key point loss calculation on the predicted selected expression image to obtain a key point loss value;

[0030] Performing face loss calculation on the predicted selected expression image to obtain a face loss value;

[0031] Performing head loss calculation on the predicted selected expression image to obtain a head loss value;

[0032] The training loss value is obtained by performing aggregation calculation based on the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value and the head loss value.

[0033] In some embodiments, the step of performing facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image to obtain the target original expression image includes:

[0034] Extracting facial muscle key point features from the expression of the reference object of the reference selected expression image to obtain reference selected key point features;

[0035] Extracting facial muscle key point features of the expression of the target object in the target initial expression image to obtain target original key point features;

[0036] Performing Euclidean distance screening on the target original key point features according to the reference selected key point features to obtain target key point features;

[0037] The target initial expression image is screened according to the target key point features to obtain the target original expression image.

[0038] In some embodiments, the step of performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain the target aligned expression feature includes:

[0039] Calculating a scaling factor based on the target original expression feature and the reference selected expression feature to obtain a target scaling factor;

[0040] The reference selected expression features are aggregated and calculated according to the target scaling factor to obtain the target aligned expression features.

[0041] In some embodiments, performing image synthesis on the target object according to the target original facial features and the target aligned three-dimensional key point features to obtain the target selected expression image of the target object includes:

[0042] Performing three-dimensional rendering on the target original face features and the target aligned three-dimensional key point features to obtain pixel distribution and density distribution;

[0043] The target object is volume rendered according to the pixel distribution and the density distribution to obtain the target selected expression image of the target object.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image synthesis device, the device comprising:

[0045] A data acquisition module, used to acquire a reference selected expression image of a reference object and a target initial expression image of a target object; wherein the reference selected expression image has the expression of the reference object, the target initial expression image has the expression of the target object, and the reference object and the target object are different;

[0046] An image screening module, used for screening facial muscle key points of the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image, to obtain a target original expression image;

[0047] An expression extraction module is used to extract expression features from the reference selected expression image based on a preset target expression migration model to obtain reference selected expression features;

[0048] A feature extraction module is used to extract migration features of the target original expression image based on the target expression migration model to obtain target original expression features, target original face features and target original head features;

[0049] An expression alignment module, used for performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain a target aligned expression feature;

[0050] A key point generation module, used for generating three-dimensional key points of the target original head feature and the target aligned expression feature based on the target expression migration model to obtain a target aligned three-dimensional key point feature;

[0051] The image synthesis module is used to perform image synthesis on the target object according to the target original face features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object.

[0052] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0053] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0054] The present application proposes an image synthesis method and device, an electronic device and a storage medium, which obtain a reference selected expression image with an expression of a reference object, obtain a target initial expression image with an expression of a target object, and the reference object and the target object are different, and then perform facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object of the reference selected expression image to obtain a target original expression image, thereby implementing the screening of facial muscle key points of the expression of the target object in the target initial expression image, and obtaining a target original expression image in the target initial expression image that is similar to the reference selected expression image in facial muscle key points, thereby providing image data for subsequent cross-person expression migration, avoiding cross-person expression migration for images with large differences in facial muscle key points, and improving the naturalness of expression of the image obtained by expression migration; further, based on a preset The target expression transfer model is used to extract expression features of the reference selected expression image to obtain the reference selected expression features, and then the target expression transfer model is used to extract transfer features of the target original expression image to obtain the target original expression features, the target original face features and the target original head features, and finally the reference selected expression features are aligned according to the target original expression features to obtain the target aligned expression features; thereby achieving cross-personal expression alignment and avoiding the situation where the expression does not match the facial features during expression transfer; further, based on the target expression transfer model, three-dimensional key points are generated for the target original head features and the target aligned expression features to obtain the target aligned three-dimensional key point features, and then the target object is synthesized according to the target original face features and the target aligned three-dimensional key point features to obtain the target selected expression image of the target object, thereby achieving the generation of more natural and realistic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of an image synthesis method provided in an embodiment of the present application;

[0056] Figure 2 is a flow chart of an image synthesis method provided by another embodiment of the present application;

[0057] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0058] Figure 4 yes Figure 2 Flow chart of step S203 in FIG.

[0059] Figure 5 yes Figure 1 Flow chart of step S102 in FIG.

[0060] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0061] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0062] Figure 8 is a structural schematic diagram of an image synthesis device provided in an embodiment of the present application;

[0063] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] First, some nouns involved in this application are analyzed:

[0068] Facial Expression Transfer: Facial Expression Transfer is a technology in the field of computer vision and artificial intelligence, which is mainly used to transfer a specific facial expression from one portrait to another. Expression transfer technology analyzes the expression features of the source portrait, such as the dynamic changes of the eyes, mouth and eyebrows, and applies these dynamic features to the target portrait, thereby achieving a natural conversion of expressions. This process usually involves deep learning models, especially generative adversarial networks, which can capture and simulate the complexity and subtle changes of facial expressions.

[0069] 3D Keypoints: 3D keypoints are an important concept in the field of computer vision and artificial intelligence, used to mark and track specific location points of objects in three-dimensional space. These points usually represent the salient features of objects, such as the corners of the eyes, the tip of the nose, the corners of the mouth, etc. on the face, or the joint positions of robots and other objects. The extraction of 3D keypoints is crucial for performing a variety of tasks, including 3D reconstruction, motion capture, posture estimation, and augmented reality. Through accurate keypoint positioning, the system can understand and analyze the spatial structure and motion of objects.

[0070] Facial Expression Features: Facial Expression Features is a technology in the field of computer vision and artificial intelligence that focuses on identifying and analyzing the dynamic changes of various parts of the face from face images or videos, which reflect the emotional state of the person. It is a branch of artificial intelligence and an interdisciplinary subject between computer science and image processing. By accurately identifying the position and movement of key facial points such as eyes, eyebrows, mouth and nose, facial expression features help machines understand human emotions and reactions. This technology is widely used in technical fields such as sentiment analysis, human-computer interaction and security systems. For example, in sentiment analysis applications, by analyzing the user's facial expressions, the system can identify the user's emotional state and provide more personalized services; in human-computer interaction, facial expression features enable robots or virtual assistants to understand and respond to the emotions and needs of human users more naturally. By simulating and extending human understanding of emotions, facial expression feature technology plays an important role in enhancing the naturalness and depth of computer-human interaction.

[0071] Head Features: Head features refer to key parameters used to describe and analyze the position, movement, and posture of a person’s head in audio-driven portrait video applications. Head features capture dynamic information such as the head’s movement trajectory, angle changes such as pitch, yaw, and roll angles, and facial expressions.

[0072] Expression transfer refers to extracting the expression of a specific object and applying it to a specific target object, so that the target object can change its expression while keeping its facial features and head posture unchanged. For example, in the promotion of insurance products, expression transfer technology can be used to generate multiple introduction videos of the same product by different speakers, so as to push recommended videos explained by speakers suitable for the user to the user based on the user's characteristics. However, existing expression transfer technology is usually applicable to the expression changes of the same person, and when transferring expressions across people, the generated images often lack a sense of naturalness. Therefore, how to generate more natural and realistic images in the expression transfer process has become an urgent problem to be solved.

[0073] Based on this, the embodiments of the present application provide an image synthesis method and device, an electronic device and a storage medium, aiming to generate more natural and realistic images.

[0074] An image synthesis method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image synthesis method in the embodiments of the present application is described.

[0075] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0077] The image synthesis method provided in the embodiment of the present application relates to the field of image processing technology and is applicable to the field of financial technology. The image synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image synthesis method, etc., but is not limited to the above forms.

[0078] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0079] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0080] Figure 1 is an optional flowchart of the image synthesis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0081] Step S101, obtaining a reference selected expression image of a reference object, and obtaining a target initial expression image of a target object; wherein the reference selected expression image has an expression of the reference object, the target initial expression image has an expression of the target object, and the reference object and the target object are not the same;

[0082] Step S102, screening facial muscle key points of the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image, to obtain the target original expression image;

[0083] Step S103, extracting expression features from the reference selected expression image based on a preset target expression transfer model to obtain reference selected expression features;

[0084] Step S104, extracting migration features of the target original expression image based on the target expression migration model to obtain target original expression features, target original face features and target original head features;

[0085] Step S105, performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain the target aligned expression feature;

[0086] Step S106, generating three-dimensional key points for the target original head features and the target aligned expression features based on the target expression transfer model to obtain the target aligned three-dimensional key point features;

[0087] Step S107, performing image synthesis on the target object according to the target original facial features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object.

[0088] Steps S101 to S107 shown in the embodiment of the present application are to obtain a reference selected expression image having the expression of a reference object, obtain a target initial expression image having the expression of a target object, and the reference object and the target object are different, and then perform facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object of the reference selected expression image to obtain a target original expression image, thereby implementing the screening of facial muscle key points of the expression of the target object in the target initial expression image, and obtaining a target original expression image in the target initial expression image that is similar to the reference selected expression image in facial muscle key points, thereby providing image data for subsequent cross-person expression migration, avoiding cross-person expression migration for images with large differences in facial muscle key points, and improving the naturalness of expression of the image obtained by expression migration; further, based on a preset target The reference expression transfer model extracts expression features from the reference selected expression image to obtain the reference selected expression features, and then based on the target expression transfer model, transfer features are extracted from the target original expression image to obtain the target original expression features, the target original face features and the target original head features, and finally, the reference selected expression features are aligned according to the target original expression features to obtain the target aligned expression features; thereby, cross-personal expression alignment is achieved to avoid the situation where the expression does not match the facial features during expression transfer; further, based on the target expression transfer model, three-dimensional key points are generated for the target original head features and the target aligned expression features to obtain the target aligned three-dimensional key point features, and then the target object is synthesized according to the target original face features and the target aligned three-dimensional key point features to obtain the target selected expression image of the target object, thereby generating a more natural and realistic image.

[0089] See also Figure 2In some embodiments, the image synthesis method further includes pre-training a target expression transfer model before step S101, which may specifically include but is not limited to steps S201 to S204:

[0090] Step S201, obtaining a reference original expression image of a reference object;

[0091] Step S202, performing expression migration on the reference original expression image based on a preset expression migration model and a reference selected expression image to obtain a predicted selected expression image;

[0092] Step S203, calculating the loss value according to the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain a training loss value;

[0093] Step S204, optimizing the parameters of the preset expression transfer model according to the training loss value to obtain the target expression transfer model.

[0094] Steps S201 to S204 shown in the embodiment of the present application are performed by obtaining a reference original expression image of a reference object, and then performing expression transfer on the reference original expression image based on a preset expression transfer model and a reference selected expression image to obtain a predicted selected expression image, and then calculating a loss value based on the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain a training loss value, and finally optimizing the parameters of the preset expression transfer model based on the training loss value to obtain a target expression transfer model, so that the target expression transfer model can perform accurate expression transfer within the same character, making the image after expression transfer more natural and realistic in expression presentation and overall quality.

[0095] In step S201 of some embodiments, the reference object refers to a specific individual selected to provide information such as facial features, expression features, and head shape, and is a facial image of a specific individual. The reference original expression image is an image of the reference object, which is image data used to show the expression state of the reference object, and records the facial and head shapes of the reference object, such as images of various expressions such as smiling, frowning, and calm.

[0096] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S306:

[0097] Step S301, extracting migration features of the reference selected expression image based on a preset expression migration model to obtain reference selected expression features, reference selected face features, reference selected head features and reference selected three-dimensional key point features;

[0098] Step S302, extracting migration features from the reference original expression image based on a preset expression migration model to obtain reference original expression features, reference original head features, and reference original three-dimensional key point features;

[0099] Step S303, mask generation is performed on the reference selected expression image and the reference original expression image based on a preset expression transfer model to obtain a face motion mask matrix;

[0100] Step S304, performing motion field calculation based on the reference selected expression features, the reference selected head features, the reference selected three-dimensional key point features, the reference original expression features, the reference original head features and the reference original three-dimensional key point features to obtain a three-dimensional motion field matrix;

[0101] Step S305, performing aggregation calculation according to the three-dimensional motion field matrix and the face motion mask matrix to obtain the transformed face features;

[0102] Step S306, feature decoding is performed on the transformed facial features and the reference selected facial features based on a preset expression transfer model to obtain a predicted selected expression image.

[0103] In the steps S301 to S306 shown in the embodiment of the present application, by extracting migration features from the reference selected expression image based on the preset expression migration model, and extracting migration features from the reference original expression image at the same time, reference selected expression features, reference selected face features, reference selected head features, reference selected three-dimensional key point features, reference original expression features, reference original head features and reference original three-dimensional key point features are obtained. Then, mask generation is performed on the reference selected expression image and the reference original expression image based on the preset expression migration model to obtain a face motion mask matrix, and motion field calculation is performed based on the reference selected expression features, reference selected head features, reference selected three-dimensional key point features, reference original expression features, reference original head features and reference original three-dimensional key point features to obtain a three-dimensional motion field matrix. Further, aggregation calculation is performed based on the three-dimensional motion field matrix and the face motion mask matrix to obtain transformed face features, and then feature decoding is performed on the transformed face features and the reference selected face features based on the preset expression migration model to obtain a predicted selected expression image, so as to realize expression migration within the same person, extract the expression in the reference selected expression image and migrate it to the reference original expression image, and obtain a predicted selected expression image.

[0104] In step S301 of some embodiments, the preset expression transfer model is a neural network model. In one embodiment, the preset expression transfer model is a GAN neural network model. The GAN neural network model includes a feature extraction module, a generator and a discriminator. The feature extraction module includes a face feature extraction module, a head feature extraction module, an expression feature extraction module, a three-dimensional key point feature extraction module and a mask estimation module.

[0105] The reference selected expression image is input into the preset expression transfer model, and the face feature extraction module is used to extract the face feature of the reference selected expression image to obtain the reference selected face feature. The head feature extraction module is used to extract the head feature of the reference selected expression image to obtain the reference selected head feature. The expression feature extraction module is used to extract the expression feature of the reference selected expression image to obtain the reference selected expression feature. The three-dimensional key point feature extraction module is used to extract the three-dimensional key point feature of the reference selected head feature and the reference selected expression feature to obtain the reference selected key point feature.

[0106] It should be noted that the extraction effect of directly using the three-dimensional key point feature extraction module to extract three-dimensional key point features from the reference selected expression image is poor. By using the three-dimensional key point feature extraction module to extract three-dimensional key point features from the reference selected head features and the reference selected expression features, the three-dimensional key point features can be accurately obtained.

[0107] In step S302 of some embodiments, the principle of extracting migration features for the reference original expression image based on the preset expression migration model is similar to the principle of extracting migration features for the reference selected expression image based on the preset expression migration model, which will not be repeated here.

[0108] In step S303 of some embodiments, mask generation is performed on the reference selected expression image and the reference original expression image based on the preset expression transfer model to obtain a face motion mask matrix. Specifically, the reference selected expression image and the reference original expression image are input into a mask estimation module in the preset expression transfer model to obtain a face motion mask matrix, as shown in formula (1):

[0109] M = MaskEstimator(I s ,I d ) (1),

[0110] Among them, MaskEstimator is the mask estimation module, the face motion mask matrix is ​​M, I s Select the expression image for reference, I d For reference, the original expression image.

[0111] In step S304 of some embodiments, a motion field calculation is performed based on the reference selected expression features, the reference selected head features, the reference selected three-dimensional key point features, the reference original expression features, the reference original head features, and the reference original three-dimensional key point features to obtain a three-dimensional motion field matrix. The motion field calculation is shown in formula (2):

[0112]

[0113] Among them, Flow is the three-dimensional motion field matrix, K d K is the reference to the original 3D key point features. s is the reference for selecting the 3D key point features, K() is the 3D key point feature extraction module, and H d To refer to the original head features, E d To refer to the original expression features, K s Select the 3D key point features for reference, H s Select the head features for reference, E s For reference, the original expression features.

[0114] In step S305 of some embodiments, an aggregation calculation is performed based on the three-dimensional motion field matrix and the face motion mask matrix to obtain the transformed face features. The aggregation calculation is shown in formula (3):

[0115] Flow masked =M⊙Flow (3),

[0116] Among them, Flow masked is to transform the facial features, and ⊙ is the element-by-element multiplication.

[0117] In step S306 of some embodiments, feature decoding is performed on the transformed facial features and the reference selected facial features based on a preset expression transfer model to obtain a predicted selected expression image. Specifically, the transformed facial features and the reference selected facial features are input into a generator in the preset expression transfer model to obtain a predicted selected expression image.

[0118] See also Figure 4 In some embodiments, step S203 may include but is not limited to steps S401 to S408:

[0119] Step S401, performing perceptual loss calculation on the predicted selected expression image and the reference selected expression image to obtain a perceptual loss value;

[0120] Step S402, performing adversarial loss calculation on the predicted selected expression image and the reference selected expression image to obtain an adversarial loss value;

[0121] Step S403, calculating the mouth consistency loss of the predicted selected expression image and the reference selected expression image to obtain the mouth consistency loss;

[0122] Step S404, performing affine transformation loss calculation on the predicted selected expression image and the reference selected expression image to obtain an affine loss value;

[0123] Step S405, performing key point loss calculation on the predicted selected expression image to obtain a key point loss value;

[0124] Step S406, calculating the face loss of the predicted selected expression image to obtain a face loss value;

[0125] Step S407, performing head loss calculation on the predicted selected expression image to obtain a head loss value;

[0126] Step S408, performing aggregation calculation based on the perception loss value, adversarial loss value, mouth consistency loss value, affine loss value, key point loss value, face loss value and head loss value to obtain a training loss value.

[0127] In steps S401 to S408 shown in the embodiment of the present application, by performing perceptual loss calculation, adversarial loss calculation, mouth consistency loss calculation and affine transformation loss calculation on the predicted selected expression image and the reference selected expression image, and performing key point loss calculation, key point loss calculation, face loss calculation and head loss calculation on the predicted selected expression image, all the loss values ​​are finally aggregated and calculated to obtain the final training loss value, so as to train the subsequent preset expression transfer model in multiple aspects and dimensions, and improve the expression transfer performance of the preset expression transfer model.

[0128] In steps S401 to S408 of some embodiments, an aggregate calculation is performed based on the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value, and the head loss value to obtain a training loss value, and the aggregate calculation is shown in formula (4):

[0129] L=L perc +L Gan +L mouth +L equiv +L kp-prior +L def-prior +L head (4)

[0130] Among them, L is the training loss value, L percis the perceptual loss value, which is implemented by the intermediate feature layer of the pre-trained VGG network. The predicted selected expression image is input into the intermediate feature layer of the pre-trained VGG network to obtain the perceptual data of the predicted selected expression image. Then, the reference selected expression image is input into the intermediate feature layer of the pre-trained VGG network to obtain the perceptual data of the reference selected expression image. Then, the difference between the perceptual data of the predicted selected expression image and the perceptual data of the reference selected expression image is calculated to obtain the perceptual loss value. L Gan The adversarial loss value is implemented by the discriminator in the preset expression transfer model. The implementation principle of adversarial loss calculation is similar to that of perceptual loss calculation, which will not be repeated here. mouth is the mouth consistency loss, which is implemented by the pre-trained SyncNet mouth feature extraction module. The implementation principle of mouth consistency loss calculation is similar to that of perceptual loss calculation, so it will not be repeated here. equiv It is the affine loss value, which is implemented by the preset affine transformation function. The implementation principle of affine transformation loss calculation is similar to that of perceptual loss calculation, so it will not be repeated here.

[0131] L kp-prior is the key point loss value, which is realized by the preset facial key point extraction network. The key point loss is shown in formula (5):

[0132]

[0133] Among them, i and j are the indices of the facial key points of the predicted selected expression image, K i To predict the i-th key point in the selected expression image, β is the preset weight, exp() is the preset exponential function, and the key point loss value is used to avoid over-concentration of facial key points in the selected expression image.

[0134] L def-prior The face loss value is realized by the face feature extraction module in the preset expression transfer model, as shown in formula (6):

[0135]

[0136] Among them, I predict To predict the selected expression image, E () is the facial feature extraction module in the preset expression transfer model, firstly, the predicted selected expression image is input into the facial feature extraction module in the preset expression transfer model, and the predicted selected facial feature E (I predict 01, and then set the first two dimensions to 0 to obtain the predicted target face feature E(I predict )2.

[0137] L headIt is the head loss value, which is implemented by the pre-trained head posture model. The predicted selected expression image is input into the pre-trained head posture model to obtain the head loss value.

[0138] In step S204 of some embodiments, parameters of the preset expression transfer model are optimized according to the training loss value. Specifically, the preset expression transfer model is subjected to random gradient descent optimization of the model parameters in the preset expression transfer model according to the training loss value to obtain a target expression transfer model capable of expression transfer within the same person.

[0139] In step S101 of some embodiments, the target object and the reference object are not the same object, the target initial expression image includes multiple expression images of the target object, the reference selected image includes the expression image of the reference object, and the facial muscles of the expressions in the target initial expression image and the reference selected image are not exactly the same.

[0140] See also Figure 5 In some embodiments, step S102 includes but is not limited to steps S501 to S504:

[0141] Step S501, extracting facial muscle key point features of the expression of the reference object of the reference selected expression image to obtain reference selected key point features;

[0142] Step S502, extracting facial muscle key point features of the target object's expression in the target initial expression image to obtain target original key point features;

[0143] Step S503, performing Euclidean distance screening on the target original key point features according to the reference selected key point features to obtain the target key point features;

[0144] Step S504, screening the target initial expression image according to the target key point features to obtain the target original expression image.

[0145] Steps S501 to S504 shown in the embodiment of the present application are to extract facial muscle key point features of the expression of the reference object of the reference selected expression image to obtain the reference selected key point features, and at the same time extract facial muscle key point features of the expression of the target object of the target initial expression image to obtain the target original key point features, and then perform Euclidean distance screening on the target original key point features according to the reference selected key point features to obtain the target key point features, and finally screen the target initial expression image according to the target key point features to obtain the target original expression image, thereby implementing the screening of the facial muscle key points of the expression of the target object in the target initial expression image, and obtaining the target original expression image in the target initial expression image that is similar to the facial muscle key points of the reference selected expression image, thereby providing image data for subsequent cross-person expression migration, avoiding cross-person expression migration of images with large differences in facial muscle key points, and improving the naturalness of the expression of the image obtained by expression migration.

[0146] In step S501 of some embodiments, the facial muscle key points are used to refer to the facial key points of the expression of the reference object on the selected expression image, and a pre-trained facial action unit (AU) network model is used to extract facial AU features, and the reference selected expression image is input into the facial action unit to obtain the reference selected key point features.

[0147] In step S502 of some embodiments, the facial muscle key points are the facial key points of the expression of the target object on the target selected expression image. A pre-trained facial action unit (AU) network model is used to extract facial AU features, and the target selected expression image is input into the facial action unit to obtain the target selected key point features.

[0148] In step S503 of some embodiments, Euclidean distance calculation is performed based on the target selected key point features and the reference selected key point features to obtain a target distance value. When the target distance value is less than a preset distance threshold, the corresponding target selected key point is used as the target key point feature.

[0149] In step S504 of some embodiments, the target initial expression image corresponding to the target key point features is acquired to obtain the target original expression image.

[0150] In step S103 of some embodiments, expression features are extracted from the reference selected expression image based on a preset target expression migration model. The principle of obtaining the reference selected expression features is similar to the principle of extracting migration features from the reference selected expression image based on the preset expression migration model, which will not be repeated here.

[0151] In step S104 of some embodiments, the principle of performing migration feature extraction on the target original expression image based on the target expression migration model to obtain the target original expression features, target original face features and target original head features is similar to the principle of performing migration feature extraction on the reference selected expression image based on the preset expression migration model, and will not be repeated here.

[0152] See also Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S602:

[0153] Step S601, calculating a scaling factor according to a target original expression feature and a reference selected expression feature to obtain a target scaling factor;

[0154] Step S602, performing aggregation calculation on the reference selected expression features according to the target scaling factor to obtain the target aligned expression features.

[0155] Steps S601 to S602 shown in the embodiment of the present application calculate the scaling factor based on the target original expression features and the reference selected expression features to obtain the target scaling factor, and then aggregate the reference selected expression features according to the target scaling factor to obtain the target aligned expression features, thereby achieving cross-character expression alignment.

[0156] In step S601 of some embodiments, a scaling factor is calculated based on the target original expression feature and the reference selected expression feature to obtain a target scaling factor. The scaling factor calculation is shown in formula (7):

[0157]

[0158] Where s is the target scaling factor, E d is the reference selected expression feature, m is the number of target original expression images, E r is the target original expression feature, i is the index of the target original expression image, is the target original expression feature of the i-th target original expression image.

[0159] In step S602 of some embodiments, the reference selected expression features are aggregated and calculated according to the target scaling factor to obtain the target aligned expression features, and the aggregate calculation is shown in formula (8):

[0160] E transform =s·E d (8)

[0161] Among them, E transform Align expression features for the target.

[0162] In step S106 of some embodiments, three-dimensional key points are generated for the target original head features and the target aligned expression features based on the target expression transfer model to obtain target aligned three-dimensional key point features. Specifically, the target original head features and the target aligned expression features are input into the three-dimensional key point feature extraction module in the target expression transfer model to obtain the target aligned three-dimensional key point features.

[0163] See also Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S702:

[0164] Step S701, performing three-dimensional rendering on the target original face features and the target aligned three-dimensional key point features to obtain pixel distribution and density distribution;

[0165] Step S702 , performing volume rendering on the target object according to the pixel distribution and density distribution to obtain a target selected expression image of the target object.

[0166] In steps S701 to S702 shown in the embodiment of the present application, pixel distribution and density distribution are obtained by performing three-dimensional rendering on the target original facial features and the target aligned three-dimensional key point features, and then volume rendering is performed on the target object according to the pixel distribution and density distribution to obtain a target selected expression image of the target object, and finally expression migration across characters is achieved to obtain a target selected expression image of the target object.

[0167] In step S701 of some embodiments, the target original facial features and the target aligned three-dimensional key point features are three-dimensionally rendered to obtain pixel distribution and density distribution. In one embodiment, the target original facial features and the target aligned three-dimensional key point features are input into a pre-trained NeRF model to obtain pixel distribution and density distribution.

[0168] In step S702 of some embodiments, the target object is volume rendered according to the pixel distribution and the density distribution to obtain a target selected expression image of the target object. The volume rendering is shown in formula (9):

[0169]

[0170] Where NeRF is the pre-trained NeRF model, o is the preset camera position, i.e., the starting point of the ray, d is the preset sight line direction unit vector, indicating the direction from the camera to a certain point in the scene, z is the depth of the ray, r(z) represents the 3D space point at the depth z on the ray, k is the target aligned 3D key point feature, f is the target original face feature, I target Select the expression image for the target, c(z) is the pixel distribution, σ(z) is the density distribution, z n is the preset proximal depth, zf is the preset far end depth, T(z) is the distance from the starting point z of the ray n The cumulative transmittance to depth z indicates the probability that the light is not completely absorbed before reaching depth z, and s is the probability of the light being absorbed from z to z. n The intermediate variable between .

[0171] See also Figure 8 The embodiment of the present application further provides an image synthesis device, which can implement the above-mentioned image synthesis method, and the device includes:

[0172] The data acquisition module 801 is used to acquire a reference selected expression image of a reference object and acquire a target initial expression image of a target object; wherein the reference selected expression image has an expression of the reference object, the target initial expression image has an expression of the target object, and the reference object and the target object are different;

[0173] An image screening module 802 is used to screen facial muscle key points of the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image, so as to obtain the target original expression image;

[0174] The expression extraction module 803 is used to extract expression features from the reference selected expression image based on a preset target expression migration model to obtain reference selected expression features;

[0175] A feature extraction module 804 is used to extract migration features of the target original expression image based on the target expression migration model to obtain target original expression features, target original face features and target original head features;

[0176] The expression alignment module 805 is used to align the reference selected expression features according to the target original expression features to obtain the target aligned expression features;

[0177] A key point generation module 806 is used to generate three-dimensional key points for the target original head features and the target aligned expression features based on the target expression transfer model to obtain the target aligned three-dimensional key point features;

[0178] The image synthesis module 807 is used to perform image synthesis on the target object according to the target original facial features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object.

[0179] The specific implementation of the image synthesis device is basically the same as the specific embodiment of the above-mentioned image synthesis method, and will not be repeated here.

[0180] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0181] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0182] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0183] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image synthesis method of the embodiment of this application;

[0184] Input / output interface 903, used to implement information input and output;

[0185] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0186] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0187] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0188] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned image synthesis method is implemented.

[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The image synthesis method, image synthesis device, electronic device and storage medium provided by the embodiments of the present application obtain a reference selected expression image having an expression of a reference object, obtain a target initial expression image having an expression of a target object, and the reference object and the target object are different, and then perform facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object of the reference selected expression image to obtain a target original expression image, thereby implementing the screening of facial muscle key points of the expression of the target object in the target initial expression image, and obtaining a target original expression image in the target initial expression image that is similar to the reference selected expression image in facial muscle key points, thereby providing image data for subsequent cross-person expression migration, avoiding cross-person expression migration for images with large differences in facial muscle key points, and improving the naturalness of expression of the image obtained by expression migration; further, based on Based on the preset target expression transfer model, expression features are extracted from the reference selected expression image to obtain the reference selected expression features, and then based on the target expression transfer model, migration features are extracted from the target original expression image to obtain the target original expression features, target original face features and target original head features, and finally, expression alignment is performed on the reference selected expression features according to the target original expression features to obtain the target aligned expression features; thereby, cross-personal expression alignment is achieved to avoid the situation where the expression does not match the facial features during expression transfer; further, based on the target expression transfer model, three-dimensional key points are generated for the target original head features and the target aligned expression features to obtain the target aligned three-dimensional key point features, and then image synthesis is performed on the target object according to the target original face features and the target aligned three-dimensional key point features to obtain the target selected expression image of the target object, thereby generating a more natural and realistic image.

[0191] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0192] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0195] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image synthesis method, characterized in that: The method comprises: Acquire a reference selected expression image of a reference object, and acquire a target initial expression image of a target object; wherein the reference selected expression image has the expression of the reference object, the target initial expression image has the expression of the target object, and the reference object and the target object are different; Performing facial muscle key point screening on the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image to obtain a target original expression image; Extracting expression features from the reference selected expression image based on a preset target expression transfer model to obtain reference selected expression features; Based on the target expression transfer model, the target original expression image is subjected to transfer feature extraction to obtain target original expression features, target original face features and target original head features; Performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain a target aligned expression feature; Based on the target expression transfer model, three-dimensional key points are generated for the target original head features and the target aligned expression features to obtain target aligned three-dimensional key point features; The target object is subjected to image synthesis according to the target original facial features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object.

2. The method according to claim 1, characterized in that The method further includes pre-training the target expression transfer model, specifically including: Obtain a reference original expression image of a reference object; Performing expression migration on the reference original expression image based on a preset expression migration model and the reference selected expression image to obtain a predicted selected expression image; Calculate the loss value according to the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain a training loss value; The preset expression transfer model is optimized in parameters according to the training loss value to obtain the target expression transfer model.

3. The method according to claim 2, characterized in that The step of performing expression migration on the reference original expression image based on the preset expression migration model and the reference selected expression image to obtain a predicted selected expression image includes: Based on the preset expression migration model, migration feature extraction is performed on the reference selected expression image to obtain reference selected expression features, reference selected face features, reference selected head features and reference selected three-dimensional key point features; Based on the preset expression migration model, the reference original expression image is subjected to migration feature extraction to obtain reference original expression features, reference original head features and reference original three-dimensional key point features; Based on the preset expression transfer model, mask generation is performed on the reference selected expression image and the reference original expression image to obtain a face motion mask matrix; Perform motion field calculation based on the reference selected expression features, the reference selected head features, the reference selected three-dimensional key point features, the reference original expression features, the reference original head features and the reference original three-dimensional key point features to obtain a three-dimensional motion field matrix; Performing aggregation calculation according to the three-dimensional motion field matrix and the face motion mask matrix to obtain transformed face features; The transformed facial features and the reference selected facial features are feature decoded based on the preset expression transfer model to obtain the predicted selected expression image.

4. The method according to claim 2, characterized in that: The calculating of the loss value according to the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain the training loss value comprises: Performing perceptual loss calculation on the predicted selected expression image and the reference selected expression image to obtain a perceptual loss value; Performing adversarial loss calculation on the predicted selected expression image and the reference selected expression image to obtain an adversarial loss value; Calculating the mouth consistency loss of the predicted selected expression image and the reference selected expression image to obtain the mouth consistency loss; Calculating the affine transformation loss of the predicted selected expression image and the reference selected expression image to obtain an affine loss value; Performing key point loss calculation on the predicted selected expression image to obtain a key point loss value; Performing face loss calculation on the predicted selected expression image to obtain a face loss value; Performing head loss calculation on the predicted selected expression image to obtain a head loss value; The training loss value is obtained by performing aggregation calculation based on the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value and the head loss value.

5. The method according to claim 1, characterized in that The step of screening facial muscle key points of the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image to obtain the target original expression image comprises: Extracting facial muscle key point features from the expression of the reference object of the reference selected expression image to obtain reference selected key point features; Extracting facial muscle key point features of the expression of the target object in the target initial expression image to obtain target original key point features; Performing Euclidean distance screening on the target original key point features according to the reference selected key point features to obtain target key point features; The target initial expression image is screened according to the target key point features to obtain the target original expression image.

6. The method according to claim 1, characterized in that The step of performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain the target aligned expression feature comprises: Calculating a scaling factor based on the target original expression feature and the reference selected expression feature to obtain a target scaling factor; The reference selected expression features are aggregated and calculated according to the target scaling factor to obtain the target aligned expression features.

7. The method according to claim 1, characterized in that The step of synthesizing the target object according to the target original face features and the target aligned three-dimensional key point features to obtain the target selected expression image of the target object includes: Performing three-dimensional rendering on the target original face features and the target aligned three-dimensional key point features to obtain pixel distribution and density distribution; The target object is volume rendered according to the pixel distribution and the density distribution to obtain the target selected expression image of the target object.

8. An image synthesis device, characterized in that: The device comprises: A data acquisition module, used to acquire a reference selected expression image of a reference object and a target initial expression image of a target object; wherein the reference selected expression image has the expression of the reference object, the target initial expression image has the expression of the target object, and the reference object and the target object are different; An image screening module, used for screening facial muscle key points of the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image, to obtain a target original expression image; An expression extraction module is used to extract expression features from the reference selected expression image based on a preset target expression migration model to obtain reference selected expression features; A feature extraction module is used to extract migration features of the target original expression image based on the target expression migration model to obtain target original expression features, target original face features and target original head features; An expression alignment module, used for performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain a target aligned expression feature; A key point generation module, used for generating three-dimensional key points of the target original head feature and the target aligned expression feature based on the target expression migration model to obtain a target aligned three-dimensional key point feature; The image synthesis module is used to perform image synthesis on the target object according to the target original facial features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image synthesis method according to any one of claims 1 to 7 is implemented.

Images