Image synthesis method and device, electronic device, and storage medium

Abstract

Embodiments of the present application provide an image synthesis method and device, electronic equipment and storage medium, belong to the technical field of image processing, and are suitable for the field of financial technology. The method comprises: obtaining a reference selected expression image and a target initial expression image; obtaining a target original expression image according to the target initial expression image and the reference selected expression image; obtaining reference selected expression features according to the reference selected expression image; obtaining target original expression features, target original face features and target original head features according to the target original expression image; obtaining target aligned expression features according to the reference selected expression features and the target original expression features; obtaining target aligned three-dimensional key point features according to the target original head features and the target aligned expression features; and obtaining a target selected expression image according to a target object, the target original face features and the target aligned three-dimensional key point features. The embodiments of the present application can generate more natural and realistic images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and is suitable for the field of financial technology, and particularly relates to an image synthesis method and device, an electronic device and a storage medium. BACKGROUND

[0002] Expression transfer refers to extracting and applying the expression of a specific object to a specific target object, so that the target object realizes expression changes while keeping the facial features and head posture unchanged. For example, in insurance product promotion, multiple introduction videos of the same product by different speakers can be generated through expression transfer technology, so as to push the recommended video of the speaker explaining suitable for the user to the user according to the characteristics of the user. However, the existing expression transfer technology is usually applicable to the expression changes of the same person, and when cross-person expression transfer is performed, the generated image often lacks naturalness. Therefore, how to generate a more natural and realistic image in the expression transfer process has become a problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide an image synthesis method and device, an electronic device and a storage medium, which aims to generate a more natural and realistic image.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides an image synthesis method, which comprises:

[0005] obtaining a reference selected expression image of a reference object and a target initial expression image of a target object; wherein the reference selected expression image has the expression of the reference object, the target initial expression image has the expression of the target object, and the reference object and the target object are not the same;

[0006] performing facial muscle key point screening on the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image, to obtain a target original expression image;

[0007] extracting expression features from the reference selected expression image based on a preset target expression transfer model, to obtain reference selected expression features;

[0008] extracting transfer features from the target original expression image based on the target expression transfer model, to obtain target original expression features, target original facial features and target original head features;

[0009] performing expression alignment on the reference selected expression features according to the target original expression features, to obtain target aligned expression features;

[0010] perform three-dimensional key point generation on the target original head feature and the target aligned expression feature based on the target expression transfer model to obtain a target aligned three-dimensional key point feature;

[0011] perform image synthesis on the target object based on the target original face feature and the target aligned three-dimensional key point feature to obtain a target selected expression image of the target object.

[0012] In some embodiments, the method further comprises pre-training the target expression transfer model, specifically comprising:

[0013] obtaining a reference original expression image of a reference object;

[0014] performing expression transfer on the reference original expression image based on a preset expression transfer model and the reference selected expression image to obtain a predicted selected expression image;

[0015] performing loss value calculation based on the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain a training loss value;

[0016] performing parameter optimization on the preset expression transfer model based on the training loss value to obtain the target expression transfer model.

[0017] In some embodiments, the performing expression transfer on the reference original expression image based on a preset expression transfer model and the reference selected expression image to obtain a predicted selected expression image comprises:

[0018] performing transfer feature extraction on the reference selected expression image based on the preset expression transfer model to obtain a reference selected expression feature, a reference selected face feature, a reference selected head feature and a reference selected three-dimensional key point feature;

[0019] performing transfer feature extraction on the reference original expression image based on the preset expression transfer model to obtain a reference original expression feature, a reference original head feature and a reference original three-dimensional key point feature;

[0020] performing mask generation on the reference selected expression image and the reference original expression image based on the preset expression transfer model to obtain a face motion mask matrix;

[0021] performing motion field calculation based on the reference selected expression feature, the reference selected head feature, the reference selected three-dimensional key point feature, the reference original expression feature, the reference original head feature and the reference original three-dimensional key point feature to obtain a three-dimensional motion field matrix;

[0022] performing aggregation calculation based on the three-dimensional motion field matrix and the face motion mask matrix to obtain a transformed face feature;

[0023] perform feature decoding on the transformed face feature and the reference selected face feature based on the preset expression transfer model to obtain the predicted selected expression image.

[0024] In some embodiments, the loss value calculation according to the reference original expression image, the predicted selected expression image and the reference selected expression image obtains a training loss value, including:

[0025] perform perceptual loss calculation on the predicted selected expression image and the reference selected expression image to obtain a perceptual loss value;

[0026] perform adversarial loss calculation on the predicted selected expression image and the reference selected expression image to obtain an adversarial loss value;

[0027] perform mouth consistency loss calculation on the predicted selected expression image and the reference selected expression image to obtain a mouth consistency loss;

[0028] perform affine transformation loss calculation on the predicted selected expression image and the reference selected expression image to obtain an affine loss value;

[0029] perform key point loss calculation on the predicted selected expression image to obtain a key point loss value;

[0030] perform face loss calculation on the predicted selected expression image to obtain a face loss value;

[0031] perform head loss calculation on the predicted selected expression image to obtain a head loss value;

[0032] perform aggregate calculation according to the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value and the head loss value to obtain the training loss value.

[0033] In some embodiments, the facial muscle key point screening according to the expression of the reference object of the reference selected expression image on the expression of the target object of the target initial expression image obtains a target original expression image, including:

[0034] perform facial muscle key point feature extraction on the expression of the reference object of the reference selected expression image to obtain a reference selected key point feature;

[0035] perform facial muscle key point feature extraction on the expression of the target object of the target initial expression image to obtain a target original key point feature;

[0036] According to the reference selected key point feature, the target original key point feature is subjected to Euclidean distance screening to obtain a target key point feature.

[0037] According to the target key point feature, the target initial expression image is screened to obtain the target original expression image.

[0038] In some embodiments, the expression alignment according to the target original expression feature and the reference selected expression feature comprises:

[0039] According to the target original expression feature and the reference selected expression feature, a scaling coefficient is calculated to obtain a target scaling coefficient.

[0040] According to the target scaling coefficient, the reference selected expression feature is subjected to aggregation calculation to obtain the target aligned expression feature.

[0041] In some embodiments, the image synthesis according to the target original face feature and the target aligned three-dimensional key point feature comprises:

[0042] The target original face feature and the target aligned three-dimensional key point feature are subjected to three-dimensional rendering to obtain a pixel distribution and a density distribution.

[0043] According to the pixel distribution and the density distribution, the target object is subjected to volume rendering to obtain the target selected expression image of the target object.

[0044] To achieve the above object, a second aspect of the embodiment of the present application proposes an image synthesis device, which comprises:

[0045] An acquisition data module is configured to acquire a reference selected expression image of a reference object and a target initial expression image of a target object, wherein the reference selected expression image has an expression of the reference object, the target initial expression image has an expression of the target object, and the reference object and the target object are not the same;

[0046] An image screening module is configured to perform facial muscle key point screening on the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image to obtain a target original expression image.

[0047] An expression extraction module is configured to perform expression feature extraction on the reference selected expression image based on a preset target expression transfer model to obtain a reference selected expression feature.

[0048] The feature extraction module is configured to perform feature extraction on the target original expression image based on the target expression transfer model, to obtain a target original expression feature, a target original face feature, and a target original head feature;

[0049] The expression alignment module is configured to perform expression alignment on the reference selected expression feature according to the target original expression feature, to obtain a target aligned expression feature.

[0050] The key point generation module is configured to perform three-dimensional key point generation on the target original head feature and the target aligned expression feature based on the target expression transfer model, to obtain a target aligned three-dimensional key point feature.

[0051] The image synthesis module is configured to perform image synthesis on the target object according to the target original face feature and the target aligned three-dimensional key point feature, to obtain a target selected expression image of the target object.

[0052] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0053] To achieve the above object, a fourth aspect of embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0054] The image synthesis method and device, electronic equipment and storage medium provided by the present application obtain a reference selected expression image having an expression of a reference object, obtain a target initial expression image having an expression of a target object, and the reference object and the target object are not the same, then perform facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image, to obtain a target original expression image, so as to realize the facial muscle key point screening on the expression of the target object in the target initial expression image, and obtain the target original expression image similar to the facial muscle key point of the reference selected expression image in the target initial expression image, so as to provide image data for subsequent cross-person expression migration, avoid cross-person expression migration on the image with a large difference in facial muscle key point, and improve the expression naturalness of the image obtained through expression migration. Further, the expression feature of the reference selected expression image is extracted based on a preset target expression migration model to obtain a reference selected expression feature, then the migration feature of the target original expression image is extracted based on the target expression migration model to obtain a target original expression feature, a target original face feature and a target original head feature, and finally the expression alignment is performed on the reference selected expression feature based on the target original expression feature to obtain a target aligned expression feature, so as to realize cross-person expression alignment and avoid the situation that the expression does not conform to the facial feature during expression migration. Further, the three-dimensional key point generation is performed on the target original head feature and the target aligned expression feature based on the target expression migration model to obtain a target aligned three-dimensional key point feature, then the image synthesis is performed on the target object based on the target original face feature and the target aligned three-dimensional key point feature to obtain a target selected expression image of the target object, so as to realize the generation of a more natural and realistic image. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of the image synthesis method provided by the embodiment of the present application;

[0056] Figure 2 is a flowchart of the image synthesis method provided by another embodiment of the present application;

[0057] Figure 3 is a flowchart of step S202 in Figure 2 ;

[0058] Figure 4 is a flowchart of step S203 in Figure 2 ;

[0059] Figure 5 is a flowchart of step S102 in Figure 1 ;

[0060] Figure 6 is a flowchart of step S105 in Figure 1 ;

[0061] Figure 7 is Figure 1 the flowchart of step S107 in

[0062] Figure 8 is a structural schematic diagram of an image synthesis device provided by an embodiment of the present application;

[0063] Figure 9 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0065] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0067] First, several terms involved in the present application are analyzed:

[0068] Facial Expression Transfer: Facial expression transfer is a technique in the field of computer vision and artificial intelligence, mainly used to transfer a specific facial expression from one person to another. Facial expression transfer technology analyzes the expression features of the source person, such as the dynamic changes of eyes, mouth and eyebrows, and applies these dynamic features to the target person, thereby realizing the natural conversion of expressions. This process usually involves deep learning models, especially generative adversarial networks, which can capture and simulate the complexity and subtle changes of facial expressions.

[0069] Three-Dimensional Keypoints (3D Keypoints): Three-dimensional keypoints are an important concept in computer vision and artificial intelligence, used to mark and track specific points of objects in three-dimensional space. These points typically represent significant features of an object, such as the corners of the eyes, the tip of the nose, and the corners of the mouth on a human face, or the joint positions of robots and other objects. The extraction of three-dimensional keypoints is crucial for performing a variety of tasks, including three-dimensional reconstruction, motion capture, pose estimation, and augmented reality. Through accurate keypoint positioning, systems can understand and analyze the spatial structure and motion of objects.

[0070] Facial Expression Features: Facial expression features are a technology in the field of computer vision and artificial intelligence that focuses on identifying and analyzing the dynamic changes in various parts of the face from facial images or videos, which reflect the emotional state of a person. It is a branch of artificial intelligence and an interdisciplinary subject of computer science and image processing. By accurately identifying the positions and movements of facial key points such as eyes, eyebrows, mouth, and nose, facial expression features help machines understand human emotions and reactions. This technology is widely used in fields such as emotional analysis, human-computer interaction, and security systems. For example, in emotional analysis applications, by analyzing the facial expressions of users, the system can identify the emotional state of the user, thereby providing more personalized services; in human-computer interaction, facial expression features enable robots or virtual assistants to more naturally understand and respond to the emotions and needs of human users. By simulating and expanding human understanding of emotions, facial expression feature technology plays an important role in enhancing the naturalness and depth of computer-human interaction.

[0071] Head Features: Head features refer to key parameters used to describe and analyze information such as the position, movement, and posture of a person's head in audio-driven portrait video applications. Head features capture dynamic information such as head movement trajectories, angle changes such as pitch, yaw, and roll angles, and facial expressions.

[0072] Expression transfer refers to the extraction and application of a specific object's expression to a specific target object, allowing the target object to change its expression while maintaining the facial features and head posture unchanged. For example, in insurance product promotion, multiple different speakers can be generated to introduce the same product through expression transfer technology, so that the user can be recommended to watch the video of the speaker explaining the product that suits the user. However, existing expression transfer technology is usually suitable for changing the expression of the same person, and when transferring expressions across different people, the generated images often lack naturalness. Therefore, how to generate more natural and realistic images during expression transfer has become a problem to be solved.

[0073] Based on this, the embodiment of the application provides an image synthesis method and device, electronic equipment and storage medium, aiming to generate more natural and realistic images.

[0074] The image synthesis method and device, electronic equipment and storage medium provided by the embodiment of the application are specifically described through the following embodiments. First, the image synthesis method in the embodiment of the application is described.

[0075] The embodiment of the application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0076] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0077] The image synthesis method provided by the embodiment of the application relates to the field of image processing and is suitable for the field of financial technology. The image synthesis method provided by the embodiment of the application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN and basic cloud computing services such as big data and artificial intelligence platform; and the software can be an application for implementing the image synthesis method, but is not limited to the above forms.

[0078] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0079] It should be noted that in each specific embodiment of the present application, when it is necessary to perform relevant processing according to user information, user behavior data, user history data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to operate normally will be obtained.

[0080] Figure 1 is an optional flowchart of an image synthesis method provided by the embodiments of the present application, Figure 1 The method in the above step S101-S107 can include but is not limited to the steps S101-S107.

[0081] In step S101, a reference selected expression image of a reference object is obtained, and a target initial expression image of a target object is obtained; wherein the reference selected expression image has an expression of the reference object, the target initial expression image has an expression of the target object, and the reference object and the target object are not the same;

[0082] In step S102, facial muscle key point screening is performed on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image, to obtain a target original expression image;

[0083] In step S103, expression feature extraction is performed on the reference selected expression image based on a preset target expression migration model, to obtain reference selected expression features;

[0084] Step S104, performing migration feature extraction on the target original expression image based on the target expression migration model to obtain a target original expression feature, a target original face feature, and a target original head feature;

[0085] Step S105, performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain a target aligned expression feature;

[0086] Step S106, performing three-dimensional key point generation on the target original head feature and the target aligned expression feature based on the target expression migration model to obtain a target aligned three-dimensional key point feature;

[0087] Step S107, performing image synthesis on the target object according to the target original face feature and the target aligned three-dimensional key point feature to obtain a target selected expression image of the target object.

[0088] The steps S101 to S107 shown in the embodiments of the present application, by obtaining a reference selected expression image having an expression of a reference object, obtaining a target initial expression image having an expression of a target object, and the reference object and the target object being different, then performing facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image to obtain a target original expression image, thereby realizing facial muscle key point screening on the expression of the target object in the target initial expression image, obtaining a target original expression image similar to the facial muscle key point of the reference selected expression image in the target initial expression image, thereby providing image data for subsequent cross-person expression migration, avoiding cross-person expression migration on images with large differences in facial muscle key points, and improving the expression naturalness of the images obtained by expression migration; further, performing expression feature extraction on the reference selected expression image based on a preset target expression migration model to obtain a reference selected expression feature, then performing migration feature extraction on the target original expression image based on the target expression migration model to obtain a target original expression feature, a target original face feature, and a target original head feature, and finally performing expression alignment on the reference selected expression feature according to the target original expression feature to obtain a target aligned expression feature; thereby realizing cross-person expression alignment and avoiding the case that the expression does not conform to the facial feature during expression migration; further, performing three-dimensional key point generation on the target original head feature and the target aligned expression feature based on the target expression migration model to obtain a target aligned three-dimensional key point feature, then performing image synthesis on the target object according to the target original face feature and the target aligned three-dimensional key point feature to obtain a target selected expression image of the target object, and realizing generation of more natural and realistic images.

[0089] Please refer to Figure 2In some embodiments, the image synthesis method further comprises, before step S101, pre-training a target expression transfer model, which can specifically include but is not limited to steps S201 to S204:

[0090] In step S201, a reference original expression image of a reference object is obtained.

[0091] In step S202, expression transfer is performed on the reference original expression image based on a preset expression transfer model and a reference selected expression image, to obtain a predicted selected expression image.

[0092] In step S203, a training loss value is obtained by performing loss value calculation on the reference original expression image, the predicted selected expression image, and the reference selected expression image.

[0093] In step S204, the preset expression transfer model is parameter-optimized based on the training loss value, to obtain a target expression transfer model.

[0094] The steps S201 to S204 shown in the embodiments of the present application obtain a reference original expression image of a reference object, then perform expression transfer on the reference original expression image based on a preset expression transfer model and a reference selected expression image, to obtain a predicted selected expression image, then perform loss value calculation on the reference original expression image, the predicted selected expression image, and the reference selected expression image, to obtain a training loss value, and finally parameter-optimize the preset expression transfer model based on the training loss value, to obtain a target expression transfer model, so that the target expression transfer model can perform accurate expression transfer within the same person, and the image after expression transfer is more natural and realistic in expression presentation and overall quality.

[0095] In step S201 of some embodiments, the reference object refers to a specific individual selected to provide facial features, expression features, and head morphology, and is a facial image of the specific individual. The reference original expression image is an image of the reference object, which is used to show the expression state of the reference object and records the morphology of the face and head of the reference object, such as images of various expressions such as smiling, frowning, and calm.

[0096] Please refer to Figure 3 In some embodiments, step S202 can include but is not limited to steps S301 to S306:

[0097] In step S301, reference selected expression features, reference selected face features, reference selected head features, and reference selected three-dimensional key point features are obtained by performing migration feature extraction on the reference selected expression image based on the preset expression transfer model.

[0098] Step S302, performing migration feature extraction on the reference original expression image based on the preset expression migration model to obtain reference original expression features, reference original head features, and reference original three-dimensional key point features;

[0099] Step S303, performing mask generation on the reference selected expression image and the reference original expression image based on the preset expression migration model to obtain a face motion mask matrix;

[0100] Step S304, performing motion field calculation according to the reference selected expression features, the reference selected head features, the reference selected three-dimensional key point features, the reference original expression features, the reference original head features, and the reference original three-dimensional key point features to obtain a three-dimensional motion field matrix;

[0101] Step S305, performing aggregation calculation according to the three-dimensional motion field matrix and the face motion mask matrix to obtain a transformed face feature;

[0102] Step S306, performing feature decoding on the transformed face feature and the reference selected face feature based on the preset expression migration model to obtain a predicted selected expression image.

[0103] The steps S301 to S306 shown in the embodiments of the present application perform migration feature extraction on the reference selected expression image and the reference original expression image based on the preset expression migration model to obtain reference selected expression features, reference selected face features, reference selected head features, reference selected three-dimensional key point features, reference original expression features, reference original head features, and reference original three-dimensional key point features. Then, mask generation is performed on the reference selected expression image and the reference original expression image based on the preset expression migration model to obtain a face motion mask matrix, and motion field calculation is performed according to the reference selected expression features, the reference selected head features, the reference selected three-dimensional key point features, the reference original expression features, the reference original head features, and the reference original three-dimensional key point features to obtain a three-dimensional motion field matrix. Further, aggregation calculation is performed according to the three-dimensional motion field matrix and the face motion mask matrix to obtain a transformed face feature, and then feature decoding is performed on the transformed face feature and the reference selected face feature based on the preset expression migration model to obtain a predicted selected expression image, so as to realize expression migration within the same person, extract the expression in the reference selected expression image, and migrate the expression to the reference original expression image to obtain a predicted selected expression image.

[0104] In step S301 of some embodiments, the preset expression transfer model is a neural network model. In one embodiment, the preset expression transfer model is a GAN neural network model. The GAN neural network model includes a feature extraction module, a generator, and a discriminator. The feature extraction module includes a face feature extraction module, a head feature extraction module, an expression feature extraction module, a three-dimensional key point feature extraction module, and a mask estimation module.

[0105] The selected reference facial expression image is input into a preset facial expression transfer model. The facial feature extraction module extracts facial features from the selected reference facial expression image to obtain the selected reference facial features. The head feature extraction module extracts head features from the selected reference facial expression image to obtain the selected reference head features. The facial expression feature extraction module extracts facial expression features from the selected reference facial expression image to obtain the selected reference facial features. Finally, the 3D keypoint feature extraction module extracts 3D keypoint features from the selected head features and the selected reference facial expression features to obtain the selected reference keypoint features.

[0106] It should be noted that directly using the 3D key point feature extraction module to extract 3D key point features from the selected reference expression image yields poor results. However, by using the 3D key point feature extraction module to extract 3D key point features from the selected reference head features and selected reference expression features, the 3D key point features can be obtained accurately.

[0107] In step S302 of some embodiments, the principle of extracting transfer features from the reference original expression image based on the preset expression transfer model is similar to that of extracting transfer features from the reference selected expression image based on the preset expression transfer model, and will not be repeated here.

[0108] In step S303 of some embodiments, a face motion mask matrix is ​​generated by generating a mask for a reference selected expression image and a reference original expression image based on a preset expression transfer model. Specifically, the reference selected expression image and the reference original expression image are input into the mask estimation module in the preset expression transfer model to obtain the face motion mask matrix, as shown in equation (1):

[0109] M = MaskEstimator(I) s ,I d (1),

[0110] MaskEstimator is the mask estimation module, and the face motion mask matrix is ​​M, I s To select a facial expression image for reference, I d For reference, the original facial expression image is used.

[0111] In step S304 of some embodiments, a flow field matrix is obtained by performing flow field calculation according to the reference selected expression feature, the reference selected head feature, the reference selected three-dimensional key point feature, the reference original expression feature, the reference original head feature, and the reference original three-dimensional key point feature, and the flow field calculation is shown in formula (2):

[0112]

[0113] wherein, Flow is the flow field matrix, K d is the reference original three-dimensional key point feature, K s is the reference selected three-dimensional key point feature, K() is a three-dimensional key point feature extraction module, H d is the reference original head feature, E d is the reference original expression feature, K s is the reference selected three-dimensional key point feature, H s is the reference selected head feature, E s is the reference original expression feature.

[0114] In step S305 of some embodiments, an aggregated calculation is performed according to the flow field matrix and a face motion mask matrix to obtain a transformed face feature, and the aggregated calculation is shown in formula (3):

[0115] Flow masked = M Flow (3),

[0116] wherein, Flow masked is the transformed face feature, and is an element-wise multiplication.

[0117] In step S306 of some embodiments, a feature decoding is performed on the transformed face feature and the reference selected face feature based on a preset expression transfer model to obtain a predicted selected expression image, specifically, the transformed face feature and the reference selected face feature are input into a generator in the preset expression transfer model to obtain the predicted selected expression image.

[0118] Please refer to Figure 4 In some embodiments, step S203 can include, but is not limited to, steps S401 to S408:

[0119] Step S401, a perceptual loss calculation is performed on the predicted selected expression image and the reference selected expression image to obtain a perceptual loss value;

[0120] Step S402, an adversarial loss calculation is performed on the predicted selected expression image and the reference selected expression image to obtain an adversarial loss value;

[0121] Step S403, mouth consistency loss calculation is performed on the predicted selected expression image and the reference selected expression image to obtain a mouth consistency loss;

[0122] Step S404, affine transformation loss calculation is performed on the predicted selected expression image and the reference selected expression image to obtain an affine loss value;

[0123] Step S405, key point loss calculation is performed on the predicted selected expression image to obtain a key point loss value;

[0124] Step S406, face loss calculation is performed on the predicted selected expression image to obtain a face loss value;

[0125] Step S407, head loss calculation is performed on the predicted selected expression image to obtain a head loss value;

[0126] Step S408, the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value and the head loss value are aggregated to obtain a training loss value.

[0127] The steps S401 to S408 shown in the embodiments of the present application perform perceptual loss calculation, adversarial loss calculation, mouth consistency loss calculation and affine transformation loss calculation on the predicted selected expression image and the reference selected expression image, and perform key point loss calculation, key point loss calculation, face loss calculation and head loss calculation on the predicted selected expression image. Finally, all the loss values obtained are aggregated to obtain the final training loss value, so as to train the subsequent preset expression transfer model from multiple aspects and multiple dimensions, and improve the expression transfer performance of the preset expression transfer model.

[0128] In steps S401 to S408 of some embodiments, the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value and the head loss value are aggregated to obtain a training loss value, and the aggregation calculation is shown in formula (4):

[0129] L = L perc + L Gan + L mouth + L equiv + L kp-prior + L def-prior + L head (4),

[0130] Wherein, L is the training loss value, L percTo perceive the loss value, the middle feature layer of the pre-trained VGG network is used, the predicted selected expression image is input into the middle feature layer of the pre-trained VGG network to obtain the perception data of the predicted selected expression image, then the reference selected expression image is input into the middle feature layer of the pre-trained VGG network to obtain the perception data of the reference selected expression image, and then the difference between the perception data of the predicted selected expression image and the perception data of the reference selected expression image is calculated to obtain the perception loss value. Gan To obtain the adversarial loss value, the discriminator in the preset expression transfer model is used, and the implementation principle of the adversarial loss calculation is similar to that of the perception loss calculation, which will not be repeated here. mouth To obtain the mouth consistency loss, the mouth feature extraction module of the pre-trained SyncNet is used, and the implementation principle of the mouth consistency loss calculation is similar to that of the perception loss calculation, which will not be repeated here. equiv To obtain the affine loss value, the preset affine transformation function is used, and the implementation principle of the affine transformation loss calculation is similar to that of the perception loss calculation, which will not be repeated here.

[0131] L kp-prior To obtain the key point loss value, the preset face key point extraction network is used, and the key point loss is shown in formula (5):

[0132]

[0133] Wherein, i and j are indexes of the face key points of the predicted selected expression image, K i is the i-th key point in the predicted selected expression image, β is a preset weight, exp() is a preset exponential function, and the key point loss value is used to avoid the face key points in the predicted selected expression image from being too concentrated.

[0134] L def-prior To obtain the face loss value, the face feature extraction module in the preset expression transfer model is used, and the face loss is shown in formula (6):

[0135]

[0136] Wherein, I predict is the predicted selected expression image, E() is the face feature extraction module in the preset expression transfer model, the predicted selected expression image is first input into the face feature extraction module in the preset expression transfer model to obtain the predicted selected face feature E(I predict 01, and then the first two dimensions are set to 0 to obtain the predicted target face feature E(I predict )2.

[0137] L headThe head loss value is obtained by inputting the predicted selected expression image into the pre-trained head pose model.

[0138] In step S204 of some embodiments, the preset expression transfer model is parameter-optimized according to the training loss value. Specifically, the model parameters in the preset expression transfer model are optimized by random gradient descent according to the training loss value, to obtain a target expression transfer model capable of expression transfer within the same person.

[0139] In step S101 of some embodiments, the target object and the reference object are not the same object, the target initial expression image includes multiple expression images of the target object, the reference selected image includes expression images of the reference object, and the facial muscles of the expressions in the target initial expression image and the reference selected image are not completely the same.

[0140] Please refer to Figure 5 In some embodiments, step S102 includes but is not limited to steps S501 to S504:

[0141] Step S501: facial muscle key point feature extraction is performed on the expression of the reference object in the reference selected expression image, to obtain reference selected key point features;

[0142] Step S502: facial muscle key point feature extraction is performed on the expression of the target object in the target initial expression image, to obtain target original key point features;

[0143] Step S503: Euclidean distance screening is performed on the target original key point features according to the reference selected key point features, to obtain target key point features;

[0144] Step S504: screening is performed on the target initial expression image according to the target key point features, to obtain a target original expression image.

[0145] The steps S501 to S504 shown in the embodiments of the present application are as follows: facial muscle key point features of the expression of the reference object in the reference selected expression image are extracted, reference selected key point features are obtained, facial muscle key point features of the expression of the target object in the target initial expression image are extracted, target original key point features are obtained, the target original key point features are filtered according to the reference selected key point features, target key point features are obtained, and the target initial expression image is filtered according to the target key point features, so that the target original expression image is obtained. In this way, the facial muscle key points of the expression of the target object in the target initial expression image are filtered, the target original expression image similar to the facial muscle key points of the reference selected expression image is obtained, and image data is provided for subsequent cross-person expression migration, so that cross-person expression migration is avoided for images with large differences in facial muscle key points, and the expression naturalness of the image obtained through expression migration is improved.

[0146] In step S501 of some embodiments, the facial muscle key point is a facial key point of the expression of the reference object in the reference selected expression image. A pre-trained facial action unit (AU) network model is used to extract facial AU features. The reference selected expression image is input into the facial action unit to obtain reference selected key point features.

[0147] In step S502 of some embodiments, the facial muscle key point is a facial key point of the expression of the target object in the target selected expression image. A pre-trained facial action unit (AU) network model is used to extract facial AU features. The target selected expression image is input into the facial action unit to obtain target selected key point features.

[0148] In step S503 of some embodiments, the target distance value is obtained by performing Euclidean distance calculation according to the target selected key point features and the reference selected key point features. When the target distance value is less than a preset distance threshold, the corresponding target selected key point is taken as a target key point feature.

[0149] In step S504 of some embodiments, the target initial expression image corresponding to the target key point feature is obtained to obtain a target original expression image.

[0150] In step S103 of some embodiments, the principle of extracting expression features of the reference selected expression image based on the preset target expression migration model is similar to that of extracting migration features of the reference selected expression image based on the preset expression migration model, which will not be described here.

[0151] In step S104 of some embodiments, the target original expression image is subjected to migration feature extraction based on the target expression migration model, to obtain target original expression features, target original face features and target original head features. The principle is similar to that of migration feature extraction of the reference selected expression image based on the preset expression migration model, and will not be described here.

[0152] Please refer to Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S602:

[0153] In step S601, scaling factor calculation is performed according to the target original expression features and the reference selected expression features, to obtain target scaling factors.

[0154] In step S602, aggregation calculation is performed on the reference selected expression features according to the target scaling factors, to obtain target aligned expression features.

[0155] The steps S601 to S602 shown in the embodiments of the present application realize cross-person expression alignment by performing scaling factor calculation according to the target original expression features and the reference selected expression features to obtain target scaling factors, and then performing aggregation calculation on the reference selected expression features according to the target scaling factors to obtain target aligned expression features.

[0156] In step S601 of some embodiments, scaling factor calculation is performed according to the target original expression features and the reference selected expression features, to obtain target scaling factors. The scaling factor calculation is shown in formula (7):

[0157]

[0158] Where s is the target scaling factor, E d is the reference selected expression feature, m is the number of images of the target original expression image, E r is the target original expression feature, i is the index of the target original expression image, is the target original expression feature of the i-th target original expression image.

[0159] In step S602 of some embodiments, aggregation calculation is performed on the reference selected expression features according to the target scaling factors, to obtain target aligned expression features. The aggregation calculation is shown in formula (8):

[0160] E transform = s·E d (8),

[0161] Where E transform is the target aligned expression feature.

[0162] In step S106 of some embodiments, the target original head feature and the target aligned expression feature are subjected to three-dimensional key point generation based on a target expression transfer model to obtain a target aligned three-dimensional key point feature. Specifically, the target original head feature and the target aligned expression feature are input into a three-dimensional key point feature extraction module in the target expression transfer model to obtain the target aligned three-dimensional key point feature.

[0163] Please refer to Figure 7 In some embodiments, step S107 can include but is not limited to steps S701 to S702:

[0164] In step S701, the target original face feature and the target aligned three-dimensional key point feature are subjected to three-dimensional rendering to obtain a pixel distribution and a density distribution.

[0165] In step S702, the target object is subjected to volume rendering according to the pixel distribution and the density distribution to obtain a target selected expression image of the target object.

[0166] The steps S701 to S702 shown in the embodiments of the present application, by subjecting the target original face feature and the target aligned three-dimensional key point feature to three-dimensional rendering to obtain a pixel distribution and a density distribution, and then subjecting the target object to volume rendering according to the pixel distribution and the density distribution to obtain a target selected expression image of the target object, finally realize cross-person expression transfer to obtain a target selected expression image of the target object.

[0167] In step S701 of some embodiments, the target original face feature and the target aligned three-dimensional key point feature are subjected to three-dimensional rendering to obtain a pixel distribution and a density distribution. In one embodiment, the target original face feature and the target aligned three-dimensional key point feature are input into a pre-trained NeRF model to obtain the pixel distribution and the density distribution.

[0168] In step S702 of some embodiments, the target object is subjected to volume rendering according to the pixel distribution and the density distribution to obtain a target selected expression image of the target object. The volume rendering is shown in formula (9):

[0169]

[0170] wherein NeRF is a pre-trained NeRF model, o is a preset position of a camera, i.e. the starting point of a ray, d is a preset unit vector of a line of sight, indicating the direction from the camera to a certain point in the scene, z is the depth of the ray, r(z) represents a three-dimensional space point on the ray with a depth of z, k is the target aligned three-dimensional key point feature, f is the target original face feature, I target is a target selected expression image, c(z) is a pixel distribution, σ(z) is a density distribution, z n is a preset near-end depth, zf T(z) is the cumulative transmittance from the ray origin z to depth z, representing the probability that a ray has not been completely absorbed before reaching depth z, and s is an intermediate variable between z and z n n to depth z, representing the probability that a ray has not been completely absorbed before reaching depth z, and s is an intermediate variable between z and z

[0171] Please refer to Figure 8 The embodiment of the present application also provides an image synthesis device, which can implement the above image synthesis method, and the device comprises:

[0172] An acquisition data module 801 is configured to acquire a reference selected expression image of a reference object and a target initial expression image of a target object, wherein the reference selected expression image has an expression of the reference object, the target initial expression image has an expression of the target object, and the reference object and the target object are different;

[0173] An image screening module 802 is configured to perform facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image, to obtain a target original expression image;

[0174] An expression extraction module 803 is configured to perform expression feature extraction on the reference selected expression image based on a preset target expression transfer model, to obtain reference selected expression features;

[0175] A feature extraction module 804 is configured to perform transfer feature extraction on the target original expression image based on the target expression transfer model, to obtain target original expression features, target original facial features and target original head features;

[0176] An expression alignment module 805 is configured to perform expression alignment on the reference selected expression features according to the target original expression features, to obtain target aligned expression features;

[0177] A key point generation module 806 is configured to perform three-dimensional key point generation on the target original head features and the target aligned expression features based on the target expression transfer model, to obtain target aligned three-dimensional key point features;

[0178] An image synthesis module 807 is configured to perform image synthesis on the target object according to the target original facial features and the target aligned three-dimensional key point features, to obtain a target selected expression image of the target object.

[0179] The specific implementation of the image synthesis device is basically the same as the specific embodiment of the above image synthesis method, and will not be repeated here.

[0180] ​The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the image synthesis method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0181] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0182] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0183] The memory 902 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the image synthesis method of the embodiments of the present application.

[0184] The input / output interface 903 is used to realize information input and output.

[0185] The communication interface 904 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0186] The bus 905 is used to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0187] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize communication connection between them in the device.

[0188] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image synthesis method.

[0189] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The image synthesis method, image synthesis device, electronic equipment and storage medium provided by the embodiments of the present application obtain a reference selected expression image having an expression of a reference object, obtain a target initial expression image having an expression of a target object, and the reference object and the target object are not the same, then perform facial muscle key point screening on the expression of the target object of the target initial expression image according to the expression of the reference object of the reference selected expression image, to obtain a target original expression image, thereby realizing screening of the facial muscle key points of the expression of the target object in the target initial expression image, obtaining the target original expression image similar to the facial muscle key points of the reference selected expression image, thereby providing image data for subsequent cross-person expression migration, avoiding cross-person expression migration of images with large differences in facial muscle key points, and improving the expression naturalness of the images obtained by expression migration; further, expression features of the reference selected expression image are extracted based on a preset target expression migration model to obtain reference selected expression features, then migration features of the target original expression image are extracted based on the target expression migration model to obtain target original expression features, target original face features and target original head features, and finally, the reference selected expression features are aligned according to the target original expression features to obtain target aligned expression features; thereby realizing cross-person expression alignment and avoiding the case that the expression does not conform to the facial features during expression migration; further, three-dimensional key points are generated based on the target expression migration model for the target original head features and the target aligned expression features to obtain target aligned three-dimensional key point features, and then the target object is synthesized according to the target original face features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object, thereby realizing generation of a more natural and realistic image.

[0191] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0192] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0194] Those skilled in the art can understand that all or some steps in the above disclosed method, functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0195] The terms "first", "second", "third", "fourth" and the like in the description of the present application and the above-mentioned figures (if any) are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0197] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0199] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0200] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0201] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. An image compositing method characterized by, The method comprises: obtaining a reference selected expression image of a reference object and an initial expression image of a target object; wherein the reference selected expression image has an expression of the reference object, and the initial expression image of the target object has an expression of the target object, and the reference object and the target object are not the same; performing facial muscle key point screening on the expression of the target object in the initial expression image of the target object according to the expression of the reference object in the reference selected expression image to obtain a target original expression image; extracting expression features from the reference selected expression image based on a preset target expression transfer model to obtain reference selected expression features; extracting transfer features from the target original expression image based on the target expression transfer model to obtain target original expression features, target original face features and target original head features; aligning the target original expression features with the reference selected expression features to obtain target aligned expression features; generating three-dimensional key points based on the target original head features and the target aligned expression features to obtain target aligned three-dimensional key point features based on the target expression transfer model; performing image synthesis on the target object according to the target original face features and the target aligned three-dimensional key point features to obtain a target selected expression image of the target object; The method further comprises pre-training the target expression transfer model, specifically comprising: obtaining a reference original expression image of a reference object; transferring expressions from the reference original expression image based on a preset expression transfer model and the reference selected expression image to obtain a predicted selected expression image; calculating a loss value according to the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain a training loss value; optimizing parameters of the preset expression transfer model according to the training loss value to obtain the target expression transfer model.

2. The method of claim 1, wherein, The method further comprises pre-training the target expression transfer model, specifically comprising: extracting transfer features from the reference selected expression image based on the preset expression transfer model to obtain reference selected expression features, reference selected face features, reference selected head features and reference selected three-dimensional key point features; ​ ​ ​ 3. The method of claim 2, wherein, ​ ​ extracting a reference original expression feature, a reference original head feature and a reference original three-dimensional key point feature from the reference original expression image based on the preset expression migration model; generating a face motion mask matrix based on the preset expression migration model and the reference selected expression image and the reference original expression image; calculating a three-dimensional motion field matrix based on the reference selected expression feature, the reference selected head feature, the reference selected three-dimensional key point feature, the reference original expression feature, the reference original head feature and the reference original three-dimensional key point feature; calculating a transformation face feature based on the three-dimensional motion field matrix and the face motion mask matrix; decoding the transformation face feature and the reference selected face feature based on the preset expression migration model to obtain the predicted selected expression image.

4. The method of claim 2, wherein, The loss value calculation based on the reference original expression image, the predicted selected expression image and the reference selected expression image to obtain the training loss value, comprising: calculating a perceptual loss value based on the predicted selected expression image and the reference selected expression image; calculating an adversarial loss value based on the predicted selected expression image and the reference selected expression image; calculating a mouth consistency loss based on the predicted selected expression image and the reference selected expression image; calculating an affine loss value based on the predicted selected expression image; calculating a key point loss value based on the predicted selected expression image; calculating a face loss value based on the predicted selected expression image; calculating a head loss value based on the predicted selected expression image; calculating the training loss value based on the perceptual loss value, the adversarial loss value, the mouth consistency loss, the affine loss value, the key point loss value, the face loss value and the head loss value.

5. The method of claim 1, wherein, The expression alignment of the reference selected expression feature based on the target original expression feature to obtain a target aligned expression feature, comprising: calculating a target scaling coefficient based on the target original expression feature and the reference selected expression feature; calculating the target aligned expression feature based on the target scaling coefficient and the reference selected expression feature.

6. The method of claim 1, wherein, The image synthesis of the target object based on the target original face feature and the target aligned three-dimensional key point feature to obtain a target selected expression image of the target object, comprising: performing three-dimensional rendering on the target original face feature and the target aligned three-dimensional key point feature to obtain a pixel distribution and a density distribution; performing volume rendering on the target object based on the pixel distribution and the density distribution to obtain the target selected expression image of the target object.

7. An image synthesizing apparatus characterized by comprising: The device comprises: The acquisition data module is used for acquiring a reference selected expression image of a reference object and a target initial expression image of a target object; wherein the reference selected expression image has an expression of the reference object, the target initial expression image has an expression of the target object, and the reference object and the target object are not the same; The image screening module is used for performing facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image, to obtain a target original expression image; The expression extraction module is used for performing expression feature extraction on the reference selected expression image based on a preset target expression transfer model, to obtain reference selected expression features; The feature extraction module is used for performing transfer feature extraction on the target original expression image based on the target expression transfer model, to obtain target original expression features, target original face features and target original head features; The expression alignment module is used for performing expression alignment on the reference selected expression features according to the target original expression features, to obtain target aligned expression features; The key point generation module is used for performing three-dimensional key point generation on the target original head features and the target aligned expression features based on the target expression transfer model, to obtain target aligned three-dimensional key point features; The image synthesis module is used for performing image synthesis on the target object according to the target original face features and the target aligned three-dimensional key point features, to obtain a target selected expression image of the target object; The facial muscle key point screening on the expression of the target object in the target initial expression image according to the expression of the reference object in the reference selected expression image to obtain a target original expression image includes: Performing facial muscle key point feature extraction on the expression of the reference object in the reference selected expression image to obtain reference selected key point features; Performing facial muscle key point feature extraction on the expression of the target object in the target initial expression image to obtain target original key point features; Performing Euclidean distance screening on the target original key point features according to the reference selected key point features to obtain target key point features; Performing screening on the target initial expression image according to the target key point features to obtain the target original expression image.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image synthesis method of any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the image synthesis method of any one of claims 1-6.

Images