Expression migration method and device, expression model training method and device, equipment and storage medium

Abstract

The invention provides expression migration and model training methods and devices, equipment and a storage medium, and relates to the technical field of expression migration. The method comprises the following steps: acquiring expression features of a sample reference expression image and identity features of a target virtual face image; generating a sample virtual face expression image according to the expression features and the identity features; according to the sample virtual face expression image and the sample reference expression image, calculating a loss function value by adopting a calculation mode corresponding to the face type of the sample reference expression image; the face type is a virtual face or a real face; and performing model training according to the loss function value to obtain a target expression migration model of the target virtual face image. Compared with the prior art, the problem that expression migration cannot be accurately completed is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of expression migration, and in particular to an expression migration, model training method, device, equipment and storage medium. Background Art

[0002] Facial expression transfer technology refers to transferring the reference expression in the input image to the target face. Game facial expression transfer can be used for facial animation transfer / generation of game or animation characters, greatly improving the efficiency of game animation generation.

[0003] It has a very broad application prospect in the fields of human-computer interaction, film and television or animation production. In recent years, many domestic and foreign scholars have proposed many related technologies. With the development of deep learning, the expression transfer technology based on deep learning has also made great progress. Most of the existing technologies use traditional manual or professional equipment to capture faces. For example, it is manually produced by professionals and captured by professional motion capture equipment.

[0004] However, with this method of expression transfer, the accuracy of expression transfer cannot be guaranteed, and expression transfer cannot be completed accurately. Summary of the invention

[0005] The purpose of this application is to provide an expression transfer, model training method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem that expression transfer cannot be accurately completed in the prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for training an expression transfer model, the method comprising:

[0008] Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image;

[0009] Generating a sample virtual facial expression image according to the expression feature and the identity feature;

[0010] According to the sample virtual human face expression image and the sample reference expression image, a loss function value is calculated using a calculation method corresponding to a face type of the sample reference expression image; the face type is used for a virtual face or a real face;

[0011] Model training is performed according to the loss function value to obtain a target expression migration model of the target virtual face image.

[0012] In a second aspect, another embodiment of the present application provides an expression migration method, the method comprising:

[0013] Obtain target reference expression image;

[0014] The target expression transfer model of the target virtual facial image is adopted to generate a target virtual facial expression image according to the target reference expression image; wherein the target expression transfer model is trained by adopting any of the methods described in the first aspect above.

[0015] In a third aspect, another embodiment of the present application provides a training device for an expression transfer model, the device comprising: an acquisition module, a generation module, a calculation module and a training module, wherein:

[0016] The acquisition module is used to acquire the expression features of the sample reference expression image and the identity features of the target virtual face image;

[0017] The generating module is used to generate a sample virtual facial expression image according to the expression feature and the identity feature;

[0018] The calculation module is used to calculate the loss function value according to the sample virtual human face expression image and the sample reference expression image, using a calculation method corresponding to the face type of the sample reference expression image; the face type is for a virtual face or a real face;

[0019] The training module is used to perform model training according to the loss function value to obtain a target expression migration model of the target virtual face image.

[0020] In a fourth aspect, another embodiment of the present application provides an expression migration device, the device comprising: an acquisition module and a generation module, wherein:

[0021] The acquisition module is used to acquire a target reference expression image;

[0022] The generation module is used to adopt the target expression transfer model of the target virtual facial image and generate a target virtual facial expression image according to the target reference expression image; wherein the target expression transfer model is trained using any of the methods described in the first aspect above.

[0023] In the fifth aspect, another embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the method described in any one of the first aspect or the second aspect above.

[0024] In a sixth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspect or the second aspect are executed.

[0025] The beneficial effect of the present application is: by adopting the training method of the expression transfer model provided by the present application, after obtaining the expression features of the sample reference expression image and the identity features of the target facial image, a sample virtual facial expression image can be generated according to the expression features and the identity features, and then the loss function value is determined based on the sample virtual facial expression image and the sample reference expression image, and the model is trained based on the loss function value to obtain a target expression transfer model of the target virtual facial image; the target expression transfer model obtained in this way not only has the ability to transfer the facial expressions on the real facial expression image to the target virtual facial image, but also has the ability to transfer the facial expressions on the virtual facial expression features to the target virtual facial image, and it does not require human participation, greatly improves the training efficiency, and solves the problem that virtual facial expression features cannot be transferred in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A flowchart of a method for training an expression transfer model provided in one embodiment of the present application;

[0028] Figure 2 A flowchart of a method for training an expression transfer model provided in another embodiment of the present application;

[0029] Figure 3 A flowchart of a method for training an expression transfer model provided in another embodiment of the present application;

[0030] Figure 4 A flowchart of a method for training an expression transfer model provided in another embodiment of the present application;

[0031] Figure 5 A flowchart of an expression transfer method provided in another embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of a training device for an expression transfer model provided in one embodiment of the present application;

[0033] Figure 7 A schematic diagram of the structure of an expression transfer device provided in another embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0036] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0037] The training method of the expression transfer model in one embodiment of the present application can be run on a local terminal device or a server. When the training method of the expression transfer model is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0038] In an optional embodiment, various cloud applications can be run under the cloud interaction system, such as cloud games. Taking cloud games as an example, cloud games refer to a game mode based on cloud computing. In the operation mode of cloud games, the operating body of the game program and the main body of the game screen presentation are separated. The storage and operation of the training method of the expression transfer model are completed on the cloud game server. The role of the client device is used for receiving and sending data and presenting the game screen. For example, the client device can be a display device with data transmission function close to the player side, such as a mobile terminal, a TV, a computer, a handheld computer, etc.; but the cloud game server in the cloud is used for information processing. When playing the game, the player operates the client device to send an operation instruction to the cloud game server. The cloud game server runs the game according to the operation instruction, encodes and compresses the game screen and other data, and returns it to the client device through the network. Finally, the client device decodes and outputs the game screen.

[0039] In an optional embodiment, taking a game as an example, a local terminal device stores a game program and is used to present a game screen. The local terminal device is used to interact with the player through a graphical user interface, that is, the game program is downloaded and installed by an electronic device and run conventionally. The local terminal device may provide the graphical user interface to the player in a variety of ways, for example, it may be rendered and displayed on a display screen of the terminal, or provided to the player through a holographic projection. For example, the local terminal device may include a display screen and a processor, the display screen is used to present a graphical user interface, the graphical user interface includes a game screen, and the processor is used to run the game, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0040] In addition, the flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of this application, and can also remove one or more operations from the flowchart.

[0041] To facilitate understanding of this application, some terms involved in this application are explained as follows:

[0042] Optical flow field: Optical flow is the instantaneous speed of an object moving in space reflected as a pixel on the camera imaging plane (a vector containing size and direction). If the optical flow of each pixel on the phase plane is recorded, an optical flow field can be obtained. It is represented by the relative position of a specific pixel (such as the pixel at the corner of the mouth) in two frames of a video image sequence. For example, the position of the pixel at the corner of the mouth in the tth frame of the video is (x1, y1) in the image. When the person smiles in the t+1th frame, the position of the corner of the mouth changes, and the coordinates are (x2, y2). Then, the relative position difference (ux, vy) = (x2, y2) - (x1, y1) can be calculated to determine whether the pixel at the corner of the mouth has shifted, as well as the direction and magnitude of the displacement.

[0043] The following is an explanation of a training method for an expression transfer model provided in an embodiment of the present application in combination with a number of specific application examples. Figure 1 A flowchart of a method for training an expression transfer model provided in one embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0044] S101: Acquire expression features of a sample reference expression image and identity features of a target virtual face image.

[0045] In an embodiment of the present application, an expression transfer model is only an expression transfer model for a target virtual face, wherein the target virtual face can be, for example, a target cartoon face. If the user needs to transfer expressions to virtual faces other than the non-target virtual face, it is necessary to re-train the expression transfer model for the other virtual faces based on the other virtual faces, so that other virtual faces can be subsequently transferred based on the trained expression transfer model.

[0046] In one embodiment of the present application, the expression migration model includes: a first feature extraction module, a second feature extraction model module, and an expression generation module. That is, in the embodiment of the present application, the first feature extraction module can be used to extract the expression features of the sample reference expression image to obtain the expression features of the sample reference expression image; wherein the first feature extraction module can, for example, be a reference expression feature extraction module, which can extract expression feature information related to facial expressions in the reference expression image from the input reference expression image, and is used as a reference for guiding the deformation direction of the target virtual facial image for the subsequent expression generation module to obtain the sample virtual facial expression image.

[0047] The second feature extraction module is used to extract the identity feature of the target virtual face image to obtain the identity feature of the target virtual face image; wherein the second feature extraction module may be, for example, a virtual face identity feature extraction module.

[0048] In an embodiment of the present application, in order to ensure that the identity of the final sample virtual facial expression image is consistent with the identity of the target virtual facial image, it is necessary to extract identity features from the target virtual facial image and provide the extracted identity features of the target virtual facial image to the expression generation module, so that the identity in the sample virtual facial expression image generated by the subsequent expression generation module is consistent with the target virtual facial image.

[0049] Among them, the identity features can be, for example, some specific features corresponding to the target virtual face, such as the skin color features, face shape features, facial features, or other makeup features (scar features, facial hand-painted features, facial mole features), etc. of the target virtual face; it should be understood that the above embodiments are only exemplary descriptions, and the feature content included in the identity features of the specific target virtual face can be flexibly adjusted according to user needs, and is not limited to the above embodiments.

[0050] In an embodiment of the present application, the second feature extraction module may be composed of an eight-layer convolutional network, and the output of the second feature extraction module is a multi-scale feature map, which includes feature maps of 7 scales in total.

[0051] For example, the size information of feature maps of 7 sizes may be respectively: [128x128x128], [256x64x64], [512x32x32], [512x16x16], [512x8x8], [512x4x4], [512x1x1].

[0052] Among them, the feature map of the last scale [512x1x1] is subjected to a global pooling and a fully connected layer to obtain a 512-dimensional latent space feature vector, and the feature vector is added to the expression feature extracted by the first feature extraction module in the previous step, and then input into the subsequent expression generation module as the final expression feature; it should be understood that the above embodiment is only an exemplary description, and the module structure of the specific second feature extraction module can be flexibly adjusted according to user needs, and it can also include feature maps of 6 scales, or 8 or any integer, and the present application does not impose any restrictions on this.

[0053] S102: Generate a sample virtual facial expression image according to the expression features and identity features.

[0054] In an embodiment of the present application, an expression generation module is used to generate a sample virtual facial expression image based on expression features and identity features.

[0055] In an embodiment of the present application, an expression generation module may be composed of a multi-layer deconvolution network, for example, and the expression generation module is used to fuse the expression features of a reference expression image into a target virtual face image, so as to generate a sample virtual face expression image based on the reference expression image and the target virtual face image, wherein the identity features of the sample virtual face expression image are consistent with those of the target virtual face image, and the expression features of the sample virtual face expression image are consistent with those of the reference expression image, that is, the virtual face in the generated sample virtual face expression image has both the identity features of the target virtual face image and the expression features of the sample virtual face expression image.

[0056] S103: Calculate the target loss function value according to the sample virtual facial expression image and the sample reference facial expression image using a calculation method corresponding to the face type of the sample reference facial expression image.

[0057] Face type: virtual face or real face; in the embodiments of the present application, in order to ensure the accuracy of subsequent calculations, different target loss function values ​​are calculated in different ways for different face types, that is, the target loss function value calculation method when the face type is a virtual face is different from the target loss function calculation method when the face type is a real face. This calculation method enables the model to be trained in a targeted manner regardless of whether the face type is a real face or a virtual face, thereby improving the migration effect of the target expression transfer model obtained by subsequent training in subsequent applications.

[0058] That is to say, the present application can not only transfer the facial expressions on a real facial expression image to a target virtual facial image to obtain a sample virtual facial expression image; it can also transfer the facial expressions on a virtual facial expression feature to a target virtual facial image to obtain a sample virtual facial expression image; thereby achieving that regardless of whether the face type in the reference expression image is a virtual face or a real face, facial expressions can be transferred to the target virtual face, solving the problem in the prior art that it is impossible to transfer facial expressions on virtual facial expression features to a target virtual facial image.

[0059] S104: Perform model training according to the target loss function value to obtain a target expression transfer model of the target virtual face image.

[0060] In an embodiment of the present application, model training is performed, that is, the expression generation module is trained. The expression generation module is trained according to the target loss function value until the target loss function value converges (the target loss function value is less than or equal to the preset threshold), and the target expression migration model of the target virtual face image can be obtained.

[0061] By adopting the training method of the expression transfer model provided by the present application, after obtaining the expression features of the sample reference expression image and the identity features of the target facial image, a sample virtual facial expression image can be generated according to the expression features and the identity features, and then the target loss function value is determined based on the sample virtual facial expression image and the sample reference expression image. Based on the target loss function value, the model is trained to obtain a target expression transfer model of the target virtual facial image; the target expression transfer model obtained in this way not only has the ability to transfer the facial expressions on the real facial expression image to the target virtual facial image, but also has the ability to transfer the facial expressions on the virtual facial expression features to the target virtual facial image, and it does not require human intervention, which greatly improves the training efficiency and solves the problem that virtual facial expression features cannot be transferred in the prior art.

[0062] Optionally, based on the above embodiment, the embodiment of the present application can also provide a training method for an expression transfer model. The following is an example of the implementation process of calculating the target loss function value in the above method in combination with the accompanying drawings. Figure 2 A flowchart of a method for training an expression transfer model provided in another embodiment of the present application is shown in FIG. Figure 2 As shown, S103 may include:

[0063] If the face type is a real face, execute S111.

[0064] S111: Calculate a first image distance loss function value according to the distance between the sample virtual facial expression image and the sample reference expression image.

[0065] Among them, the first image distance loss function value L exp 1 is used to constrain the sample virtual facial expression image and the sample reference expression image, and the calculation method thereof may be, for example:

[0066] L exp1 =||E(x d )-E(x s→d )||2

[0067] Among them, x d is the sample reference expression image, x s→d is a sample virtual facial expression image, E is a preset expression feature pre-training network, which is used to extract the expression feature information in the sample reference expression image and the expression feature vector of the sample virtual facial expression image, and calculate the distance between the two feature vectors. That is to say, E(x) in the above formula represents feature extraction of the input image x based on the expression feature pre-training network E, and obtains the expression feature vector corresponding to the input image x.

[0068] That is to say, the above formula is to calculate the distance between the sample reference expression image and the sample virtual face expression image to obtain the first image distance loss function L exp1 , L exp1 The smaller it is, the more similar the sample reference expression image and the sample virtual face expression image are. exp1 The larger it is, the less similar the sample reference expression image is to the sample virtual face expression image.

[0069] S112: Calculate the unsaturated opponent loss function value according to the sample virtual facial expression image.

[0070] In the embodiment of the present application, in order to further obtain a more realistic and high-quality generated image, this solution uses an unsaturated opponent loss function to supervise the above expression feature pre-training network E to determine whether the face type of the current sample reference expression image is a real face or a virtual face, wherein the unsaturated opponent loss function L exp2 The calculation method of is as follows:

[0071] L exp2 =E[-log(D(x s→d ))]

[0072] Where D is a preset discriminator network, which is used to determine whether the face type of the sample reference expression image is a real face or a virtual face, log(D(x s→d )) represents the discriminator network for the sample virtual facial expression image x s→dThe output result of the logarithmic operation is input into the expression feature training network E to obtain x s→d is the discrimination result of the sample virtual facial expression image.

[0073] S113: Determine a target loss function value according to the first image distance loss function value and the unsaturated counterpart loss function value.

[0074] That is, after obtaining the target loss function value by summing the first image distance loss function value and the unsaturated counterpart loss function, the model is iteratively trained based on the target loss function value until the target loss function value converges to within a preset value, and then it is determined that the iterative training of the model is completed.

[0075] Optionally, based on the above embodiment, the embodiment of the present application can also provide a training method for an expression transfer model. The following is an example of the implementation process of calculating the target loss function value in the above method in combination with the accompanying drawings. Figure 3 A flowchart of a method for training an expression transfer model provided in another embodiment of the present application is shown in FIG. Figure 3 As shown, S103 may include:

[0076] If the face type is a virtual face, execute S121.

[0077] S121: Calculate a second image distance loss function value according to the distance between the sample virtual facial expression image and the sample reference expression image.

[0078] In an embodiment of the present application, the method for calculating the second image distance loss function value can, for example, be: extracting image features of the sample virtual facial expression image at multiple first preset scales; summing the image features of the sample virtual facial expression image at multiple first preset scales and the image features of the sample reference expression image to obtain the second image distance loss function value.

[0079] In an embodiment of the present application, the number of multiple first preset scales and the scale parameters of the multiple first preset scales can be flexibly adjusted according to user needs. In some possible embodiments, for example, the number of multiple first preset scales can be 7, wherein the scale data of the 7 first preset scales can be, for example, respectively: [128*128*128], [256*64*64], [512*32*32], [512*16*16], [512*8*8], [512*4*4], [512*1*1]; it should be understood that the above embodiment is only an exemplary description, and the specific number of the first preset scales can be 4, 5, 6, 8 or even more, which can be flexibly adjusted according to user needs and is not limited to the above embodiment.

[0080] That is, in the embodiment of the present application, in order to enhance the quality of image generation, a second image distance loss function L is added. pecp , used for perceptual loss to supervise the generated sample virtual facial expression images and sample reference expression images:

[0081]

[0082] Among them, x d is the sample reference expression image, x s→d is a sample virtual facial expression image, V is a pre-trained convolutional neural network (CNN) model (for example, it can be a VGG19 network); it is used to extract image features of the sample virtual facial expression image and the sample reference expression image at N first preset scales, and calculate the distance between the image features of the sample virtual facial expression image and the sample reference expression image at each first preset scale, and sum the distances between the corresponding image features at each first preset scale.

[0083] It is worth noting that, in the embodiment of the present application, although the first image distance loss function and the second image distance loss function are both calculated based on the sample virtual facial expression image and the sample reference expression image, their calculation methods are essentially different:

[0084] The first image distance loss function is based on the expression feature training network E(x), extracts the expression feature vectors of the sample facial expression image and the sample reference image respectively, and determines the distance between the sample facial expression image and the sample reference expression based on the extracted expression feature vector of the sample facial expression image and the expression feature vector of the sample reference image.

[0085] The specific loss function of the second image is to calculate the distance between the image features of the sample virtual facial expression image and the sample reference expression image at each scale respectively, and sum the distances between the corresponding image features at each scale to determine the distance between the sample facial expression image and the sample reference expression based on the summation result.

[0086] It can be seen that although the first image distance loss function value and the second image specific loss function value are both determined based on the distance between the sample facial expression image and the sample reference expression, the determination method and the dimension / angle of determining the distance are different.

[0087] S122: Calculate a pixel loss function according to the pixel values ​​of the sample virtual facial expression image and the sample reference expression image.

[0088] Among them, the pixel loss function L rec The calculation method of can be, for example:

[0089]

[0090] Where i and j are the coordinate positions of the height direction and width direction of each expression image respectively. is the pixel value of the sample reference expression image at position (i, j), is the pixel value of the sample virtual facial expression image at position (i, j).

[0091] S123: Calculate a target loss function value according to the second image distance loss function value and the pixel loss function value.

[0092] That is, after obtaining the target loss function value by summing the two-image distance loss function value and the calculated pixel loss function, the model is iteratively trained based on the target loss function value until the target loss function value converges to within the preset value.

[0093] In an embodiment of the present application, since the face type may be a virtual face or a real face, the method provided by the present application is adopted, and different face types correspond to different methods of calculating the target loss function value. Only after the target loss function value under each face type converges, that is, the target loss function value of the virtual face converges, and the target loss function value of the real face converges, is the model iteration completed. In this way, the target expression transfer model obtained by training can obtain good transfer effect no matter which face type corresponds to the expression being transferred.

[0094] Optionally, based on the above embodiment, the embodiment of the present application can also provide a training method for an expression transfer model. The implementation process of the above method is illustrated below in conjunction with the accompanying drawings. Figure 4 A flowchart of a method for training an expression transfer model provided in another embodiment of the present application is shown in FIG. Figure 4 As shown, S101 may include:

[0095] S131: Acquire expression features of a sample reference expression image and identity features of a target virtual face image at a plurality of second preset scales.

[0096] In an embodiment of the present application, the number of second preset scales may be the same as the first preset scale, and the scale corresponding to each second preset scale may also be the same as the scale data of multiple first preset scales. That is to say, the number of multiple second preset scales may also be 7, for example, wherein the scale data of the 7 second preset scales may be, for example, respectively: [128*128*128], [256*64*64], [512*32*32], [512*16*16], [512*8*8], [512*4*4], [512*1*1]; of course, the number of second preset scales may also be 6 or any other integer, and the scale data of the second preset scale may also be flexibly adjusted according to user needs. The specific number of second preset scales and the scale data may be flexibly adjusted according to user needs, and are not limited to those given in the above embodiments.

[0097] S102 may include:

[0098] S132: performing upsampling processing and convolution processing on the expression features in sequence to obtain a plurality of motion fields of a second preset scale.

[0099] In an embodiment of the present application, the motion field is the optical flow field; the sample reference expression image and the target virtual face image can be equivalent to two frames in a video, and the brightness change at a specific coordinate point on the two frames of the image is often defined as the optical flow vector of the coordinate point, and the optical flow vector is calculated for each coordinate point on the image, thus forming the optical flow field of the image; the optical flow field can be used as a guide to guide the expression generation module to deform the target virtual face image to achieve the same pixel change as the sample reference expression image, thereby obtaining a sample reference expression image with the same expression as the sample reference expression image.

[0100] In an embodiment of the present application, the expression generation module is composed of a multi-layer deconvolution network, the main purpose of which is to fuse the expression features of the sample reference expression image to the target virtual face image, and finally output the migrated sample virtual face expression image. Specifically, the expression generation module takes the expression features of the facial expression image extracted by the first feature extraction module at multiple second preset scales as input, and generates multiple motion fields corresponding to the second preset scales after upsampling and convolution module processing; then, the motion field is used as a guide to perform feature deformation on the expression features at multiple second preset scales extracted by the second feature extraction module, and then a multi-scale expression migration image is output through a convolution module.

[0101] During the generation process, the size of the motion field of each scale corresponds to the corresponding second preset scale feature size (e.g., when the feature size of the feature map of the second preset scale is [512x8x8], the corresponding motion field size is [8x8]; when the feature size of the feature map of the second preset scale is [512x16x16], the corresponding motion field size is [16x16]), and the feature size is generated step by step from small to large during the generation process; the generated feature deformation and migration images will be sent to the next scale generation process after upsampling. Finally, a sample virtual facial expression image of the same size as the original image is generated.

[0102] S133: Based on the motion fields of the second preset scales, feature deformation is performed on the identity features corresponding to the second preset scales to generate a sample virtual human facial expression image.

[0103] In an embodiment of the present application, for example, the deformed features at multiple second preset scales can be subjected to convolution processing to obtain expression migration images corresponding to multiple second preset scales; the expression migration images corresponding to multiple second preset scales can be upsampled to obtain sample virtual facial expression images.

[0104] Optionally, based on the above embodiment, the embodiment of the present application may also provide an expression migration method, and the implementation process of the above method is illustrated below in conjunction with the accompanying drawings. Figure 5 A flowchart of an expression migration method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method may include:

[0105] S201: Acquire a target reference expression image.

[0106] In an embodiment of the present application, the target reference expression image can be, for example, a real face reference expression image or a virtual face reference expression image. The face type of the specific target reference expression image can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0107] S202: Using the target expression transfer model of the target virtual facial image, and generating a target virtual facial expression image according to the target reference expression image.

[0108] Among them, the target expression transfer model adopts the above Figure 1-Figure 4Any method can be used for training. This method of generating a target virtual facial expression image, since the target expression transfer model is obtained by training based on a large number of real faces and virtual faces, not only has the ability to transfer the facial expressions on the real facial expression image to the target virtual facial image, but also has the ability to transfer the facial expressions on the virtual facial expression features to the target virtual facial image. That is to say, no matter whether the face type of the target reference expression image is a virtual face or a real face, the above-mentioned target expression transfer model can well transfer the facial expressions on the target reference expression image to the target virtual facial image, and no human intervention is required in the middle, thereby greatly reducing the manpower dependence on professionals in the expression transfer process, improving the efficiency of expression transfer, and saving migration costs and additional labor costs.

[0109] The following is an explanation of the training device of the expression transfer model provided by the present application in conjunction with the accompanying drawings. The training device of the expression transfer model can execute the above Figure 1-Figure 4 The specific implementation and beneficial effects of the training method of any expression transfer model are as mentioned above and will not be repeated below.

[0110] Figure 6 A schematic diagram of the structure of a training device for an expression transfer model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 301, a generation module 302, a calculation module 303 and a training module 304, wherein:

[0111] An acquisition module 301 is used to acquire expression features of a sample reference expression image and identity features of a target virtual face image;

[0112] A generating module 302, for generating a sample virtual human face expression image according to expression features and identity features;

[0113] A calculation module 303 is used to calculate the target loss function value according to the sample virtual human face expression image and the sample reference human face expression image, using a calculation method corresponding to the human face type of the sample reference human face expression image; the human face type is used for a virtual human face or a real human face;

[0114] The training module 304 is used to perform model training according to the target loss function value to obtain a target expression transfer model of the target virtual face image.

[0115] Optionally, the calculation module 303 is specifically used to calculate the first image distance loss function value according to the distance between the sample virtual facial expression image and the sample reference expression image if the face type is a real face; calculate the unsaturated opponent loss function value according to the sample virtual facial expression image; and determine the target loss function value according to the first image distance loss function value and the unsaturated opponent loss function value.

[0116] Optionally, the calculation module 303 is specifically used to calculate the second image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image if the face type is a virtual face; calculate the pixel loss function value according to the pixel values ​​of the sample virtual face expression image and the sample reference expression image; and calculate the target loss function value according to the second image distance loss function value and the pixel loss function value.

[0117] Optionally, the acquisition module 301 is specifically used to extract image features of the sample virtual facial expression image at multiple first preset scales; extract image features of the sample reference expression image at multiple first preset scales; sum the image features of the sample virtual facial expression images at multiple first preset scales and the image features of the sample reference expression image to obtain a second image distance loss function value.

[0118] Optionally, the acquisition module 301 is specifically used to acquire the expression features of the sample reference expression image and the identity features of the target virtual face image at multiple second preset scales;

[0119] The generation module 302 is specifically used to perform upsampling and convolution processing on the expression features in sequence to obtain multiple motion fields of second preset scales; based on the multiple motion fields of second preset scales, the identity features corresponding to each second preset scale are deformed to generate a sample virtual facial expression image.

[0120] Optionally, the generation module 302 is specifically used to perform convolution processing on the deformed features at multiple second preset scales to obtain expression migration images corresponding to multiple second preset scales; and upsample the expression migration images corresponding to multiple second preset scales to obtain sample virtual facial expression images.

[0121] Optionally, the expression transfer model includes: a first feature extraction module, a second feature extraction model module, an expression generation module, and an acquisition module 301, which is specifically used to use the first feature extraction module to obtain the expression features of the sample reference expression image; use the second feature extraction module to obtain the identity features of the target virtual face image;

[0122] The generation module 302 is specifically used to generate a sample virtual face expression image according to the expression characteristics and identity characteristics by using the expression generation module;

[0123] The training module 304 is specifically used to train the expression generation module according to the loss function value to obtain a target expression migration model of the target virtual face image.

[0124] The following is an explanation of the expression transfer device provided by the present application in conjunction with the accompanying drawings. The expression transfer device can perform the above Figure 5The expression migration method, its specific implementation and beneficial effects are as mentioned above, and will not be repeated below.

[0125] Figure 7 A schematic diagram of the structure of an expression transfer device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device includes: an acquisition module 401 and a generation module 402, wherein:

[0126] An acquisition module 401 is used to acquire a target reference expression image;

[0127] The generation module 402 is used to use the target expression transfer model of the target virtual face image to generate a target virtual face expression image according to the target reference expression image; wherein the target expression transfer model is based on the above Figure 1-Figure 4 Any method of training.

[0128] A training device for the expression transfer model provided by the present application can generate a sample virtual facial expression image based on the expression features and the identity features after acquiring the expression features of the sample reference expression image and the identity features of the target facial image, and then determine the loss function value based on the sample virtual facial expression image and the sample reference expression image. Based on the loss function value, the model is trained to obtain a target expression transfer model for the target virtual facial image. The target expression transfer model obtained in this way not only has the ability to transfer facial expressions on a real facial expression image to a target virtual facial image, but also has the ability to transfer facial expressions on virtual facial expression features to a target virtual facial image. It does not require human intervention, greatly improves training efficiency, and solves the problem that virtual facial expression features cannot be transferred in the prior art.

[0129] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0130] Figure 8A schematic diagram of the structure of an electronic device provided in one embodiment of the present application, which electronic device can be integrated into a terminal device or a chip of the terminal device.

[0131] like Figure 8 As shown, the electronic device includes: a processor 501 , a bus 502 and a storage medium 503 .

[0132] The processor 501 is used to store programs. The processor 501 calls the program stored in the storage medium 503 to execute the above Figure 1-Figure 5 Corresponding method embodiments.

[0133] Specifically, the processor 501 performs the steps of training the above expression transfer model including:

[0134] Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image;

[0135] Generating a sample virtual facial expression image according to the expression feature and the identity feature;

[0136] According to the sample virtual human face expression image and the sample reference expression image, a target loss function value is calculated using a calculation method corresponding to the face type of the sample reference expression image; the face type is: a virtual face or a real face;

[0137] Model training is performed according to the target loss function value to obtain a target expression transfer model for the target virtual face image.

[0138] In some embodiments, the processor 501 performs the step of calculating the target loss function value according to the sample virtual facial expression image and the reference facial expression image using the calculation method corresponding to the face type of the sample reference facial expression image, which may include:

[0139] If the face type is a real face, calculating a first image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image;

[0140] According to the sample virtual facial expression image, the unsaturated opponent loss function value is calculated; according to the first image distance loss function value and the unsaturated opponent loss function value, the target loss function value is determined.

[0141] In some embodiments, the processor 501 performs the step of calculating the target loss function value according to the sample virtual facial expression image and the reference facial expression image using the calculation method corresponding to the face type of the sample reference facial expression image, which may include:

[0142] If the face type is a virtual face, calculating a second image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image;

[0143] Calculating a pixel loss function according to pixel values ​​of the sample virtual facial expression image and the sample reference expression image;

[0144] The target loss function value is calculated according to the second image distance loss function value and the pixel loss function value.

[0145] In some embodiments, the processor 501 performs the step of calculating the second image distance loss function value according to the distance between the sample virtual facial expression image and the sample reference expression image, which may include:

[0146] Extracting image features of the sample virtual human facial expression image at a plurality of first preset scales;

[0147] Extracting image features of the sample reference expression image at a plurality of the first preset scales;

[0148] The image features of the plurality of sample virtual facial expression images at the first preset scale and the image features of the sample reference expression image are summed to obtain the second image distance loss function value.

[0149] In some embodiments, the processor 501 executes the step of acquiring the expression features of the sample reference expression image and the identity features of the target virtual face image, which may include:

[0150] Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image at multiple second preset scales;

[0151] The step of determining the migrated virtual facial expression image according to the expression feature and the identity feature may include:

[0152] Based on the motion fields of the plurality of second preset scales, feature deformation is performed on the identity features corresponding to each of the second preset scales to generate the sample virtual human facial expression image.

[0153] In some embodiments, the processor 501 executes the step of deforming the identity features corresponding to each of the second preset scales based on the motion fields of the plurality of second preset scales to generate the sample virtual facial expression image, which may include:

[0154] The deformed features at the plurality of second preset scales are subjected to convolution processing to obtain expression migration images corresponding to the plurality of second preset scales;

[0155] The expression transition images corresponding to the plurality of second preset scales are up-sampled to obtain the sample virtual human face expression images.

[0156] In some embodiments, the expression transfer model includes: a first feature extraction module, a second feature extraction model module, and an expression generation module. The processor 501 performs the steps of obtaining the expression features of the sample reference expression image and the identity features of the target virtual face image, which may include:

[0157] Using the first feature extraction module, acquiring the expression features of the sample reference expression image;

[0158] Using the second feature extraction module to obtain the identity features of the target virtual face image;

[0159] The step of generating a sample virtual facial expression image according to the expression feature and the identity feature may include:

[0160] Using the expression generation module, generating a sample virtual facial expression image according to the expression features and the identity features;

[0161] The step of performing model training according to the loss function value to obtain the target expression transfer model of the target virtual face image may include:

[0162] The expression generation module is trained according to the loss function value to obtain a target expression migration model of the target virtual face image.

[0163] The processor 501 executes the above expression migration method including:

[0164] Obtain target reference expression image;

[0165] The target expression transfer model of the target virtual face image is used to generate a target virtual face expression image according to the target reference expression image; wherein the target expression transfer model is used to generate a target virtual face expression image according to the target reference expression image; Figure 1-Figure 4 Any method of training.

[0166] The training device of the expression transfer model provided by the present application can generate a sample virtual facial expression image according to the expression features and the identity features after acquiring the expression features of the sample reference expression image and the identity features of the target facial image, and then determine the loss function value based on the sample virtual facial expression image and the sample reference expression image, and train the model based on the loss function value to obtain a target expression transfer model of the target virtual facial image; the target expression transfer model obtained in this way not only has the ability to transfer the facial expressions on the real facial expression image to the target virtual facial image, but also has the ability to transfer the facial expressions on the virtual facial expression features to the target virtual facial image, and it does not require human intervention, which greatly improves the training efficiency and solves the problem that the virtual facial expression features cannot be transferred in the prior art.

[0167] Optionally, the present application also provides a program product, such as a storage medium, on which a computer program is stored, including a program that, when executed by a processor, executes an embodiment corresponding to the above method.

[0168] Specifically, the processor performs the steps of training the above expression transfer model including:

[0169] Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image;

[0170] Generating a sample virtual facial expression image according to the expression feature and the identity feature;

[0171] According to the sample virtual human face expression image and the sample reference expression image, a target loss function value is calculated using a calculation method corresponding to the face type of the sample reference expression image; the face type is: a virtual face or a real face;

[0172] Model training is performed according to the target loss function value to obtain a target expression transfer model for the target virtual face image.

[0173] In some embodiments, the processor performs the step of calculating the target loss function value according to the sample virtual facial expression image and the reference facial expression image using the calculation method corresponding to the facial type of the sample reference facial expression image, which may include:

[0174] If the face type is a real face, calculating a first image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image;

[0175] Calculating a non-saturated opponent loss function value according to the sample virtual facial expression image;

[0176] The target loss function value is determined according to the first image distance loss function value and the unsaturated opponent loss function value.

[0177] In some embodiments, the processor performs the step of calculating the target loss function value according to the sample virtual facial expression image and the reference facial expression image using the calculation method corresponding to the facial type of the sample reference facial expression image, which may include:

[0178] If the face type is a virtual face, calculating a second image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image;

[0179] Calculating a pixel loss function according to pixel values ​​of the sample virtual facial expression image and the sample reference expression image;

[0180] The target loss function value is calculated according to the second image distance loss function value and the pixel loss function value.

[0181] In some embodiments, the processor performs the step of calculating the second image distance loss function value according to the distance between the sample virtual facial expression image and the sample reference expression image, which may include:

[0182] Extracting image features of the sample virtual human facial expression image at a plurality of first preset scales;

[0183] Extracting image features of the sample reference expression image at a plurality of the first preset scales;

[0184] The image features of the plurality of sample virtual facial expression images at the first preset scale and the image features of the sample reference expression image are summed to obtain the second image distance loss function value.

[0185] In some embodiments, the processor performs the step of acquiring the expression features of the sample reference expression image and the identity features of the target virtual face image, which may include:

[0186] Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image at multiple second preset scales;

[0187] The step of determining the migrated virtual facial expression image according to the expression feature and the identity feature may include:

[0188] Based on the motion fields of the plurality of second preset scales, feature deformation is performed on the identity features corresponding to each of the second preset scales to generate the sample virtual human facial expression image.

[0189] In some embodiments, the processor executes the step of deforming the identity features corresponding to each of the second preset scales based on the motion fields of the plurality of second preset scales to generate the sample virtual facial expression image, which may include:

[0190] The deformed features at the plurality of second preset scales are subjected to convolution processing to obtain expression migration images corresponding to the plurality of second preset scales;

[0191] The expression transition images corresponding to the plurality of second preset scales are up-sampled to obtain the sample virtual human face expression images.

[0192] In some embodiments, the expression transfer model includes: a first feature extraction module, a second feature extraction model module, and an expression generation module. The processor performs the steps of obtaining the expression features of the sample reference expression image and the identity features of the target virtual face image, which may include:

[0193] Using the first feature extraction module, acquiring the expression features of the sample reference expression image;

[0194] Using the second feature extraction module to obtain the identity features of the target virtual face image;

[0195] The step of generating a sample virtual facial expression image according to the expression feature and the identity feature may include:

[0196] Using the expression generation module, generating a sample virtual facial expression image according to the expression features and the identity features;

[0197] The step of performing model training according to the loss function value to obtain the target expression transfer model of the target virtual face image may include:

[0198] The expression generation module is trained according to the loss function value to obtain a target expression migration model of the target virtual face image.

[0199] The processor executes the above expression migration method in the following steps:

[0200] Obtain target reference expression image;

[0201] The target expression transfer model of the target virtual face image is used to generate a target virtual face expression image according to the target reference expression image; wherein the target expression transfer model is used to generate a target virtual face expression image according to the target reference expression image; Figure 1-Figure 4 Any method of training.

[0202] By using the storage medium provided by the present application, after obtaining the expression features of the sample reference expression image and the identity features of the target facial image, a sample virtual facial expression image can be generated according to the expression features and the identity features, and then the loss function value is determined based on the sample virtual facial expression image and the sample reference expression image. Based on the loss function value, the model is trained to obtain a target expression migration model of the target virtual facial image; the target expression migration model obtained in this way not only has the ability to migrate the facial expressions on the real facial expression image to the target virtual facial image, but also has the ability to migrate the facial expressions on the virtual facial expression features to the target virtual facial image. It does not require human intervention, greatly improves the training efficiency, and solves the problem that virtual facial expression features cannot be migrated in the prior art.

[0203] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0204] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0206] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A training method for an expression transfer model, characterized in that: The method comprises: Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image; Generating a sample virtual facial expression image according to the expression feature and the identity feature; According to the sample virtual human face expression image and the sample reference expression image, a target loss function value is calculated using a calculation method corresponding to the face type of the sample reference expression image; the face type is: a virtual face or a real face; Model training is performed according to the target loss function value to obtain a target expression transfer model for the target virtual face image.

2. The method according to claim 1, characterized in that The step of calculating the target loss function value according to the sample virtual human face expression image and the reference expression image and using a calculation method corresponding to the face type of the sample reference expression image comprises: If the face type is a real face, calculating a first image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image; Calculating a non-saturated opponent loss function value according to the sample virtual facial expression image; The target loss function value is determined according to the first image distance loss function value and the unsaturated opponent loss function value.

3. The method according to claim 1, characterized in that The loss is calculated according to the sample virtual human face expression image and the reference expression image, using a calculation method corresponding to the face type of the sample reference expression image to calculate a target loss function value, including: If the face type is a virtual face, calculating a second image distance loss function value according to the distance between the sample virtual face expression image and the sample reference expression image; Calculating a pixel loss function value according to pixel values ​​of the sample virtual facial expression image and the sample reference expression image; The target loss function value is calculated according to the second image distance loss function value and the pixel loss function value.

4. The method according to claim 3, characterized in that The calculating the second image distance loss function value according to the distance between the sample virtual facial expression image and the sample reference expression image comprises: Extracting image features of the sample virtual human facial expression image at a plurality of first preset scales; Extracting image features of the sample reference expression image at a plurality of the first preset scales; The image features of the plurality of sample virtual facial expression images at the first preset scale and the image features of the sample reference expression image are summed to obtain the second image distance loss function value.

5. The method according to claim 1, characterized in that The step of acquiring the expression features of the sample reference expression image and the identity features of the target virtual face image includes: Acquire the expression features of the sample reference expression image and the identity features of the target virtual face image at multiple second preset scales; The step of determining the migrated virtual facial expression image according to the expression feature and the identity feature includes: performing upsampling processing and convolution processing on the expression features in sequence to obtain a plurality of motion fields of the second preset scale; Based on the motion fields of the plurality of second preset scales, feature deformation is performed on the identity features corresponding to each of the second preset scales to generate the sample virtual human facial expression image.

6. The method according to claim 5, characterized in that The step of deforming the identity features corresponding to each of the second preset scales based on the motion fields of the plurality of second preset scales to generate the sample virtual human facial expression image comprises: The deformed features at the plurality of second preset scales are subjected to convolution processing to obtain expression migration images corresponding to the plurality of second preset scales; The expression transition images corresponding to the plurality of second preset scales are up-sampled to obtain the sample virtual human face expression images.

7. A method for expression transfer, characterized in that: The method comprises: Obtain target reference expression image; The target expression transfer model of the target virtual facial image is used to generate a target virtual facial expression image according to the target reference expression image; wherein the target expression transfer model is trained using any of the methods described in claims 1-6 above.

8. A training device for an expression transfer model, characterized in that: The device comprises: an acquisition module, a generation module, a calculation module and a training module, wherein: The acquisition module is used to acquire the expression features of the sample reference expression image and the identity features of the target virtual face image; The generating module is used to generate a sample virtual facial expression image according to the expression feature and the identity feature; The calculation module is used to calculate the loss function value according to the sample virtual human face expression image and the sample reference expression image, using a calculation method corresponding to the face type of the sample reference expression image; the face type is for a virtual face or a real face; The training module is used to perform model training according to the loss function value to obtain a target expression migration model of the target virtual face image.

9. An expression transfer device, characterized in that: The device comprises: an acquisition module and a generation module, wherein: The acquisition module is used to acquire a target reference expression image; The generation module is used to adopt the target expression transfer model of the target virtual facial image and generate a target virtual facial expression image according to the target reference expression image; wherein the target expression transfer model is trained using the method described in any one of claims 1-6 above.

10. An electronic device, characterized in that: The device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 7.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images