Expression generation method and device, and training method and device of expression generation model

Abstract

The present disclosure relates to an expression generation method and device, and a training method and device of an expression generation model, and relates to the technical field of computers. The method of the present disclosure comprises: acquiring feature information of each frame of image in an original video, feature information of a face key point, and classification information of an original expression; fusing the feature information of each frame of image, the feature information of the face key point, and the classification information of the original expression with preset classification information corresponding to a target expression to obtain feature information of a fused image corresponding to each frame of image; and generating a fused image corresponding to each frame of image according to the feature information of the fused image corresponding to each frame of image to obtain a target video in which a face expression formed by the fused images corresponding to all images is the target expression.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to an expression generation method and device, and a training method and device of an expression generation model. BACKGROUND

[0002] With the driving of the new technology wave such as artificial intelligence and virtual reality, the digital human production process is effectively simplified, and the performance in all aspects is greatly improved, and gradually deepens from the digitalization of appearance to the interaction of behavior and the intelligence of emotion. The digital human represented by virtual anchors and virtual employees has successfully entered the public view, and has made great achievements in many fields such as film and television, games, media, tourism, and finance with a variety of poses.

[0003] The interactive digital human image customization pursues realism and individualization. Under the requirement of photographic super-realism, every detail of the character image will be concerned by the user. In the production of the photographic super-realistic interactive digital human, a model can first record a video, and then edit the expression and action of the model based on the video to generate expressions and actions matching the interactive scene. SUMMARY

[0004] One of the technical problems to be solved by the present disclosure is how to edit the expression of a character in a video in the digital human production process to generate a video of a digital human with a target expression corresponding to a scene, and to make the generated video stable and clear.

[0005] According to some embodiments of the present disclosure, an expression generation method is provided, which includes: obtaining feature information of each frame of image, feature information of facial key points, and classification information of original expression in an original video; fusing the feature information of each frame of image, the feature information of facial key points, and the classification information of original expression with preset classification information corresponding to a target expression to obtain feature information of a fusion image corresponding to each frame of image; generating the fusion image corresponding to each frame of image according to the feature information of the fusion image corresponding to each frame of image to obtain a target video in which facial expressions corresponding to all images are the target expression.

[0006] In some embodiments, obtaining the feature information of each frame of image and the feature information of facial key points in the original video includes: inputting each frame of image in the original video into a face feature extraction model to obtain output feature information of each frame of image; inputting the feature information of each frame of image into a face key point detection model to obtain coordinate information of facial key points of each frame of image; and performing dimension reduction on the coordinate information of all facial key points by principal component analysis to obtain information of a preset dimension as the feature information of facial key points.

[0007] In some embodiments, the obtaining of the classification information of the original expression of each frame of image in the original video comprises: inputting the feature information of each frame of image into an expression classification model to obtain the classification information of the original expression of each frame of image.

[0008] In some embodiments, the fusing of the feature information of each frame of image, the feature information of the facial key point, the classification information of the original expression, and the preset classification information corresponding to the target expression comprises: summing and averaging the classification information of the original expression of each frame of image and the preset classification information corresponding to the target expression to obtain the classification information of the fused expression corresponding to each frame of image; and splicing the feature information of the facial key point of each frame of image multiplied by the first weight obtained through training, the feature information of each frame of image multiplied by the second weight obtained through training, and the classification information of the fused expression corresponding to each frame of image.

[0009] In some embodiments, the generating of the fused image corresponding to each frame of image according to the feature information of the fused image corresponding to each frame of image comprises: inputting the feature information of the fused image corresponding to each frame of image into a decoder to output the generated fused image corresponding to each frame of image, wherein the face feature extraction model comprises a convolution layer, and the decoder comprises a deconvolution layer.

[0010] According to some other embodiments of the present disclosure, a training method of an expression generation model is provided, comprising: obtaining training pairs composed of frames of images of an original training video and frames of images of a target training video; inputting the frames of images of the original training video into a first generator to obtain feature information of the frames of images of the original training video, feature information of facial key points, and classification information of an original expression, fusing the feature information of the frames of images of the original training video, the feature information of the facial key points, the classification information of the original expression, and preset classification information corresponding to a target expression to obtain feature information of each frame of fused image corresponding to the original training video, and obtaining the frames of fused image corresponding to the original training video output by the first generator according to the feature information of the frames of fused image corresponding to the original training video; inputting the frames of images of the target training video into a second generator to obtain feature information of the frames of images of the target training video, feature information of facial key points, and classification information of a target expression, fusing the feature information of the frames of images of the target training video, the feature information of the facial key points, the classification information of the target expression, and preset classification information corresponding to the original expression to obtain feature information of each frame of fused image corresponding to the target training video, and obtaining the frames of fused image corresponding to the target training video output by the second generator according to the feature information of the frames of fused image corresponding to the target training video; determining an adversarial loss and a cycle consistency loss according to the frames of fused image corresponding to the original training video and the frames of fused image corresponding to the target training video; and training the first generator and the second generator according to the adversarial loss and the cycle consistency loss.

[0011] In some embodiments, the method further comprises: determining a pixel-to-pixel loss according to a pixel difference between each two adjacent fusion images corresponding to the original training video and a pixel difference between each two adjacent fusion images corresponding to the target training video; and training the first generator and the second generator according to the adversarial loss, the cycle consistency loss, and the pixel-to-pixel loss.

[0012] In some embodiments, determining the adversarial loss according to the fusion images corresponding to the original training video and the fusion images corresponding to the target training video comprises: inputting the fusion images corresponding to the original training video into the first discriminator to obtain first discrimination results of the fusion images corresponding to the original training video; inputting the fusion images corresponding to the target training video into the second discriminator to obtain second discrimination results of the fusion images corresponding to the target training video; determining a first adversarial loss according to the first discrimination results of the fusion images corresponding to the original training video, and determining a second adversarial loss according to the second discrimination results of the fusion images corresponding to the target training video.

[0013] In some embodiments, inputting the fusion images corresponding to the original training video into the first discriminator to obtain the first discrimination results of the fusion images corresponding to the original training video comprises: inputting the fusion images corresponding to the original training video into a first facial feature extraction model in the first discriminator to obtain feature information of the fusion images corresponding to the original training video; and inputting the feature information of the fusion images corresponding to the original training video into a first expression classification model in the first discriminator to obtain classification information of expressions of the fusion images corresponding to the original training video as the first discrimination results; and inputting the fusion images corresponding to the target training video into the second discriminator to obtain the second discrimination results of the fusion images corresponding to the target training video comprises: inputting the fusion images corresponding to the target training video into a second facial feature extraction model in the second discriminator to obtain feature information of the fusion images corresponding to the target training video; and inputting the feature information of the fusion images corresponding to the target training video into a second expression classification model in the second discriminator to obtain classification information of expressions of the fusion images corresponding to the target training video as the second discrimination results.

[0014] In some embodiments, the cycle consistency loss is determined by: inputting the fusion images corresponding to the original training video into the second generator to generate reconstructed images of the original training video, and inputting the fusion images corresponding to the target training video into the first generator to generate reconstructed images of the target training video; and determining the cycle consistency loss according to a difference between the reconstructed images of the original training video and the images of the original training video, and a difference between the reconstructed images of the target training video and the images of the target training video.

[0015] In some embodiments, the pixel-to-pixel loss is determined by the following method: for each position in each adjacent two-frame fusion image corresponding to the original training video, the distance between the representation vectors of the two pixels at the position in the adjacent two-frame fusion image is determined, and the distances corresponding to all positions are summed to obtain a first loss; for each position in each adjacent two-frame fusion image corresponding to the target training video, the distance between the representation vectors of the two pixels at the position in the adjacent two-frame fusion image is determined, and the distances corresponding to all positions are summed to obtain a second loss; the first loss and the second loss are summed to obtain the pixel-to-pixel loss.

[0016] In some embodiments, obtaining the feature information of each frame image of the original training video and the feature information of the facial key points includes: inputting each frame image in the original training video into a third facial feature extraction model in the first generator to obtain the output feature information of each frame image; inputting the feature information of each frame image into a first facial key point detection model in the first generator to obtain the coordinate information of the facial key points of each frame image; using principal component analysis to reduce the dimension of the coordinate information of all facial key points to obtain first information of a preset dimension as the feature information of the facial key points of each frame image of the original training video; obtaining the feature information of each frame image of the target training video and the feature information of the facial key points includes: inputting each frame image in the target training video into a fourth facial feature extraction model in the second generator to obtain the output feature information of each frame image; inputting the feature information of each frame image into a second facial key point detection model in the second generator to obtain the coordinate information of the facial key points of each frame image; using principal component analysis to reduce the dimension of the coordinate information of all facial key points to obtain second information of a preset dimension as the feature information of the facial key points of each frame image of the target training video.

[0017] In some embodiments, obtaining the classification information of the original expression of each frame image in the original training video includes: inputting the feature information of each frame image in the original training video into a third expression classification model in the first generator to obtain the classification information of the original expression of each frame image in the original training video; obtaining the classification information of the target expression of each frame image in the target training video includes: inputting the feature information of each frame image in the target training video into a fourth expression classification model in the second generator to obtain the classification information of the target expression of each frame image in the target training video.

[0018] In some embodiments, the fusing the feature information of each frame image of the original training video, the feature information of the facial key points, the classification information of the original expression, and the preset classification information corresponding to the target expression includes: summing and averaging the classification information of the original expression of each frame image of the original training video and the preset classification information corresponding to the target expression to obtain the classification information of the fused expression corresponding to each frame image of the original training video; and splicing the feature information of the facial key points of each frame image of the original training video multiplied by the first weight to be trained, the feature information of each frame image of the original training video multiplied by the second weight to be trained, and the classification information of the fused expression corresponding to each frame image of the original training video. The fusing the feature information of each frame image of the target training video, the feature information of the facial key points, the classification information of the target expression, and the preset classification information corresponding to the original expression includes: summing and averaging the classification information of the target expression of each frame image of the target training video and the preset classification information corresponding to the original expression to obtain the classification information of the fused expression corresponding to each frame image of the target training video; and splicing the feature information of the facial key points of each frame image of the target training video multiplied by the third weight to be trained, the feature information of each frame image of the target training video multiplied by the fourth weight to be trained, and the classification information of the fused expression corresponding to each frame image of the target training video.

[0019] In some embodiments, the training the first generator and the second generator according to the adversarial loss, the cycle consistency loss, and the pixel-to-pixel loss includes: summing and weighting the adversarial loss, the cycle consistency loss, and the pixel-to-pixel loss to obtain a total loss; and training the first generator and the second generator according to the total loss.

[0020] According to still other embodiments of the present disclosure, an expression generation apparatus is provided, which includes: an acquisition module configured to acquire feature information of each frame image, feature information of facial key points, and classification information of an original expression in an original video; a fusion module configured to fuse the feature information of each frame image, the feature information of the facial key points, the classification information of the original expression, and preset classification information corresponding to a target expression to obtain feature information of a fused image corresponding to each frame image; and a generation module configured to generate a fused image corresponding to each frame image according to the feature information of the fused image corresponding to each frame image, so that a facial expression formed by all the fused images is a target video in which the target expression is the target expression.

[0021] According to still another embodiment of the present disclosure, a training device of an expression generation model is provided, comprising: an acquisition module configured to acquire a training pair composed of each frame image of an original training video and each frame image of a target training video; a first generation module configured to input each frame image of the original training video into a first generator, acquire feature information of each frame image of the original training video, feature information of a facial key point, and classification information of an original expression, fuse the feature information of each frame image of the original training video, the feature information of the facial key point, the classification information of the original expression, and preset classification information corresponding to a target expression, obtain feature information of each frame fusion image corresponding to the original training video, and obtain each frame fusion image output by the first generator according to the feature information of each frame fusion image corresponding to the original training video; a second generation module configured to input each frame image of the target training video into a second generator, acquire feature information of each frame image of the target training video, feature information of a facial key point, and classification information of a target expression, fuse the feature information of each frame image of the target training video, the feature information of the facial key point, the classification information of the target expression, and preset classification information corresponding to the original expression, obtain feature information of each frame fusion image corresponding to the target training video, and obtain each frame fusion image output by the second generator according to the feature information of each frame fusion image corresponding to the target training video; a determination module configured to determine an adversarial loss and a cycle consistency loss according to each frame fusion image corresponding to the original training video and each frame fusion image corresponding to the target training video; and a training module configured to train the first generator and the second generator according to the adversarial loss and the cycle consistency loss.

[0022] According to still another embodiment of the present disclosure, a training device of an expression generation model is provided, comprising: an acquisition module configured to acquire a training pair composed of each frame image of an original training video and each frame image of a target training video; a first generation module configured to input each frame image of the original training video into a first generator, acquire feature information of each frame image of the original training video, feature information of a facial key point, and classification information of an original expression, fuse the feature information of each frame image of the original training video, the feature information of the facial key point, the classification information of the original expression, and preset classification information corresponding to a target expression, obtain feature information of each frame fusion image corresponding to the original training video, and obtain each frame fusion image output by the first generator according to the feature information of each frame fusion image corresponding to the original training video; a second generation module configured to input each frame image of the target training video into a second generator, acquire feature information of each frame image of the target training video, feature information of a facial key point, and classification information of a target expression, fuse the feature information of each frame image of the target training video, the feature information of the facial key point, the classification information of the target expression, and preset classification information corresponding to the original expression, obtain feature information of each frame fusion image corresponding to the target training video, and obtain each frame fusion image output by the second generator according to the feature information of each frame fusion image corresponding to the target training video; a determination module configured to determine an adversarial loss and a cycle consistency loss according to each frame fusion image corresponding to the original training video and each frame fusion image corresponding to the target training video; and a training module configured to train the first generator and the second generator according to the adversarial loss and the cycle consistency loss.

[0023] According to still another embodiment of the present disclosure, a training device of an expression generation model is provided, comprising: an acquisition module configured to acquire a training pair composed of each frame image of an original training video and each frame image of a target training video; a first generation module configured to input each frame image of the original training video into a first generator, acquire feature information of each frame image of the original training video, feature information of a facial key point, and classification information of an original expression, fuse the feature information of each frame image of the original training video, the feature information of the facial key point, the classification information of the original expression, and preset classification information corresponding to a target expression, obtain feature information of each frame fusion image corresponding to the original training video, and obtain each frame fusion image output by the first generator according to the feature information of each frame fusion image corresponding to the original training video; a second generation module configured to input each frame image of the target training video into a second generator, acquire feature information of each frame image of the target training video, feature information of a facial key point, and classification information of a target expression, fuse the feature information of each frame image of the target training video, the feature information of the facial key point, the classification information of the target expression, and preset classification information corresponding to the original expression, obtain feature information of each frame fusion image corresponding to the target training video, and obtain each frame fusion image output by the second generator according to the feature information of each frame fusion image corresponding to the target training video; a determination module configured to determine an adversarial loss and a cycle consistency loss according to each frame fusion image corresponding to the original training video and each frame fusion image corresponding to the target training video; and a training module configured to train the first generator and the second generator according to the adversarial loss and the cycle consistency loss.

[0024] According to still another embodiment of the present disclosure, a training device of an expression generation model is provided, comprising: an acquisition module configured to acquire a training pair composed of each frame image of an original training video and each frame image of a target training video; a first generation module configured to input each frame image of the original training video into a first generator, acquire feature information of each frame image of the original training video, feature information of a facial key point, and classification information of an original expression, fuse the feature information of each frame image of the original training video, the feature information of the facial key point, the classification information of the original expression, and preset classification information corresponding to a target expression, obtain feature information of each frame fusion image corresponding to the original training video, and obtain each frame fusion image output by the first generator according to the feature information of each frame fusion image corresponding to the original training video; a second generation module configured to input each frame image of the target training video into a second generator, acquire feature information of each frame image of the target training video, feature information of a facial key point, and classification information of a target expression, fuse the feature information of each frame image of the target training video, the feature information of the facial key point, the classification information of the target expression, and preset classification information corresponding to the original expression, obtain feature information of each frame fusion image corresponding to the target training video, and obtain each frame fusion image output by the second generator according to the feature information of each frame fusion image corresponding to the target training video; a determination module configured to determine an adversarial loss and a cycle consistency loss according to each frame fusion image corresponding to the original training video and each frame fusion image corresponding to the target training video; and a training module configured to train the first generator and the second generator according to the adversarial loss and the cycle consistency loss.

[0025] The scheme of the present disclosure extracts feature information of each frame of image in the original video, feature information of face key points and classification information of the original expression, fuses the extracted information with preset classification information corresponding to the target expression, obtains feature information of a fused image corresponding to each frame of image, and then generates a fused image corresponding to each frame of image according to the feature information of the fused image corresponding to each frame of image. All the fused images can form a target video in which the face expression is the target expression. In the present disclosure, the feature information of the face key points is extracted and used for feature fusion, so that the expression in the fused image is more real and smooth. Through the fusion of the preset classification information corresponding to the target expression, the generation of the target expression is directly realized, and the target expression is compatible with the facial action and mouth shape of the character in the original image, does not affect the mouth shape and head action of the character, does not affect the definition of the original image, and makes the generated video stable, clear and smooth.

[0026] Other features of the present disclosure and its advantages will become more apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0028] Figure 1 A flowchart of an expression generation method of some embodiments of the present disclosure is shown.

[0029] Figure 2 A schematic diagram of an expression generation method of some embodiments of the present disclosure is shown.

[0030] Figure 3 A flowchart of a training method of an expression generation model of some embodiments of the present disclosure is shown.

[0031] Figure 4 A schematic diagram of a training method of an expression generation model of some embodiments of the present disclosure is shown.

[0032] Figure 5 A structural schematic diagram of an expression generation device of some embodiments of the present disclosure is shown.

[0033] Figure 6 A structural schematic diagram of a training device of an expression generation model of some embodiments of the present disclosure is shown.

[0034] Figure 7 A structural schematic diagram of an electronic device of some embodiments of the present disclosure is shown.

[0035] Figure 8 A structural schematic diagram of an electronic device showing some embodiments of the present disclosure.

[0036] Figure 9 A structural schematic diagram of an expression generation system showing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present disclosure.

[0038] The present disclosure provides an expression generation method, which will be described below in combination with Figures 1-2 .

[0039] Figure 1 A flowchart of some embodiments of the expression generation method of the present disclosure. As shown in the figure, the method of this embodiment includes steps S102-S106. Figure 1

[0040] In step S102, the feature information of each frame of image, the feature information of the face key point, and the classification information of the original expression in the original video are obtained.

[0041] The original video can be a pre-recorded video, and the facial expression in the video is the original expression. The facial expression in each frame of image can be basically consistent, for example, the facial expression in each frame of image is mainly a calm expression, that is, the proportion of the original expression (preset expression) in each frame of image exceeds a preset proportion.

[0042] In some embodiments, each frame of image in the original video is input into a face feature extraction model to obtain the output feature information of each frame of image; the feature information of each frame of image is input into a face key point detection model to obtain the coordinate information of the face key point of each frame of image; principal component analysis (PCA) is used to reduce the dimension of the coordinate information of all face key points to obtain information of a preset dimension as the feature information of the face key point. The feature information of each frame of image is input into an expression classification model to obtain the classification information of the original expression of each frame of image.

[0043] ​The overall expression generation model includes an encoder and a decoder. The encoder can include a face feature extraction model, a face key point detection model, and an expression classification model. The face feature extraction model is connected to the face key point detection model and the expression classification model. The face feature extraction model can use an existing model, such as a deep learning model with feature extraction function, such as VGG-19, ResNet, Transformer, etc. The part before the block 5 of VGG-19 can be used as the face feature extraction model. The face key point detection model and the expression classification model can also use an existing model, such as MLP (Multi-Layer Perceptron), etc. Specifically, it can be a 3-layer MLP. The expression generation model is used to generate expressions after training. The training process will be described in detail later.

[0044] The feature information of each frame of image in the original video is, for example, a feature map (FeatureMap) output by the face feature extraction model. The key points include, for example, 68 key points such as chin, eyebrow center, and mouth corner. Each key point is represented as the horizontal and vertical coordinates of the location. After obtaining the coordinate information of each key point by the face key point detection model, in order to reduce redundant information and improve efficiency, the coordinate information of all face key points is dimensionally reduced by PCA to obtain information of a preset dimension (for example, 6 dimensions, which can achieve the best effect), as the feature information of the face key points. The expression classification model can output the classification of several expressions such as neutral, happy, and sad. One-hot encoding vector can be used for representation. The classification information of the original expression can be the one-hot encoding of the classification of the original expression in each frame of image in the original video obtained by the expression classification model.

[0045] In step S104, the feature information of each frame of image, the feature information of the face key points, and the classification information of the original expression are fused with the preset classification information corresponding to the target expression to obtain the feature information of the fusion image corresponding to each frame of image.

[0046] In some embodiments, the classification information of the original expression of each frame of image and the preset classification information corresponding to the target expression are added and averaged to obtain the classification information of the fusion expression corresponding to each frame of image. The feature information of the face key points of each frame of image multiplied by the first weight obtained by training, the feature information of each frame of image multiplied by the second weight obtained by training, and the classification information of the fusion expression corresponding to each frame of image are spliced.

[0047] The target expression is different from the original expression, for example, a smiling expression, and the preset classification information corresponding to the target expression is, for example, a preset one-hot encoding of the target expression. The preset classification information does not need to be obtained by the model, but can be directly encoded by using a preset encoding rule (one-hot). For example, a calm expression is encoded as 1000, and a smiling expression is encoded as 0100. The classification information of the foregoing original expression is obtained by the expression classification model, and the classification information can be different from the preset classification information corresponding to the original expression. For example, the original expression is a calm expression, and the preset one-hot encoding is 1000, but the one-hot encoding obtained by the expression classification model can be 0.8 0.2 0 0.

[0048] The encoder can further include a feature fusion model. The feature fusion model is configured to input the feature information of each frame of image, the feature information of the face key point, the classification information of the original expression, and the preset classification information corresponding to the target expression into the feature fusion model for fusion. The parameters that need to be trained in the feature fusion model include the first weight and the second weight. For each frame of image, the first weight obtained by training is multiplied by the feature information of the face key point of the image to obtain a first feature vector, and the second weight obtained by training is multiplied by the feature information of the image to obtain a second feature vector. The first feature vector, the second feature vector, and the classification information of the fusion expression corresponding to the image are spliced to obtain the feature information of the fusion image corresponding to the image. The first weight and the second weight can unify the value range of the three kinds of information.

[0049] In step S106, the fusion image corresponding to each frame of image is generated according to the feature information of the fusion image corresponding to each frame of image, and the target video in which the face expression formed by the fusion images corresponding to all images is the target expression is obtained.

[0050] In some embodiments, the feature information of the fusion image corresponding to each frame of image is input into the decoder, and the generated fusion image corresponding to each frame of image is output. The face feature extraction model includes a convolutional layer, and the decoder includes a deconvolutional layer, and the image can be generated based on the feature. The decoder is, for example, block 5 of VGG-19, and the last convolutional layer is replaced by a deconvolutional layer. The fusion image is the image in which the face expression is the target expression, and the target video is formed by the fusion images.

[0051] The following will be described in combination with Figure 2 Some embodiments of the present disclosure are described.

[0052] As Figure 2As shown, a frame from the original video is used to extract features to obtain a feature map. Based on the feature map, facial keypoint detection and expression classification are performed. The feature information of each keypoint obtained from facial keypoint detection is subjected to PCA to reduce the dimensionality to a preset dimension as keypoint features. The original expression classification information is one-hot encoded and fused with the preset classification information corresponding to the target expression to obtain an expression classification vector (fused expression classification information). Then, the facial feature map, expression classification vector and keypoint features are fused to obtain the feature information of the fused image. The feature information of the fused image is then decoded to obtain the facial image of the target expression.

[0053] The above embodiment extracts feature information, facial key point feature information, and original expression classification information from each frame of the original video. The extracted information is then fused with preset classification information corresponding to the target expression to obtain feature information for each frame of the fused image. Based on this feature information, a fused image is generated for each frame, and all fused images form the target video containing the target facial expression. In this embodiment, by extracting feature information from facial key points and using it for feature fusion, the expressions in the fused images are more realistic and fluid. By fusing the preset classification information corresponding to the target expression, the target expression is directly generated, and it is compatible with the facial movements and lip movements in the original image, without affecting the lip movements, head movements, etc., or the clarity of the original image, resulting in a stable, clear, and smooth video.

[0054] The following is combined Figure 3 Describe the training method for the facial expression generation model.

[0055] Figure 3 Flowcharts of some embodiments of the training method for the facial expression generation model disclosed herein. For example... Figure 3 As shown, the method of this embodiment includes steps S302 to S310.

[0056] In step S302, a training pair is obtained, consisting of each frame of the original training video and each frame of the target training video.

[0057] The original training video contains videos with the original facial expressions, while the target training video contains videos with the target facial expressions. Frames in the original training video do not need to correspond one-to-one with frames in the target training video. The classification information for both the original and target facial expressions is labeled.

[0058] In step S304, the frames of the original training video are input into the first generator to obtain the feature information of the frames of the original training video, the feature information of the facial key points, and the classification information of the original expression. The feature information of the frames of the original training video, the feature information of the facial key points, the classification information of the original expression, and the preset classification information corresponding to the target expression are fused to obtain the feature information of the frames of the original training video corresponding to the fusion image. The feature information of the frames of the original training video corresponding to the fusion image is obtained based on the feature information of the frames of the original training video corresponding to the fusion image, and the first generator outputs the frames of the original training video corresponding to the fusion image.

[0059] After the first generator is trained, the first generator is used as an expression generation model. In some embodiments, the feature information of the frames of the original training video is input into the third facial feature extraction model in the first generator to obtain the output feature information of the frames; the feature information of the frames is input into the first facial key point detection model in the first generator to obtain the coordinate information of the facial key points of the frames; the coordinate information of all the facial key points is reduced in dimension by principal component analysis to obtain first information of a preset dimension as the feature information of the facial key points of the frames of the original training video; and the feature information of the frames of the original training video is input into the third expression classification model in the first generator to obtain the classification information of the original expression of the frames of the original training video.

[0060] The coordinate information of the facial key points is subjected to principal component analysis (PCA), and the coordinate information of the key points is reduced to 6 dimensions (6 dimensions are the best effect obtained through a large number of experiments). PCA does not involve training parameters (the feature extraction of PCA and the corresponding relationship between the feature dimensions before and after the PCA do not change with training. When the gradient is back propagated, only the feature corresponding relationship obtained by the initial PCA is used to transmit the gradient to the front parameters).

[0061] In some embodiments, the classification information of the original expression of the frames of the original training video and the preset classification information corresponding to the target expression are added and averaged to obtain the classification information of the fusion expression corresponding to the frames of the original training video; the feature information of the facial key points of the frames of the original training video multiplied by the first weight to be trained, the feature information of the frames of the original training video multiplied by the second weight to be trained, and the classification information of the fusion expression corresponding to the frames of the original training video are spliced to obtain the feature information of the frames of the original training video corresponding to the fusion image.

[0062] The first generator includes a first feature fusion model, and the first weight and the second weight are parameters to be trained in the first feature fusion model. The above feature extraction and feature fusion process can refer to the foregoing embodiments.

[0063] The first generator comprises a first encoder and a first decoder, and the first encoder comprises a third face feature extraction model, a first face key point detection model, a third expression classification model, and a first feature fusion model.

[0064] In step S306, the target training video frames are input into the second generator to obtain the feature information of the target training video frames, the feature information of the face key points, and the classification information of the target expression. The feature information of the target training video frames, the feature information of the face key points, the classification information of the target expression, and the preset classification information corresponding to the original expression are fused to obtain the feature information of the target training video corresponding to each frame of the fused image. According to the feature information of the target training video corresponding to each frame of the fused image, the second generator outputs the target training video corresponding to each frame of the fused image.

[0065] The second generator is the same or similar in structure to the first generator, and the training target of the second generator is to generate a video with the same expression as the original training video based on the target training video.

[0066] In some embodiments, the fourth face feature extraction model in the second generator is input with the target training video frames to obtain the output feature information of the frames; the second face key point detection model in the second generator is input with the feature information of the frames to obtain the coordinate information of the face key points of the frames; the principal component analysis method is used to reduce the dimension of the coordinate information of all face key points to obtain second information of a preset dimension as the feature information of the face key points of the target training video frames. The fourth expression classification model in the second generator is input with the feature information of the target training video frames to obtain the classification information of the target expression of the target training video frames.

[0067] The feature information of the face key points of the target training video frames is the same in dimension as the feature information of the face key points of the original training video frames, for example, 6 dimensions.

[0068] In some embodiments, the classification information of the target expression of the target training video frames and the preset classification information corresponding to the original expression are summed and averaged to obtain the classification information of the fused expression corresponding to the target training video frames; the feature information of the face key points of the target training video frames multiplied by the third weight to be trained, the feature information of the target training video frames multiplied by the fourth weight to be trained, and the classification information of the fused expression corresponding to the target training video frames are spliced to obtain the feature information of the target training video corresponding to each frame of the fused image.

[0069] The preset classification information corresponding to the original expression does not need to be obtained through a model, but can be directly encoded by using a preset encoding rule. The second generator includes a second feature fusion model, and the third weight and the fourth weight are parameters to be trained in the second feature fusion model. The feature extraction and feature fusion processes can refer to the foregoing embodiments, and will not be described again.

[0070] The second generator includes a second encoder and a second decoder. The second encoder includes a fourth face feature extraction model, a second face key point detection model, a fourth expression classification model, and a second feature fusion model. The feature information of each frame of the fused image corresponding to the target training video is input into the second decoder to obtain each frame of the generated fused image corresponding to the target training video.

[0071] In step S308, the adversarial loss and the cycle consistency loss are determined according to the each frame of the fused image corresponding to the original training video and the each frame of the fused image corresponding to the target training video.

[0072] The end-to-end training based on the generative adversarial learning and the cross-domain transfer learning can improve the accuracy of the model and improve the training efficiency.

[0073] In some embodiments, the adversarial loss is determined by the following method: the each frame of the fused image corresponding to the original training video is input into the first discriminator to obtain the first discrimination result of the each frame of the fused image corresponding to the original training video; the each frame of the fused image corresponding to the target training video is input into the second discriminator to obtain the second discrimination result of the each frame of the fused image corresponding to the target training video; the first adversarial loss is determined according to the first discrimination result of the each frame of the fused image corresponding to the original training video, and the second adversarial loss is determined according to the second discrimination result of the each frame of the fused image corresponding to the target training video.

[0074] Further, in some embodiments, the each frame of the fused image corresponding to the original training video is input into the first face feature extraction model in the first discriminator to obtain the feature information of the each frame of the fused image corresponding to the original training video; the feature information of the each frame of the fused image corresponding to the original training video is input into the first expression classification model in the first discriminator to obtain the classification information of the expression of the each frame of the fused image corresponding to the original training video as the first discrimination result; the each frame of the fused image corresponding to the target training video is input into the second face feature extraction model in the second discriminator to obtain the feature information of the each frame of the fused image corresponding to the target training video; and the feature information of the each frame of the fused image corresponding to the target training video is input into the second expression classification model in the second discriminator to obtain the classification information of the expression of the each frame of the fused image corresponding to the target training video as the second discrimination result.

[0075] The overall model in the training process includes two sets of generators plus discriminators. The first discriminator and the second discriminator have the same or similar structures, both including a face feature extraction model and an expression classification model. The first face feature extraction model and the second face feature extraction model have the same or similar structures as the third face feature extraction model and the fourth face feature extraction model, and the first expression classification model and the second expression classification model have the same or similar structures as the third expression classification model and the fourth expression classification model.

[0076] For example, the data of the original video is represented by X = {x i} and the data of the target video is represented by Y = {y i}. The first generator G is used to realize X→Y, and the training is to make G(X) as close to Y as possible. The first discriminator D Y is used to distinguish the true or false of the fusion image corresponding to the original training video. The first adversarial loss can be represented by the following formula:

[0077]

[0078] The second generator F is used to realize Y→X, and the training is to make F(Y) as close to X as possible. The second discriminator D X is used to distinguish the true or false of the fusion image corresponding to the target training video. The second adversarial loss can be represented by the following formula:

[0079]

[0080] In some embodiments, the cycle consistency loss is determined by the following method: inputting the fusion image corresponding to the original training video into the second generator to generate the reconstructed image of each frame of the original training video, and inputting the fusion image corresponding to the target training video into the first generator to generate the reconstructed image of each frame of the target training video; and determining the cycle consistency loss according to the difference between the reconstructed image of each frame of the original training video and the image of each frame of the original training video, and the difference between the reconstructed image of each frame of the target training video and the image of each frame of the target training video.

[0081] In order to further improve the accuracy of the model, the image generated by the first generator is input into the second generator to obtain the reconstructed image of each frame of the original training video, and it is expected that the reconstructed image of each frame of the original training video generated by the second generator is consistent with the image of each frame of the original training video, i.e. F(G(x))≈x. The image generated by the second generator is input into the first generator to obtain the reconstructed image of each frame of the target training video, and it is expected that the reconstructed image of each frame of the target training video generated by the first generator is consistent with the image of each frame of the target training video, i.e. G(F(y))≈y.

[0082] The difference between each frame of the reconstructed image of the original training video and each frame of the original training video can be determined by the following method: for each frame of the reconstructed image of the original training video and the image of the original training video corresponding to the reconstructed image, the distance (such as the Euclidean distance) between the representation vectors of the pixels at the same position of the reconstructed image and the corresponding image is determined, and the sum of all distances is summed.

[0083] The difference between each frame of the reconstructed image of the target training video and each frame of the target training video can be determined by the following method: for each frame of the reconstructed image of the target training video and the image of the target training video corresponding to the reconstructed image, the distance (such as the Euclidean distance) between the representation vectors of the pixels at the same position of the reconstructed image and the corresponding image is determined, and the sum of all distances is summed.

[0084] In step S310, the first generator and the second generator are trained according to the adversarial loss and the cycle consistency loss.

[0085] The first adversarial loss, the second adversarial loss and the cycle consistency loss can be weighted and summed to obtain a total loss, and the first generator and the second generator are trained according to the total loss. For example, the total loss can be determined by the following formula:

[0086] L = L GAN (G, D Y , X, Y) + L GAN (F, D X , Y, X) + λL cyc (G, F) (3)

[0087] Wherein, L cyc (G, F) represents the cycle consistency loss, and λ is a weight, which can be obtained by experiment.

[0088] In order to further improve the accuracy of the model and ensure the stability and continuity of the output video result, the loss caused by the pixel difference between two frames of video is added in the training process. In some embodiments, according to the pixel difference between each adjacent two frames of the fusion image corresponding to the original training video and the pixel difference between each adjacent two frames of the fusion image corresponding to the target training video, a pixel-to-pixel loss is determined, and the first generator and the second generator are trained according to the adversarial loss, the cycle consistency loss and the pixel-to-pixel loss.

[0089] Further, in some embodiments, for each position in each adjacent two-frame fusion image corresponding to the original training video, a distance between representation vectors of two pixels at the position in the adjacent two-frame fusion image is determined, distances corresponding to all positions are added to obtain a first loss; for each position in each adjacent two-frame fusion image corresponding to the target training video, a distance between representation vectors of two pixels at the position in the adjacent two-frame fusion image is determined, distances corresponding to all positions are added to obtain a second loss; the first loss and the second loss are added to obtain a pixel-to-pixel loss. The pixel-to-pixel loss can make the change of adjacent two frames of the generated video not too large.

[0090] In some embodiments, the adversarial loss, the cycle consistency loss and the pixel-to-pixel loss are weighted and summed to obtain a total loss; the first generator and the second generator are trained according to the total loss. For example, the total loss can be determined by the following formula:

[0091] L = L GAN (G, D Y , X, Y) + L GAN (F, D X , Y, X) + λ1L cyc (G, F) + λ2L P2P (G(x i ), G(x i+1 )) + λ3L P2P (F(y j ), F(y j+1 )) (4)

[0092] wherein λ1, λ2, λ3 are weights, which can be obtained by experiments, L P2P (G(x i ), G(x i+1 )) represents the first loss, L P2P (F(y j ), F(y j,1 )) represents the second loss.

[0093] As Figure 4As shown, before end-to-end training is performed, the model of each part can be pre-trained, for example, a large amount of open source face recognition data is first selected to pre-train the face recognition model, and the part before the output feature map is selected as the face feature extraction model (the part method is not unique, for example, vgg-19 is selected before block5, which can output an 8x8x512-dimensional feature map). After that, the face feature extraction model and the parameters are fixed, and the latter is divided into two branches, and the two branches are the face key point detection model and the expression classification model, and the face key point detection data set and the expression classification data are used to fine-tune the parameters in the model structure of the two parts. The face key point detection model is not unique, as long as the model based on the convolutional network model can obtain accurate key points and can be connected to the scheme; the expression classification model is a single label classification task based on the convolutional network model. After pre-training, the end-to-end training process can be performed based on the foregoing embodiments. In this way, the training efficiency can be improved.

[0094] The method of the above embodiment adopts the adversarial loss, the cycle consistency loss, and the pixel loss between adjacent two frames of the video to train the overall model, which can improve the accuracy of the model, and the end-to-end training process can improve the efficiency and save the computing resources.

[0095] The scheme of the present disclosure is suitable for editing the facial expression in a single image. The present disclosure adopts a unique deep learning model, combines expression recognition, key point detection and other technologies, learns the rules of the movement of the facial key points under different expressions through data training, and finally controls the facial expression state output by the model by inputting the classification information of the target expression to the model. The expression only exists as a style state, and when the character speaks or makes a head tilt or eye blink action, the effect can be well superimposed, so that the final output character facial action video is natural and not uncomfortable. The output result can have the same resolution and detail level as the input image, and the output result remains stable, clear and flawless under 1080p or even 2k resolution.

[0096] The present disclosure also provides an expression generation device, which will be described below in conjunction with Figure 5

[0097] Figure 5 The structure diagram of some embodiments of the expression generation device of the present disclosure is shown in FIG. 1. As shown in the figure, the device 50 of the embodiment includes an acquisition module 510, a fusion module 520, and a generation module 530. Figure 5

[0098] The acquisition module 510 is configured to acquire the feature information of each frame of image in the original video, the feature information of the face key points, and the classification information of the original expression.

[0099] ​​In some embodiments, the acquisition module 510 is used to input each frame of the original video into the face feature extraction model to obtain the feature information of each frame of the output image; input the feature information of each frame of the image into the face key point detection model to obtain the coordinate information of the face key points of each frame of the image; and use principal component analysis to reduce the dimensionality of the coordinate information of all face key points to obtain information of a preset dimension as the feature information of the face key points.

[0100] In some embodiments, the acquisition module 510 is used to input the feature information of each frame image into the expression classification model to obtain the original expression classification information of each frame image.

[0101] The fusion module 520 is used to fuse the feature information of each frame image, the feature information of facial key points, the classification information of the original expression, and the preset classification information corresponding to the target expression to obtain the feature information of the fused image corresponding to each frame image.

[0102] In some embodiments, the fusion module 520 is used to sum and average the classification information of the original expression of each frame image with the preset classification information corresponding to the target expression to obtain the classification information of the fused expression corresponding to each frame image; and to concatenate the feature information of the facial key points of each frame image after multiplying with the first weight obtained by training, the feature information of each frame image after multiplying with the second weight obtained by training, and the classification information of the fused expression corresponding to each frame image.

[0103] The generation module 530 is used to generate a fused image corresponding to each frame image based on the feature information of the fused image corresponding to each frame image, and obtain a target video in which the facial expression formed by the fused images corresponding to all images is the target expression.

[0104] In some embodiments, the generation module 530 is used to input the feature information of the fused image corresponding to each frame image into the decoder and output the generated fused image corresponding to each frame image, wherein the face feature extraction model includes a convolutional layer and the decoder includes a deconvolutional layer.

[0105] This disclosure also provides a training device for an expression generation model, which is described below in conjunction with... Figure 6 Describe it.

[0106] Figure 6 Structural diagrams of some embodiments of the training apparatus for the facial expression generation model of this disclosure. For example... Figure 6 As shown, the apparatus 60 in this embodiment includes: an acquisition module 610, a first generation module 620, a second generation module 630, a determination module 640, and a training module 650.

[0107] The acquisition module 610 is used to acquire training pairs consisting of each frame of the original training video and each frame of the target training video.

[0108] The first generation module 620 is configured to input each frame image of the original training video into the first generator, obtain feature information of each frame image of the original training video, feature information of facial key points, and classification information of an original expression, fuse the feature information of each frame image of the original training video, the feature information of the facial key points, the classification information of the original expression, and preset classification information corresponding to a target expression, obtain feature information of each frame fusion image corresponding to the original training video, and obtain each frame fusion image corresponding to the original training video output by the first generator according to the feature information of each frame fusion image corresponding to the original training video.

[0109] In some embodiments, the first generation module 620 is configured to input each frame image in the original training video into a third facial feature extraction model in the first generator, obtain feature information of the output frame image, input the feature information of each frame image into a first facial key point detection model in the first generator, obtain coordinate information of facial key points of each frame image, perform dimension reduction on the coordinate information of all facial key points by using principal component analysis, obtain first information of a preset dimension as feature information of the facial key points of each frame image of the original training video, and input the feature information of each frame image in the original training video into a third expression classification model in the first generator, to obtain classification information of an original expression of each frame image in the original training video.

[0110] In some embodiments, the first generation module 620 is configured to add and average the classification information of the original expression of each frame image of the original training video and the preset classification information corresponding to the target expression, to obtain classification information of a fusion expression corresponding to each frame image of the original training video, splice the feature information of the facial key points of each frame image of the original training video multiplied by the first weight to be trained, the feature information of each frame image of the original training video multiplied by the second weight to be trained, and the classification information of the fusion expression corresponding to each frame image of the original training video.

[0111] The second generation module 630 is configured to input each frame image of the target training video into the second generator, obtain feature information of each frame image of the target training video, feature information of facial key points, and classification information of a target expression, fuse the feature information of each frame image of the target training video, the feature information of the facial key points, the classification information of the target expression, and preset classification information corresponding to an original expression, obtain feature information of each frame fusion image corresponding to the target training video, and obtain each frame fusion image of the target training video output by the second generator according to the feature information of each frame fusion image corresponding to the target training video.

[0112] In some embodiments, the second generation module 630 is configured to input each frame image in the target training video into a fourth face feature extraction model in the second generator to obtain feature information of the output frame image; input the feature information of each frame image into a second face key point detection model in the second generator to obtain coordinate information of the face key points of each frame image; perform dimension reduction on the coordinate information of all face key points by principal component analysis to obtain second information of a preset dimension as the feature information of the face key points of each frame image of the target training video; and input the feature information of each frame image of the target training video into a fourth expression classification model in the second generator to obtain classification information of the target expression of each frame image of the target training video.

[0113] In some embodiments, the second generation module 630 is configured to add and average the classification information of the target expression of each frame image of the target training video and the preset classification information corresponding to the original expression to obtain classification information of a fusion expression corresponding to each frame image of the target training video; and splice the feature information of the face key points of each frame image of the target training video multiplied by the third weight, the feature information of each frame image of the target training video multiplied by the fourth weight, and the classification information of the fusion expression corresponding to each frame image of the target training video.

[0114] The determination module 640 is configured to determine an adversarial loss and a cycle consistency loss according to the each frame fusion image corresponding to the original training video and the each frame fusion image corresponding to the target training video.

[0115] The training module 650 is configured to train the first generator and the second generator according to the adversarial loss and the cycle consistency loss.

[0116] In some embodiments, the determination module 640 is configured to determine a pixel-to-pixel loss according to a pixel difference between each adjacent two frame fusion images corresponding to the original training video and a pixel difference between each adjacent two frame fusion images corresponding to the target training video; and the training module 650 is configured to train the first generator and the second generator according to the adversarial loss, the cycle consistency loss, and the pixel-to-pixel loss.

[0117] In some embodiments, the determination module 640 is configured to input each frame fusion image corresponding to the original training video into the first discriminator to obtain a first discrimination result of each frame fusion image corresponding to the original training video; input each frame fusion image corresponding to the target training video into the second discriminator to obtain a second discrimination result of each frame fusion image corresponding to the target training video; determine a first adversarial loss according to the first discrimination result of each frame fusion image corresponding to the original training video; and determine a second adversarial loss according to the second discrimination result of each frame fusion image corresponding to the target training video.

[0118] In some embodiments, the determining module 640 is configured to input each frame of the fusion image corresponding to the original training video into a first face feature extraction model in the first discriminator to obtain feature information of the output of each frame of the fusion image corresponding to the original training video; input the feature information of each frame of the fusion image corresponding to the original training video into a first expression classification model in the first discriminator to obtain classification information of the expression of each frame of the fusion image corresponding to the original training video as a first discrimination result; input each frame of the fusion image corresponding to the target training video into a second face feature extraction model in the second discriminator to obtain feature information of the output of each frame of the fusion image corresponding to the target training video; and input the feature information of each frame of the fusion image corresponding to the target training video into a second expression classification model in the second discriminator to obtain classification information of the expression of each frame of the fusion image corresponding to the target training video as a second discrimination result.

[0119] In some embodiments, the determining module 640 is configured to input each frame of the fusion image corresponding to the original training video into the second generator to generate each frame of the reconstructed image of the original training video, and input each frame of the fusion image corresponding to the target training video into the first generator to generate each frame of the reconstructed image of the target training video; and determine a cycle consistency loss according to a difference between each frame of the reconstructed image of the original training video and each frame of the image of the original training video, and a difference between each frame of the reconstructed image of the target training video and each frame of the image of the target training video.

[0120] In some embodiments, the determining module 640 is configured to, for each position in each pair of adjacent frames of the fusion image corresponding to the original training video, determine a distance between representation vectors of two pixels at the position in the pair of adjacent frames of the fusion image, sum distances corresponding to all positions to obtain a first loss; for each position in each pair of adjacent frames of the fusion image corresponding to the target training video, determine a distance between representation vectors of two pixels at the position in the pair of adjacent frames of the fusion image, sum distances corresponding to all positions to obtain a second loss; and sum the first loss and the second loss to obtain a pixel-to-pixel loss.

[0121] In some embodiments, the training module 650 is configured to sum the adversarial loss, the cycle consistency loss, and the pixel-to-pixel loss to obtain a total loss; and train the first generator and the second generator according to the total loss.

[0122] The expression generation apparatus and the training apparatus of the expression generation model in the embodiments of the present disclosure can each be implemented by various computing devices or computer systems, which are described below in conjunction with Figure 7 and Figure 8 .

[0123] Figure 7 is a structural diagram of some embodiments of the electronic device of the present disclosure. As Figure 7As shown, the electronic device 70 of this embodiment includes a memory 710 and a processor 720 coupled to the memory 710. The processor 720 is configured to execute the expression generation method or expression generation model training method in any of the embodiments of this disclosure based on instructions stored in the memory 710.

[0124] The memory 710 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0125] Figure 8 These are structural diagrams of some other embodiments of the electronic device disclosed herein. For example... Figure 8 As shown, the electronic device 80 of this embodiment includes a memory 810 and a processor 820, which are similar to the memory 710 and processor 720, respectively. It may also include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850, and the memory 810 and processor 820 can be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 840 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0126] This disclosure also provides an expression generation system, such as Figure 9 As shown, the expression generation system 9 includes the expression generation device 50 of any of the foregoing embodiments, and the expression generation model training device 60 of any of the foregoing embodiments.

[0127] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart

[0129] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart

[0130] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the steps listed in the flowchart

[0131] The above description is only preferred embodiments of the present disclosure, not intended to limit the present disclosure, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An expression generation method, comprising: obtaining feature information of each frame of image in an original video, feature information of facial key points, and classification information of an original expression, including: inputting each frame of image in the original video into a facial feature extraction model to obtain output feature information of each frame of image, inputting the feature information of each frame of image into a facial key point detection model to obtain coordinate information of facial key points of each frame of image, and performing dimension reduction on the coordinate information of all facial key points by principal component analysis to obtain information of a preset dimension as the feature information of the facial key points, wherein the classification information of the original expression is one-hot encoding of the classification of the original expression; adding and averaging the classification information of the original expression of each frame of image and preset classification information corresponding to a target expression to obtain classification information of a fusion expression corresponding to each frame of image, and splicing the feature information of the facial key points of each frame of image multiplied by a first weight obtained by training, the feature information of each frame of image multiplied by a second weight obtained by training, and the classification information of the fusion expression corresponding to each frame of image to obtain feature information of a fusion image corresponding to each frame of image; generating a fusion image corresponding to each frame of image according to the feature information of the fusion image corresponding to each frame of image to obtain a target video in which a facial expression formed by all images corresponding to the fusion image is a target expression.

2. The expression generation method according to claim 1, wherein The obtaining of the classification information of the original expression of each frame of image in the original video includes: inputting the feature information of each frame of image into an expression classification model to obtain the classification information of the original expression of each frame of image.

3. The expression generation method according to claim 1, wherein The generating of the fusion image corresponding to each frame of image according to the feature information of the fusion image corresponding to each frame of image includes: inputting the feature information of the fusion image corresponding to each frame of image into a decoder to output the generated fusion image corresponding to each frame of image. The facial feature extraction model includes a convolution layer, and the decoder includes a deconvolution layer.

4. A training method of an expression generation model, comprising: obtaining a training pair composed of frames of image of an original training video and frames of image of a target training video; inputting the frames of image of the original training video into a first generator to obtain and add and average classification information of an original expression of the frames of image of the original training video and preset classification information corresponding to a target expression to obtain classification information of a fusion expression corresponding to the frames of image of the original training video; splicing the feature information of facial key points of the frames of image of the original training video multiplied by a first weight to be trained, the feature information of the frames of image of the original training video multiplied by a second weight to be trained, and the classification information of the fusion expression corresponding to the frames of image of the original training video to obtain feature information of each frame of fusion image corresponding to the original training video, and obtaining the frames of fusion image corresponding to the original training video output by the first generator according to the feature information of each frame of fusion image corresponding to the original training video, wherein the classification information of the original expression is one-hot encoding of the classification of the original expression. inputting each frame image of the target training video into the second generator, obtaining classification information of a target expression of each frame image of the target training video, and averaging the classification information of the target expression and preset classification information corresponding to the original expression to obtain classification information of a fusion expression corresponding to each frame image of the target training video; concatenating feature information of a face key point of each frame image of the target training video multiplied by a third weight to be trained, feature information of each frame image of the target training video multiplied by a fourth weight to be trained, and the classification information of the fusion expression corresponding to each frame image of the target training video to obtain feature information of each frame fusion image corresponding to the target training video, and obtaining each frame fusion image corresponding to the target training video output by the second generator according to the feature information of each frame fusion image corresponding to the target training video, wherein the classification information of the target expression is one-hot encoding of classification of the target expression; determining an adversarial loss and a cycle consistency loss according to the each frame fusion image corresponding to the original training video and the each frame fusion image corresponding to the target training video; determining a pixel-to-pixel loss according to a pixel difference between each adjacent two frame fusion images corresponding to the original training video and a pixel difference between each adjacent two frame fusion images corresponding to the target training video; training the first generator and the second generator according to the adversarial loss, the cycle consistency loss and the pixel-to-pixel loss.

5. The training method of claim 4, wherein, The determining the adversarial loss according to the each frame fusion image corresponding to the original training video and the each frame fusion image corresponding to the target training video includes: inputting the each frame fusion image corresponding to the original training video into a first discriminator to obtain a first discrimination result of the each frame fusion image corresponding to the original training video; inputting the each frame fusion image corresponding to the target training video into a second discriminator to obtain a second discrimination result of the each frame fusion image corresponding to the target training video; determining a first adversarial loss according to the first discrimination result of the each frame fusion image corresponding to the original training video, and determining a second adversarial loss according to the second discrimination result of the each frame fusion image corresponding to the target training video.

6. The training method of claim 5, wherein, The inputting the each frame fusion image corresponding to the original training video into the first discriminator to obtain the first discrimination result of the each frame fusion image corresponding to the original training video includes: inputting the each frame fusion image corresponding to the original training video into a first face feature extraction model in the first discriminator to obtain feature information of the each frame fusion image corresponding to the original training video output by the first face feature extraction model; inputting the feature information of the each frame fusion image corresponding to the original training video into a first expression classification model in the first discriminator to obtain classification information of an expression of the each frame fusion image corresponding to the original training video as the first discrimination result; The inputting the each frame fusion image corresponding to the target training video into the second discriminator to obtain the second discrimination result of the each frame fusion image corresponding to the target training video includes: inputting the feature information of each frame of the fusion image corresponding to the target training video into a second expression classification model in the second discriminator to obtain classification information of the expression of each frame of the fusion image corresponding to the target training video as a second discrimination result. The cycle-consistent loss is determined by the following method:

7. The training method of claim 4, wherein, inputting each frame of the fusion image corresponding to the original training video into the second generator to generate each frame of the reconstructed image of the original training video, and inputting each frame of the fusion image corresponding to the target training video into the first generator to generate each frame of the reconstructed image of the target training video; determining a cycle-consistent loss according to the difference between each frame of the reconstructed image of the original training video and each frame of the image of the original training video, and the difference between each frame of the reconstructed image of the target training video and each frame of the image of the target training video. The pixel-to-pixel loss is determined by the following method:

8. The training method of claim 4, wherein, for each position in each pair of adjacent frames of the fusion image corresponding to the original training video, determining the distance between the representation vectors of the two pixels at the position in the pair of adjacent frames of the fusion image, and adding the distances corresponding to all positions to obtain a first loss; for each position in each pair of adjacent frames of the fusion image corresponding to the target training video, determining the distance between the representation vectors of the two pixels at the position in the pair of adjacent frames of the fusion image, and adding the distances corresponding to all positions to obtain a second loss; adding the first loss and the second loss to obtain the pixel-to-pixel loss. The feature information of each frame of the image of the original training video and the feature information of the facial key point include:

9. The training method of claim 4, wherein, inputting each frame of the image in the original training video into a third facial feature extraction model in the first generator to obtain the output feature information of the frame of the image; inputting the feature information of the frame of the image into a first facial key point detection model in the first generator to obtain the coordinate information of the facial key point of the frame of the image; and performing dimension reduction on the coordinate information of all facial key points by principal component analysis to obtain first information of a preset dimension as the feature information of the facial key point of each frame of the image of the original training video; The feature information of each frame of the image of the original training video and the feature information of the facial key point include: inputting each frame of the image in the original training video into a third facial feature extraction model in the first generator to obtain the output feature information of the frame of the image; inputting the feature information of the frame of the image into a first facial key point detection model in the first generator to obtain the coordinate information of the facial key point of the frame of the image; and performing dimension reduction on the coordinate information of all facial key points by principal component analysis to obtain first information of a preset dimension as the feature information of the facial key point of each frame of the image of the original training video; The classification information of the original expression of each frame of the image in the original training video includes:

10. The training method of claim 9, wherein, ​ inputting the feature information of each frame image in the original training video into a third expression classification model in the first generator to obtain classification information of an original expression of each frame image in the original training video; obtaining the classification information of the target expression of each frame image in the target training video includes: inputting the feature information of each frame image in the target training video into a fourth expression classification model in the second generator to obtain the classification information of the target expression of each frame image in the target training video.

11. The training method of claim 4, wherein, the training of the first generator and the second generator according to the adversarial loss, the cycle consistency loss and the pixel-to-pixel loss includes: weighting and summing the adversarial loss, the cycle consistency loss and the pixel-to-pixel loss to obtain a total loss; training the first generator and the second generator according to the total loss.

12. An expression generation apparatus, comprising: an acquisition module configured to acquire feature information of each frame image in an original video, feature information of a face key point and classification information of an original expression, including: inputting each frame image in the original video into a face feature extraction model to obtain output feature information of the each frame image, inputting the feature information of the each frame image into a face key point detection model to obtain coordinate information of the face key point of the each frame image, and performing dimension reduction on the coordinate information of all face key points by principal component analysis to obtain information of a preset dimension as the feature information of the face key point, wherein the classification information of the original expression is one-hot encoding of classification of the original expression; a fusion module configured to sum and average the classification information of the original expression of the each frame image and preset classification information corresponding to a target expression to obtain classification information of a fusion expression corresponding to the each frame image, and concatenate the feature information of the face key point of the each frame image multiplied by a first weight obtained by training, the feature information of the each frame image multiplied by a second weight obtained by training, and the classification information of the fusion expression corresponding to the each frame image to obtain feature information of a fusion image corresponding to the each frame image; a generation module configured to generate a fusion image corresponding to the each frame image according to the feature information of the fusion image corresponding to the each frame image, so that a face expression formed by all images corresponding to the fusion image is a target video in which the target expression is a target expression.

13. An expression generation model training apparatus, comprising: an acquisition module configured to acquire a training pair composed of each frame image of an original training video and each frame image of a target training video; a first generation module configured to input each frame image of the original training video into a first generator, acquire classification information of an original expression of each frame image of the original training video, and sum and average the classification information of the original expression and preset classification information corresponding to a target expression to obtain classification information of a fusion expression corresponding to each frame image of the original training video; The feature information of the facial key points of each frame image of the original training video multiplied by the first weight to be trained is spliced with the feature information of each frame image of the original training video multiplied by the second weight to be trained and the classification information of the fusion expression corresponding to each frame image of the original training video, to obtain the feature information of each frame fusion image corresponding to the original training video. The first generator outputs the feature information of each frame fusion image corresponding to the original training video, wherein the classification information of the original expression is one-hot encoding of the classification of the original expression. The second generation module is configured to input each frame image of the target training video into a second generator, add and average the classification information of the target expression of each frame image of the target training video and the preset classification information corresponding to the original expression to obtain the classification information of the fusion expression corresponding to each frame image of the target training video, splice the feature information of the facial key points of each frame image of the target training video multiplied by the third weight to be trained, the feature information of each frame image of the target training video multiplied by the fourth weight to be trained and the classification information of the fusion expression corresponding to each frame image of the target training video to obtain the feature information of each frame fusion image corresponding to the target training video, and output each frame fusion image corresponding to the target training video of the second generator according to the feature information of each frame fusion image corresponding to the target training video, wherein the classification information of the target expression is one-hot encoding of the classification of the target expression. The determination module is configured to determine an adversarial loss and a cycle consistency loss according to the each frame fusion image corresponding to the original training video and the each frame fusion image corresponding to the target training video, and determine a pixel-to-pixel loss according to the pixel difference between each adjacent two frame fusion images corresponding to the original training video and the pixel difference between each adjacent two frame fusion images corresponding to the target training video. The training module is configured to train the first generator and the second generator according to the adversarial loss, the cycle consistency loss and the pixel-to-pixel loss.

14. An electronic device comprising: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform the expression generation method of any one of claims 1-3 or the training method of the expression generation model of any one of claims 4-11. The program is executed by the processor to implement the expression generation method of any one of claims 1-3 or the training method of the expression generation model of any one of claims 4-11.

15. A non-transitory computer readable storage medium having stored thereon a computer program, wherein, The expression generation apparatus of claim 12 and the training apparatus of the expression generation model of claim 13.

16. An expression generation system comprising: ​

Images