Training method for image generation model, image generation method, apparatus and device

Abstract

To provide a training method for an image generation model, an image generation method, apparatus, and a device.SOLUTION: A training method includes: extracting reference keypoints of a person from sample audio data, a sample reference image, and annotation image data; based on a model to be trained, performing motion estimation using the sample audio data and the reference keypoints to obtain predicted keypoints that match the sample audio data; performing parameter estimation using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints; performing prior motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; performing image prediction using the sample reference image and dense optical flow to obtain predicted image data that matches the sample audio data; and performing model training using the predicted image data and annotated image data to obtain an image generation model.SELECTED DRAWING: Figure 1a

Description

[Technical Field]

[0001] The present disclosure relates to the computer field, particularly to the technical field of artificial intelligence, specifically to technical fields such as augmented reality (AR), virtual reality (VR), computer vision, and deep learning, and is applicable to scenes such as the metaverse and virtual digital humans. Specifically, the present disclosure relates to a method for training an image generation model, an image generation method, a device, and an apparatus. [Background technology]

[0002] A virtual human, also known as a digital human, is a virtual person with a digitized appearance, exists depending on the display device, and has human appearance, human behavior (they can speak and raise their hands), and human thoughts. Virtual human generation is an important technology in the metaverse, intelligent customer service, e-commerce, etc. Its core is the generation of continuous, realistic digital human images, which has a wide range of application scenarios and business needs.

[0003] How to generate a digital human image is very important. Summary of the Invention

[0004] The present disclosure provides a method for training an image generation model, an image generation method, an apparatus, and a device.

[0005] According to one aspect of the present disclosure, obtaining sample audio data, a sample reference image, and annotation image data, and extracting reference keypoints of a person from the sample reference image; Based on the training model, employing the sample speech data and the reference keypoints to perform motion estimation to obtain predicted keypoints matching the sample speech data; According to the training model, employ the reference keypoints and the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, and employ the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain optical flow of non-key pixel points; According to the training model, performing image prediction by adopting a sample reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points, to obtain predicted image data matching the sample audio data; and performing model training using the predicted image data and the annotation image data to obtain an image generation model. A method for training an image generation model is provided.

[0006] According to one aspect of the present disclosure, Acquiring target audio data and a target reference image, and extracting reference keypoints of a person from the target reference image; Based on an image generation model, perform motion estimation using the target audio data and reference keypoints to obtain predicted keypoints that match the target audio data; According to the image generation model, employ the reference keypoints and the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, and employ the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain optical flow of non-key pixel points; According to the image generation model, performing image prediction using a target reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data matching the target audio data; An image generation method is provided.

[0007] According to one aspect of the present disclosure, a reference keypoint module for receiving sample audio data, a sample reference image, and annotation image data, and extracting reference keypoints of a person from the sample reference image; a predicted keypoint module for performing motion estimation based on the training model, employing sample speech data and reference keypoints, to obtain predicted keypoints matching the sample speech data; an optical flow estimation module for performing parameter estimation based on the training model using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing a priori motion estimation based on the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; an image prediction module for performing image prediction based on the training model by adopting a sample reference image and a dense optical flow including an optical flow of predicted key points and an optical flow of non-key pixel points to obtain predicted image data matching the sample audio data; a model training module for employing the predicted image data and the annotation image data to perform model training to obtain an image generation model; A training device for an image generation model is provided.

[0008] According to one aspect of the present disclosure, a reference keypoint module for obtaining target audio data and a target reference image and extracting reference keypoints of a person from the target reference image; a predicted keypoint module for performing motion estimation based on an image generation model, employing target audio data and reference keypoints, to obtain predicted keypoints matching the target audio data; an optical flow estimation module for performing parameter estimation based on the image generation model using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing a priori motion estimation based on the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; an image prediction module for performing image prediction based on the image generation model by employing a target reference image and a dense optical flow including an optical flow of predicted key points and an optical flow of non-key pixel points to obtain predicted image data matching the target audio data; An image generation device is provided.

[0009] According to another aspect of the present disclosure, at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions providing electronic equipment that, when executed by the at least one processor, enables the at least one processor to perform a method according to any embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform a method according to any embodiment of the present disclosure.

[0011] According to a further aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method according to any embodiment of the present disclosure.

[0012] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become more readily apparent from the following specification. [Brief explanation of the drawings]

[0013] The drawings are used for a better understanding of the present embodiments and are not intended to limit the present disclosure.

[0014] [Figure 1a]1 is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure. [Figure 1b] FIG. 1 is a schematic diagram of the training principle of an image generation model according to an embodiment of the present disclosure. [Figure 2] 1 is a flowchart of another method for training an image generation model according to an embodiment of the present disclosure. [Figure 3a] 10 is a flowchart of a method for training a further image generation model according to an embodiment of the present disclosure. [Figure 3b] FIG. 10 is a processing schematic diagram of a sample reference image according to an embodiment of the present disclosure. [Figure 3c] FIG. 1 is a schematic diagram of the training principle of an image generation model according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of an image generation method according to an embodiment of the present disclosure. [Figure 5] 10 is a flowchart of another image generation method according to an embodiment of the present disclosure. [Figure 6a] 10 is a flowchart of a further image generation method according to an embodiment of the present disclosure. [Figure 6b] 1 is a flowchart illustrating the principle of image generation according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a structural schematic diagram of a training device for an image generation model according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a structural schematic diagram of another image generating device according to an embodiment of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an electronic device for implementing a method for training an image generation model or a method for generating an image according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] FIG. 1a is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure. The method is applied to training an image generation model based on motion priors. The method can be executed by an image generation model training device, which can be implemented in software and / or hardware and integrated into electronic equipment. As shown in FIG. 1a, the image generation model training method according to this embodiment includes the following steps: S101: obtaining sample voice data, a sample reference image, and annotation image data, and extracting reference key points of a person from the sample reference image; S102: based on the training model, employing sample speech data and reference keypoints to perform motion estimation to obtain predicted keypoints matching the sample speech data; S103: based on the training model, employ the reference keypoints and the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, and employ the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain optical flow of non-key pixel points; S104: based on the training model, perform image prediction by adopting the sample reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data matching the sample audio data; The method may further include performing model training using the predicted image data and the annotation image data to obtain an image generation model (S105).

[0016] In an embodiment of the present disclosure, a method for training an image generation model based on motion priors is presented, which introduces motion priors into the model training process, greatly reducing the difficulty of model learning and improving model training efficiency. The image generation model is used to generate a digital human image that matches audio, with its facial and body movements matching the audio, especially the mouth shape matching the audio. The image generation model takes continuous audio data and reference images as input, and outputs a video with a continuous flow of mouth shapes and body movements, where the person's movements in the video match the audio.

[0017] Sample audio data, sample reference images, and annotation image data are obtained, and the duration of each sample audio data is a predetermined fixed value, for example, 800 ms. Each sample audio data can correspond to one frame of sample reference image and annotation image data, the duration of which is also a fixed value, and the movements of the person in the annotation image data are matched to the sample audio data. To ensure that the images generated by the model are continuous, there is overlap between different sample audio data and between different annotation image data, thereby improving the quality of image generation.

[0018] In the embodiments of the present disclosure, the sample reference image can be preprocessed to obtain reference keypoints of the person in the sample reference image. Specifically, keypoint detection can be performed on the sample reference image to obtain reference keypoints. The reference keypoints provide the initial position of the person in the model training process, and inform the model of the rough spatial positions of the person's face, limbs, etc., so as to improve the learning quality of the model.

[0019] Referring to FIG. 1b, the training model may include a motion estimation network 11, a parameter estimation network 22, and an image generation network 33. The motion estimation network 11 predicts a predicted keypoint at time t based on sample audio data at time t and a reference keypoint. The reference keypoint and the predicted keypoint at time t are input to the parameter estimation network 22 to obtain motion parameters for the predicted keypoint at time t. The motion parameters of the predicted keypoint are used to perform a priori motion estimation to obtain the optical flow of non-key pixel points, that is, the optical flow of general pixel points around the predicted keypoint. The motion parameters of the predicted keypoint include not only the optical flow of the predicted keypoint but also the influence parameters of the predicted keypoint on surrounding pixel points. The optical flow of each predicted keypoint and the optical flow of each non-key pixel point can form a dense optical flow at time t.

[0020] The image generation network 33 obtains predicted image data at time t by employing a sample reference image and dense optical flow at time t, which is used to match the sample audio data. The predicted image data is compared with the annotation image data, and a loss function is constructed based on the comparison results. The loss function is then used to update the learnable parameters in the model to obtain an image generation model. The loss function and hyperparameters, such as the learning rate, are not specifically limited. For example, the loss function can employ VGG perceptual loss, GAN (adversarial network) loss, etc. The image generation network simultaneously generates the facial and mouth shapes and limb movements of a human figure, improving the rhythmic matching between the facial and mouth shapes and limb movements, thereby enhancing the realism of the digital human. The motion parameters of the predicted keypoints are determined, and the motion parameters of the predicted keypoints are used to perform motion estimation to obtain the optical flow of the non-key pixel points, thereby obtaining the dense optical flow at time t. The dense optical flow is used as the a priori motion information for image prediction, and the sample reference image and the dense optical flow are used to perform image prediction, thereby greatly reducing the difficulty of image prediction, improving the efficiency of image prediction, and improving the training efficiency of the model.

[0021] The technical solution of the embodiments of the present disclosure determines motion parameters of predicted keypoints, and uses the motion parameters of the predicted keypoints to perform motion estimation to obtain the optical flow of non-key pixel points, i.e., obtains dense optical flow, and uses the sample reference image and the dense optical flow to generate predicted image data, and uses the predicted image data and annotation image data to perform model training to obtain an image generation model, and uses the dense optical flow as a priori motion information for image generation, thereby greatly reducing the difficulty of image prediction, improving the efficiency of image prediction, and improving the efficiency of model training.

[0022] In a preferred embodiment, performing motion estimation employing sample speech data and reference keypoints to obtain predicted keypoints matching the sample speech data according to a training-ready model includes: encoding the sample speech data to obtain speech features according to a training-ready model; and performing motion estimation employing the reference keypoints and the speech features to obtain predicted keypoints matching the sample speech data.

[0023] The motion estimation network 11 may include a speech encoder and a motion estimation unit. Specifically, sample speech data at time t is input to the speech encoder to obtain speech features, and the speech features and reference keypoints are input to the motion estimation unit to obtain predicted keypoints at time t. The network structures of the speech encoder and the motion estimation unit are not specifically limited. For example, the speech features may be conventional speech feature MFCCs (Mel-Frequency Cepstrum Coefficients) or features based on deep learning, such as features based on a Wav2Vec 2.0 network. The motion estimation unit may employ networks such as ResNet (deep residual network), Unet (U-shaped network), or Transformer. By performing motion estimation in combination with the speech features and reference keypoints, predicted keypoints at time t are obtained. The predicted keypoints include mouth shape keypoints and limb movement keypoints. Therefore, the predicted keypoints not only match the sample speech data, but also the mouth shape and limb movement, thereby further improving the image generation quality.

[0024] 2 is a flowchart of another image generation model training method according to an embodiment of the present disclosure. Referring to FIG. 2, the image generation model training method of this embodiment includes: S201: obtaining sample voice data, a sample reference image, and annotation image data, and extracting reference keypoints of a person from the sample reference image; S202: based on the training model, employing sample speech data and reference keypoints to perform motion estimation to obtain predicted keypoints matching the sample speech data; S203: based on the training model, adopt the coordinates of the predicted keypoints and the coordinates of the reference keypoints to obtain the optical flow of the predicted keypoints; S204: employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints; selecting auxiliary keypoints for non-key pixel points from the predicted keypoints; and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain optical flow of the non-key pixel points; S205: based on the training model, perform image prediction by adopting the sample reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data matching the sample audio data; The method may further include performing model training using the predicted image data and the annotation image data to obtain an image generation model (S206).

[0025] The training model may include a motion estimation network, a parameter estimation network, and an image generation network, where the motion estimation network can use the sample speech data and reference keypoints at time t to perform motion estimation to obtain predicted keypoints at time t; the parameter estimation network can use the coordinates of the predicted keypoints and the coordinates of the reference keypoints to obtain sparse optical flow of the predicted keypoints;

number

number

number

[0026] The motion parameters of the predicted keypoint are used to represent the influence of the predicted keypoint on the motion of the surrounding non-key pixel points. The parameter estimation network further employs the optical flow of the predicted keypoint to perform parameter estimation to obtain the motion parameters of the predicted keypoint, whose coordinates are (x p ,y p ), for a non-key pixel point p where p is a distance between the non-key pixel point and the predicted keypoint, an auxiliary keypoint can be further selected for the non-key pixel point from each predicted keypoint, for example, by determining the distance between the non-key pixel point and the predicted keypoint, and selecting the one with the closest distance as the auxiliary keypoint, for example, by selecting the one with the shortest distance as the auxiliary keypoint, and then employing the optical flow and motion parameters of the auxiliary keypoint to perform motion estimation to obtain the optical flow of the non-key pixel point. By selecting auxiliary keypoints for the non-key pixel points and employing the optical flow and motion parameters of the auxiliary keypoint to perform a priori motion estimation to obtain the optical flow of the non-key pixel point, the accuracy of the optical flow of the non-key pixel can be improved and the quality of model training can be improved.

[0027] In one preferred embodiment, employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and performing a priori motion estimation using the optical flow and motion parameters of the auxiliary keypoints to obtain the optical flow of the non-key pixel points includes employing the optical flow of the predicted keypoints to determine a motion function to be followed by the predicted keypoints, calculating derivatives of the motion function based on a Taylor distribution to obtain first and second partial derivatives of the horizontal and vertical motion of the predicted keypoints, and performing a priori motion estimation using the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the first and second partial derivatives of the auxiliary keypoints in the horizontal and vertical directions to obtain the optical flow of the non-key pixel points.

[0028] Based on the motion estimation network, the optical flow of the predicted keypoints is used to determine the motion function that the predicted keypoints follow, and the derivative of the motion function is calculated based on the Taylor distribution to obtain the partial derivative of the predicted keypoints as the motion parameters of the predicted keypoints, and the motion parameters of the auxiliary keypoints are used to perform motion estimation to obtain the optical flow of the non-key pixel points. By expanding the motion function based on the Taylor distribution to obtain the motion parameters of the auxiliary keypoints and using the motion parameters of the auxiliary keypoints to determine the optical flow of the non-key pixel points, the motion between the predicted keypoints and the non-key pixel points conforms to the same distribution and is on the same motion curve, and there is local consistency and correlation between the predicted keypoints and the non-key pixel points, for example, there is local consistency between the optical flow of the palm center and the optical flow of the keypoints on the palm, and during the process of spreading the hand, the entire palm and wrist all move in the same direction. Based on the local motion consistency and relevance, the partial derivatives of auxiliary keypoints are used to perform motion estimation and obtain the optical flow of non-key pixel points, thereby improving the accuracy of motion estimation, i.e., improving the quality of dense optical flow. Subsequently, the local motion consistency and relevance are introduced into the image generation process as a priori knowledge, which further significantly reduces the difficulty of model learning and improves the model training efficiency.

[0029] Specifically, a priori motion estimation is performed using the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions to obtain the optical flow of the non-key pixel points;

number

number

number

number

number

number

number

[0030] In a preferred embodiment, after obtaining the optical flow of the non-key pixel points, the method further includes: determining influence weights of the non-key pixel points that the auxiliary keypoints receive according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints and the learnable influence radius based on a Gaussian distribution; and adopting the influence weights of the non-key pixel points that the auxiliary keypoints receive to scale the optical flow of the non-key pixel points, thereby obtaining scaled optical flow of the non-key pixel points.

[0031] The influence radius is used to represent the degree of influence of the auxiliary keypoint on the non-key pixel point, and the influence degree decreases with the increase of the influence radius. Specifically, the influence weight of the auxiliary keypoint on the non-key pixel point can be determined by the following formula:

number

number

number

number

number

number

number

[0032] In one preferred embodiment, after obtaining the optical flow of the non-key pixel points, the method further includes adopting a learnable optical flow offset amount to modify the optical flow of the non-key pixel points, and obtaining a modified optical flow of the non-key pixel points.

[0033] In the embodiment of the present disclosure, a learnable optical flow offset is further introduced, and the initial value of the optical flow offset is (0,0). The learnable optical flow offset is adopted to modify the optical flow of non-key pixel points according to the following formula:

number

number

number

[0034] The technical solution of the embodiments of the present disclosure introduces the local consistency and correlation of motion as a priori knowledge into the model learning process based on second-order Taylor expansion, introduces the attenuation of motion tendency as a priori knowledge into the model learning process based on Gaussian distribution, and introduces motion diversity into the model learning process based on learnable offset amount, thereby significantly reducing the difficulty of model learning and improving the training efficiency of the model, and further improving the accuracy of optical flow and the quality of image generation.

[0035] In one preferred embodiment, performing image prediction based on the training wait model using a sample reference image and dense optical flow to obtain predicted image data that matches the sample audio data includes: masking the dense optical flow based on the training wait model to obtain a masked dense optical flow; and performing image prediction based on the sample reference image and the masked dense optical flow to obtain predicted image data that matches the sample audio data.

[0036] In an embodiment of the present disclosure, probabilistic masking can further be performed on the dense optical flow using the following formula:

number

number

[0037] 3a is a flowchart of a further image generation model training method according to an embodiment of the present disclosure. Referring to FIG. 3a, the image generation model training method of this embodiment includes: S301: obtaining sample voice data, a sample reference image and annotation image data, extracting reference key points, a reference person image and a background image from the sample reference image, and supplementing the background image to obtain a supplemented background image; S302: based on the training model, employing sample speech data and reference keypoints to perform motion estimation to obtain predicted keypoints matching the sample speech data; S303: based on the training model, employ the reference keypoints and the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, and employ the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain optical flow of non-key pixel points; S304: Encoding a reference person image based on the training model to obtain reference person image features; S305: based on the training model, decode the reference human image features and the dense optical flow including the optical flow of the predicted key points and the optical flow of the non-key pixel points to obtain predicted human image data; S306: fusing the predicted person image data with the supplemented background image to obtain predicted image data matching the sample voice data; The method may include S307: performing model training using the predicted image data and the annotation image data to obtain an image generation model.

[0038] Referring to Figure 3b, not only is a reference keypoint 012 extracted from a sample reference image 011 by keypoint detection, but a person image segmentation is also performed on the sample reference image 011 to obtain a reference person image 014 and a background image 013, and the background image 013 is supplemented to obtain a supplemented background image 015. By separating the foreground reference person image from the background image, the model does not consider the background during the learning process and only focuses on the foreground person image, which not only reduces the learning difficulty of the model, but also avoids the problem of unnatural interconnections between the person image and the background in the predicted image data by fusing the subsequently generated predicted person image data with the supplemented background image to obtain predicted image data.

[0039] The sample speech data at time t is subjected to speech encoding to obtain speech features, the speech features and reference keypoints are input into a motion estimation network to obtain predicted keypoints at time t, the predicted keypoints and reference keypoints are input into a parameter estimation network to obtain motion parameters of the predicted keypoints, and the motion parameters of the predicted keypoints are used to estimate the optical flow of non-key pixel points to obtain dense optical flow, the reference person image is input into the corresponding encoder to obtain reference person image features, and the reference person image features and dense optical flow are input into an image generation network to obtain predicted person image data at time t, and the predicted person image data at time t is fused with the supplemented background image to obtain predicted image data at time t.

[0040] In the embodiment of the present disclosure, the alpha channel of the all-black part in the predicted person image data is set to 0, and the alpha channel of the non-all-black part is set to 1. Then, the predicted person image data and the supplemented background image are blended based on the alpha channel to achieve the background overlay effect. Specifically, the foreground-background blending can be performed using the following formula:

number

[0041] In one preferred embodiment, performing image prediction based on the training wait model using a sample reference image and dense optical flow to obtain predicted image data that matches the sample audio data includes: masking the dense optical flow based on the training wait model to obtain a masked dense optical flow; and performing image prediction based on the sample reference image and the masked dense optical flow to obtain predicted image data that matches the sample audio data.

[0042] In an embodiment of the present disclosure, probabilistic masking is further performed on the dense optical flow to obtain a masked dense optical flow, and image prediction is performed using the sample reference image and the masked dense optical flow to obtain predicted image data that matches the sample audio data. Specifically, the reference person image is input to a person image encoder to obtain reference person image features, and the reference person image features and the masked dense optical flow are input to an image generation network to obtain predicted person image data at time t. The predicted person image data is then fused with the supplemented background image to obtain predicted image data. By performing probabilistic masking on the dense optical flow, the information provided to the model is reduced, increasing the learning difficulty and improving the robustness of the model.

[0043] Referring to Figure 3c, in the training phase, sample speech data is input to a speech encoder to obtain speech features, the reference keypoints and speech features are input to a motion estimation network to obtain predicted keypoints, the reference keypoints and predicted keypoints are input to a parameter estimation network to obtain motion parameters of the predicted keypoints, such as the influence radius range and first and second partial derivatives, and the coordinates of the reference keypoints, the coordinates of the predicted keypoints, and the motion parameters of the predicted keypoints are used to perform second-order dense motion estimation to obtain dense optical flow. Probabilistic masking is performed on the dense optical flow to obtain masked dense optical flow. Then, identity encoding is performed on the reference person image to obtain reference person image features, and the reference person image features and masked dense optical flow are input to a decoder to obtain predicted person image data, and the supplemented background image and predicted person image data are subjected to alpha channel fusion to obtain predicted image data.

[0044] The technical solution of the embodiments of the present disclosure determines the motion parameters of predicted keypoints, and uses the motion parameters of the predicted keypoints to perform motion estimation to obtain the optical flow of non-key pixel points, thereby obtaining dense optical flow, and introducing the dense optical flow into the image generation process as a priori knowledge, thereby improving the efficiency of image generation. In this method, a training method for an image generation model based on second-order motion priors is proposed, which can generate a digital human that matches voice data through learning, and can improve the matching between the digital human's facial and limb movements and the voice data.

[0045] FIG. 4 is a flowchart of an image generation method according to an embodiment of the present disclosure. The method is applied to generating image data matching audio data. The method can be executed by an image generation device, which can be implemented in software and / or hardware and integrated into electronic equipment. As shown in FIG. 4, the image generation method according to this embodiment includes: S401: obtaining target voice data and a target reference image, and extracting reference keypoints of a person from the target reference image; S402: based on an image generation model, perform motion estimation using the target audio data and reference keypoints to obtain predicted keypoints matching the target audio data; S403: based on the image generation model, perform parameter estimation using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and perform a priori motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; The method may include S404, which performs image prediction based on the image generation model by employing a target reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data.

[0046] In an embodiment of the present disclosure, a method for image generation based on motion priors is presented, which introduces motion priors to significantly reduce the difficulty of model training and improve the efficiency of image generation. The image generation model is used to generate a digital human image that matches the voice, with its facial and body movements matching the voice, and in particular, its mouth shape matching the voice. The input of the image generation model is continuous voice data and a reference image, and it outputs a video with a continuous flow of mouth shapes and body movements, and the person's movements in the video match the voice.

[0047] After obtaining the input target voice data and the target reference image, the target reference image can be pre-processed to obtain reference keypoints of the person in the target reference image. Specifically, keypoint detection can be performed on the target reference image to obtain reference keypoints, which provide the initial position of the person and inform the model of the rough spatial positions of the person's face, limbs, etc., so as to improve the image generation quality.

[0048] The image generation model may include a motion estimation network, a parameter estimation network, and an image generation network. The motion estimation network predicts a predicted keypoint at time t based on the target audio data and reference keypoints at time t. The reference keypoint and the predicted keypoint at time t are input to the parameter estimation network to obtain motion parameters of the predicted keypoint at time t. The motion parameters of the predicted keypoint are used to perform a priori motion estimation to obtain the optical flow of non-key pixel points, that is, the optical flow of general pixel points around the predicted keypoint. The motion parameters of the predicted keypoint include not only the optical flow of the predicted keypoint but also the influence parameters of the predicted keypoint on surrounding pixel points. The optical flow of each predicted keypoint and the optical flow of each non-key pixel point can form a dense optical flow at time t.

[0049] By using an image generation network, predicted image data at time t can be obtained by adopting a target reference image and dense optical flow at time t, which can be used as predicted image data matching the target audio data.Movement parameters of the predicted keypoints are determined, and the motion parameters of the predicted keypoints are used to perform motion estimation to obtain the optical flow of non-key pixel points, thereby obtaining the dense optical flow at time t.The dense optical flow is used as the a priori motion information for image prediction, and by adopting the target reference image and dense optical flow to perform image prediction, the difficulty of image prediction is greatly reduced and the efficiency of image prediction is improved.

[0050] The technical solution of the embodiments of the present disclosure determines the motion parameters of the predicted keypoints, and then uses the motion parameters of the predicted keypoints to perform motion estimation to obtain the optical flow of non-key pixel points, thereby obtaining the dense optical flow. The predicted image data is generated using the target reference image and the dense optical flow. That is, by using the dense optical flow as the a priori motion information for image generation, the difficulty of image prediction is greatly reduced and the efficiency of image prediction is improved.

[0051] In one preferred embodiment, performing motion estimation using target audio data and reference keypoints based on an image generation model to obtain predicted keypoints matching the target audio data includes: encoding the target audio data to obtain audio features based on an image generation model; and performing motion estimation using the reference keypoints and the audio features to obtain predicted keypoints matching the target audio data.

[0052] The motion estimation network may include a speech encoder and a motion estimation unit. Specifically, target speech data at time t is input to the speech encoder to obtain speech features, and the speech features and reference keypoints are input to the motion estimation unit to obtain predicted keypoints at time t. Motion estimation is performed in combination with the speech features and reference keypoints to obtain predicted keypoints at time t so that the predicted keypoints match the target speech data, thereby improving the matching degree between subsequent predicted image data and the target speech data.

[0053] In one preferred embodiment, the method further includes obtaining keypoints of a customized action, and fusing the keypoints of the customized action with predicted keypoints that match the target audio data to obtain new predicted keypoints.

[0054] And further support the sequence of key points of user customized movements. Specifically, obtain the key points of user customized movements, and combine the customized key points with predicted key points to obtain new predicted key points. For example, obtain the body movement key points customized by the user, and use predicted mouth shape key points that match the target voice data to supplement the customized body movement key points to obtain new predicted key points. Thus, the new predicted key points can simultaneously consider the flexibility of body movements, the consistency and smoothness of mouth shape. Then, use the new predicted key points to determine dense optical flow.

[0055] 5 is a flowchart of a further image generation method according to an embodiment of the present disclosure. Referring to FIG. 5, the image generation method of this embodiment includes: S501: obtaining target voice data and a target reference image, and extracting reference keypoints of a person from the target reference image; S502: based on an image generation model, employing target speech data and reference keypoints to perform motion estimation to obtain predicted keypoints matching the target speech data; S503: obtaining an optical flow of the predicted keypoints by adopting the coordinates of the predicted keypoints and the coordinates of the reference keypoints according to the image generation model; S504: employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints; selecting auxiliary keypoints for non-key pixel points from the predicted keypoints; and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain optical flow of the non-key pixel points; The method may include S505, which performs image prediction based on the image generation model by employing a target reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data.

[0056] Wherein, the image generation module may include a motion estimation network, a parameter estimation network, and an image generation network, and the motion estimation network can use the target speech data and reference keypoints at time t to perform motion estimation to obtain predicted keypoints at time t. The parameter estimation network can use the coordinates of the predicted keypoints and the coordinates of the reference keypoints to obtain sparse optical flow of the predicted keypoints.

number

number

number

[0057] The motion parameters of the predicted keypoint are used to represent the influence of the predicted keypoint on the motion of the surrounding non-key pixel points. The parameter estimation network further employs the optical flow of the predicted keypoint to perform parameter estimation to obtain the motion parameters of the predicted keypoint, whose coordinates are (x p ,y p ), for a non-key pixel point p, an auxiliary keypoint can be further selected from each predicted keypoint for the non-key pixel point, for example, by determining the distance between the non-key pixel point and the predicted keypoint, and selecting the closest auxiliary keypoint, and using the optical flow and motion parameters of the auxiliary keypoint to perform motion estimation to obtain the optical flow of the non-key pixel point. By selecting auxiliary keypoints for the non-key pixel points and using the optical flow and motion parameters of the auxiliary keypoint to perform a priori motion estimation to obtain the optical flow of the non-key pixel point, the accuracy of the optical flow of the non-key pixel can be improved and the quality of model training can be improved.

[0058] In one preferred embodiment, employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and performing a priori motion estimation using the optical flow and motion parameters of the auxiliary keypoints to obtain the optical flow of the non-key pixel points includes employing the optical flow of the predicted keypoints to determine a motion function to be followed by the predicted keypoints, calculating derivatives of the motion function based on a Taylor distribution to obtain first and second partial derivatives of the horizontal and vertical motion of the predicted keypoints, and performing a priori motion estimation using the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the first and second partial derivatives of the auxiliary keypoints in the horizontal and vertical directions to obtain the optical flow of the non-key pixel points.

[0059] Based on the motion estimation network, the optical flow of the predicted keypoints is used to determine the motion function that the predicted keypoints follow, and the derivative of the motion function is calculated based on the Taylor distribution to obtain the partial derivative of the predicted keypoints as the motion parameters of the predicted keypoints, and the motion parameters of the auxiliary keypoints are used to perform motion estimation to obtain the optical flow of the non-key pixel points. By expanding the motion function based on the Taylor distribution to obtain the motion parameters of the auxiliary keypoints and using the motion parameters of the auxiliary keypoints to determine the optical flow of the non-key pixel points, the motion between the predicted keypoints and the non-key pixel points conforms to the same distribution and is on the same motion curve, and there is local consistency and correlation between the predicted keypoints and the non-key pixel points, for example, there is local consistency between the optical flow of the palm center and the optical flow of the keypoints on the palm, and during the process of spreading the hand, the entire palm and wrist all move in the same direction. Based on the local motion consistency and relevance, the partial derivatives of auxiliary keypoints are used to perform motion estimation and obtain the optical flow of non-key pixel points, thereby improving the accuracy of motion estimation, i.e., improving the quality of dense optical flow. Subsequently, the local motion consistency and relevance are introduced into the image generation process as a priori knowledge, which further significantly reduces the difficulty of model learning and improves the model training efficiency.

[0060] Specifically, a priori motion estimation is performed using the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions to obtain the optical flow of the non-key pixel points;

number

number

number

number

number

number

[0061] In a preferred embodiment, after obtaining the optical flow of the non-key pixel points, the method further includes: determining influence weights of the non-key pixel points that the auxiliary keypoints receive according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints and the learnable influence radius based on a Gaussian distribution; and adopting the influence weights of the non-key pixel points that the auxiliary keypoints receive to scale the optical flow of the non-key pixel points, thereby obtaining scaled optical flow of the non-key pixel points.

[0062] The influence radius is used to represent the degree of influence of the auxiliary keypoint on the non-key pixel point, and the influence degree decreases with the increase of the influence radius. Specifically, the influence weight of the auxiliary keypoint on the non-key pixel point can be determined by the following formula:

number

number

number

number

number

number

number

[0063] In one preferred embodiment, after obtaining the optical flow of the non-key pixel points as described above, the method further includes adopting a learnable optical flow offset amount to modify the optical flow of the non-key pixel points, and obtaining a modified optical flow of the non-key pixel points.

[0064] In the embodiments of the present disclosure, a learnable optical flow offset is further introduced, and the learnable optical flow offset is adopted to correct the optical flow of non-key pixel points according to the following formula:

number

number

number

[0065] 6a is a flowchart of a further image generation method according to an embodiment of the present disclosure. Referring to FIG. 6a, the image generation method of this embodiment includes: S601: obtaining target voice data and a target reference image, and extracting reference key points and a reference person image from the target reference image; S602: based on an image generation model, employing target speech data and reference keypoints to perform motion estimation to obtain predicted keypoints matching the target speech data; S603: based on the image generation model, perform parameter estimation using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and perform a priori motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; S604: encoding a reference person image based on the image generation model to obtain reference person image features; S605: based on the image generation model, decode the reference human image features and the dense optical flow including the optical flow of the predicted key points and the optical flow of the non-key pixel points to obtain predicted human image data; The method may include a step S606 of fusing the predicted person image data with a target background image to obtain predicted image data that matches the target voice data.

[0066] In addition to extracting reference keypoints from the target reference image through keypoint detection, the system also performs human image segmentation on the target reference image to obtain the reference human image. By separating the foreground reference human image from the background image, the model can ignore the background during the prediction process and only focus on the foreground human image, reducing the difficulty of model training.

[0067] The target speech data at time t is encoded to obtain speech features, the speech features and reference keypoints are input into a motion estimation network to obtain predicted keypoints at time t, the predicted keypoints and reference keypoints are input into a parameter estimation network to obtain motion parameters of the predicted keypoints, and the motion parameters of the predicted keypoints are used to estimate the optical flow of non-key pixel points to obtain dense optical flow, the reference person image is input into a person image encoder to obtain reference person image features, and the reference person image features and dense optical flow are input into an image generation network to obtain predicted person image data at time t.

[0068] Then, a target background image is determined, and the generated predicted person image data is fused with the target background image to obtain predicted image data. Specifically, the alpha channel of the all-black part of the predicted person image data is set to 0, and the alpha channel of the non-all-black part is set to 1. The predicted person image data and the target background image are fused based on the alpha channel to achieve a background overlay effect. Specifically, foreground-background fusion can be performed using the following formula:

number

[0069] In one preferred embodiment, the method further comprises extracting a background image from the target reference image, filling the extracted background image, and making the filled background image the target background image.

[0070] In the process of performing person segmentation on the target reference image, not only the reference person image but also the background image is obtained, and the background image is supplemented to obtain the supplemented background image, which is the target background image. By fusing the generated predicted person image data with the supplemented background image to obtain predicted image data, the interconnectivity between the person image and the background in the predicted image data can be further improved, and the image generation quality can be further improved.

[0071] In one preferred embodiment, the method further includes obtaining a customized background image as the target background image.

[0072] The embodiments of the present disclosure further support customized background images, use the customized background images as target background images, and fuse the generated predicted person image data with the customized background images to obtain predicted image data, thereby further improving the flexibility of image generation.

[0073] Referring to Figure 6b, in the image generation stage, predicted keypoints are determined, and the predicted keypoints and reference keypoints are input into a parameter estimation network to obtain the motion parameters of the predicted keypoints. Then, the coordinates of the predicted keypoints, the motion parameters of the predicted keypoints, and the coordinates of the reference keypoints are used to perform second-order dense motion estimation to obtain dense optical flow. Then, identity encoding is performed on the reference person image to obtain reference person image features. The reference person image features and the dense optical flow are input into a decoder to obtain predicted person image data. Alpha channel fusion is performed on the target background image and the predicted person image data to obtain predicted image data. Note that probabilistic masking is not required for the dense optical flow during the image generation process, and the probabilistic masking module can be omitted during the training process.

[0074] The technical solution of the embodiments of the present disclosure determines the motion parameters of predicted keypoints, and uses the motion parameters of the predicted keypoints to perform motion estimation to obtain the optical flow of non-key pixel points, thereby obtaining dense optical flow, and then introducing the dense optical flow into the image generation process as a priori knowledge, thereby improving the efficiency of image generation. That is, a method for generating digital human faces and movements based on second-order motion priors is proposed, which uses an end-to-end network to simultaneously generate the digital human's facial mouth shape and limb movements, and further supports user-customized movements and backgrounds, and has a wide range of application scenarios.

[0075] FIG. 7 is a structural schematic diagram of an image generation model training device according to an embodiment of the present disclosure. This embodiment is applied to training an image generation model based on motor priors. The device can be implemented in software and / or hardware and can be integrated into electronic devices. As shown in FIG. 7, the image generation model training device 700 of this embodiment includes: a reference keypoint module 710 for obtaining sample audio data, a sample reference image and annotation image data, and extracting reference keypoints of people from the sample reference image; a predicted keypoint module 720 for performing motion estimation based on the training model, employing sample speech data and reference keypoints to obtain predicted keypoints matching the sample speech data; an optical flow estimation module 730 for performing parameter estimation based on the training model using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing a priori motion estimation based on the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; an image prediction module 740 for performing image prediction based on the training model by employing a sample reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data matching the sample audio data; and a model training module 750 for employing the predicted image data and the annotation image data to perform model training to obtain an image generation model.

[0076] In one preferred embodiment, the optical flow estimation module 730: a key optical flow unit for obtaining an optical flow of the predicted keypoints by adopting the coordinates of the predicted keypoints and the coordinates of the reference keypoints according to the training model; a pixel optical flow unit for employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, selecting auxiliary keypoints from the predicted keypoints for the non-key pixel points, and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points.

[0077] In one preferred embodiment, the pixel optical flow unit comprises: Specifically, the optical flow of the predicted keypoint is used to determine the motion function that the predicted keypoint follows, and the derivative of the motion function is calculated based on the Taylor distribution to obtain the first and second partial derivatives of the horizontal and vertical motion of the predicted keypoint; The apparatus includes a first optical flow subunit, which is used to perform a priori motion estimation by employing the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions, and to obtain the optical flow of the non-key pixel points.

[0078] In one preferred embodiment, the pixel optical flow unit comprises: Specifically, determine the influence weight of the non-key pixel points from the auxiliary key points according to the coordinates of the non-key pixel points, the coordinates of the auxiliary key points and the learnable influence radius based on the Gaussian distribution; The apparatus further includes a second optical flow subunit, which is used to scale the optical flow of the non-key pixel points by adopting the influence weights that the non-key pixel points receive from the auxiliary key points, and obtain the scaled optical flow of the non-key pixel points.

[0079] In one preferred embodiment, the pixel optical flow unit comprises: Specifically, the present invention includes a third optical flow subunit, which is used to adopt a learnable optical flow offset amount to modify the optical flow of the non-key pixel points and obtain the modified optical flow of the non-key pixel points.

[0080] In one preferred embodiment, the reference keypoint module 710 is specifically used to extract reference keypoints, reference person images and background images from a sample reference image, and fill in the background image to obtain a filled background image; The image prediction module 740 a person image encoding unit for encoding a reference person image to obtain reference person image features according to the training-ready model; The system includes a person image decoding unit for decoding the reference person image features and dense optical flow based on the training model to obtain predicted person image data, and a fusion unit for fusing the predicted person image data with a supplemented background image to obtain predicted image data that matches the sample voice data.

[0081] In one preferred embodiment, the fusion unit is Specifically, based on the training wait model, the dense optical flow is masked to obtain a masked dense optical flow; Image prediction is performed employing the sample reference image and masked dense optical flow to obtain predicted image data that matches the sample audio data.

[0082] In one preferred embodiment, the reference keypoint module 710: Specifically, based on the training model, sample speech data is encoded to obtain speech features; The reference keypoints and speech features are employed to perform motion estimation and are used to obtain predicted keypoints that match the sample speech data.

[0083] The technical solution of the embodiments of the present disclosure determines the motion parameters of predicted keypoints, and uses the motion parameters of the predicted keypoints to perform motion estimation to obtain the optical flow of non-key pixel points, thereby obtaining dense optical flow, and introducing the dense optical flow into the image generation process as prior knowledge, thereby improving the efficiency of image generation.In this method, a training method for an image generation model based on second-order motion priors is proposed, which can generate a digital human that matches voice data through learning, and can improve the matching between the digital human's facial and limb movements and voice data.

[0084] FIG. 8 is a structural schematic diagram of an image generating device according to an embodiment of the present disclosure. This embodiment is applied to generating image data matching audio data. The device can be implemented in software and / or hardware and can be integrated into electronic devices. As shown in FIG. 8, the image generating device 800 of this embodiment includes: a reference keypoint module 810 for obtaining target audio data and a target reference image and extracting reference keypoints of a person from the target reference image; a predicted keypoint module 820 for performing motion estimation based on an image generation model, employing target audio data and reference keypoints, to obtain predicted keypoints matching the target audio data; an optical flow estimation module 830 for performing parameter estimation based on the image generation model using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing a priori motion estimation based on the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; The image generation module 840 may also include an image prediction module for performing image prediction based on the image generation model by employing a target reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data.

[0085] In one preferred embodiment, the optical flow estimation module 830: a key optical flow unit for obtaining an optical flow of the predicted keypoints by adopting the coordinates of the predicted keypoints and the coordinates of the reference keypoints based on an image generation model; a pixel optical flow unit for employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, selecting auxiliary keypoints from the predicted keypoints for the non-key pixel points, and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points.

[0086] In one preferred embodiment, the pixel optical flow unit comprises: Specifically, the optical flow of the predicted keypoint is used to determine the motion function that the predicted keypoint follows, and the derivative of the motion function is calculated based on the Taylor distribution to obtain the first and second partial derivatives of the horizontal and vertical motion of the predicted keypoint; The apparatus includes a first optical flow subunit, which is used to perform a priori motion estimation by employing the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions, and to obtain the optical flow of the non-key pixel points.

[0087] In one preferred embodiment, the pixel optical flow unit comprises: Specifically, determine the influence weight of the non-key pixel points from the auxiliary key points according to the coordinates of the non-key pixel points, the coordinates of the auxiliary key points and the learnable influence radius based on the Gaussian distribution; The apparatus further includes a second optical flow subunit, which is used to scale the optical flow of the non-key pixel points by adopting the influence weights that the non-key pixel points receive from the auxiliary key points, and obtain the scaled optical flow of the non-key pixel points.

[0088] In one preferred embodiment, the pixel optical flow unit comprises: Specifically, the present invention further includes a third optical flow subunit, which is used to adopt a learnable optical flow offset amount to modify the optical flow of the non-key pixel points, and obtain the modified optical flow of the non-key pixel points.

[0089] In one preferred embodiment, the reference keypoint module 810 is specifically used to extract reference keypoints and reference figures from the target reference image; The image prediction module 840 a person image encoding unit for encoding a reference person image to obtain reference person image features based on the image generation model; a person image encoding unit for decoding reference person image features and dense optical flow based on the image generation model to obtain predicted person image data; and an image fusion unit for fusing the predicted person image data with a target background image to obtain predicted image data that matches the target voice data.

[0090] In one preferred embodiment, the image prediction module 840: Specifically, a background image is extracted from the target reference image, the extracted background image is supplemented, and the supplemented background image is used as the target background image; Alternatively, the device may further include a target background unit used to obtain a customized background image and set it as the target background image.

[0091] In one preferred embodiment, the predict keypoints module 820: a speech encoding unit for encoding the target speech data to obtain speech features based on an image generation model; a keypoint prediction unit for performing motion estimation using the reference keypoints and speech features to obtain predicted keypoints matching the target speech data.

[0092] In one preferred embodiment, the predict keypoints module 820 further comprises: Capture key points of customization behavior, The keypoints of the customized action are fused with the predicted keypoints that match the target speech data and are used to obtain new predicted keypoints.

[0093] The technical solution of the embodiments of the present disclosure determines the motion parameters of predicted keypoints, and uses the motion parameters of the predicted keypoints to perform motion estimation to obtain the optical flow of non-key pixel points, thereby obtaining dense optical flow, and introducing the dense optical flow into the image generation process as a priori knowledge, thereby improving the efficiency of image generation. That is, a method for generating digital human faces and movements based on second-order motion priors is proposed, which uses an end-to-end network to simultaneously generate the digital human's facial mouth shape and limb movements, and further supports user-customized movements and backgrounds, and has a wide range of application scenarios.

[0094] In the technical solution disclosed herein, the acquisition, storage, application, etc. of users' personal information all comply with the provisions of relevant laws and regulations and do not violate public order and morals.

[0095] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0096] FIG. 9 is a block diagram of an electronic device for implementing the image generation model training method or image generation method according to an embodiment of the present disclosure.

[0097] 9 illustrates a schematic block diagram of an exemplary electronic device 900 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various types of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely exemplary and are not intended to limit the practice of the present disclosure as described and / or claimed herein.

[0098] 9, the electronic device 900 includes a computing unit 901 that can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 can also store various programs and data required for the operation of the electronic device 900. The computing unit 901, the ROM 902, and the RAM 903 are connected to one another by a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0099] The I / O interface 905 connects to several components in the electronic device 900, including an input unit 906, e.g., a keyboard, a mouse, etc., an output unit 907, e.g., various types of displays, speakers, etc., a storage unit 908, e.g., a magnetic disk, an optical disk, etc., and a communication unit 909, e.g., a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 enables the electronic device 900 to exchange information / data with other devices, e.g., via a computer network of the Internet and / or various telecommunication networks.

[0100] The computing unit 901 may be a general-purpose and / or special-purpose processing component having various processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute algorithms for machine learning models, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs each of the methods and processes described above, such as the image generation model training method or the image generation method. For example, in some embodiments, the image generation model training method or the image generation method can be embodied as a computer software program tangibly included in a machine-readable medium, such as the storage unit 908. In some embodiments, the computer program can be partially or completely loaded and / or installed in the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, it can perform one or more steps of the image generation model training method or the image generation method described above. Alternatively, in other embodiments, the computing unit 901 is configured to perform the image generation model training method or the image generation method in any other suitable manner (eg, by firmware).

[0101] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific general purpose packages (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that can be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, capable of receiving data and instructions from, and transmitting data and instructions to, a memory system, at least one input device, and at least one output device.

[0102] Program code for implementing the methods of the present disclosure may be written using any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program code may be executed entirely on the device, partially on the device, as a separate software package, partially on the device and partially on a remote device, or entirely on a remote device or server.

[0103] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use in or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection of one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical memory device, a magnetic memory device, or any suitable combination of the foregoing.

[0104] To provide for user interaction, the systems and techniques described herein can be implemented on a computer that has a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can be used to provide for user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form (including sound, speech, or tactile input).

[0105] The systems and techniques described herein may be implemented in a computing system that includes background components (e.g., as a data server), or in a computing system that includes middleware components (e.g., an application server), or in a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a network browser through which a user can interact with embodiments of the systems and techniques described herein), or in a computing system that includes any combination of such background components, middleware components, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0106] The computer system may include a client terminal and a server. The client terminal and the server are generally remote from each other and typically interact via a communication network. The relationship between the client terminal and the server is established by a computer program running on a corresponding computer and having a client terminal-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0107] Artificial intelligence is a field that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, etc., while artificial intelligence software technologies mainly include several directions, such as computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0108] Cloud computing is a technology system that allows access to a shared, elastic, and scalable pool of physical or virtual resources via a network, and allows resources to be deployed and managed on a service-by-service basis according to needs. Resources may include servers, operating systems, networks, software, applications, and storage devices. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, and for model training.

[0109] It should be understood that various types of flows shown above can be used, and steps can be rearranged, added, or deleted. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and are not limited thereto in this specification.

[0110] The above specific embodiments do not limit the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions are possible based on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. obtaining sample audio data, a sample reference image, and annotation image data, and extracting reference keypoints of a person from the sample reference image; Based on the training model, employing the sample speech data and the reference keypoints to perform motion estimation to obtain predicted keypoints matching the sample speech data; According to the training model, employ the reference keypoints and the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, and employ the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain optical flow of non-key pixel points; According to the training model, performing image prediction by adopting a sample reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points, to obtain predicted image data matching the sample audio data; and performing model training using the predicted image data and the annotation image data to obtain an image generation model. How to train image generation models.

2. According to the training model, employing the reference keypoints and the predicted keypoints to perform parameter estimation to obtain the motion parameters of the predicted keypoints, and employing the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points; According to the training model, the coordinates of the predicted keypoints and the coordinates of the reference keypoints are adopted to obtain the optical flow of the predicted keypoints; employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints; selecting auxiliary keypoints from the predicted keypoints for the non-key pixel points; and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points. The method of claim 1.

3. Employing the optical flow of the predicted keypoints to perform parameter estimation to obtain the motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points; Employing the optical flow of the predicted keypoints to determine a motion function that the predicted keypoints will follow, and calculating derivatives of the motion function based on a Taylor distribution to obtain first and second partial derivatives of the horizontal and vertical motion of the predicted keypoints; and performing a priori motion estimation using the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions to obtain optical flow of the non-key pixel points. The method of claim 2.

4. After obtaining the optical flow of non-key pixel points, Determine the influence weights of the non-key pixel points from the auxiliary keypoints according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints and the learnable influence radius based on Gaussian distribution; and further comprising: adopting influence weights of the non-key pixel points received from the auxiliary key points to scale the optical flow of the non-key pixel points, thereby obtaining scaled optical flow of the non-key pixel points. The method of claim 3.

5. After obtaining the optical flow of non-key pixel points, The method further includes: employing a learnable optical flow offset to modify the optical flow of the non-key pixel points, and obtaining a modified optical flow of the non-key pixel points. The method according to claim 3 or 4.

6. Extracting reference keypoints of a person from the sample reference image includes extracting reference keypoints, a reference person image, and a background image from the sample reference image, and filling the background image to obtain a filled background image; According to the training model, performing image prediction by adopting sample reference images and dense optical flow to obtain predicted image data matching the sample audio data, encoding a reference person image based on the trained model to obtain reference person image features; Decoding the reference human image features and dense optical flow based on the trained model to obtain predicted human image data; fusing the predicted person image data with the supplemented background image to obtain predicted image data that matches the sample voice data; The method of claim 1.

7. According to the training model, performing image prediction by adopting sample reference images and dense optical flow to obtain predicted image data matching the sample audio data, Masking the dense optical flow based on the training-wait model to obtain a masked dense optical flow; and performing image prediction employing the sample reference image and the masked dense optical flow to obtain predicted image data matching the sample audio data. The method according to claim 1 or 6.

8. According to the training model, performing motion estimation using sample speech data and reference keypoints to obtain predicted keypoints matching the sample speech data; Encoding sample speech data to obtain speech features based on the trained model; employing the reference keypoints and audio features to perform motion estimation to obtain predicted keypoints that match the sample audio data; The method of claim 1.

9. Acquiring target audio data and a target reference image, and extracting reference keypoints of a person from the target reference image; Based on an image generation model, perform motion estimation using the target audio data and reference keypoints to obtain predicted keypoints that match the target audio data; According to the image generation model, employ the reference keypoints and the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints, and employ the motion parameters of the predicted keypoints to perform a priori motion estimation to obtain optical flow of non-key pixel points; According to the image generation model, performing image prediction using a target reference image and a dense optical flow including the optical flow of predicted key points and the optical flow of non-key pixel points to obtain predicted image data matching the target audio data; Image generation method.

10. According to the image generation model, the reference keypoints and the predicted keypoints are used to perform parameter estimation to obtain the motion parameters of the predicted keypoints, and the motion parameters of the predicted keypoints are used to perform a priori motion estimation to obtain the optical flow of the non-key pixel points; According to an image generation model, the coordinates of the predicted keypoints and the coordinates of the reference keypoints are adopted to obtain the optical flow of the predicted keypoints; employing the optical flow of the predicted keypoints to perform parameter estimation to obtain motion parameters of the predicted keypoints; selecting auxiliary keypoints from the predicted keypoints for the non-key pixel points; and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points.

10. The method of claim 9.

11. Employing the optical flow of the predicted keypoints to perform parameter estimation to obtain the motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and employing the optical flow and motion parameters of the auxiliary keypoints to perform a priori motion estimation to obtain the optical flow of the non-key pixel points; Employing the optical flow of the predicted keypoints to determine a motion function that the predicted keypoints will follow, and calculating derivatives of the motion function based on a Taylor distribution to obtain first and second partial derivatives of the horizontal and vertical motion of the predicted keypoints; and performing a priori motion estimation using the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions to obtain optical flow of the non-key pixel points. The method of claim 10.

12. After obtaining the optical flow of non-key pixel points, Determine the influence weights of the non-key pixel points from the auxiliary keypoints according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints and the learnable influence radius based on Gaussian distribution; The method further includes: adopting the influence weights that the non-key pixel points receive from the auxiliary key points to scale the optical flow of the non-key pixel points, and obtaining scaled optical flow of the non-key pixel points; The method of claim 11.

13. After obtaining the optical flow of non-key pixel points, The method further includes: employing a learnable optical flow offset to modify the optical flow of the non-key pixel points, and obtaining a modified optical flow of the non-key pixel points.

13. The method of claim 11 or 12.

14. Extracting reference keypoints of a person from the target reference image includes extracting reference keypoints and a reference person image from the target reference image; According to the image generation model, image prediction is performed by employing a target reference image and dense optical flow to obtain predicted image data that matches the target audio data; encoding a reference person image based on the image generation model to obtain reference person image features; Decoding the reference person image features and dense optical flow based on the image generation model to obtain predicted person image data; and fusing the predicted person image data with a target background image to obtain predicted image data that matches the target voice data.

10. The method of claim 9.

15. extracting a background image from the target reference image, filling in the extracted background image, and setting the filled background image as the target background image; Alternatively, the method may further include obtaining a customized background image and setting the customized background image as the target background image.

15. The method of claim 14.

16. According to an image generation model, performing motion estimation using target audio data and reference keypoints to obtain predicted keypoints matching the target audio data; Encoding the target speech data based on the image generation model to obtain speech features; employing the reference keypoints and the audio features to perform motion estimation to obtain predicted keypoints that match the target audio data; 10. The method of claim 9.

17. To capture key points of customization behavior, and fusing the keypoints of the customized action with the predicted keypoints matching the target speech data to obtain new predicted keypoints.

17. The method of claim 9 or 16.

18. a reference keypoint module for receiving sample audio data, a sample reference image, and annotation image data, and extracting reference keypoints of a person from the sample reference image; a predicted keypoint module for performing motion estimation based on the training model, employing sample speech data and reference keypoints, to obtain predicted keypoints matching the sample speech data; an optical flow estimation module for performing parameter estimation based on the training model using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing a priori motion estimation based on the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; an image prediction module for performing image prediction based on the training model by adopting a sample reference image and a dense optical flow including an optical flow of predicted key points and an optical flow of non-key pixel points to obtain predicted image data matching the sample audio data; a model training module for employing the predicted image data and the annotation image data to perform model training to obtain an image generation model; A training device for image generation models.

19. a reference keypoint module for obtaining target audio data and a target reference image and extracting reference keypoints of a person from the target reference image; a predicted keypoint module for performing motion estimation based on an image generation model, employing target audio data and reference keypoints, to obtain predicted keypoints matching the target audio data; an optical flow estimation module for performing parameter estimation based on the image generation model using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints, and performing a priori motion estimation based on the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; an image prediction module for performing image prediction based on the image generation model by employing a target reference image and a dense optical flow including an optical flow of predicted key points and an optical flow of non-key pixel points to obtain predicted image data matching the target audio data; Image generating device.

20. at least one processor; a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor is capable of performing the method of any one of claims 1 to 4, 6, 8 to 12 and 14 to 16; electronic equipment.

21. A computer having stored thereon computer instructions for causing a computer to carry out the method according to any one of claims 1 to 4, 6, 8 to 12 and 14 to 16. A non-transitory computer-readable storage medium.

22. When executed by a processor, the method according to any one of claims 1 to 4, 6, 8 to 12 and 14 to 16 is implemented. Computer program.

Images