Image generation model training method, image generation method, apparatus and equipment

Abstract

To provide a training method for an image generation model, an image generation method, apparatus, and a device.SOLUTION: A training method includes: extracting reference keypoints of a person from sample audio data, a sample reference image, and annotation image data; based on a model to be trained, performing motion estimation using the sample audio data and the reference keypoints to obtain predicted keypoints that match the sample audio data; performing parameter estimation using the reference keypoints and the predicted keypoints to obtain motion parameters of the predicted keypoints; performing prior motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points; performing image prediction using the sample reference image and dense optical flow to obtain predicted image data that matches the sample audio data; and performing model training using the predicted image data and annotated image data to obtain an image generation model.SELECTED DRAWING: Figure 1a

Description

Technical Field

[0001] The present disclosure relates to the field of computers, particularly to the field of artificial intelligence technology, specifically to the fields of technologies such as augmented reality (AR), virtual reality (VR), computer vision, deep learning, etc., and is applicable to scenes such as the metaverse, virtual digital humans, etc. Specifically, it relates to a method for training an image generation model, an image generation method, an apparatus, and a device.

Background Art

[0002] A virtual human, also called a digital human, refers to a virtual character with a digitized appearance, exists depending on a display device, and has a human appearance, human behavior (such as being able to talk or raise a hand), and human thoughts. The generation of virtual humans is an important technology in scenes such as the metaverse, intelligent customer service, e-commerce, etc. Its core is to generate continuous real digital human images, and there are broad application scenes and business needs.

[0003] How to generate a digital human image is very important.

Summary of the Invention

[0004] The present disclosure provides a method for training an image generation model, an image generation method, an apparatus, and a device.

[0005] According to one aspect of the present disclosure, obtain sample audio data, sample reference images, and annotation image data, and extract reference keypoints of a person from the sample reference images; perform motion estimation by adopting the sample audio data and the reference keypoints based on a model waiting for training to obtain predicted keypoints that match the sample audio data; Based on the aforementioned training-waiting model, parameter estimation is performed using reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed using the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points. Based on the aforementioned training model, image prediction is performed using a sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the sample audio data. This includes obtaining an image generation model by training the model using the aforementioned predicted image data and the aforementioned annotation image data, This provides a method for training image generation models.

[0006] According to one aspect of this disclosure, The process involves acquiring target audio data and target reference images, and extracting keypoints of the person from the target reference images. Based on an image generation model, motion estimation is performed using target audio data and reference keypoints to obtain predicted keypoints that match the target audio data. Based on the image generation model, parameter estimation is performed using reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed using the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points. Based on the aforementioned image generation model, the process includes performing image prediction using a target reference image and a dense optical flow that includes the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data. This provides an image generation method.

[0007] According to one aspect of this disclosure, A reference keypoint module for acquiring sample audio data, sample reference images, and annotation image data, and for extracting reference keypoints of people from the sample reference images, A prediction keypoint module for obtaining prediction keypoints that match the sample audio data by performing motion estimation using sample audio data and reference keypoints based on a training waiting model, An optical flow estimation module for obtaining the motion parameters of the predicted key points by employing the aforementioned training waiting model, employing the reference key points and predicted key points, employing the motion parameters of the predicted key points, and performing prior motion estimation to obtain the optical flow of non-key pixel points, An image prediction module for obtaining predicted image data that matches sample audio data by performing image prediction using a sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points, based on the aforementioned training model, The system includes a model training module for obtaining an image generation model by performing model training using the aforementioned predicted image data and the aforementioned annotation image data. We provide a training device for image generation models.

[0008] According to one aspect of this disclosure, A reference keypoint module for acquiring target audio data and target reference images, and for extracting reference keypoints of a person from the target reference image, A prediction keypoint module for obtaining prediction keypoints that match the target audio data by performing motion estimation based on an image generation model, using target audio data and reference keypoints, An optical flow estimation module for obtaining the motion parameters of predicted key points by employing reference key points and predicted key points based on the image generation model, and obtaining the optical flow of non-key pixel points by employing the motion parameters of predicted key points and performing prior motion estimation, and The system includes an image prediction module that, based on the image generation model, performs image prediction by employing a target reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data. We provide an image generation device.

[0009] According to another aspect of this disclosure, At least one processor, The system comprises memory that is communicated to at least one of the processors, The memory stores instructions that can be executed by the at least one processor, and the instructions provide electronic equipment that can be executed by the at least one processor so that the at least one processor can perform a method according to any embodiment of the present disclosure.

[0010] According to another aspect of this disclosure, a non-temporary computer-readable storage medium storing computer instructions for causing a computer to execute a method relating to any embodiment of this disclosure.

[0011] Further aspects of this disclosure provide a computer program product comprising a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0012] It should be understood that the information described in this section is not intended to identify any essential or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure are made clearer by the following specification. [Brief explanation of the drawing]

[0013] The drawings are provided for a better understanding of this embodiment and do not limit the disclosure.

[0014] [Figure 1a]It is a flowchart of a method for training an image generation model according to an embodiment of the present disclosure. [Figure 1b] It is a schematic diagram of a training principle of an image generation model according to an embodiment of the present disclosure. [Figure 2] It is a flowchart of another method for training an image generation model according to an embodiment of the present disclosure. [Figure 3a] It is a flowchart of a further method for training an image generation model according to an embodiment of the present disclosure. [Figure 3b] It is a schematic diagram of a processing mode of a sample reference image according to an embodiment of the present disclosure. [Figure 3c] It is a schematic diagram of a training principle of an image generation model according to an embodiment of the present disclosure. [Figure 4] It is a flowchart of an image generation method according to an embodiment of the present disclosure. [Figure 5] It is a flowchart of another image generation method according to an embodiment of the present disclosure. [Figure 6a] It is a flowchart of a further image generation method according to an embodiment of the present disclosure. [Figure 6b] It is a flowchart of a principle of image generation according to an embodiment of the present disclosure. [Figure 7] It is a schematic diagram of a structure of a training apparatus for an image generation model according to an embodiment of the present disclosure. [Figure 8] It is a schematic diagram of a structure of another image generation apparatus according to an embodiment of the present disclosure. [Figure 9] It is a block diagram of an electronic device for realizing a method for training an image generation model or an image generation method according to an embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0015] Figure 1a is a flowchart of a training method for an image generation model according to an embodiment of the present disclosure. The method is applicable when training an image generation model based on motion priorities. The method can be performed by an image generation model training device, which can be implemented in software and / or hardware and can be integrated into electronic equipment. As shown in Figure 1a, the training method for an image generation model according to this embodiment is S101 acquires sample audio data, sample reference images, and annotation image data, and extracts reference keypoints of a person from the sample reference image. S102 performs motion estimation based on a training waiting model, using sample audio data and reference keypoints to obtain predicted keypoints that match the sample audio data. Based on the aforementioned training waiting model, parameter estimation is performed by adopting reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed by adopting the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points S103, Based on the aforementioned training model, S104 performs image prediction using a sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the sample audio data. S105 may also include the step of training a model using the predicted image data and the annotation image data to obtain an image generation model.

[0016] In embodiments of this disclosure, a method for training an image generation model based on motor priori information is provided, which introduces motor priori information into the model training process, significantly reducing the difficulty of model learning and improving the training efficiency of the model. The image generation model is used to generate digital human images that match speech, with the facial and limb movements matching the speech, and in particular the mouth shape matching the speech. The input to the image generation model is continuous audio data and reference images, and it outputs a video in which the flow of mouth shape and limb movements is continuously transmitted, with the movements of the person in the video matching the speech.

[0017] Sample audio data, sample reference images, and annotation image data are acquired. The duration of a single sample audio data is a predetermined fixed value, for example, 800ms. A single sample audio data can correspond to a single frame of sample reference image and annotation image data, which also has a fixed duration. The movements of the person in the annotation image data are matched to the sample audio data. The image generation quality is improved by creating overlaps between different sample audio data and between different annotation image data, so that the screens of the images generated by the model are continuous.

[0018] In the embodiments of this disclosure, a sample reference image can be preprocessed to obtain reference keypoints of a person in the sample reference image. Specifically, keypoint detection can be performed on the sample reference image to obtain reference keypoints. These reference keypoints are used to provide the initial position of a person during the model training process, to inform the model of the approximate spatial position of the person's face, limbs, etc., and to improve the learning quality of the model.

[0019] Referring to Figure 1b, the training waiting model may include a motion estimation network 11, a parameter estimation network 22, and an image generation network 33. The motion estimation network 11 predicts a predicted keypoint at time t based on sample audio data and a reference keypoint at time t. The reference keypoint and the predicted keypoint at time t are input to the parameter estimation network 22 to obtain the motion parameters of the predicted keypoint at time t. The motion parameters of the predicted keypoint are then used to perform prior motion estimation to obtain the optical flow of non-key pixel points, that is, the optical flow of general pixel points around the predicted keypoint can be obtained. The motion parameters of the predicted keypoint include not only the optical flow of the predicted keypoint but also the influence parameters of the predicted keypoint on surrounding pixel points. The optical flow of each predicted keypoint and the optical flow of each non-key pixel point can constitute a dense optical flow at time t.

[0020] The image generation network 33 obtains predicted image data at time t by employing a dense optical flow between the sample reference image and time t, and uses this predicted image data to match the sample audio data. The predicted image data and the annotation image data are then compared, a loss function is constructed according to the comparison result, and the learnable parameters in the model are updated using the loss function to obtain an image generation model. The loss function and hyperparameters such as the learning rate are not specifically limited; for example, the loss function can be a VGG Perceptual Loss, a GAN loss, etc. The image generation network simultaneously generates the mouth shape and limb movements of the human face, improving the degree of rhythmic matching between the mouth shape and limb movements, thereby improving the realism of the digital human. By determining the motion parameters of the predicted keypoints and using these parameters to perform motion estimation, we obtain the optical flow of non-key pixel points, thereby obtaining a dense optical flow at time t. This dense optical flow is used as prior motion information for image prediction, and by using the sample reference image and the dense optical flow to perform image prediction, the difficulty of image prediction is significantly reduced, the efficiency of image prediction is improved, and the training efficiency of the model is enhanced.

[0021] The technical method according to the embodiment of this disclosure determines the motion parameters of the predicted keypoints, obtains the optical flow of non-key pixel points by performing motion estimation using the motion parameters of the predicted keypoints, that is, obtains a dense optical flow, generates predicted image data by adopting the sample reference image and the dense optical flow, obtains an image generation model by training the model using the predicted image data and annotation image data, and uses the dense optical flow as prior motion information for image generation, thereby significantly reducing the difficulty of image prediction, improving the efficiency of image prediction, and improving the training efficiency of the model.

[0022] In one preferred embodiment, obtaining predicted keypoints that match the sample audio data by performing motion estimation based on a training-wait model, using sample audio data and reference keypoints, includes encoding the sample audio data to obtain audio features based on the training-wait model, and performing motion estimation using reference keypoints and audio features to obtain predicted keypoints that match the sample audio data.

[0023] The motion estimation network 11 may include an audio encoder and a motion estimation unit. Specifically, sample audio data at time t is input to the audio encoder to obtain audio features, and the audio features and reference keypoints are input to the motion estimation unit to obtain predicted keypoints at time t. The network structure of the audio encoder and the motion estimation unit is not specifically limited; for example, the audio features may be conventional audio features MFCC (Mel-frequency cepstrum coefficients), or features based on deep learning, such as features based on a Wav2Vec 2.0 network. The motion estimation unit can employ networks such as ResNet (deep residual network), Unet (U-shaped network), or Transformer (transformer). By performing motion estimation in conjunction with the audio features and reference keypoints, predicted keypoints at time t are obtained. Since the predicted keypoints include mouth shape keypoints and limb movement keypoints, the predicted keypoints not only match the sample audio data, but also the mouth shape and limb movement, thereby further improving the image generation quality.

[0024] Figure 2 is a flowchart of a training method for another image generation model according to an embodiment of the present disclosure. Referring to Figure 2, the training method for the image generation model in this embodiment is as follows: S201 acquires sample audio data, sample reference images, and annotation image data, and extracts reference keypoints of people from the sample reference images. S202 performs motion estimation based on a training waiting model, using sample audio data and reference keypoints to obtain predicted keypoints that match the sample audio data. Based on the aforementioned training waiting model, S203 obtains the optical flow of the predicted keypoint by adopting the coordinates of the predicted keypoint and the coordinates of the reference keypoint, S204 involves using the optical flow of predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and using the optical flow and motion parameters of the auxiliary keypoints to perform prior motion estimation and obtain the optical flow of the non-key pixel points. S205 obtains predicted image data that matches the sample audio data by performing image prediction using the sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points, based on the aforementioned training waiting model. S206 may also include the process of obtaining an image generation model by training the model using the predicted image data and the annotation image data.

[0025] The training-waiting model may include a motion estimation network, a parameter estimation network, and an image generation network. The motion estimation network can perform motion estimation by adopting sample audio data and a reference keypoint at time t to obtain a predicted keypoint at time t. The parameter estimation network can obtain a sparse optical flow of the predicted keypoint by adopting the coordinates of the predicted keypoint and the coordinates of the reference keypoint.

number

number

number

[0026] Among these, the motion parameters of the predicted keypoints are used to represent the influence of the predicted keypoints on the motion of surrounding non-key pixel points. The parameter estimation network further employs the optical flow of the predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints, where the coordinates are (x p ,y p For a non-key pixel point p, an auxiliary keypoint can be selected from each predicted keypoint for that non-key pixel point. For example, the distance between the non-key pixel point and the predicted keypoint can be determined, and the closest one can be selected as an auxiliary keypoint. Then, the optical flow and motion parameters of the auxiliary keypoint can be used to perform motion estimation and obtain the optical flow of the non-key pixel point. By selecting an auxiliary keypoint for the non-key pixel point and using the optical flow and motion parameters of the auxiliary keypoint to perform prior motion estimation and obtain the optical flow of the non-key pixel point, the accuracy of the optical flow of the non-key pixel can be improved, and the quality of model learning can be improved.

[0027] In one preferred embodiment, obtaining the motion parameters of the predicted keypoints by employing the optical flow of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and obtaining the optical flow of the non-key pixel points by employing the optical flow and motion parameters of the auxiliary keypoints and performing prior motion estimation, includes determining the motion function followed by the predicted keypoints by employing the optical flow of the predicted keypoints, finding derivatives of the motion function based on a Taylor distribution to obtain first and second partial derivatives of the predicted keypoints moving in the horizontal and vertical directions, and performing prior motion estimation to obtain the optical flow of the non-key pixel points by employing the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the first and second partial derivatives of the auxiliary keypoints in the horizontal and vertical directions.

[0028] Based on a motion estimation network, the optical flow of predicted keypoints is adopted to determine the motion function followed by the predicted keypoints. Based on a Taylor distribution, the derivative of the motion function is obtained to obtain the partial derivative of the predicted keypoints, which are used as the motion parameters of the predicted keypoints. Motion estimation is then performed using the motion parameters of auxiliary keypoints to obtain the optical flow of non-key pixel points. By expanding the motion function based on a Taylor distribution to obtain the motion parameters of auxiliary keypoints, and adopting the motion parameters of auxiliary keypoints to determine the optical flow of non-key pixel points, the motion between predicted keypoints and non-key pixel points conforms to the same distribution and lies on the same motion curve. There is local motion agreement and correlation between predicted keypoints and non-key pixel points. For example, there is local agreement between the optical flow of the palm center and the optical flow of keypoints in the palm, and in the process of spreading the hand, the entire palm and wrist all move in one direction. By employing partial derivatives of auxiliary keypoints to estimate motion based on the local consistency and relevance of motion and obtaining the optical flow of non-key pixel points, the accuracy of motion estimation is improved, i.e., the quality of dense optical flow is enhanced. Subsequently, by introducing the local consistency and relevance of motion as prior knowledge into the image generation process, the difficulty of model learning can be significantly reduced, and the training efficiency of the model can be improved.

[0029] Specifically, by employing the coordinates of non-key pixel points, the coordinates of auxiliary keypoints, and the first and second partial derivatives of the auxiliary keypoints in the horizontal and vertical directions, prior motion estimation can be performed to obtain the optical flow of non-key pixel points.

number

number

number

number

number

number

number

[0030] In one preferred embodiment, after obtaining the optical flow of the non-key pixel points as described above, the process further includes determining the influence weights that the non-key pixel points receive from the auxiliary key points based on a Gaussian distribution, depending on the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the learnable influence radius, and scaling the optical flow of the non-key pixel points by adopting the influence weights that the non-key pixel points receive from the auxiliary key points to obtain the scaled optical flow of the non-key pixel points.

[0031] Of these, the influence radius is used to represent the degree to which a non-key pixel point is influenced by an auxiliary keypoint, and the degree of influence decreases as the influence radius increases. Specifically, the influence weight of a non-key pixel point from an auxiliary keypoint can be determined by the following formula:

number

number

number

number

number

number

number

[0032] In one preferred embodiment, after obtaining the optical flow of non-key pixel points, it is further included to correct the optical flow of non-key pixel points by employing a learnable optical flow offset amount to obtain the corrected optical flow of non-key pixel points.

[0033] In the embodiments of this disclosure, a learnable optical flow offset is further introduced, with an initial value of (0,0). The optical flow of non-key pixel points is corrected by adopting the learnable optical flow offset using the following formula:

number

number

number

[0034] The technical method according to the embodiments of this disclosure introduces the local consistency and correlation of motion as prior knowledge into the model learning process based on a second-order Taylor expansion, introduces the decay of motion tendency as prior knowledge into the model learning process based on a Gaussian distribution, and introduces the diversity of motion as learnable offset amounts into the model learning process, thereby significantly reducing the difficulty of model learning, improving the training efficiency of the model, and further enhancing the accuracy of optical flow and improving the quality of image generation.

[0035] In one preferred embodiment, obtaining predicted image data that matches sample audio data by performing image prediction using a sample reference image and a dense optical flow based on the training-wait model includes masking the dense optical flow and obtaining the masked dense optical flow based on the training-wait model, and obtaining predicted image data that matches sample audio data by performing image prediction using the sample reference image and the masked dense optical flow.

[0036] In the embodiments of this disclosure, stochastic masking can be performed on dense optical flow using the following formula:

number

number

[0037] Figure 3a is a flowchart of a further image generation model training method according to an embodiment of the present disclosure. Referring to Figure 3a, the image generation model training method of this embodiment is: S301 acquires sample audio data, sample reference images, and annotation image data, extracts reference keypoints, reference person images, and background images from the sample reference images, supplements the background images, and obtains the supplemented background images. S302 performs motion estimation based on a training waiting model, using sample audio data and reference keypoints to obtain predicted keypoints that match the sample audio data. S303 involves using the aforementioned training waiting model to perform parameter estimation by adopting reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and then using the motion parameters of the predicted keypoints to perform prior motion estimation to obtain the optical flow of non-key pixel points. Based on the aforementioned training model, S304 encodes the reference person profile to obtain the reference person profile features, S305 decodes the reference person profile features and the dense optical flow, which includes the optical flow of predicted keypoints and the optical flow of non-key pixel points, based on the aforementioned training-waiting model, to obtain predicted person profile data. S306 fuses the aforementioned predicted person image data with the supplemented background image to obtain predicted image data that matches the sample audio data. S307 may also include the step of training a model using the predicted image data and the annotation image data to obtain an image generation model.

[0038] Referring to Figure 3b, not only is a reference keypoint 012 extracted from the sample reference image 011 by keypoint detection, but a person image segmentation is performed on the sample reference image 011 to obtain a reference person image 014 and a background image 013. The background image 013 is then supplemented to obtain the supplemented background image 015. By separating the foreground reference person image from the background image, the model focuses only on the foreground person image and does not consider the background during the learning process, which not only reduces the difficulty of learning the model, but also avoids the problem of unnatural interconnections between the person image and the background in the predicted image data by fusing the subsequently generated predicted person image data with the supplemented background image.

[0039] The sample audio data at time t is encoded to obtain audio features. These audio features and reference keypoints are input into a motion estimation network to obtain predicted keypoints at time t. These predicted keypoints and reference keypoints are input into a parameter estimation network to obtain motion parameters for the predicted keypoints. These motion parameters are used to estimate the optical flow of non-key pixel points, resulting in a dense optical flow. A reference person image is input into the corresponding encoder to obtain reference person image features. These reference person image features and dense optical flow are input into an image generation network to obtain predicted person image data at time t. The predicted person image data at time t and the supplemented background image are fused to obtain predicted image data at time t.

[0040] In the embodiments of this disclosure, the alpha channel of the all-black areas in the predicted human figure data is set to 0, the alpha channel of the non-all-black areas is set to 1, and the predicted human figure data and the supplemented background image are fused based on the alpha channel to achieve a background superposition effect. Specifically, foreground-background fusion can be performed using the following formula:

number

[0041] In one preferred embodiment, obtaining predicted image data that matches sample audio data by performing image prediction using a sample reference image and a dense optical flow based on the training-wait model includes masking the dense optical flow and obtaining the masked dense optical flow based on the training-wait model, and obtaining predicted image data that matches sample audio data by performing image prediction using the sample reference image and the masked dense optical flow.

[0042] In the embodiments of this disclosure, probabilistic masking is further applied to the dense optical flow to obtain the masked dense optical flow, and image prediction is performed using the sample reference image and the masked dense optical flow to obtain predicted image data that matches the sample audio data. Specifically, a reference person image is input to a person image encoder to obtain reference person image features, and the reference person image features and the masked dense optical flow are input to an image generation network to obtain predicted person image data at time t. Then, the predicted person image data and the supplemented background image are fused to obtain predicted image data. By applying probabilistic masking to the dense optical flow, the information provided to the model is reduced, increasing the difficulty of learning and improving the robustness of the model.

[0043] Referring to Figure 3c, during the training phase, sample audio data is input to an audio encoder to obtain audio features, the reference keypoints and audio features are input to a motion estimation network to obtain predicted keypoints, the reference keypoints and predicted keypoints are input to a parameter estimation network to obtain motion parameters of the predicted keypoints, such as the influence radius range and first and second-order partial derivatives, and a second-order priori dense motion estimation is performed using the coordinates of the reference keypoints, the coordinates of the predicted keypoints, and the motion parameters of the predicted keypoints to obtain a dense optical flow. Probabilistic masking is performed on the dense optical flow to obtain a masked dense optical flow. Then, identity encoding is performed on the reference person image to obtain reference person image features, the reference person image features and the masked dense optical flow are input to a decoder to obtain predicted person image data, and the Alpha channel is fused with the supplemented background image and the predicted person image data to obtain predicted image data.

[0044] The technical proposal according to the embodiment of this disclosure improves the efficiency of image generation by determining the motion parameters of predicted keypoints, performing motion estimation using the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points, thereby obtaining a dense optical flow, and introducing the dense optical flow as prior knowledge into the image generation process. In this method, a training method for an image generation model based on second-order motion prior knowledge has been submitted, and through learning, it is possible to generate a digital human that matches audio data and improve the matching between the facial and limb movements of the digital human and the audio data.

[0045] Figure 4 is a flowchart of an image generation method according to an embodiment of the present disclosure. The method is applied when generating image data to match audio data. The method can be performed by an image generation device, which can be implemented in software and / or hardware and can be integrated into electronic equipment. As shown in Figure 4, the image generation method of this embodiment is S401 acquires target audio data and target reference images, and extracts reference keypoints of a person from the target reference image. S402 performs motion estimation based on an image generation model, using target audio data and reference keypoints to obtain predicted keypoints that match the target audio data. S403 involves using the aforementioned image generation model to perform parameter estimation by adopting reference keypoints and predicted keypoints to obtain motion parameters for the predicted keypoints, and then using the motion parameters for the predicted keypoints to perform prior motion estimation to obtain the optical flow of non-key pixel points. S404 may include, based on the image generation model, performing image prediction using a target reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data.

[0046] In the embodiments of this disclosure, a motion priori-based image generation method is provided, which significantly reduces the difficulty of model training and improves the efficiency of image generation by introducing motion priori information. The image generation model is used to generate digital human images that match speech, in which the facial and limb movements match the speech, and in particular the mouth shape matches the speech. The input to the image generation model is continuous audio data and a reference image, and it outputs a video in which the flow of mouth shape and limb movements is continuously transmitted, and the movements of the person in the video are matched to the speech.

[0047] Upon acquiring the input target audio data and target reference image, the target reference image can be preprocessed to obtain the reference keypoints of the person in the target reference image. Specifically, keypoint detection can be performed on the target reference image to obtain the reference keypoints. These reference keypoints provide the initial position of the person, informing the model of the approximate spatial position of the person's face, limbs, etc., and are used to improve the image generation quality.

[0048] The image generation model may include a motion estimation network, a parameter estimation network, and an image generation network. The motion estimation network predicts the predicted keypoint at time t based on the target audio data and reference keypoint at time t. The reference keypoint and the predicted keypoint at time t are input to the parameter estimation network to obtain the motion parameters of the predicted keypoint at time t. The motion parameters of the predicted keypoint are then used to perform prior motion estimation to obtain the optical flow of non-key pixel points, that is, the optical flow of general pixel points around the predicted keypoint can be obtained. The motion parameters of the predicted keypoint include not only the optical flow of the predicted keypoint but also the influence parameters of the predicted keypoint on surrounding pixel points. The optical flow of each predicted keypoint and the optical flow of each non-key pixel point can constitute a dense optical flow at time t.

[0049] By using an image generation network, predicted image data at time t is obtained by employing the target reference image and the dense optical flow at time t, which can then be used as predicted image data that matches the target audio data. By determining the motion parameters of the predicted keypoints and performing motion estimation using these parameters to obtain the optical flow of non-key pixel points, a dense optical flow at time t is obtained. This dense optical flow is used as prior motion information for image prediction, and by employing the target reference image and the dense optical flow to perform image prediction, the difficulty of image prediction is significantly reduced, and the efficiency of image prediction is improved.

[0050] The technical method according to the embodiment of this disclosure determines the motion parameters of the predicted keypoints, uses the motion parameters of the predicted keypoints to perform motion estimation and obtain the optical flow of non-key pixel points, thereby obtaining a dense optical flow, and uses the target reference image and the dense optical flow to generate predicted image data. In other words, by using the dense optical flow as prior motion information for image generation, the difficulty of image prediction is greatly reduced and the efficiency of image prediction is improved.

[0051] In one preferred embodiment, obtaining predicted keypoints that match the target audio data by performing motion estimation based on an image generation model, using target audio data and reference keypoints, includes encoding the target audio data to obtain audio features based on an image generation model, and performing motion estimation using reference keypoints and audio features to obtain predicted keypoints that match the target audio data.

[0052] The motion estimation network may include an audio encoder and a motion estimation unit. Specifically, target audio data at time t can be input to the audio encoder to obtain audio features, and these audio features and reference keypoints can be input to the motion estimation unit to obtain predicted keypoints at time t. By performing motion estimation in conjunction with the audio features and reference keypoints so that the predicted keypoints match the target audio data, the degree of matching between subsequent predicted image data and the target audio data is improved.

[0053] In one preferred embodiment, the method further includes obtaining keypoints of a customized operation and fusing the keypoints of the customized operation with predicted keypoints that match target speech data to obtain new predicted keypoints.

[0054] Furthermore, it provides additional support for the sequence of key points of user-customized actions. Specifically, it obtains key points of user-customized actions, fuses the customized action key points with predicted key points to obtain new predicted key points. For example, it can obtain user-customized limb movement key points, adopt predicted mouth shape key points that match the target speech data, and supplement the customized limb movement key points to obtain new predicted key points. This allows the new predicted key points to simultaneously consider the flexibility of limb movements, mouth shape consistency, and smoothness. Subsequently, the new predicted key points are adopted to determine a dense optical flow.

[0055] Figure 5 is a flowchart of a further image generation method according to an embodiment of the present disclosure. Referring to Figure 5, the image generation method of this embodiment is as follows: S501 acquires target audio data and target reference images, and extracts reference keypoints of a person from the target reference image. S502 performs motion estimation based on an image generation model, using target audio data and reference keypoints to obtain predicted keypoints that match the target audio data. S503 obtains the optical flow of the predicted keypoints by adopting the coordinates of the predicted keypoints and the coordinates of the reference keypoints based on the image generation model. S504 involves using the optical flow of predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and using the optical flow and motion parameters of the auxiliary keypoints to perform prior motion estimation and obtain the optical flow of the non-key pixel points. S505 may include, based on the image generation model, performing image prediction using a target reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data.

[0056] The image generation module may include a motion estimation network, a parameter estimation network, and an image generation network. The motion estimation network can perform motion estimation by adopting target audio data and a reference keypoint at time t to obtain a predicted keypoint at time t. The parameter estimation network can obtain a sparse optical flow of the predicted keypoint by adopting the coordinates of the predicted keypoint and the coordinates of the reference keypoint.

number

number

number

[0057] Among these, the motion parameters of the predicted keypoints are used to represent the influence of the predicted keypoints on the motion of surrounding non-key pixel points. The parameter estimation network further employs the optical flow of the predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints, where the coordinates are (x p ,y p For a non-key pixel point p, an auxiliary keypoint can be selected from each predicted keypoint for that non-key pixel point. For example, the distance between the non-key pixel point and the predicted keypoint can be determined, and the closest one can be selected as an auxiliary keypoint. The optical flow and motion parameters of the auxiliary keypoint can then be used to perform motion estimation and obtain the optical flow of the non-key pixel point. By selecting an auxiliary keypoint for the non-key pixel point and performing prior motion estimation using the optical flow and motion parameters of the auxiliary keypoint to obtain the optical flow of the non-key pixel point, the accuracy of the optical flow of the non-key pixel can be improved, and the quality of model learning can be enhanced.

[0058] In one preferred embodiment, obtaining the motion parameters of the predicted keypoints by employing the optical flow of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and obtaining the optical flow of the non-key pixel points by employing the optical flow and motion parameters of the auxiliary keypoints and performing prior motion estimation, includes determining the motion function followed by the predicted keypoints by employing the optical flow of the predicted keypoints, finding derivatives of the motion function based on a Taylor distribution to obtain first and second partial derivatives of the predicted keypoints moving in the horizontal and vertical directions, and performing prior motion estimation to obtain the optical flow of the non-key pixel points by employing the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the first and second partial derivatives of the auxiliary keypoints in the horizontal and vertical directions.

[0059] Based on a motion estimation network, the optical flow of predicted keypoints is adopted to determine the motion function followed by the predicted keypoints. Based on a Taylor distribution, the derivative of the motion function is obtained to obtain the partial derivative of the predicted keypoints, which are used as the motion parameters of the predicted keypoints. Motion estimation is then performed using the motion parameters of auxiliary keypoints to obtain the optical flow of non-key pixel points. By expanding the motion function based on a Taylor distribution to obtain the motion parameters of auxiliary keypoints, and adopting the motion parameters of auxiliary keypoints to determine the optical flow of non-key pixel points, the motion between predicted keypoints and non-key pixel points conforms to the same distribution and lies on the same motion curve. There is local motion agreement and correlation between predicted keypoints and non-key pixel points. For example, there is local agreement between the optical flow of the palm center and the optical flow of keypoints in the palm, and in the process of spreading the hand, the entire palm and wrist all move in one direction. By employing partial derivatives of auxiliary keypoints to estimate motion based on the local consistency and relevance of motion and obtaining the optical flow of non-key pixel points, the accuracy of motion estimation is improved, i.e., the quality of dense optical flow is enhanced. Subsequently, by introducing the local consistency and relevance of motion as prior knowledge into the image generation process, the difficulty of model learning can be significantly reduced, and the training efficiency of the model can be improved.

[0060] Specifically, by employing the coordinates of non-key pixel points, the coordinates of auxiliary keypoints, and the first and second partial derivatives of the auxiliary keypoints in the horizontal and vertical directions, prior motion estimation can be performed to obtain the optical flow of non-key pixel points.

number

number

number

number

number

number

[0061] In one preferred embodiment, after obtaining the optical flow of the non-key pixel points as described above, the process further includes determining the influence weights that the non-key pixel points receive from the auxiliary key points based on a Gaussian distribution, depending on the coordinates of the non-key pixel points, the coordinates of the auxiliary key points, and the learnable influence radius, and scaling the optical flow of the non-key pixel points by adopting the influence weights that the non-key pixel points receive from the auxiliary key points to obtain the scaled optical flow of the non-key pixel points.

[0062] Of these, the influence radius is used to represent the degree to which a non-key pixel point is influenced by an auxiliary keypoint, and the degree of influence decreases as the influence radius increases. Specifically, the influence weight of a non-key pixel point from an auxiliary keypoint can be determined by the following formula:

number

number

number

number

number

number

number

[0063] In one preferred embodiment, after obtaining the optical flow of the non-key pixel points as described above, it is further included to correct the optical flow of the non-key pixel points by employing a learnable optical flow offset amount to obtain the corrected optical flow of the non-key pixel points.

[0064] In the embodiments of this disclosure, a learnable optical flow offset is further introduced, and the optical flow of non-key pixel points is corrected by adopting the learnable optical flow offset using the following formula:

number

number

number

[0065] Figure 6a is a flowchart of a further image generation method according to an embodiment of the present disclosure. Referring to Figure 6a, the image generation method of this embodiment is: S601 acquires target audio data and target reference images, and extracts reference key points and reference person images from the target reference images. S602 performs motion estimation based on an image generation model, using target audio data and reference keypoints to obtain predicted keypoints that match the target audio data. Based on the image generation model, S603 involves taking reference keypoints and predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints, and then taking the motion parameters of the predicted keypoints to perform prior motion estimation and obtain the optical flow of non-key pixel points. Based on the aforementioned image generation model, S604 encodes a reference person image to obtain reference person image features, Based on the image generation model, S605 decodes the reference person image features and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted person image data. The system may also include S606, which fuses the predicted person profile data with the target background image to obtain predicted image data that matches the target voice data.

[0066] Keypoint detection not only extracts reference keypoints from the target reference image, but also performs person segmentation on the target reference image to obtain reference person images. By separating the foreground reference person image from the background image, the model focuses only on the foreground person image and does not consider the background during the prediction process, thereby reducing the difficulty of training the model.

[0067] The target audio data at time t is encoded to obtain audio features. These audio features and reference keypoints are input into a motion estimation network to obtain predicted keypoints at time t. These predicted keypoints and reference keypoints are input into a parameter estimation network to obtain motion parameters for the predicted keypoints. These motion parameters are used to estimate the optical flow of non-key pixel points, resulting in a dense optical flow. A reference person image is input into a person image encoder to obtain reference person image features. These reference person image features and dense optical flow are input into an image generation network to obtain predicted person image data at time t.

[0068] Then, the target background image is determined, and the generated predicted human figure data and the target background image are fused to obtain predicted image data. Specifically, the alpha channel of the all-black areas in the predicted human figure data is set to 0, the alpha channel of the non-all-black areas is set to 1, and the predicted human figure data and the target background image are fused based on the alpha channels to achieve a background superposition effect. Specifically, foreground-background fusion can be performed using the following formula:

number

[0069] In one preferred embodiment, the method further includes extracting a background image from a target reference image, supplementing it with the extracted background image, and using the supplemented background image as the target background image.

[0070] In the process of segmenting a person's image onto a target reference image, not only the reference person's image but also the background image is obtained. The background image is then supplemented, and the supplemented background image is used as the target background image. By fusing the generated predicted person's image data with the supplemented background image to obtain predicted image data, the interoperability between the person's image and the background in the predicted image data can be further improved, thereby further enhancing the image generation quality.

[0071] In one preferred embodiment, the method further includes obtaining a customized background image and setting it as the target background image.

[0072] The embodiments of this disclosure further support customized background images, and by using the customized background image as the target background image and fusing the generated predicted human figure data with the customized background image to obtain predicted image data, the flexibility of image generation can be further improved.

[0073] Referring to Figure 6b, in the image generation stage, the predicted keypoints are determined, the predicted keypoints and reference keypoints are input into a parameter estimation network to obtain the motion parameters of the predicted keypoints, and a second-order a priori dense motion estimation is performed using the coordinates of the predicted keypoints, the motion parameters of the predicted keypoints, and the coordinates of the reference keypoints to obtain a dense optical flow. Then, the identity is coded for the reference person image to obtain the reference person image features, the reference person image features and dense optical flow are input into a decoder to obtain predicted person image data, and the Alpha channel is fused with the target background image and the predicted person image data to obtain predicted image data. Note that probabilistic masking of the dense optical flow is not required in the image generation process, and the probabilistic masking module in the training process may be removed.

[0074] The technical solution according to the embodiment of this disclosure improves the efficiency of image generation by determining the motion parameters of predicted keypoints, performing motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points, thereby obtaining dense optical flow, and introducing the dense optical flow as prior knowledge into the image generation process. In other words, a method for generating the face and movements of a digital human based on second-order motion prior knowledge has been submitted, and through an end-to-end network, it simultaneously generates the mouth shape and limb movements of the digital human's face, further supports user-customized movements and backgrounds, and has a wide range of application scenarios.

[0075] Figure 7 is a schematic diagram of the structure of an image generation model training device according to an embodiment of the present disclosure. This embodiment is applicable when training an image generation model based on motion priori. The device can be implemented in software and / or hardware and can be integrated into electronic equipment. As shown in Figure 7, the image generation model training device 700 of this embodiment is A reference keypoint module 710 acquires sample audio data, sample reference images, and annotation image data, and extracts reference keypoints of a person from the sample reference image. A prediction keypoint module 720 for performing motion estimation based on a training waiting model, using sample audio data and reference keypoints, and obtaining prediction keypoints that match the sample audio data, An optical flow estimation module 730 is used to obtain the motion parameters of the predicted key points by employing the aforementioned training waiting model, and to obtain the optical flow of non-key pixel points by employing the motion parameters of the predicted key points and performing prior motion estimation. An image prediction module 740 for obtaining predicted image data that matches sample audio data by performing image prediction using a sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points, based on the aforementioned training model, The system may also include a model training module 750 for obtaining an image generation model by training the model using the predicted image data and the annotation image data.

[0076] In one preferred embodiment, the optical flow estimation module 730 is A key optical flow unit for obtaining the optical flow of a predicted key point by adopting the coordinates of the predicted key point and the coordinates of the reference key point based on the aforementioned training waiting model, The system includes a pixel optical flow unit for obtaining motion parameters of predicted keypoints by employing the optical flow of predicted keypoints and performing parameter estimation, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and obtaining the optical flow of non-key pixel points by employing the optical flow and motion parameters of the auxiliary keypoints and performing prior motion estimation.

[0077] In one preferred embodiment, the pixel optical flow unit is Specifically, the optical flow of the predicted keypoint is used to determine the motion function followed by the predicted keypoint, the derivatives of the motion function are obtained based on the Taylor distribution, and the first and second partial derivatives of the predicted keypoint's motion in the horizontal and vertical directions are obtained. It includes a first optical flow subunit used to obtain the optical flow of non-key pixel points by performing prior motion estimation using the coordinates of non-key pixel points, the coordinates of auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions.

[0078] In one preferred embodiment, the pixel optical flow unit is Specifically, based on a Gaussian distribution, the influence weights that non-key pixel points receive from auxiliary keypoints are determined according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the learnable influence radius. It includes a second optical flow subunit used to scale the optical flow of non-key pixel points by employing influence weights received by non-key pixel points from auxiliary keypoints, thereby obtaining the scaled optical flow of non-key pixel points.

[0079] In one preferred embodiment, the pixel optical flow unit is Specifically, it includes a third optical flow subunit used to correct the optical flow of non-key pixel points by employing a learnable optical flow offset amount, thereby obtaining the corrected optical flow of the non-key pixel points.

[0080] In one preferred embodiment, the reference keypoint module 710 is specifically used to extract reference keypoints, reference person images and background images from a sample reference image, to supplement the background image, and to obtain a supplemented background image. The aforementioned image prediction module 740 is Based on the aforementioned training-waiting model, a person image encoding unit is provided to encode a reference person image and obtain reference person image features, The system includes a person image decoding unit for decoding reference person image features and dense optical flow based on the aforementioned training model to obtain predicted person image data, and a fusion unit for fusing the predicted person image data with supplemented background images to obtain predicted image data that matches the sample audio data.

[0081] In one preferred embodiment, the fusion unit is Specifically, based on the aforementioned training waiting model, dense optical flow is masked, and a masked dense optical flow is obtained. This method uses a sample reference image and a masked, dense optical flow to perform image prediction and obtain predicted image data that matches the sample audio data.

[0082] In one preferred embodiment, the reference keypoint module 710 is Specifically, based on the training model, sample audio data is encoded to obtain audio features, This is used to perform motion estimation by employing reference keypoints and speech features to obtain predicted keypoints that match the sample speech data.

[0083] The technical proposal of the embodiment of this disclosure improves the efficiency of image generation by determining the motion parameters of predicted keypoints, performing motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points, thereby obtaining dense optical flow, and introducing the dense optical flow as prior knowledge into the image generation process. In this method, a training method for an image generation model based on second-order motion prior knowledge is provided, and through learning, it is possible to generate a digital human that matches audio data and improve the matching between the facial and limb movements of the digital human and the audio data.

[0084] Figure 8 is a schematic diagram of the structure of an image generation device according to an embodiment of the present disclosure. This embodiment is applied when generating image data to match audio data. The device can be implemented in software and / or hardware and can be integrated into electronic equipment. As shown in Figure 8, the image generation device 800 of this embodiment is A reference keypoint module 810 for acquiring target audio data and target reference images, and for extracting reference keypoints of a person from the target reference image, A prediction keypoint module 820 for performing motion estimation based on an image generation model, employing target audio data and reference keypoints, and obtaining prediction keypoints that match the target audio data, An optical flow estimation module 830 is used to obtain the motion parameters of the predicted key points by employing the aforementioned image generation model, and to obtain the optical flow of non-key pixel points by employing the motion parameters of the predicted key points and performing prior motion estimation. The system may also include an image prediction module 840 that performs image prediction based on the image generation model described above, using a target reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points, in order to obtain predicted image data that matches the target audio data.

[0085] In one preferred embodiment, the optical flow estimation module 830 is A key optical flow unit for obtaining the optical flow of predicted key points by adopting the coordinates of predicted key points and reference key points based on an image generation model, The system includes a pixel optical flow unit for obtaining motion parameters of predicted keypoints by employing the optical flow of predicted keypoints and performing parameter estimation, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and obtaining the optical flow of non-key pixel points by employing the optical flow and motion parameters of the auxiliary keypoints and performing prior motion estimation.

[0086] In one preferred embodiment, the pixel optical flow unit is Specifically, the optical flow of the predicted keypoint is used to determine the motion function followed by the predicted keypoint, the derivatives of the motion function are obtained based on the Taylor distribution, and the first and second partial derivatives of the predicted keypoint's motion in the horizontal and vertical directions are obtained. It includes a first optical flow subunit used to obtain the optical flow of non-key pixel points by performing prior motion estimation using the coordinates of non-key pixel points, the coordinates of auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions.

[0087] In one preferred embodiment, the pixel optical flow unit is Specifically, based on a Gaussian distribution, the influence weights that non-key pixel points receive from auxiliary keypoints are determined according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the learnable influence radius. The system further includes a second optical flow subunit used to scale the optical flow of non-key pixel points by employing influence weights from auxiliary keypoints, thereby obtaining the scaled optical flow of non-key pixel points.

[0088] In one preferred embodiment, the pixel optical flow unit is Specifically, it further comprises a third optical flow subunit used to correct the optical flow of non-key pixel points by employing a learnable optical flow offset amount, thereby obtaining the corrected optical flow of the non-key pixel points.

[0089] In one preferred embodiment, the reference keypoint module 810 is specifically used to extract reference keypoints and reference person images from a target reference image. The aforementioned image prediction module 840 is Based on the aforementioned image generation model, a person image encoding unit for encoding a reference person image to obtain reference person image features, Based on the aforementioned image generation model, a person image coding unit decodes reference person image features and dense optical flow to obtain predicted person image data, The system includes an image fusion unit for fusing the predicted person profile data with a target background image to obtain predicted image data that matches the target voice data.

[0090] In one preferred embodiment, the image prediction module 840 is Specifically, a background image is extracted from the target reference image, the extracted background image is used to supplement the target background image, and the supplemented background image is used as the target background image. Alternatively, the system further comprises a target background unit that acquires a customized background image and uses it as the target background image.

[0091] In one preferred embodiment, the predictive keypoint module 820 is Based on an image generation model, a speech coding unit is used to encode target audio data and obtain audio features, The system includes a keypoint prediction unit that performs motion estimation by employing reference keypoints and speech features, and obtains predicted keypoints that match the target speech data.

[0092] In one preferred embodiment, the predictive keypoint module 820 further includes: Obtain key points for customized behavior, It is used to merge the key points of the customized operation with the predicted key points that match the target audio data, in order to obtain new predicted key points.

[0093] The technical solution of the embodiment of this disclosure improves the efficiency of image generation by determining the motion parameters of predicted keypoints, performing motion estimation using the motion parameters of the predicted keypoints to obtain optical flow of non-key pixel points, thereby obtaining dense optical flow, and introducing the dense optical flow as prior knowledge into the image generation process. In other words, a method for generating the face and movements of a digital human based on second-order motion prior knowledge has been submitted, which simultaneously generates the mouth shape and limb movements of the digital human face using an end-to-end network, further supports user-customized movements and backgrounds, and has a wide range of application scenarios.

[0094] In the proposed technology disclosed herein, the acquisition, storage, and use of such user personal information are all in accordance with the provisions of relevant laws and regulations and do not violate public order and morals.

[0095] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, readable storage media, and computer programs.

[0096] Figure 9 is a block diagram of electronic equipment for implementing the image generation model training method or image generation method of the embodiment of this disclosure.

[0097] Figure 9 shows a schematic block diagram of an exemplary electronic device 900 that can be used to carry out embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may further represent various forms of mobile devices, such as personal digital processing devices, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are illustrative and are not intended to limit the implementation of the present disclosure as described and / or requested herein.

[0098] As shown in Figure 9, the electronic device 900 includes a computing unit 901 capable of performing various appropriate operations and processes based on computer programs stored in read-only memory (ROM) 902 or computer programs loaded from storage unit 908 into random access memory (RAM) 903. The RAM 903 can also store various programs and data necessary for operating the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are connected to each other by a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0099] Multiple components of an electronic device 900 are connected to the I / O interface 905, which includes, for example, an input unit 906 such as a keyboard or mouse, an output unit 907 such as various types of displays or speakers, a storage unit 908 such as a magnetic disk or optical disk, and a communication unit 909 such as a network card, modem, or wireless communication transceiver. The communication unit 909 enables the electronic device 900 to exchange information / data with other devices, for example, via the Internet computer network and / or various telecommunication networks.

[0100] The computing unit 901 may be a general-purpose and / or dedicated processing component having various processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs each of the methods and processes described above, for example, a method for training an image generation model or a method for generating an image. For example, in some embodiments, the method for training an image generation model or a method for generating an image can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, for example, a storage unit 908. In some embodiments, the computer program may be loaded and / or installed in the electronic device 900 in part or in whole via ROM 902 and / or a communication unit 909. Once the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the above-described method for training an image generation model or a method for generating an image can be executed. Alternatively, in other embodiments, the computing unit 901 is configured to perform a training method for an image generation model or an image generation method by any other suitable method (for example, by firmware).

[0101] In this specification, various embodiments of the systems and technologies described above can be implemented as digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific general-purpose products (ASSPs), systems-on-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a memory system, at least one input device, and at least one output device, and transmitting data and instructions to the memory system, the at least one input device, and the at least one output device.

[0102] Program code for carrying out the methods of this disclosure may be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations defined in the flowcharts and / or block diagrams are performed. The program code may be executed entirely on a device, partially on a device, as a standalone software package, partially on a device and partially on a remote device, or entirely on a remote device or server.

[0103] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or can store programs used in or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatus, or any appropriate combination of the above. More specific examples of machine-readable storage media include one or more line-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical memory devices, magnetic memory devices, or any appropriate combination of the above.

[0104] To provide user interaction, the systems and technologies described herein can be implemented on a computer, which may have a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard or pointing device (e.g., a mouse or trackball), and the user may provide input to the computer via the keyboard or pointing device. Other types of devices may be used to provide user interaction, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein may be implemented in a computing system including background components (e.g., as a data server), or in a computing system including middleware components (e.g., an application server), or in a computing system including front-end components (a user computer having an image user interface or a network browser, through which the user can interact with embodiments of the systems and technologies described herein), or in a computing system including any combination of such background components, middleware components, or front-end components. The components of the system may be connected to one another by digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0106] A computer system may include client terminals and servers. Client terminals and servers are generally geographically separated and typically interact via a communication network. The relationship between client terminals and servers is established by corresponding computer programs that run on the same computer and have a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology.

[0107] Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent actions (e.g., learning, reasoning, thinking, planning, etc.), and it encompasses both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing, while artificial intelligence software technologies mainly include several areas such as computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0108] Cloud computing is a technological framework that allows access to a shared, resilient, and scalable pool of physical or virtual resources via a network, enabling the deployment and management of resources as needed, from a service-oriented approach. These resources may include servers, operating systems, networks, software, applications, and storage devices. Cloud computing technology can provide efficient and robust data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0109] It should be understood that the steps can be rearranged, added, or deleted using the various forms of flows described above. For example, each step described herein may be performed in parallel, sequentially, or in a different order, as long as the desired results of the proposed technology disclosed herein are achieved.

[0110] The specific embodiments described above do not limit the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. The process involves acquiring sample audio data, sample reference images, and annotation image data, and extracting reference keypoints of a person from the sample reference images. Based on a training-waiting model, motion estimation is performed using sample audio data and reference keypoints to obtain predicted keypoints that match the sample audio data. Based on the aforementioned training-waiting model, parameter estimation is performed using reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed using the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points. Based on the aforementioned training model, image prediction is performed using a sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the sample audio data. This includes obtaining an image generation model by training the model using the aforementioned predicted image data and the aforementioned annotation image data, Based on the aforementioned training-waiting model, parameter estimation is performed by adopting reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed by adopting the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points. Based on the aforementioned training-waiting model, the coordinates of the predicted keypoints and the coordinates of the reference keypoints are used to obtain the optical flow of the predicted keypoints. This includes: employing the optical flow of predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints; selecting auxiliary keypoints from the predicted keypoints for non-key pixel points; and employing the optical flow and motion parameters of the auxiliary keypoints to perform prior motion estimation and obtain the optical flow of the non-key pixel points. Training methods for image generation models.

2. The process involves using the optical flow of predicted keypoints to estimate parameters and obtain the motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and then using the optical flow and motion parameters of the auxiliary keypoints to perform prior motion estimation and obtain the optical flow of the non-key pixel points. The optical flow of the predicted keypoint is used to determine the motion function followed by the predicted keypoint, and the derivatives of the motion function are obtained based on the Taylor distribution to obtain the first and second partial derivatives of the predicted keypoint moving in the horizontal and vertical directions. This includes performing prior motion estimation by employing the coordinates of non-key pixel points, the coordinates of auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions, and obtaining the optical flow of the non-key pixel points. The method according to claim 1.

3. After obtaining the optical flow for non-key pixel points, Based on a Gaussian distribution, the influence weights that non-key pixel points receive from auxiliary keypoints are determined according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the learnable influence radius. This further includes scaling the optical flow of non-key pixel points by adopting the influence weights they receive from auxiliary keypoints, and obtaining the scaled optical flow of non-key pixel points. The method according to claim 2.

4. After obtaining the optical flow for non-key pixel points, This further includes correcting the optical flow of non-key pixel points by employing a learnable optical flow offset amount, and obtaining the corrected optical flow of non-key pixel points. The method according to claim 2 or 3.

5. Extracting reference keypoints of a person from a sample reference image includes extracting reference keypoints, a reference person image, and a background image from the sample reference image, supplementing the background image, and obtaining the supplemented background image. Based on the aforementioned training model, image prediction is performed using a sample reference image and a dense optical flow to obtain predicted image data that matches the sample audio data. Based on the aforementioned training model, the reference person profile is encoded to obtain the reference person profile features, Based on the aforementioned training model, the reference person profile features and dense optical flow are decoded to obtain predicted person profile data. This includes fusing the aforementioned predicted person profile data with supplemented background images to obtain predicted image data that matches the sample audio data. The method according to claim 1.

6. Based on the aforementioned training model, image prediction is performed using a sample reference image and a dense optical flow to obtain predicted image data that matches the sample audio data. Based on the aforementioned training-waiting model, the dense optical flow is masked, and the masked dense optical flow is obtained. This includes performing image prediction using a sample reference image and a masked dense optical flow to obtain predicted image data that matches the sample audio data, The method according to claim 1 or 5.

7. Based on a training-waiting model, motion estimation is performed using sample audio data and reference keypoints to obtain predicted keypoints that match the sample audio data. Based on the training model, sample audio data is encoded to obtain audio features, This includes performing motion estimation using reference keypoints and speech features to obtain predicted keypoints that match sample speech data, The method according to claim 1.

8. The process involves acquiring target audio data and target reference images, and extracting keypoints of the person from the target reference images. Based on an image generation model, motion estimation is performed using target audio data and reference keypoints to obtain predicted keypoints that match the target audio data. Based on the image generation model, parameter estimation is performed using reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed using the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points. Based on the image generation model, the process includes performing image prediction using a target reference image and a dense optical flow that includes the optical flow of predicted keypoints and the optical flow of non-key pixel points to obtain predicted image data that matches the target audio data. Based on an image generation model, parameter estimation is performed using reference keypoints and predicted keypoints to obtain the motion parameters of the predicted keypoints, and prior motion estimation is performed using the motion parameters of the predicted keypoints to obtain the optical flow of non-key pixel points. Based on the image generation model, the optical flow of the predicted keypoint is obtained by adopting the coordinates of the predicted keypoint and the coordinates of the reference keypoint, This includes: employing the optical flow of predicted keypoints to perform parameter estimation and obtain the motion parameters of the predicted keypoints; selecting auxiliary keypoints from the predicted keypoints for non-key pixel points; and employing the optical flow and motion parameters of the auxiliary keypoints to perform prior motion estimation and obtain the optical flow of the non-key pixel points. Image generation method.

9. The process involves using the optical flow of predicted keypoints to estimate parameters and obtain the motion parameters of the predicted keypoints, selecting auxiliary keypoints for non-key pixel points from the predicted keypoints, and then using the optical flow and motion parameters of the auxiliary keypoints to perform prior motion estimation and obtain the optical flow of the non-key pixel points. The optical flow of the predicted keypoint is used to determine the motion function followed by the predicted keypoint, and the derivatives of the motion function are obtained based on the Taylor distribution to obtain the first and second partial derivatives of the predicted keypoint moving in the horizontal and vertical directions. This includes performing prior motion estimation by employing the coordinates of non-key pixel points, the coordinates of auxiliary key points, and the first and second partial derivatives of the auxiliary key points in the horizontal and vertical directions, and obtaining the optical flow of the non-key pixel points. The method according to claim 8.

10. After obtaining the optical flow for non-key pixel points, Based on a Gaussian distribution, the influence weights that non-key pixel points receive from auxiliary keypoints are determined according to the coordinates of the non-key pixel points, the coordinates of the auxiliary keypoints, and the learnable influence radius. This further includes scaling the optical flow of non-key pixel points by adopting the influence weights they receive from auxiliary keypoints, and obtaining the scaled optical flow of non-key pixel points. The method according to claim 9.

11. After obtaining the optical flow for non-key pixel points, This further includes correcting the optical flow of non-key pixel points by employing a learnable optical flow offset amount, and obtaining the corrected optical flow of non-key pixel points. The method according to claim 9 or 10.

12. Extracting reference keypoints of a person from a target reference image includes extracting reference keypoints and a reference person's image from the target reference image. Based on the aforementioned image generation model, obtaining predicted image data that matches the target audio data by performing image prediction using a target reference image and dense optical flow is possible. Based on the aforementioned image generation model, the reference person image is encoded to obtain the reference person image features, Based on the aforementioned image generation model, the reference person image features and dense optical flow are decoded to obtain predicted person image data. This includes fusing the aforementioned predicted person profile data with a target background image to obtain predicted image data that matches the target audio data. The method according to claim 8.

13. Extract a background image from the target reference image, supplement with the extracted background image, and use the supplemented background image as the target background image. Alternatively, the process may further include obtaining a customized background image and using it as the target background image. The method according to claim 12.

14. Based on an image generation model, motion estimation is performed by employing target audio data and reference keypoints to obtain predicted keypoints that match the target audio data. Based on an image generation model, the target audio data is encoded to obtain audio features, This includes performing motion estimation by employing reference keypoints and audio features, and obtaining predicted keypoints that match the target audio data. The method according to claim 8.

15. To obtain key points for customized behavior, This further includes fusing key points of customized actions with predicted key points that match target audio data to obtain new predicted key points, The method according to claim 8 or 14.

16. A reference keypoint module for acquiring sample audio data, sample reference images, and annotation image data, and for extracting reference keypoints of people from the sample reference images, A prediction keypoint module for obtaining prediction keypoints that match the sample audio data by performing motion estimation using sample audio data and reference keypoints based on a training waiting model, An optical flow estimation module for obtaining the motion parameters of the predicted key points by employing the aforementioned training waiting model, employing the reference key points and predicted key points, employing the motion parameters of the predicted key points, and performing prior motion estimation to obtain the optical flow of non-key pixel points, An image prediction module for obtaining predicted image data that matches sample audio data by performing image prediction using a sample reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points, based on the aforementioned training model, The system includes a model training module for obtaining an image generation model by training the model using the aforementioned predicted image data and the aforementioned annotation image data. The optical flow estimation module is Based on the aforementioned training model, the coordinates of the predicted keypoints and the coordinates of the reference keypoints are used to obtain the optical flow of the predicted keypoints. The optical flow of the predicted keypoints is used to estimate parameters and obtain the motion parameters of the predicted keypoints. Auxiliary keypoints are selected from the predicted keypoints for non-key pixel points. The optical flow and motion parameters of the auxiliary keypoints are used to perform prior motion estimation and obtain the optical flow of the non-key pixel points. Furthermore, the following methods are employed: A training device for image generation models.

17. A reference keypoint module for acquiring target audio data and target reference images, and for extracting reference keypoints of a person from the target reference image, A prediction keypoint module for obtaining prediction keypoints that match the target audio data by performing motion estimation based on an image generation model, using target audio data and reference keypoints, An optical flow estimation module for obtaining the motion parameters of predicted key points by employing reference key points and predicted key points based on the image generation model, and obtaining the optical flow of non-key pixel points by employing the motion parameters of predicted key points and performing prior motion estimation, and Based on the aforementioned image generation model, the system includes an image prediction module for obtaining predicted image data that matches the target audio data by employing a target reference image and a dense optical flow including the optical flow of predicted keypoints and the optical flow of non-key pixel points, thereby performing image prediction. The optical flow estimation module is Based on the image generation model, the optical flow of the predicted keypoint is obtained by adopting the coordinates of the predicted keypoint and the coordinates of the reference keypoint, The optical flow of the predicted keypoints is used to estimate parameters and obtain the motion parameters of the predicted keypoints. Auxiliary keypoints are selected from the predicted keypoints for non-key pixel points. The optical flow and motion parameters of the auxiliary keypoints are used to perform prior motion estimation and obtain the optical flow of the non-key pixel points. Furthermore, the following methods are employed: Image generation device.

18. At least one processor, The system comprises a memory that is communicated to at least one of the processors, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor such that the at least one processor can perform the method according to any one of claims 1 to 3, 5, 7 to 10 and 12 to 14. electronic equipment.

19. Computer instructions for causing a computer to perform the method according to any one of claims 1 to 3, 5, 7 to 10 and 12 to 14 are stored. A non-temporary computer-readable storage medium.

20. When executed by a processor, it realizes the method according to any one of claims 1 to 3, 5, 7 to 10 and 12 to 14. Computer program.

Images