Key point prediction model training and video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a key point prediction model training method and device, a video generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring audio features of a sample video and a first face key point set of a target object in the sample video; performing jitter removal processing on the first face key points in the first face key point set to obtain a second face key point set; performing key point prediction on the target object based on the audio features through a key point prediction model to obtain a third face key point set; determining a loss value of the key point prediction model based on the second face key point set and the third face key point set; and performing model parameter updating on the key point prediction model based on the loss value to obtain an updated key point prediction model. According to the application, the model performance of the key point prediction model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a key point prediction model training and video generation method, device, electronic device and storage medium. Background Art

[0002] Nowadays, digital human generation technology has been gradually applied. In the training process of key point prediction models, related technologies use training data sets or manually annotated training data sets, and do not consider the quality of label data in the training data sets. In addition, the annotation efficiency is low, which will seriously affect the model reliability and effectiveness of the key point prediction model. Summary of the invention

[0003] The embodiments of the present application provide a key point prediction model training and video generation method, device, electronic device and storage medium, which can improve the model performance of the key point prediction model.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] An embodiment of the present application provides a key point prediction model training method, the method comprising: obtaining audio features of a sample video and a first facial key point set of a target object in the sample video; based on a first facial key point subset corresponding to each of two adjacent image frames in the first facial key point set, de-jittering the first facial key point in the first facial key point set to obtain a second facial key point set; using a key point prediction model, performing key point prediction on the target object based on the audio features to obtain a third facial key point set; determining a loss value of the key point prediction model based on the second facial key point set and the third facial key point set; and updating model parameters of the key point prediction model based on the loss value to obtain an updated key point prediction model.

[0006] An embodiment of the present application provides a video generation method, which includes: using a key point prediction model to perform key point prediction on audio features of a target audio to obtain a fifth facial key point set; the key point prediction model is trained using the key point prediction model training method; based on the fifth facial key point set, a plurality of frames of continuous facial contour images are generated; using a pre-trained digital object generation model, image processing is performed on the facial contour image to obtain a plurality of frames of continuous digital object images; and a digital object video is generated based on the plurality of frames of continuous digital object images.

[0007] An embodiment of the present application provides a key point prediction model training device, including: an acquisition module, used to acquire audio features of a sample video and a first facial key point set of a target object in the sample video; a de-jitter processing module, used to perform de-jitter processing on the first facial key point in the first facial key point set based on a first facial key point subset corresponding to each of two adjacent image frames in the first facial key point set, so as to obtain a second facial key point set; a key point prediction module, used to perform key point prediction on the target object based on the audio features through a key point prediction model, so as to obtain a third facial key point set; a determination module, used to determine a loss value of the key point prediction model based on the second facial key point set and the third facial key point set; and an update module, used to update model parameters of the key point prediction model based on the loss value, so as to obtain an updated key point prediction model.

[0008] In the above scheme, the de-jitter processing module is also used to: based on the first facial key point subsets corresponding to each of two adjacent image frames, perform a first de-jitter processing on the first facial key point in the first facial key point set to obtain a fourth facial key point set; wherein the fourth facial key point set includes fourth facial key point subsets corresponding to multiple adjacent image frames; the fourth facial key point subset corresponding to each image frame includes multiple local key point subsets; based on the local key point subsets corresponding to each of two adjacent image frames, perform a second de-jitter processing on the fourth facial key point in the fourth facial key point set to obtain the second facial key point set.

[0009] In the above scheme, the de-jitter processing module is also used to: determine, for the first facial key point subset corresponding to the i-th image frame in the first facial key point set, an offset value between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame; i is an integer greater than 1; when the offset value is less than a preset first offset threshold, replace the first facial key point subset corresponding to the i-th image frame with the first facial key point subset corresponding to the i-1-th image frame to obtain the fourth facial key point set; when the offset value is greater than or equal to the first offset threshold, update the first facial key point subset corresponding to the i-th image frame based on the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame to obtain the fourth facial key point set.

[0010] In the above scheme, the de-jitter processing is also used to: determine a first updated subset based on a preset first proportion and a first facial key point subset corresponding to the i-th image frame; determine a second updated subset based on a second proportion and a first facial key point subset corresponding to the i-1-th image frame; the second proportion is determined based on the first proportion; determine an updated first facial key point subset corresponding to the i-th image frame based on the first updated subset and the second updated subset; determine the fourth facial key point set based on the updated first facial key point subset.

[0011] In the above scheme, the de-jitter processing module is also used to: determine the number of local key points that have position offsets between the first local key point subset and the second local key point subset; determine the proportion of local offset key points based on the number and the total number of local key points in the first local key point subset; wherein the first local key point subset is the jth local key point subset corresponding to the i-th image frame, and the second local key point subset is the jth local key point subset corresponding to the i-1th image frame; i is an integer greater than 1, j is an integer greater than 1 and less than or equal to a first value, and the first value is equal to the number of local key point subsets of each image frame; based on the proportion of local offset key points, perform a second de-jitter processing on the fourth facial key point in the fourth facial key point set to obtain the second facial key point set.

[0012] In the above scheme, the de-jitter processing module is also used to: when the proportion of the local offset key points is less than a preset proportion threshold, determine the fourth facial key point set as the second facial key point set; when the proportion of the local offset key points is greater than or equal to the proportion threshold, replace the jth local key point subset corresponding to the i-1th image frame with the jth local key point subset corresponding to the i-1th image frame to obtain the second facial key point set.

[0013] In the above scheme, the second facial key point set includes a second facial key point subset corresponding to each of a plurality of adjacent image frames; the third facial key point set includes a third facial key point subset corresponding to each of a plurality of adjacent image frames; the determination module is further used to: determine a first loss value of the key point prediction model based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set, and the third facial key point subset corresponding to the i-th image frame in the third facial key point set; i is an integer greater than 1; determine a second loss value of the key point prediction model based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set, the second facial key point subset corresponding to the i-1-th image frame, the third facial key point subset corresponding to the i-th image frame in the third facial key point set, and the third facial key point subset corresponding to the i-1-th image frame; and determine the loss value based on the first loss value and the second loss value.

[0014] In the above scheme, the determination module is also used to: determine the absolute value of the offset value between the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame; when the absolute value is less than a preset second offset threshold, determine the first loss value based on the absolute value and the second offset threshold; when the absolute value is greater than or equal to the second offset threshold, determine the first loss value based on the absolute value.

[0015] In the above scheme, the determination module is also used to: determine a first offset between a second facial key point subset corresponding to the i-th image frame and a second facial key point subset corresponding to the i-1-th image frame, and a second offset between a third facial key point subset corresponding to the i-th image frame and a third facial key point subset corresponding to the i-1-th image frame; and determine the second loss value based on the first offset and the second offset.

[0016] In the above scheme, the acquisition module is also used to: extract audio features from the speech data of the target object in the sample video through a pre-trained audio feature extraction module to obtain the audio features; and predict facial key points from the image data of the target object in the sample video through a pre-trained facial key point prediction module to obtain the first facial key point set.

[0017] The embodiment of the present application provides a video generation device, including: a prediction module, used to perform key point prediction on the audio features of the target audio through a key point prediction model to obtain a fifth facial key point set; the key point prediction model is trained using the key point prediction model training method provided in the embodiment of the present application; a first generation module, used to generate a plurality of frames of continuous facial contour images based on the fifth facial key point set; an image processing module, used to perform image processing on the facial contour image through a digital object generation model to obtain a plurality of frames of continuous digital object images; and a second generation module, used to generate a digital object video based on the plurality of frames of continuous digital object images.

[0018] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the key point prediction model training method provided in the embodiment of the present application, or implementing the video generation method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the key point prediction model training method provided in the embodiment of the present application when executed by a processor, or implementing the video generation method provided in the embodiment of the present application.

[0020] An embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the computer executable instructions from the computer-readable storage medium and executes the computer executable instructions, the key point prediction model training method provided in the embodiment of the present application is implemented, or the video generation method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] When the key point prediction model is trained, the first facial key point set obtained is the original model label for calculating the loss value of the key point prediction model. Before the first facial key point set is used as the model label of the key point prediction model, the first facial key point in the first facial key point set is de-jittered in consideration of the jitter problem of the first facial key point in the first facial key point set, and the second facial key point set after the de-jittering is used as the model label for calculating the loss value of the key point prediction model, thereby improving the stability of the model label of the key point prediction model; and the loss value of the key point prediction model is calculated based on the third facial key point set obtained by the model prediction and the second facial key point set after the de-jittering. In this way, since there are no jittered facial key points in the second facial key point set after the de-jittering, the loss value of the key point prediction model is the difference between the second facial key point set after the de-jittering and the third facial key point set. Therefore, based on the stable model label, the loss value of the key point prediction model can be accurately calculated, thereby accurately updating the model parameters of the key point prediction model based on the loss value, and improving the model performance of the key point prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is an optional flowchart of the key point prediction model training method provided in the embodiment of the present application;

[0024] Figure 2 is a schematic diagram of determining the audio features and the first facial key point set provided in an embodiment of the present application;

[0025] Figure 3 is an optional schematic diagram for determining the second facial key point set provided in an embodiment of the present application;

[0026] Figure 4 is an optional schematic diagram for determining the fourth facial key point set provided in an embodiment of the present application;

[0027] Figure 5 is another optional schematic diagram for determining the fourth facial key point set provided in an embodiment of the present application;

[0028] Figure 6 is another optional schematic diagram for determining the second facial key point set provided in an embodiment of the present application;

[0029] Figure 7 It is a schematic diagram for determining the loss value of the key point prediction model provided in an embodiment of the present application;

[0030] Figure 8 It is a schematic diagram for determining the first loss value of the key point prediction model provided in an embodiment of the present application;

[0031] Fig. 9 It is a schematic diagram for determining the second loss value of the key point prediction model provided in an embodiment of the present application;

[0032] Fig.10 This is an optional flowchart of the video generation method provided in the embodiment of the present application;

[0033] Fig.11 It is a flowchart of a key point-guided voice-driven digital human method provided in an embodiment of the present application;

[0034] Fig.12 It is a schematic diagram of the structure of the Audio2Landmarks network provided in an embodiment of the present application;

[0035] Fig.13 It is a schematic diagram comparing the WingLoss and L2 functions provided in the embodiments of the present application;

[0036] Fig.14 It is a schematic diagram of the process of generating a digital human image by the Pix2pix model provided in an embodiment of the present application;

[0037] Fig.15 It is a structural block diagram of a key point prediction model training device provided in an embodiment of the present application;

[0038] Fig.16 It is a structural block diagram of a video generation provided by an embodiment of the present application;

[0039] Fig.17 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] The embodiment of the present application provides a key point prediction model training method. When the key point prediction model is trained, the first facial key point set is the original model label used to calculate the loss value of the key point prediction model. Before the first facial key point set is used as the model label of the key point prediction model, the first facial key point in the first facial key point set is de-jittered in consideration of the jitter problem of the first facial key point in the first facial key point set, and the second facial key point set after the de-jittering is used as the model label for calculating the loss value of the key point prediction model, thereby improving the stability of the model label of the key point prediction model. Thus, based on the third facial key point set predicted by the model and the loss value calculated by the second facial key point set after the de-jittering, the model parameters of the key point prediction model are accurately updated, thereby improving the model performance of the key point prediction model.

[0044] The key point prediction model training method and video generation method provided in the embodiments of the present application can be applied to electronic devices such as robots, laptops, tablet computers, desktop computers, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, smart home appliances, and smart vehicle-mounted devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0045] The key point prediction model training method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0046] Figure 1 1 is an optional flow chart of a key point prediction model training method provided in an embodiment of the present application, which can be applied to electronic devices. Figure 1 As shown, the method includes the following steps S101 to S105:

[0047] Step S101, obtaining audio features of a sample video and a first facial key point set of a target object in the sample video.

[0048] The sample video is a sample data set for training a key point prediction model. The sample video includes the voice data and image data of the target object. The object appearing in the sample video can be used as the target object. The voice data and image data of the target object are obtained by decomposing the video file of the sample video into an audio stream and a video stream. The decomposition of the video file can be completed by a video processing library or tool, such as FFmpeg, OpenCV, etc. The audio processing library can be used to read the voice data containing multiple continuous audio segments from the audio stream, and the image processing library can be used to read the image data containing multiple continuous image frames from the video stream. The voice data and image data in the sample video are synchronized, so each image frame in the sample video corresponds to a specific audio segment in the sample video. For example, synchronization can be achieved in the following ways: using the timestamp and frame rate of the sample video to determine the time period of each image frame corresponding to the audio, or, according to the sampling rate of the audio and the frame rate of the sample video, calculating the range of the audio segment corresponding to each image frame.

[0049] See also Figure 2 , Figure 2 It is shown that in step S101, obtaining the audio features of the sample video and the first facial key point set of the target object in the sample video can be achieved by the following steps S1011 to S1012:

[0050] Step S1011, extracting audio features from the speech data of the target object in the sample video using a pre-trained audio feature extraction module to obtain audio features.

[0051] In an embodiment of the present application, the pre-trained audio feature extraction module is a neural network that has been trained by a model and can be used to accurately extract audio features from the speech data of the target object. For example, the pre-trained audio feature extraction module can be an adversarial training network module (AT-Net). The network structure of the audio feature extraction module is not specifically limited here. After obtaining the speech data of the target object in the sample video, the speech data is directly input into the pre-trained audio feature extraction module, and the audio feature extraction is performed on the speech data through the pre-trained audio feature extraction module to obtain the audio features of the sample video.

[0052] Step S1012: perform facial key point prediction on the image data of the target object in the sample video using a pre-trained facial key point prediction module to obtain a first facial key point set.

[0053] In an embodiment of the present application, the pre-trained facial key point prediction module is a neural network that has been trained by a model and can be used to predict facial key points of the image data of the target object. For example, the pre-trained facial key point prediction module can be a two-dimensional facial alignment network module (2D Face Alignment Network, 2DFAN). The network structure of the facial key point prediction module is not specifically limited here. The first facial key point set refers to a series of key points on the face of the target object detected and located by the facial key point prediction module. These key points represent the important feature positions of the face, usually including the key positions of the eyes, nose, mouth, eyebrows and other parts.

[0054] After obtaining the image data of the target object in the sample video, the image data is directly input into the pre-trained facial key point prediction module, and the pre-trained facial key point prediction module is used to perform audio feature extraction and facial key point prediction on the image data to obtain the first facial key point set of the target object in the sample video.

[0055] Here, through the audio feature extraction capability of the pre-trained audio feature extraction module and the facial key point prediction capability of the facial key point prediction module, the input data (i.e., audio features) and original label data (i.e., the first facial key point set of the target object) used to train the key point prediction model can be accurately obtained.

[0056] Step S102, based on the first facial key point subsets corresponding to two adjacent image frames in the first facial key point set, de-jittering is performed on the first facial key point set to obtain a second facial key point set.

[0057] In the embodiment of the present application, the first facial key point set includes a first facial key point subset corresponding to each of a plurality of adjacent image frames, that is, each image frame corresponds to a first facial key point subset. Since the first facial key point set is predicted by a pre-trained facial key point prediction module, there may be first facial key points with positional offset (i.e., jitter) between the first facial key point subsets corresponding to each of two adjacent image frames in the first facial key point set, that is, the first facial key points with positional offset in the first facial key point subset corresponding to the current image frame relative to the first facial key point subset corresponding to the previous image frame. Positional offset refers to the positional deviation of the positional coordinates of the first facial key point in the first facial key point subset corresponding to the current image frame relative to the positional coordinates of the first facial key point in the first facial key point subset corresponding to the previous image frame. Therefore, before using the first facial key point set as the label data of the key point prediction model, it is necessary to perform de-jitter processing on the first facial key point in the first facial key point set, and use the first facial key point set after de-jitter processing as the label data of the key point prediction model. That is, de-jittering is the process of eliminating the first facial key points that exist in the first facial key point set. The purpose of de-jittering is to convert the unstable original label data of the key point prediction model (i.e., the first facial key point set) into stable label data (i.e., the second facial key point set).

[0058] In some embodiments, see Figure 3 , Figure 3 It is shown that in step S102, the first facial key point in the first facial key point set is debounced to obtain a second facial key point set, which can be achieved by the following steps S1021 to S1022:

[0059] Step S1021, based on the first facial key point subsets corresponding to each of the two adjacent image frames, a first de-jittering process is performed on the first facial key point in the first facial key point set to obtain a fourth facial key point set.

[0060] In the embodiment of the present application, the fourth facial key point set refers to the facial key point set obtained after the first de-jittering process is performed on the first facial key point in the first facial key point set, and the fourth facial key point set includes a fourth facial key point subset corresponding to each of a plurality of adjacent image frames, that is, each image frame corresponds to a fourth facial key point subset. The first de-jittering process is mainly used to determine whether there is jitter of facial key points between the first facial key point subsets corresponding to each of two adjacent image frames, and the first de-jittering process is a de-jittering process performed on the facial key points of each image frame as a whole. If jitter exists, it means that the first facial key point in the first facial key point set is not stable and cannot be directly used as label data of the key point prediction model, and the first facial key point in the first facial key point set needs to be de-jittered.

[0061] In some embodiments, see Figure 4 , Figure 4 It is shown that in step S1021, based on the first facial key point subsets corresponding to the two adjacent image frames, the first facial key point in the first facial key point set is subjected to the first de-jittering process to obtain the fourth facial key point set, which can be achieved by the following steps S401 to S403:

[0062] Step S401, for a first facial key point subset corresponding to an i-th image frame in a first facial key point set, determine an offset value between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame.

[0063] In the embodiment of the present application, i is an integer greater than 1; the first facial key point in the first facial key point subset corresponding to the i-th image frame and the first facial key point in the first facial key point subset corresponding to the i-1-th image frame are arranged according to the same sorting rule, so there is a one-to-one correspondence between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame. The offset value between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame indicates that there are first facial key points with inconsistent position coordinates in the first facial key point subsets corresponding to the two adjacent image frames, the i-th image frame and the i-1-th image frame, that is, there is jitter.

[0064] For the first facial key point subset corresponding to the i-th image frame in the first facial key point set, calculate the sum of the position coordinate differences between all the first facial key points in the first facial key point subset corresponding to the i-th image frame and all the first facial key points in the first facial key point subset corresponding to the i-1-th image frame, that is, the offset value.

[0065] Step S402, when the offset value is less than a preset first offset threshold, the first facial key point subset corresponding to the i-th image frame is replaced with the first facial key point subset corresponding to the i-1-th image frame to obtain a fourth facial key point set.

[0066] In the embodiment of the present application, the preset first offset threshold refers to a value preset to measure whether the jitter between the first facial key point subsets corresponding to two adjacent image frames is too large. When the offset value between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1th image frame is less than the preset first offset threshold, it means that the jitter between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1th image frame is small, and the first facial key point subset corresponding to the i-th image frame can be directly replaced with the first facial key point subset corresponding to the i-th image frame, that is, the first facial key point subset corresponding to the i-th image frame is replaced with the first facial key point subset corresponding to the i-1th image frame to eliminate the jitter between the first facial key point subsets corresponding to the two adjacent image frames. The first facial key point set after replacement is the fourth facial key point set, and there is no jitter between the first facial key point subsets corresponding to the two adjacent image frames in the fourth facial key point set.

[0067] For example, assuming that the first facial key point subsets corresponding to the multiple adjacent image frames included in the first facial key point set are A, B, C, D, E and F respectively, and the offset value between the first facial key point subset E corresponding to the 5th image frame and the first facial key point subset D corresponding to the 4th image frame is less than a preset first offset threshold, then the first facial key point subset E corresponding to the 5th image frame is replaced with the first facial key point subset D corresponding to the 4th image frame, and the facial key point subsets corresponding to the multiple adjacent image frames included in the fourth facial key point set are A, B, C, D, D and F respectively.

[0068] Step S403, when the offset value is greater than or equal to the first offset threshold, based on the first facial key point subset corresponding to the ith image frame and the first facial key point subset corresponding to the (i-1)th image frame, the first facial key point subset corresponding to the ith image frame is updated to obtain a fourth facial key point set.

[0069] In an embodiment of the present application, when the offset value between the first facial key point subset corresponding to the ith image frame and the first facial key point subset corresponding to the i-1th image frame is greater than or equal to a preset first offset threshold, it indicates that the jitter between the first facial key point subset corresponding to the ith image frame and the first facial key point subset corresponding to the i-1th image frame is large, and the first facial key point subset corresponding to the ith image frame needs to be updated to reduce the jitter between the first facial key point subsets corresponding to two adjacent image frames in the updated first facial key point set (i.e., the fourth facial key point set).

[0070] In some embodiments, see Figure 5 , Figure 5 It is shown that in step S403, based on the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame, the first facial key point subset corresponding to the i-th image frame is updated to obtain a fourth facial key point set, which can be achieved by the following steps S4031 to S4034:

[0071] Step S4031: Determine a first updated subset based on a first facial key point subset having a preset first ratio and corresponding to the i-th image frame.

[0072] In the embodiment of the present application, the preset first proportion refers to the proportion of the first facial key point subset corresponding to the i-th image frame, which is preset under a specific situation, relative to the first facial key point subset corresponding to the i-1th image frame. The specific situation refers to updating the first facial key point subset corresponding to the i-th image frame based on the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1th image frame. When the first proportion and the first facial key point subset corresponding to the i-th image frame are determined, the first facial key point subset corresponding to the i-th image frame is weighted by the first proportion to obtain the weighted first facial key point subset corresponding to the i-th image frame, i.e., the first updated subset.

[0073] Step S4032: Determine a second updated subset based on the first facial key point subset having a second proportion and corresponding to the (i-1)th image frame.

[0074] In the embodiment of the present application, the second proportion is determined based on the first proportion. For example, when the first proportion is α, the first proportion is 1-α. The second proportion refers to the proportion of the first facial key point subset corresponding to the i-1th image frame determined by the first proportion relative to the first facial key point subset corresponding to the i-th image frame under specific circumstances. The specific situation refers to updating the first facial key point subset corresponding to the i-th image frame based on the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1th image frame. When the second proportion and the first facial key point subset corresponding to the i-1th image frame are determined, the first facial key point subset corresponding to the i-1th image frame is weighted by the second proportion to obtain the weighted first facial key point subset corresponding to the i-1th image frame, i.e., the second updated subset.

[0075] Step S4033: Determine an updated first facial key point subset corresponding to the i-th image frame based on the first updated subset and the second updated subset.

[0076] In the embodiment of the present application, the updated first facial key point subset corresponding to the i-th image frame is a subset obtained by summing the first updated subset and the second updated subset. After obtaining the first updated subset corresponding to the i-th image frame and the second updated subset corresponding to the i-1-th image frame, the first updated subset and the second updated subset are added to obtain the updated first facial key point subset corresponding to the i-th image frame.

[0077] Step S4034: Determine a fourth facial key point set based on the updated first facial key point subset.

[0078] In an embodiment of the present application, the updated first facial key point subset corresponding to the i-th image frame replaces the original first facial key point subset corresponding to the i-th image frame in the first facial key point set to obtain an updated first facial key point set, i.e., the fourth facial key point set.

[0079] Here, by combining the first facial key point subset corresponding to the ith image frame and the first facial key point subset corresponding to the i-1th image frame, and adjusting the proportions of the above two first facial key point subsets, the first facial key point subset corresponding to the ith image frame can be effectively updated.

[0080] By comparing the offset value between the first facial key point subsets corresponding to two adjacent image frames with a preset first offset threshold, it is determined whether the jitter between the first facial key point subsets corresponding to the two adjacent image frames is large. When the jitter is small, the first facial key point subset corresponding to the previous image frame is directly used to replace the first facial key point subset corresponding to the current image frame to quickly eliminate the jitter; when the jitter is large, the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame are combined to calculate the updated first facial key point subset corresponding to the i-th image frame, thereby achieving an effective update of the first facial key point subset corresponding to the i-th image frame, thereby effectively reducing the jitter.

[0081] Step S1022: Based on the local key point subsets corresponding to two adjacent image frames, a second de-jittering process is performed on the fourth facial key point in the fourth facial key point set to obtain a second facial key point set.

[0082] In the embodiment of the present application, the fourth facial key point set includes a fourth facial key point subset corresponding to each of a plurality of adjacent image frames; the fourth facial key point subset corresponding to each image frame includes a plurality of local key point subsets. A local key point subset refers to a series of key points corresponding to a certain part of the face of a target object. For example, a local key point subset may be an eye key point subset, a nose key point subset, a mouth key point subset, etc. of the target object, which respectively represent key position sets of eyes, nose, mouth, etc. The second de-jittering process is mainly used to determine whether there is jitter of facial key points between the local key point subsets corresponding to each of two adjacent image frames. The second de-jittering process is a de-jittering process performed on the local facial key points of each image frame. If there is jitter, it means that the fourth facial key point in the fourth facial key point set is not stable and cannot be directly used as label data of the key point prediction model. It is also necessary to perform de-jittering on the fourth facial key point in the fourth facial key point set obtained by the first de-jittering process.

[0083] In some embodiments, see Figure 6 , Figure 6 It is shown that in step S1022, based on the local key point subsets corresponding to two adjacent image frames, the fourth facial key point in the fourth facial key point set is subjected to a second de-jittering process to obtain a second facial key point set, which can be achieved by the following steps S601 to S603:

[0084] Step S601: determine the number of local key points with position offsets between the first local key point subset and the second local key point subset.

[0085] Step S602: Determine the proportion of local offset key points based on the number and the total number of local key points in the first local key point subset.

[0086] In the embodiment of the present application, the first local key point subset is the jth local key point subset corresponding to the i-th image frame, and the second local key point subset is the jth local key point subset corresponding to the i-1th image frame; i is an integer greater than 1, j is an integer greater than 1 and less than or equal to a first value, and the first value is equal to the number of local key point subsets of each image frame. First, the number of local key points with position offset (i.e., jitter) between the jth local key point subset corresponding to the i-th image frame and the jth local key point subset corresponding to the i-1th image frame is counted, that is, the number of local key points with position offset in the jth local key point subset corresponding to the i-th image frame relative to the jth local key point subset corresponding to the i-1th image frame. Position offset refers to the position deviation of the position coordinates of the local key point in the jth local key point subset corresponding to the i-th image frame relative to the position coordinates of the local key point in the jth local key point subset corresponding to the i-1th image frame. Then, the ratio between the number of local key points with positional offset (i.e., jitter) and the total number of local key points in the jth local key point subset corresponding to the i-th image frame is determined, and the ratio is determined as the local offset key point ratio.

[0087] Step S603: Based on the proportion of local offset key points, a second de-jittering process is performed on the fourth facial key point in the fourth facial key point set to obtain a second facial key point set.

[0088] In the embodiment of the present application, based on the proportion of local offset key points, the fourth facial key points in the fourth facial key point set are subjected to a second de-jittering process to obtain a second facial key point set, which can be achieved in the following two ways: In the first way, when the proportion of local offset key points is less than a preset proportion threshold, the fourth facial key point set is determined as the second facial key point set. The preset proportion threshold refers to a preset size used to measure the jitter of the local key points in the fourth facial key point set. When the proportion of local offset key points is less than the preset proportion threshold, it means that the local key points in the jth local key point subset corresponding to the i-th image frame in the fourth facial key point set have a smaller local jitter than the local key points in the jth local key point subset corresponding to the i-1th image frame, that is, the local key point jitter in the fourth facial key point set is smaller, the local key point jitter can be ignored, and there is no need to continue the de-jittering process. The fourth facial key point set can be directly used as the label data of the key point prediction model, that is, the second facial key point set.

[0089] The second method is to replace the jth local key point subset corresponding to the i-th image frame with the jth local key point subset corresponding to the i-1th image frame when the proportion of local offset key points is greater than or equal to the proportion threshold, and obtain the second facial key point set. When the proportion of local offset key points is greater than or equal to the preset proportion threshold, it means that the local key points in the jth local key point subset corresponding to the i-th image frame in the fourth facial key point set have a larger local jitter than the local key points in the jth local key point subset corresponding to the i-1th image frame, that is, the local key point jitter in the fourth facial key point set is larger, and the local key point jitter cannot be ignored, and further de-jittering processing is required.

[0090] For example, assuming that the jth local key point subset is a mouth key point subset, the mouth key point subset includes 10 mouth key points, and the number of mouth key points that are positionally offset in the mouth key point subset corresponding to the ith image frame relative to the mouth key point subset corresponding to the i-1th image frame is 5. The proportion of local offset key points is 50%, and when the preset proportion threshold is 30%, the preset proportion threshold is greater than the proportion threshold, and de-jitter processing needs to continue.

[0091] When the proportion of local offset key points is greater than or equal to the proportion threshold, the jth local key point subset corresponding to the i-th image frame is replaced with the jth local key point subset corresponding to the i-1-th image frame, so as to quickly and effectively eliminate the local jitter of the local key points in the fourth facial key point set. Here, by performing the first de-jitter processing and the second de-jitter processing on the first facial key point in the first facial key point set, the overall key point jitter and the local key point jitter of the first facial key point subset corresponding to each of the multiple adjacent image frames in the first facial key point set are comprehensively considered, and the jitter of the first facial key point in the first facial key point set can be effectively eliminated or reduced, thereby improving the stability of the label data (i.e., the second facial key point set) of the key point prediction model, so as to subsequently improve the model performance of the key point prediction model based on the stable label data.

[0092] Step S103: using a key point prediction model, predicting key points of the target object based on audio features, to obtain a third facial key point set.

[0093] In the embodiment of the present application, the key point prediction model refers to a key point prediction model that needs to be trained. The key point prediction model can automatically identify and predict the facial key point positions of the target object by analyzing the given input data (i.e., the audio features of the target object). The key point prediction model is commonly used for the conversion of speech to facial key points, and then generates facial expressions synchronized with the speech. For example, the key point prediction model can be an Audio2Landmarks network, etc. The model structure of the key point prediction model is not specifically limited here. The audio features are the input data of the key point prediction model, and the third facial key point set is the output data predicted by the key point prediction model.

[0094] Step S104, determining a loss value of a key point prediction model based on the second facial key point set and the third facial key point set.

[0095] In the embodiment of the present application, the second facial key point set is stable label data, and the third facial key point set is output data predicted by the key point prediction model. The second facial key point set includes a second facial key point subset corresponding to each of a plurality of adjacent image frames; the third facial key point set includes a third facial key point subset corresponding to each of a plurality of adjacent image frames. Based on the label data and the predicted output data of the key point prediction model, a loss function of the key point prediction model is constructed, and the loss is calculated based on the loss function to obtain the loss value of the key point prediction model.

[0096] The loss function of the key point prediction model includes two parts, and the corresponding loss values ​​include a first loss value used to characterize the difference between the second facial key point subset corresponding to the i-th image frame in the second facial key point set and the third facial key point subset corresponding to the i-th image frame in the third facial key point set, and a second loss value used to characterize the error between the difference between the second facial key point subset corresponding to the i-th image frame in the second facial key point set and the second facial key point subset corresponding to the i-1-th image frame, and the difference between the third facial key point subset corresponding to the i-th image frame in the third facial key point set and the third facial key point subset corresponding to the i-1-th image frame.

[0097] In some embodiments, see Figure 7 , Figure 7 It is shown that in step S104, based on the second facial key point set and the third facial key point set, the loss value of the key point prediction model is determined, which can be achieved by the following steps S1041 to S1043:

[0098] Step S1041, determining a first loss value of the key point prediction model based on a second facial key point subset corresponding to the i-th image frame in the second facial key point set and a third facial key point subset corresponding to the i-th image frame in the third facial key point set.

[0099] In the embodiment of the present application, i is an integer greater than 1. The difference between the second facial key point subset corresponding to the i-th image frame in the second facial key point set and the third facial key point subset corresponding to the i-th image frame in the third facial key point set is calculated, that is, the error between the label data and the output data of the key point prediction model is calculated, and the error is determined as the first loss value.

[0100] In some embodiments, see Figure 8 , Figure 8 It is shown that in step S1041, based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set and the third facial key point subset corresponding to the i-th image frame in the third facial key point set, the first loss value of the key point prediction model is determined, which can be achieved by the following steps S801 to S803:

[0101] Step S801, determining the absolute value of the offset between the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame.

[0102] In an embodiment of the present application, the offset value between the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame refers to the sum of the differences between the respective facial key point position coordinates in the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame, and the absolute value of the offset value is calculated.

[0103] Step S802: When the absolute value is less than a preset second offset threshold, a first loss value is determined based on the absolute value and the second offset threshold.

[0104] In the embodiment of the present application, the preset second offset threshold is used to distinguish the calculation method of the first loss value of the key point prediction model. When the absolute value of the offset value between the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame is less than the preset second offset threshold, the first loss value is determined based on the absolute value and the second offset threshold. At this time, the specific calculation formula of the first loss value can refer to the first formula in formula (4) below.

[0105] Step S803: when the absolute value is greater than or equal to the second offset threshold, determine the first loss value based on the absolute value.

[0106] In the embodiment of the present application, when the absolute value of the offset value between the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame is greater than or equal to a preset second offset threshold, a first loss value is determined based on the absolute value. At this time, the specific calculation formula of the first loss value can be found in the second formula in formula (4) below.

[0107] Here, the first loss value of the key point prediction model can be accurately determined by the absolute value of the offset value between the second facial key point subset corresponding to the i-th image frame in the second facial key point set and the third facial key point subset corresponding to the i-th image frame in the third facial key point set, that is, the difference between the prediction result of the key point prediction model and the label data, so that the first loss value can be used to correctly guide the parameter update of the key point prediction model in the subsequent process.

[0108] Step S1042, determines a second loss value of the key point prediction model based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set, the second facial key point subset corresponding to the i-1-th image frame, the third facial key point subset corresponding to the i-th image frame in the third facial key point set, and the third facial key point subset corresponding to the i-1-th image frame.

[0109] In some embodiments, see Fig. 9 , Fig. 9 It is shown that in step S1042, based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set, the second facial key point subset corresponding to the i-1-th image frame, the third facial key point subset corresponding to the i-th image frame in the third facial key point set, and the third facial key point subset corresponding to the i-1-th image frame, determining the second loss value of the key point prediction model can be achieved by following the steps S901 to S902:

[0110] Step S901, determining a first offset between a second facial key point subset corresponding to the i-th image frame and a second facial key point subset corresponding to the i-1-th image frame, and a second offset between a third facial key point subset corresponding to the i-th image frame and a third facial key point subset corresponding to the i-1-th image frame.

[0111] In this embodiment of the present application, the first offset is the sum of the differences between the second facial key point position coordinates of the second facial key point subset corresponding to the i-th image frame and the second facial key point subset corresponding to the i-1-th image frame, and the second offset is the sum of the differences between the third facial key point position coordinates of the third facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-1-th image frame.

[0112] Step S902: Determine a second loss value based on the first offset and the second offset.

[0113] In an embodiment of the present application, after obtaining the first offset between the second facial key point subset corresponding to the i-th image frame and the second facial key point subset corresponding to the i-1-th image frame, and the second offset between the third facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-1-th image frame, the difference between the first offset and the second offset is calculated, and then the squared L2 norm (i.e., Euclidean norm) of the difference is calculated to obtain a second loss value.

[0114] Here, by calculating the offset between the second facial key point subset corresponding to the i-th image frame in the second facial key point set and the second facial key point subset corresponding to the i-1-th image frame, as well as the offset between the third facial key point subset corresponding to the i-th image frame in the third facial key point set and the third facial key point subset corresponding to the i-1-th image frame, the error of the facial key points between two adjacent image frames in the label data and the error of the facial key points between two adjacent image frames in the prediction result of the key point prediction model are determined, and the difference between the two errors is determined as the second loss value of the key point prediction model, so as to achieve accurate determination of the second loss value, so as to correctly guide the parameter update of the key point prediction model through the first loss value later.

[0115] Step S1043: determine the loss value based on the first loss value and the second loss value.

[0116] In an embodiment of the present application, after obtaining the first loss value and the second loss value of the key point prediction model, the first loss value and the second loss value are weighted and summed using a preset loss weight to obtain the loss value of the key point prediction model.

[0117] Here, the loss value is used to measure the inconsistency between the prediction results of the key point prediction model and the label data for the facial key points corresponding to the same image frame, and the inconsistency between the prediction results of the key point prediction model and the label data for the facial key points corresponding to the same two adjacent image frames, that is, the difference between the forward calculation results of each iteration of the calculation model and the true value, so as to guide the next step of training of the key point prediction model in the right direction.

[0118] Step S105: based on the loss value, update the model parameters of the key point prediction model to obtain an updated key point prediction model.

[0119] In an embodiment of the present application, based on the loss value, the model parameters can be updated by an optimization algorithm such as gradient descent, so as to train the optimal key point prediction model. In the process of updating the model parameters in the key point prediction model based on the loss value, the gradient of the model parameters of the key point prediction model can be calculated according to the loss value of the key point prediction model. For example, the gradient of the model parameters of the key point prediction model to the loss value of the key point prediction model can be calculated by the back propagation algorithm, that is, the model parameters of the key point prediction model are updated in the direction that can reduce the loss value of the key point prediction model, so that the loss value gradually decreases and the model performance of the key point prediction model is gradually optimized. Finally, the key point prediction model is repeatedly iterated and trained. The above steps can be repeated to continuously update the model parameters of the key point prediction model and continuously optimize the model performance of the key point prediction model until the model converges or reaches the preset iteration condition, stop updating the model parameters in the key point prediction model, and save the updated key point prediction model. The preset iteration condition can be a preset iteration time or number of iterations.

[0120] Here, an updated key point prediction model is obtained through a model training process of the key point prediction model, so that the updated key point prediction model can be directly used in the subsequent model use process to perform key point prediction on the audio features of the input target audio to obtain an accurate fifth facial key point set.

[0121] The key point prediction model training method provided in the embodiment of the present application is that when the key point prediction model is trained, the first facial key point set is the original model label used to calculate the loss value of the key point prediction model. Before the first facial key point set is used as the model label of the key point prediction model, the first facial key point in the first facial key point set is de-jittered in consideration of the jitter problem of the first facial key point in the first facial key point set, and the second facial key point set after the de-jittering is used as the model label for calculating the loss value of the key point prediction model, thereby improving the stability of the model label of the key point prediction model. Thus, based on the third facial key point set predicted by the model and the loss value calculated by the second facial key point set after the de-jittering, the model parameters of the key point prediction model are accurately updated, thereby improving the model performance of the key point prediction model.

[0122] The video generation method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0123] Fig.10 FIG. 1 is an optional flow chart of a video generation method provided in an embodiment of the present application, and the method can be applied to electronic devices. Fig.10 As shown, the method includes the following steps S1001 to S1004:

[0124] Step S1001: perform key point prediction on the audio features of the target audio using a key point prediction model to obtain a fifth facial key point set.

[0125] In the embodiment of the present application, the key point prediction model is trained by the key point prediction model training method provided by the above steps S101 to S105 of the embodiment of the present application. The audio feature of the target audio refers to the audio feature that needs to be predicted by the pre-trained key point prediction model. The audio feature is the input data of the pre-trained key point prediction model during the use of the model. The audio feature can be obtained by extracting the audio feature of the speech data of the target audio through the pre-trained audio feature extraction module. The fifth facial key point set is the facial key point set corresponding to the audio feature of the target audio obtained by the pre-trained key point prediction model for key point prediction, that is, the output data of the key point prediction model. The fifth facial key point set refers to a series of key points identified and detected by the pre-trained key point prediction model based on the audio features of the target audio. These key points represent the important feature positions of the face, usually including the key positions of the eyes, nose, mouth, eyebrows and other parts. The audio features of the target audio are directly input into the pre-trained key point prediction model, and the audio features of the target audio are predicted by the pre-trained key point prediction model, so as to obtain the fifth facial key point set output by the key point prediction model.

[0126] Step S1002: Generate multiple frames of continuous facial contour images based on the fifth facial key point set.

[0127] In the embodiment of the present application, the fifth facial key point set includes a fifth facial key point subset corresponding to each of the plurality of adjacent image frames. The fifth facial key point subset corresponding to each of the plurality of adjacent image frames can be converted into a plurality of continuous facial contour maps by a facial contour map generation algorithm. For example, the facial contour map generation algorithm can be a contour tracking method, i.e., using an edge detection algorithm (e.g., Sobel, Canny operator) to detect contours around the key points of each image frame, and tracking these contours to form continuous contour lines, thereby generating a plurality of continuous facial contour maps.

[0128] Step S1003, performing image processing on the facial contour image through the pre-trained digital object generation model to obtain multiple frames of continuous digital object images.

[0129] In an embodiment of the present application, the digital object generation model is a digital object generation model that is trained through a model training process and is used to convert a facial contour map into a digital object image. For example, the digital object generation model may be a Pix2pix model. The model structure of the digital object generation model is not specifically limited herein. The digital object image refers to a static image of a virtual object generated based on a facial contour map by a pre-trained digital object generation model. For example, the virtual object may be a virtual anchor, etc. Multiple frames of continuous facial contour maps are input into the pre-trained digital object generation model. The pre-trained digital object generation model is used to perform image processing on the multiple frames of continuous facial contour maps to obtain multiple frames of continuous digital object images.

[0130] Step S1004: Generate a digital object video based on multiple frames of continuous digital object images.

[0131] In the embodiment of the present application, the digital object video refers to a dynamic video of a virtual object, and multiple frames of continuous digital object images can be spliced ​​together in time sequence to obtain the digital object video.

[0132] The video generation method provided in the embodiment of the present application obtains the fifth facial key point set corresponding to the audio features of the target audio based on the pre-trained key point prediction model, and there is no jitter between the facial key points in the fifth facial key point set. Therefore, after performing facial contour extraction, image processing and other operations on the facial key point set, a natural and realistic digital object video can be generated.

[0133] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0134] The embodiment of the present application provides a key point guided voice driven digital human method, the overall solution flow chart of the method is as follows: Fig.11 First, a speech recognition feature processing method (e.g., Mel-scale Frequency Cepstral Coefficients (MFCC), Fbank, etc.) is used to extract speech data a t The speech spectrum f audio , and then the speech spectrum f audio Send it to the AT-Net network (i.e. the pre-trained audio feature extraction module mentioned above) to obtain the speech feature h with time sequence information t (i.e., the audio features of the target audio above). Combined with the speech features h t The key point prediction model 1100 is used to predict the key point information (i.e., the fifth facial key point set mentioned above). For the predicted key points, the following is generated: Fig.11Finally, the key point contour map 1101 is sent to the U-Net network to obtain the speech data a t The corresponding digital human image 1102 (ie the above-mentioned digital object image).

[0135] Voice feature h t The conversion of speech spectrum to key points is the top priority in the entire digital human generation process. First, we need to select the speech feature extraction network related to key points. Among them, the AT-Net network has this feature. audio Converted to 256-dimensional speech feature h t ,h t The image frame is aligned with 25 frames, that is, 1 second of speech will generate (25, 256) features. The key point prediction model 1100 (for example, the Audio2Landmarks network) can be used to predict facial key points. The structural diagram of the Audio2Landmarks network is shown in Fig.12 As shown, due to the speech feature h t It is related to facial key points, so there is no need to design a very complex network to predict facial key points.

[0136] The key to a good prediction by the Audio2Landmarks network is that the label data (i.e., the first set of facial key points mentioned above) used for model training needs to be stable and accurate. The label data can be the 68 facial key points predicted by the 2DFAN model (i.e., the pre-trained facial key point prediction module mentioned above). However, the facial key points obtained by the 2DFAN model will shake, which will cause the lips of the final generated digital person to tremble unnaturally. Therefore, it is necessary to first de-jitter the predicted facial key points, and then use the de-jittered facial key points (i.e., the second set of facial key points mentioned above) as Groudtruth to train the Audio2Landmarks model.

[0137] The de-jittering process mainly combines the exponential moving average (EMA) de-jittering mechanism (i.e., the first de-jittering process mentioned above) and the local Part de-jittering mechanism (i.e., the second de-jittering process mentioned above). The EMA de-jittering mechanism establishes a connection between the offset of the facial key points of the current image frame and the facial key points of the previous image frame, and records P t is the facial key point of the current image frame, P t-1 is the facial key point of the previous image frame. The offset value calculation formula is shown in formula (1). If the offset value Dis(P t ,P t-1) is small (i.e., smaller than the preset offset threshold), it can be considered to be caused by jitter, that is, the facial key points of the current image frame are made equal to the facial key points of the previous image frame, as shown in formula (2). t ,P t-1 ) is larger (i.e., greater than or equal to the preset offset threshold), combined with the facial key point P of the current image frame t and the facial key points P of the previous image frame t-1 To calculate the updated facial key points of the current image frame (i.e., the updated first facial key point subset) to reduce the jitter error, as shown in formula (3).

[0138]

[0139] P t =P t-1 Dis(P t ,P t-1 ) <P (2)

[0140] P′ t =αP t +(1-α)P t-1 Dis(P t ,P t-1 )≥P (3)

[0141] Among them, P t_x Represents the facial key point P of the current t-th image frame t The horizontal axis, P t_y Represents the facial key point P of the current t-th image frame t The vertical axis, P t-1_x Represents the facial key point P of the previous (i.e., t-1th) image frame t-1 The horizontal axis, P t-1_y Represents the facial key point P of the previous image frame t-1 The vertical coordinate of . P is the preset offset threshold, α is the proportion of facial key points in the current image frame (i.e., the above-mentioned preset first proportion), and 1-α is the proportion of facial key points in the previous image frame (i.e., the above-mentioned second proportion). P′ t Represents the updated facial key points of the current image frame.

[0142] In addition, after the EMA de-jittering mechanism, the localPart de-jittering mechanism can be used to continue the de-jittering process. After the EMA de-jittering mechanism is integrated, the lightweight model still has more or less jitter. For the local key points of the local unit (that is, the above-mentioned local key point subset), for example, the eye key points, the mouth key points, etc., set a preset proportion threshold, and calculate the proportion of the jittered local key points relative to all the key points. If the proportion is less than the proportion threshold, the jittered local key points are in the minority, and this part of the jitter is ignored. If the proportion is greater than or equal to the proportion threshold, the jittered local key points are in the majority, and the local key points of the current image frame of the local key points of the previous image frame are used.

[0143] Through the above two de-jittering mechanisms, stable label data can be obtained, and the loss function of the key point prediction model 1100 can be constructed, such as Fig.12 As shown, the key point prediction model 1100 is composed of three one-dimensional convolutional layers. Since the facial key points in the facial key point set are relatively dense, it is necessary to finely optimize the error to achieve accurate positioning of the facial key points. Therefore, the WingLoss shown in formula (4) is used as the first loss Loss1 to calculate the loss value of the model (i.e., the above-mentioned first loss value).

[0144]

[0145] Among them, w is a positive number used to limit the range of the nonlinear part to the interval (-w, w), ln represents the logarithmic function, |x| is the absolute value of the difference between the output data of the key point prediction model and the label data (that is, the absolute value of the above offset value), ∈ is the curvature of the constrained nonlinear region, and C is a constant that can be used to smoothly connect the linear and nonlinear parts of the segment. Fig.13 It can be seen that the response y of ln(1+x) in WingLoss to slight differences x is much higher than that of L2 function Therefore, WingLoss can achieve more refined positioning of facial key points.

[0146] In order to make the predicted facial key points more stable during the entire timing process, such as Fig.12 As shown in Figure 1, based on WingLoss, a motion smoothness constraint Loss2 is added, that is, it is hoped that the difference in the prediction results in time series and the difference in the label data are consistent. The loss function corresponding to the motion smoothness constraint is as shown in formula (5).

[0147]

[0148] Among them, y t is the facial key point corresponding to the current t-th image frame output by the key point prediction model, y t-1The facial key points corresponding to the t-1th image frame output by the key point prediction model, that is, the third facial key point subset in the third facial key point set; is the facial key point corresponding to the current t-th image frame in the label data, is the facial key point corresponding to the t-1th image frame in the label data, that is, the second facial key point subset in the above-mentioned second facial key point set.

[0149] In summary, the total loss Loss_1 of the key point prediction model is calculated as follows:

[0150] Loss_1=ω1*Loss1+ω2*Loss2 (6)

[0151] Among them, ω1 and ω2 are the loss weights of the two losses respectively.

[0152] After the facial key points are generated by the updated key point prediction model, a key point contour map is generated based on the facial key points. At this time, the key point contour map can be constructed. For the training data of the digital human image, the Pix2pix model can be used based on the key point contour map. Generate digital human images like Fig.14 shown.

[0153] The loss of the Pix2pix model is mainly composed of adversarial loss and reconstruction loss. The adversarial loss includes the generator loss L1_G of the generator and the discriminator loss L1_D of the discriminator 1401. The calculation formula of the generator loss L1_G is shown in formula (7), the calculation formula of the discriminator loss L1_D of the discriminator 1401 is shown in formula (8), and the calculation formula of the adversarial loss Loss3 is shown in formula (9).

[0154]

[0155] Loss3=L1_G+L1_D (9)

[0156] Among them, bce represents binary cross entropy, Represents the true value image.

[0157] In addition, the calculation formula of the reconstruction loss Loss4 is shown in formula (10).

[0158]

[0159] In order to make the digital human images generated by the U-Net network The effect is better. A style consistency loss Loss5 is introduced to constrain the difference of style features. The calculation formula is shown in formula (11).

[0160]

[0161] Among them, G represents the Gram matrix, φ represents the features predicted using the pre-trained VGG16, and j represents the stage features used. The Gram matrix reflects the channel correlation of the predicted feature map and can better restore local texture features.

[0162] The calculation formula of the total loss Loss_2 of the Pix2pix model is formula (12):

[0163] Loss_2=ω3*Loss3+ω4*Loss4+ω5*Loss5 (12)

[0164] Among them, ω3, ω4 and ω5 are loss weights, and ω3 can be set to 1, ω4 = 100, and ω3 = 5.

[0165] The digital human images generated by the standard Pix2pix model will have random noise and often have holes in the content. The holes in the content are caused by the fact that the context of the generated image frames is not well associated. Therefore, the input data of the Pix2pix model needs to be converted from a single image frame to a time-series image frame. For example, 7 consecutive image frames are combined and input into the Pix2pix model for prediction. The associated time sequence can associate the content of the image frames before and after the digital human is generated, so that the generated digital human image has no holes. In addition, the training resolution can be increased from 256 to 512 to eliminate random noise and make the generated digital human clearer and more realistic.

[0166] In the embodiment of the present application, the method can establish the connection between facial key points and speech with a small amount of data, and then generate a digital human corresponding to the speech based on the facial key points using a generative network. And by using key point de-jittering technology to generate more stable label data, and then combining key point frame difference information to predict a more stable conversion from speech to facial key points, the digital human's face does not shake, and a natural and realistic digital human is generated.

[0167] Based on the key point prediction model training method described in the above embodiment, Fig.15A structural block diagram of a key point prediction model training device 1500 provided in an embodiment of the present application is shown. The key point prediction model training device may be a device in an electronic device. The key point prediction model training device may be implemented in software, which may be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 1501, a de-jitter processing module 1502, a key point prediction module 1503, a determination module 1504 and an update module 1505. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented. Among them, the acquisition module 1501 is used to obtain the audio features of the sample video and the first facial key point set of the target object in the sample video; the de-jitter processing module 1502 is used to de-jitter the first facial key point in the first facial key point set based on the first facial key point subsets corresponding to two adjacent image frames in the first facial key point set, so as to obtain a second facial key point set; the key point prediction module 1503 is used to predict the key points of the target object based on the audio features through a key point prediction model, so as to obtain a third facial key point set; the determination module 1504 is used to determine the loss value of the key point prediction model based on the second facial key point set and the third facial key point set; the update module 1505 is used to update the model parameters of the key point prediction model based on the loss value, so as to obtain an updated key point prediction model.

[0168] In some embodiments, the de-jitter processing module 1502 is also used to: based on the first facial key point subsets corresponding to each of two adjacent image frames, perform a first de-jitter processing on the first facial key point in the first facial key point set to obtain a fourth facial key point set; wherein the fourth facial key point set includes fourth facial key point subsets corresponding to multiple adjacent image frames; the fourth facial key point subset corresponding to each image frame includes multiple local key point subsets; based on the local key point subsets corresponding to each of two adjacent image frames, perform a second de-jitter processing on the fourth facial key point in the fourth facial key point set to obtain the second facial key point set.

[0169] In some embodiments, the de-jitter processing module 1502 is further used to: determine, for the first facial key point subset corresponding to the ith image frame in the first facial key point set, an offset value between the first facial key point subset corresponding to the ith image frame and the first facial key point subset corresponding to the i-1th image frame; i is an integer greater than 1; when the offset value is less than a preset first offset threshold, replace the first facial key point subset corresponding to the ith image frame with the first facial key point subset corresponding to the i-1th image frame to obtain the fourth facial key point set; when the offset value is greater than or equal to the first offset threshold, update the first facial key point subset corresponding to the ith image frame based on the first facial key point subset corresponding to the ith image frame and the first facial key point subset corresponding to the i-1th image frame to obtain the fourth facial key point set.

[0170] In some embodiments, the de-jitter processing 1502 is also used to: determine a first updated subset based on a preset first proportion and a first facial key point subset corresponding to the i-th image frame; determine a second updated subset based on a second proportion and a first facial key point subset corresponding to the i-1-th image frame; the second proportion is determined based on the first proportion; determine an updated first facial key point subset corresponding to the i-th image frame based on the first updated subset and the second updated subset; determine the fourth facial key point set based on the updated first facial key point subset.

[0171] In some embodiments, the de-jitter processing module 1502 is also used to: determine the number of local key points that have position offsets between the first local key point subset and the second local key point subset; determine the proportion of local offset key points based on the number and the total number of local key points in the first local key point subset; wherein the first local key point subset is the jth local key point subset corresponding to the i-th image frame, and the second local key point subset is the jth local key point subset corresponding to the i-1th image frame; i is an integer greater than 1, j is an integer greater than 1 and less than or equal to a first value, and the first value is equal to the number of local key point subsets of each image frame; based on the proportion of local offset key points, perform a second de-jitter processing on the fourth facial key point in the fourth facial key point set to obtain the second facial key point set.

[0172] In some embodiments, the de-jitter processing module 1502 is also used to: when the proportion of the local offset key points is less than a preset proportion threshold, determine the fourth facial key point set as the second facial key point set; when the proportion of the local offset key points is greater than or equal to the proportion threshold, replace the jth local key point subset corresponding to the i-1th image frame with the jth local key point subset corresponding to the i-1th image frame to obtain the second facial key point set.

[0173] In some embodiments, the second facial key point set includes a second facial key point subset corresponding to each of a plurality of adjacent image frames; the third facial key point set includes a third facial key point subset corresponding to each of a plurality of adjacent image frames; the determination module 1504 is further used to: determine a first loss value of the key point prediction model based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set, and the third facial key point subset corresponding to the i-th image frame in the third facial key point set; i is an integer greater than 1; determine a second loss value of the key point prediction model based on the second facial key point subset corresponding to the i-th image frame in the second facial key point set, the second facial key point subset corresponding to the i-1-th image frame, the third facial key point subset corresponding to the i-th image frame in the third facial key point set, and the third facial key point subset corresponding to the i-1-th image frame; and determine the loss value based on the first loss value and the second loss value.

[0174] In some embodiments, the determination module 1504 is also used to: determine the absolute value of the offset value between the second facial key point subset corresponding to the i-th image frame and the third facial key point subset corresponding to the i-th image frame; when the absolute value is less than a preset second offset threshold, determine the first loss value based on the absolute value and the second offset threshold; when the absolute value is greater than or equal to the second offset threshold, determine the first loss value based on the absolute value.

[0175] In some embodiments, the determination module 1504 is also used to: determine a first offset between a second facial key point subset corresponding to the i-th image frame and a second facial key point subset corresponding to the i-1-th image frame, and a second offset between a third facial key point subset corresponding to the i-th image frame and a third facial key point subset corresponding to the i-1-th image frame; and determine the second loss value based on the first offset and the second offset.

[0176] In some embodiments, the acquisition module 1501 is also used to: extract audio features from the speech data of the target object in the sample video through a pre-trained audio feature extraction module to obtain the audio features; and predict facial key points from the image data of the target object in the sample video through a pre-trained facial key point prediction module to obtain the first facial key point set.

[0177] Based on the video generation method described in the above embodiment, Fig.16 A structural block diagram of a video generating device 1600 provided in an embodiment of the present application is shown. The video generating device may be a device in an electronic device. The video generating device may be implemented in software, which may be software in the form of programs and plug-ins, etc., including the following software modules: a prediction module 1601, a first generating module 1602, an image processing module 1603 and a second generating module 1604. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0178] Among them, the prediction module 1601 is used to perform key point prediction on the audio features of the target audio through a key point prediction model to obtain a fifth facial key point set; the key point prediction model is trained using the key point prediction model training method provided in the embodiment of the present application; the first generation module 1602 is used to generate a plurality of frames of continuous facial contour images based on the fifth facial key point set; the image processing module 1603 is used to perform image processing on the facial contour image through a pre-trained digital object generation model to obtain a plurality of frames of continuous digital object images; the second generation module 1604 is used to generate a digital object video based on the plurality of frames of continuous digital object images.

[0179] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0180] An embodiment of the present application provides an electronic device, Fig.17 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig.17 As shown, the electronic device 1700 includes: at least one processor 1701 ( Fig.17 Only one is shown), a memory 1702 and executable instructions 1703 stored in the memory 1702 and executable on at least one processor 1701, when the processor 1701 executes the executable instructions 1703, the steps in any of the above-mentioned key point prediction model training method embodiments are implemented, or the steps in any of the above-mentioned video generation method embodiments are implemented.

[0181] The electronic device may include but is not limited to a processor 1701 and a memory 1702. Those skilled in the art may understand that: Fig.17 It is only an example of the electronic device 1700 and does not constitute a limitation of the electronic device 1700. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0182] The processor 1701 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0183] In some embodiments, the memory 1702 may be an internal storage unit of the electronic device 1700, such as a hard disk or memory of the electronic device 1700. In other embodiments, the memory 1702 may also be an external storage device of the electronic device 1700, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the electronic device 1700. Further, the memory 1702 may also include both an internal storage unit of the electronic device 1700 and an external storage device. The memory 1702 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program, etc. The memory 1702 may also be used to temporarily store data that has been output or is to be output.

[0184] The present application embodiment provides a computer-readable storage medium in which computer executable instructions are stored. When the computer executable instructions are executed by a processor, the processor will execute the key point prediction model training method provided by the present application embodiment, for example, Figure 1 Alternatively, the video generation method provided in the embodiment of the present application is executed, for example, Fig.10 The video generation method is shown.

[0185] The embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the key point prediction model training method described in the embodiment of the present application, or executes the video generation method provided in the embodiment of the present application.

[0186] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0187] In some embodiments, the computer executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. As an example, the computer executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files storing one or more modules, subroutines, or code portions).

[0188] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A key point prediction model training method, characterized in that: The method comprises: Acquire audio features of a sample video and a first facial key point set of a target object in the sample video; Based on first facial key point subsets corresponding to two adjacent image frames in the first facial key point set, respectively, de-jittering the first facial key points in the first facial key point set to obtain a second facial key point set; Using a key point prediction model, predicting key points of the target object based on the audio features, to obtain a third facial key point set; Determining a loss value of the key point prediction model based on the second facial key point set and the third facial key point set; Based on the loss value, the model parameters of the key point prediction model are updated to obtain an updated key point prediction model.

2. The method according to claim 1, characterized in that The step of performing de-jittering processing on the first facial key points in the first facial key point set based on the first facial key point subsets respectively corresponding to two adjacent image frames in the first facial key point set to obtain a second facial key point set includes: Based on the first facial key point subsets respectively corresponding to the two adjacent image frames, a first de-jittering process is performed on the first facial key point in the first facial key point set to obtain a fourth facial key point set; wherein the fourth facial key point set includes fourth facial key point subsets respectively corresponding to a plurality of adjacent image frames; and the fourth facial key point subset corresponding to each image frame includes a plurality of local key point subsets; Based on the local key point subsets corresponding to two adjacent image frames, a second de-jittering process is performed on the fourth facial key point in the fourth facial key point set to obtain the second facial key point set.

3. The method according to claim 2, characterized in that The step of performing a first de-jittering process on the first facial key point in the first facial key point set based on the first facial key point subsets respectively corresponding to the two adjacent image frames to obtain a fourth facial key point set includes: For a first facial key point subset corresponding to an i-th image frame in the first facial key point set, determine an offset value between the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to an i-1-th image frame; i is an integer greater than 1; When the offset value is less than a preset first offset threshold, the first facial key point subset corresponding to the i-th image frame is replaced by the first facial key point subset corresponding to the i-1-th image frame to obtain the fourth facial key point set; When the offset value is greater than or equal to the first offset threshold, the first facial key point subset corresponding to the i-th image frame is updated based on the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame to obtain the fourth facial key point set.

4. The method according to claim 3, characterized in that: The updating of the first facial key point subset corresponding to the i-th image frame based on the first facial key point subset corresponding to the i-th image frame and the first facial key point subset corresponding to the i-1-th image frame to obtain the fourth facial key point set includes: Determine a first update subset based on a first facial key point subset having a preset first proportion and corresponding to the i-th image frame; Determine a second updated subset based on a first facial key point subset having a second proportion corresponding to the (i-1)th image frame; the second proportion being determined based on the first proportion; Determine an updated first facial key point subset corresponding to the i-th image frame based on the first updated subset and the second updated subset; The fourth facial key point set is determined based on the updated first facial key point subset.

5. The method according to claim 2, characterized in that: The step of performing a second de-jittering process on the fourth facial key point in the fourth facial key point set based on the local key point subsets respectively corresponding to two adjacent image frames to obtain the second facial key point set includes: Determine the number of local key points that have a positional offset between the first subset of local key points and the second subset of local key points; Based on the number and the total number of local key points in the first local key point subset, determining the proportion of local offset key points; wherein the first local key point subset is the jth local key point subset corresponding to the i-th image frame, and the second local key point subset is the jth local key point subset corresponding to the i-1th image frame; i is an integer greater than 1, j is an integer greater than 1 and less than or equal to a first value, and the first value is equal to the number of local key point subsets of each image frame; Based on the proportion of the local offset key points, a second de-jittering process is performed on the fourth facial key points in the fourth facial key point set to obtain the second facial key point set.

6. The method according to claim 5, characterized in that The performing a second de-jittering process on the fourth facial key points in the fourth facial key point set based on the proportion of the local offset key points to obtain the second facial key point set includes: When the proportion of the local offset key points is less than a preset proportion threshold, determining the fourth facial key point set as the second facial key point set; When the proportion of the local offset key points is greater than or equal to the proportion threshold, the j-th local key point subset corresponding to the i-th image frame is replaced by the j-th local key point subset corresponding to the i-1-th image frame to obtain the second facial key point set.

7. The method according to claim 1, characterized in that The second facial key point set includes a second facial key point subset corresponding to each of a plurality of adjacent image frames; the third facial key point set includes a third facial key point subset corresponding to each of a plurality of adjacent image frames; The determining, based on the second facial key point set and the third facial key point set, a loss value of the key point prediction model comprises: Determining a first loss value of the key point prediction model based on a second facial key point subset corresponding to an i-th image frame in the second facial key point set and a third facial key point subset corresponding to an i-th image frame in the third facial key point set; i is an integer greater than 1; Determining a second loss value of the key point prediction model based on a second facial key point subset corresponding to the i-th image frame in the second facial key point set, a second facial key point subset corresponding to the i-1-th image frame, a third facial key point subset corresponding to the i-th image frame in the third facial key point set, and a third facial key point subset corresponding to the i-1-th image frame; The loss value is determined based on the first loss value and the second loss value.

8. The method according to claim 7, characterized in that The determining, based on a second facial key point subset corresponding to an i-th image frame in the second facial key point set and a third facial key point subset corresponding to an i-th image frame in the third facial key point set, a first loss value of the key point prediction model comprises: Determine an absolute value of an offset value between a second subset of facial key points corresponding to the i-th image frame and a third subset of facial key points corresponding to the i-th image frame; In a case where the absolute value is less than a preset second offset threshold, determining the first loss value based on the absolute value and the second offset threshold; In a case where the absolute value is greater than or equal to the second offset threshold, the first loss value is determined based on the absolute value.

9. The method according to claim 7, characterized in that: The determining a second loss value of the key point prediction model based on a second facial key point subset corresponding to the i-th image frame in the second facial key point set, a second facial key point subset corresponding to the i-1-th image frame, a third facial key point subset corresponding to the i-th image frame in the third facial key point set, and a third facial key point subset corresponding to the i-1-th image frame, comprises: Determining a first offset between a second subset of facial key points corresponding to the i-th image frame and a second subset of facial key points corresponding to the i-1-th image frame, and a second offset between a third subset of facial key points corresponding to the i-th image frame and a third subset of facial key points corresponding to the i-1-th image frame; The second loss value is determined based on the first offset and the second offset.

10. The method according to any one of claims 1 to 9, characterized in that: The step of obtaining the audio features of the sample video and the first facial key point set of the target object in the sample video includes: Extracting audio features from the speech data of the target object in the sample video using a pre-trained audio feature extraction module to obtain the audio features; The facial key point prediction module is used to predict the facial key points of the target object in the sample video, thereby obtaining the first facial key point set.

11. A video generation method, characterized in that: The method comprises: Using a key point prediction model, performing key point prediction on the audio features of the target audio to obtain a fifth facial key point set; the key point prediction model is trained using the key point prediction model training method provided by any one of claims 1 to 10; Based on the fifth facial key point set, generating multiple frames of continuous facial contour images; Performing image processing on the facial contour image through a digital object generation model to obtain multiple frames of continuous digital object images; A digital object video is generated based on the plurality of consecutive frames of digital object images.

12. A key point prediction model training device, characterized in that: include: An acquisition module, used to acquire audio features of a sample video and a first facial key point set of a target object in the sample video; a de-jitter processing module, configured to perform a de-jitter processing on the first facial key points in the first facial key point set based on the first facial key point subsets respectively corresponding to two adjacent image frames in the first facial key point set, so as to obtain a second facial key point set; A key point prediction module, configured to predict key points of the target object based on the audio features using a key point prediction model to obtain a third facial key point set; A determination module, configured to determine a loss value of the key point prediction model based on the second facial key point set and the third facial key point set; An updating module is used to update the model parameters of the key point prediction model based on the loss value to obtain an updated key point prediction model.

13. An electronic device, characterized in that: include: A memory for storing computer executable instructions; The processor is used to implement the key point prediction model training method described in any one of claims 1 to 10, or implement the video generation method described in claim 11 when executing the computer executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and when the computer executable instructions are executed by a processor, the key point prediction model training method described in any one of claims 1 to 10 is implemented, or the video generation method described in claim 11 is implemented.

Images