Human body action video generation method and device and medium

Abstract

The invention discloses a human motion video generation method and device and a medium, and belongs to the field of video generation, and the method comprises the steps: calculating the skeleton similarity in different video frames according to a skeleton sequence, constructing a reference key frame set based on the skeleton similarity, and predicting a shielded skeleton key point higher than a threshold value as a target based on a confidence coefficient score and the threshold value, pre-training a skeleton prediction network, correcting skeleton key points with confidence lower than a threshold value by using the skeleton prediction network, respectively extracting features of the skeleton key points with confidence scores higher than the threshold value and all corrected skeleton key points, splicing the features as a generation condition of a video model, and generating the video model by using a reference key frame set and the generation condition. Generating a video clip of the previous de-noising step number through approximate guidance, and generating all video clips by using a sliding window strategy; and circulation is carried out until the whole human body action video is generated. The equipment and the medium are used for implementing the method. According to the invention, the consistency, the generation efficiency and the robustness of long video generation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video generation, and particularly relates to a method, device, and medium for generating human action videos. Background Art

[0002] The field of human action video generation is a technology that uses computer algorithms and deep learning models to generate realistic human action videos based on given reference images and pose information. It has wide application value in many fields such as film and television production, virtual reality, etc. This field faces high challenges in many aspects such as video quality, generation efficiency, generation controllability, and the duration of the generated video.

[0003] On the one hand, for long video generation tasks, as the length of the video increases, the training and inference computing resources required will also increase sharply, and the generation efficiency of the current generation methods needs to be optimized. Secondly, when generating video segments one by one, it is easy to have inconsistent situations between segments, such as the loss of target identity information, motion artifacts, and other inconsistent problems, which affect the quality of the video. Generally, long video generation methods use a sliding window strategy to generate videos segment by segment, but it is difficult to achieve feature propagation between video frames that are far apart in time. On the other hand, existing human action video generation methods highly rely on the accuracy of the skeleton estimation method. When there are local positioning deviations in the input skeleton, such as incorrect limb orientations, key point offsets, etc., the generation model will transmit the error to the pixel space, resulting in visual defects such as limb distortion in the output video. The mainstream video generation methods are based on the ideal assumption of absolutely accurate skeletons and do not fully consider the uncertainty of the real scene, which is prone to error accumulation and leads to problems such as lack of robustness and generalization in the generation results.

[0004] The patent "Long Video Generation Method and Device Based on Text Input" (Publication No.: CN117793480A, Publication Date: 2024-03-29) applied by Paiou Cloud Computing (Shanghai) Co., Ltd. proposes a two-stage long video generation architecture based on a diffusion model, aiming to improve and enhance the generation quality of long videos. The main steps of this invention include a global key frame generation stage and a local frame filling stage. In the former, according to multiple input text prompts, multiple corresponding video key frames are generated through a diffusion model; in the latter, continuous combined position encoding is input into the model, and the spatio-temporal relationship between key frames is dynamically learned through a visual information masker to generate intermediate frames between every two consecutive key frames to fill the time gap. However, due to the use of a two-stage generation strategy, the generation efficiency of this method is low. Moreover, due to the strategy of generating global frames first and then local frames, it is impossible to consider information from other global frames at long distances during the generation process of local frames, resulting in inconsistent problems in long videos.

[0005] The patent "Video Generation Method, System, Storage Medium and Device Driven by Human Skeleton" (Publication No.: CN118609033A, Publication Date: September 6, 2024), jointly applied by Zhejiang University and Zhejiang Tonghuashun Intelligent Technology Co., Ltd., proposes a solution for video generation based on dynamic control of human skeletons. The core process of this method includes: First, by inputting the picture to be processed and the skeleton-driven video, the human body structure detection is used to separate the people and the background in the picture, and the background picture is generated; Subsequently, the corresponding coordinate information of the people in the picture when performing the corresponding postures of the skeleton-driven video is processed to obtain a two-dimensional pose sequence map; The generated dynamic pose sequence is fused into the background environment frame by frame using mask redrawing, and finally a coherent video result is generated. However, this method relies on the separate processing of the background and the foreground, and the implementation is complex. In addition, when using the dynamic information of the skeleton, the confidence information of the skeleton is not considered, and it is easily affected by incorrect skeleton sequences, and the robustness of the generation cannot be guaranteed. Summary of the Invention

[0006] In order to overcome the shortcomings of the above existing solutions, the purpose of the present invention is to provide a human action video generation method, device and medium to achieve robust and consistent generation of long human action videos; This method is based on the diffusion model framework, mainly including reference key frame selection, approximate guidance, mask-based skeleton prediction network and confidence level classification control; First, considering that the similarity of human skeleton coordinates can reflect the similarity of the appearance of people in the generated frames, reference key frame selection is adopted to select reference key frames independent of time points from the long video according to content similarity, and solve the problem that the selected reference information has no association with whether it can provide valuable reference features for other video segments; Subsequently, based on the selected set of key frames, approximate guidance is used to broadcast the predicted reference key frames to the entire long video during the denoising process, and solve the problems of inconsistent appearance across segments caused by the lack of a central representation of human features and large additional resource overhead; At the same time, a mask-based skeleton prediction network and confidence level classification control are introduced, considering the temporal continuity constraint of single-frame skeletons and the confidence information of skeletons, and solving the problem of poor robustness of the generation result.

[0007] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A human action video generation method, including the following steps:

[0009] Step 1, based on the skeleton estimation network, obtain the skeleton sequence and confidence score of the target skeleton key points, and calculate the similarity of the human skeletons in the content of different video frames according to the skeleton sequence;

[0010] Step 2, select reference key frames based on the skeleton similarity and construct a set of reference key frames;

[0011] Step 3: Set a confidence threshold. Based on the confidence scores obtained in Step 1 and the set confidence threshold, aim to predict the occluded skeleton key points above the confidence threshold, and pre-train the skeleton prediction network;

[0012] Step 4: Use the pre-trained skeleton prediction network to correct the coordinates of the skeleton key points with confidence below the threshold;

[0013] Step 5: Respectively perform feature extraction and splicing on the skeleton key points with confidence scores above the threshold obtained in Step 1 and all the skeleton key points corrected in Step 4, as the generation conditions for the video model;

[0014] Step 6: Utilize the reference key frame set constructed in Step 2 and the generation conditions in Step 5 to generate a video segment of the previous denoising step through approximate guidance;

[0015] Step 7: Apply a sliding window strategy to Step 6 until all the video segments included in the entire video of the previous denoising step are generated; Loop and execute Step 7 until the entire human action video is generated.

[0016] The specific steps of Step 1 include the following steps:

[0017] Step 1.1: Obtain the skeleton sequence of the target through the skeleton estimation network, and obtain the two-dimensional coordinate values (x f,p , y f,p ) of the p-th skeleton key point in the f-th frame based on the skeleton sequence;

[0018] Step 1.2: Calculate the weighted center coordinates of all the skeleton points within a single frame based on the two-dimensional coordinate values (x f,p , y f,p ):

[0019]

[0020] where P is the total number of skeleton points, and w p is the weight of the p-th skeleton point;

[0021] Step 1.3: Calculate the decentralized coordinates (x′ f,p , y′ f,p ) of each key point:

[0022]

[0023] Step 1.4: Calculate the skeleton similarity Similarity(i, j) between the i-th frame and the j-th frame:

[0024]

[0025] Step 2 specifically includes the following steps:

[0026] Step 2.1, initialize the reference key frame set and the reference key frame candidate set Record the reference image given by the user as the 0th frame, and define the maximum number of reference key frames m;

[0027] Step 2.2, calculate the sum of similarities S between frame j in the reference key frame candidate set and the current reference key frame set n,j :

[0028]

[0029] where Similarity(i, j) is the skeleton similarity between the ith frame and the jth frame;

[0030] Step 2.3, find the frame k with the lowest similarity between frame j and the current reference key frame set as the nth reference key frame: n

[0031]

[0032] Step 2.4, store the newly obtained reference key frame k n into the reference key frame set :

[0033]

[0034] Step 2.5, delete the newly obtained reference key frame k n from the reference key frame candidate set :

[0035]

[0036] Step 2.6, repeat steps 2.2 to 2.5 until m reference key frames are obtained, and get the final reference key frame set where k is the frame coordinate of the reference key frame.

[0037] Step 3 specifically includes the following steps:

[0038] Step 3.1, normalize the skeleton key point coordinates (x, y) to obtain (x norm , y norm );

[0039] ​Step 3.2, set the confidence threshold τ, and during training, mask the skeleton key points with a confidence score score higher than the confidence threshold τ with probability p, so that the skeleton prediction network predicts the true coordinates of the masked skeleton points based on the positions and confidence scores score of other skeleton key points in the video:

[0040] (x masked ,y masked ,score masked ) = f mask (x norm ,y norm ,score);

[0041] where f mask is a random masking function that replaces the input value (x norm ,y norm ) with a random value (x masked ,y masked ) from a normal distribution with probability p, and at the same time reduces the confidence score score of the skeleton key points masked ;

[0042] Step 3.3, use Fourier embedding to process the skeleton key point coordinates to obtain the position embedding of each skeleton key point. The formula for Fourier embedding is:

[0043] φ(x) = [cos(xω0), sin(xω0),..., cos(xω B-1 ), sin(xω B-1 )];

[0044] where B is the specified number of frequency bases, ω i = γ i / B , and γ controls the frequency band span; calculate the Fourier embedding for x masked , y masked , score masked respectively, and splice and send them into a multi-layer perceptron (MLP) for processing and dimensionality increase, finally obtaining the input skeleton key point features

[0045]

[0046] where represents the splicing operation, is the skeleton embedding, used to distinguish the skeleton key points representing different parts of the body, and F, P, C are the number of frames, the number of skeleton key points, and the embedding dimension respectively;

[0047] Step 3.4, use skeletal attention to process all skeletal key points within each frame, and use temporal attention to process the same skeletal key point on different frames. By alternately using skeletal attention and temporal attention, fuse and process the features h0 of all skeletal key points, and process the skeletal key point features h0 through N layers of Transformer blocks:

[0048] h i = TemporalTransformer(SpatialTransformer(h i-1 )) for i = 1, …, N;

[0049] Finally, obtain the processed confidence-aware feature space h N ;

[0050] Meanwhile, use rotational position encoding in the temporal attention layer. For the query q at position a a and the key k at position b b , the obtained attention score is:

[0051] AttentionScore(q a , k b ) = Re[(R a q a ) T (R b k b )];

[0052] where, is a complex-domain rotation matrix obtained from the hyperparameters of rotational position encoding;

[0053] Step 3.5, input the feature space h N into a multi-layer perceptron (MLP) to obtain the predicted coordinates of each skeletal key point The loss function uses the confidence score score of each skeletal key point predicted by the skeletal extraction network as the weight to optimize the distance between the predicted coordinates and the skeletal key point coordinates (x, y):

[0054]

[0055] where, 0 ≤ f ≤ F, 0 ≤ p ≤ P, w is the weight, and finally obtain a mask-based skeletal prediction network.

[0056] The specific steps of the said Step 4 include the following steps:

[0057] Step 4.1, during inference, obtain the coordinates (x, y) of single-frame skeleton key points with confidence score score through the skeleton estimation network, and normalize the coordinates (x, y) of the skeleton key points to obtain the coordinates (x norm , y norm );

[0058] Step 4.2, according to the coordinates (x norm , y norm ), mask the skeleton key points with confidence lower than τ to obtain the masked coordinates (x masked , y masked , score masked );

[0059] Step 4.3, use Fourier embedding to process the masked coordinates (x masked , y masked , score masked ) to obtain the position embedding of each skeleton key point, and then send it into a multi-layer perceptron (MLP) for dimensionality increase processing, and finally obtain the input skeleton key point features

[0060]

[0061] where φ represents Fourier embedding, represents the concatenation operation, is the skeleton embedding, used to distinguish the skeleton key points representing different body parts, and F, P, and C are the number of frames, the number of skeleton key points, and the embedding dimension respectively;

[0062] Step 4.4, use the pre-trained skeleton prediction network, and use the obtained F×P skeleton key point features h0 as tokens for feature processing, and finally obtain the processed confidence-aware feature space h N ;

[0063] Step 4.5, input the feature space h N into a multi-layer perceptron (MLP) to obtain the predicted coordinates of each skeleton key point.

[0064] The specific steps of the said Step 5 include the following steps:

[0065] Take the part with confidence higher than the threshold τ in the possibly noisy skeleton results extracted by the skeleton estimation network as reliable features, and send them into the high-confidence skeleton encoder to obtain the corresponding strong skeleton constraint c strong ; In addition, take the corrected skeleton key points in Step 4 as low-confidence features, send them into the low-confidence skeleton encoder to obtain the confidence-aware feature space h N , as the corresponding weak skeleton hint cweak ; The skeleton strong constraint c strong and the skeleton weak hint c weak are concatenated as the generation condition of the video model:

[0066] c input = Concat(c strong , c weak );

[0067] where Concat represents the concatenation of features.

[0068] Step 6 specifically includes the following steps:

[0069] Replace the input skeleton sequence condition with the generation condition c input , and process the reference key frame set into the form of reference images, that is, obtain the latent space that is truly referenced during the denoising process where wd = [s:e] is the window for generating the video segment, s and e are the frame coordinates of the start frame and the end frame, and the sampling process of denoising is formulated as:

[0070]

[0071] where t is the current denoising step, represents the video segment of the window wd corresponding to the noise at the t-th step, represents the skeleton condition corresponding to the window wd, r represents the reference image, rp s:e represents the reference skeleton sequence corresponding to the window wd, represents the concatenated feature obtained in Step 6; obtain an approximation by denoising the video segment containing the key reference frame in advance at the current denoising step:

[0072]

[0073] where α t is a hyperparameter of the diffusion model, ∈ θ is the human action video generation model; after denoising the frames of the time window wd, obtain the estimated value of the video segment Then extract a frame from the estimated value to obtain Finally, form the latent space where is the reference key frame condition used when generating the window wd:

[0074]

[0075] where, is the reference key frame set, the mean μ θ and variance Σ of the latent space obtained by video samplingt The method is as follows:

[0076]

[0077] Wherein, σ t is a hyperparameter of DDIM sampling, ∈ θ is a human action video generation model.

[0078] Step 7 specifically includes the following steps:

[0079] Step 7.1, define a long video L is the length of the long video, and then the long video is divided into D overlapping segments:

[0080]

[0081] Wherein, l d represents the starting frame position of the d-th segment, which is obtained by a recursive formula:

[0082] l d+1 = l d + F - Δ;

[0083] Wherein, F is the number of frames inferred by the model at one time, and Δ is the size of the overlapping area;

[0084] Step 7.2, denoise the D overlapping segments divided by the human action video model:

[0085]

[0086] Obtain all the video segments included in the entire video of the previous denoising step; wherein, G video is the sampling process of denoising, denoted as:

[0087]

[0088] Wherein, r represents the reference image, and rp s:e represents the corresponding reference skeleton sequence in the window, represents the step splicing feature;

[0089] Step 7.3, by iteratively executing each video segment, gradually denoise the entire long video, and then iteratively execute Steps 7.1 to 7.2 for multi-step denoising, and finally generate the entire human action video.

[0090] A human action video generation device, including:

[0091] A memory: used to store a computer program for implementing the human action video generation method;

[0092] Processor: It is used to implement the human motion video generation method when executing the computer program.

[0093] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the human motion video generation method.

[0094] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0095] 1. The present invention uses reference key frame selection and selects reference key frames according to the similarity of skeletons. This is different from the method of selecting reference frames at fixed positions. Frames with different human postures scattered in the video are selected as references, solving the problem of lack of feature long-distance dependence and inconsistent long video timings in the generated long videos.

[0096] 2. The present invention also uses approximate guidance, utilizes the predicted reference key frames in the denoising process of each step, broadcasts reference features on all video segments of the long video, and unifies the estimation direction of human features, solving the problem of low long video generation efficiency.

[0097] 3. The present invention simultaneously uses a mask-based skeleton prediction network and confidence level classification control to process the position information and confidence information of skeleton key points, considering the continuity of motion time and skeleton confidence, enabling the video model not to require precise skeleton estimation, and solving the problem of poor robustness of the generated results.

[0098] In summary, the present invention proposes a robust and consistent long human motion video generation method, mainly including reference key frame selection, approximate guidance, a mask-based skeleton prediction network, and confidence level classification control, improving the consistency, generation efficiency, and robustness of long video generation. Description of the Drawings

[0099] Figure 1 It is a schematic diagram of the mask-based skeleton prediction network of the present invention.

[0100] Figure 2 It is the video generation process of the mask-based skeleton prediction network and confidence level classification control of the present invention.

[0101] Figure 3 It is a schematic diagram of the confidence level classification control of the present invention.

[0102] Figure 4 It is a schematic diagram of the approximate guidance of the present invention.

[0103] Figure 5 It is a schematic diagram of the long video sampling process using approximate guidance and a sliding window of the present invention. Detailed Embodiments

[0104] The present invention will be described in detail below with reference to the accompanying drawings.

[0105] The present invention provides a method for generating a human action video, and the specific operation steps are as follows:

[0106] Step 1: Based on the skeleton estimation network, obtain the skeleton sequence and confidence score of the target skeleton key points, and calculate the similarity of the human skeletons in different video frame contents according to the skeleton sequence. Before generating the video, calculate the similarity between every two frames of skeletons to approximate the content similarity between these two frames. For example, the similarity between the skeleton points is relatively low when a person is in a half-squat and standing position, or when turning around and facing the camera, and the appearance performance of the person in the corresponding video frames also has a large difference.

[0107] Step 1.1: Obtain the skeleton sequence of the target through the skeleton estimation network (DWPose is used in this embodiment), and based on the skeleton sequence, obtain the two-dimensional coordinate values (x f,p , y f,p ) of the p-th skeleton key point in the f-th frame;

[0108] Step 1.2: Calculate the weighted center coordinates of all skeleton points within a single frame based on the two-dimensional coordinate values (x f,p , y f,p ) of the skeleton key points

[0109]

[0110] where P is the total number of skeleton points, w p is the weight of the p-th skeleton point. For the hand, since the number of skeleton points is relatively large, set the weight of the hand skeleton points through w p ;

[0111] Step 1.3: The pose of a person is only determined by the relative positions between the coordinate points and has nothing to do with the absolute position of the person in the picture. Therefore, calculate the decentralized coordinates (x′ f,p , y′ f,p ) of each key point:

[0112]

[0113] Step 1.4: Calculate the skeleton similarity Similarity(i, j) between the i-th frame and the j-th frame:

[0114]

[0115] Step 2: Select reference key frames based on the skeleton similarity and construct a set of reference key frames. After obtaining the similarity metric of different video frame contents, it is necessary to select a set of reference key frames from them Cover as many features as possible that appear in the video, where k is the frame coordinate of the reference key frame. To make the reference key frames cover more features, the similarity between the reference key frames needs to be as low as possible. For example, when a person turns 360 degrees, the features of the front and back of the person are suitable as reference features because their features are different, and at this time their skeleton similarity is also relatively large.

[0116] Step 2.1, initialize the set of reference key frames and the set of reference key frame candidates Record the reference image given by the user as the 0th frame, and define the maximum number of reference key frames m;

[0117] Step 2.2, start selecting the nth reference key frame;

[0118] Step 2.3, calculate the sum of similarities S between frame j in the set of reference key frame candidates and the current set of reference key frames n,j :

[0119]

[0120] Step 2.4, find the frame k with the lowest similarity between frame j and the current set of reference key frames as the nth reference key frame: n

[0121]

[0122] Step 2.5, store the newly obtained reference key frame k n into the set of reference key frames :

[0123]

[0124] Step 2.6, delete the newly obtained reference key frame k n from the set of reference key frame candidates :

[0125]

[0126] Step 2.7, repeat steps 2.2 to 2.6 until m reference key frames with the maximum number of reference key frames are obtained, and the final set of reference key frames is obtained where k is the frame coordinate of the reference key frame.

[0127] For example Figure 1 ​As shown in the figure, in step 3, pre-train the skeleton prediction network. After obtaining the skeleton sequence of the target through the skeleton estimation network, the confidence score score of each skeleton key point is obtained at the same time. Classify the skeleton key points according to the confidence score score. Do not process the skeleton key points with a confidence lower than the threshold τ. Randomly mask and predict the skeleton key points with a confidence higher than the threshold τ to obtain the pre-trained skeleton prediction network.

[0128] Step 3.1, normalize the coordinates (x, y) of the skeleton key points to obtain (x norm , y norm );

[0129] Step 3.2, set the confidence threshold τ, and mask the skeleton key points with a confidence score score higher than the confidence threshold τ with probability p during training, so that the skeleton prediction network predicts the true coordinates of the masked skeleton points according to the positions and confidence scores score of other skeleton key points in the video:

[0130] (x masked , y masked , score masked ) = f mask (x norm , y norm , score);

[0131] Among them, f mask is a random masking function, whose function is to replace the input value (x norm , y norm ) with a random value (x masked , y masked ) from a normal distribution with probability p, and at the same time reduce the confidence score of the skeleton key point score masked ;

[0132] Step 3.3, process the coordinates of the skeleton key points using Fourier embedding to obtain the position embedding of each skeleton key point. The formula for Fourier embedding is:

[0133] φ(x) = [cos(xω0), sin(xω0),..., cos(xω B-1 ), sin(xω B-1 )];

[0134] Among them, B is the specified number of frequency bases, ω i = γ i / B , γ controls the frequency band span; substitute x masked , y masked , score maskedCalculate the Fourier embeddings separately, concatenate them and feed them into a multi-layer perceptron (MLP) for dimensionality increase processing, and finally obtain the input skeleton key point features

[0135]

[0136] Among them, represents the concatenation operation, is the skeleton embedding, which is used to distinguish the skeleton key points representing different parts of the body. F, P, and C are the number of frames, the number of skeleton key points, and the embedding dimension respectively;

[0137] Step 3.4, Use the Transformer architecture to process the obtained F×P skeleton key point features h0 as tokens, and perform spatial modeling and temporal modeling simultaneously to handle the relationships of skeleton key points within a single frame and the continuity of cross-frame motion; Use skeleton attention to process all the skeleton key points within each frame, and use temporal attention to process the same skeleton key point on different frames. By alternately using skeleton attention and temporal attention, fuse and process all the skeleton key point features h0 to reduce the computational complexity, and process the skeleton key point features h0 through N layers of Transformer blocks:

[0138] h i = TemporalTransformer(SpatialTransformer(h i-1 )) for i = 1,…,N;

[0139] Finally, obtain the processed confidence-aware feature space h N ;

[0140] At the same time, use rotational position encoding in the temporal attention layer. For the query q a at position a and the key k b at position b, the obtained attention score is:

[0141] AttentionScore(q a , k b ) = Re[(R a q a ) T (R b k b )];

[0142] Among them, is the complex domain rotation matrix obtained from the hyperparameters of the rotational position encoding;

[0143] Step 3.5, Feed the feature space h NInput the multi-layer perceptron (MLP) to obtain the predicted coordinates of each skeleton key point The loss function optimizes the predicted coordinates with the confidence score score of each skeleton key point predicted by the skeleton extraction network as the weight And the distance from the skeleton key point coordinates (x, y):

[0144]

[0145] Among them, 0 ≤ f ≤ F, 0 ≤ p ≤ P, w is the weight, and finally a mask-based skeleton prediction network is obtained

[0146] Step 4, during inference, obtain the single-frame skeleton key point coordinates (x, y) with the confidence score score through the skeleton estimation network. For the skeleton key points with low confidence, use the skeleton prediction network in Step 3 to correct the coordinates. After being processed by the skeleton prediction network that fully considers the temporal continuity information and confidence information, the skeleton key point coordinates closer to the true distribution are obtained

[0147] Step 4.1, normalize the skeleton key point coordinates (x, y) to obtain the skeleton key point coordinates (x norm , y norm );

[0148] Step 4.2, according to the skeleton key point coordinates (x norm , y norm ), mask the skeleton key points with a confidence lower than τ to obtain the masked skeleton key point coordinates (x masked , y masked , score masked );

[0149] Step 4.3, use Fourier embedding to process the masked skeleton key point coordinates (x masked , y masked , score masked ) to obtain the position embedding of each skeleton key point, and then send it into the multi-layer perceptron (MLP) for processing and dimensionality increase, and finally obtain the input skeleton key point features

[0150]

[0151] Among them, φ represents Fourier embedding, represents the concatenation operation, is the skeleton embedding, which is used to distinguish the skeleton key points representing different parts of the body. F, P, and C are the number of frames, the number of skeleton key points, and the embedding dimension respectively

[0152] Step 4.4: Using the pre-trained skeleton prediction network, take the obtained F×P skeleton key-point features h0 as tokens for feature processing, and finally obtain the processed confidence-aware feature space h N ;

[0153] Step 4.5: Input the feature space h N into a multi-layer perceptron (MLP) to obtain the predicted coordinates of each skeleton key point.

[0154] As Figure 2 、 Figure 3 shown, in Step 5, use confidence grading to provide guidance information with different credibility levels. For the part of the skeleton result that may contain noise extracted by the skeleton estimation network and whose confidence is higher than the threshold τ, take it as the reliable feature and send it into the high-confidence skeleton encoder to obtain the corresponding strong skeleton constraint c strong ; In addition, take the skeleton key points corrected in Step 4 as low-credibility features, send them into the low-confidence skeleton encoder to obtain the confidence-aware feature space h N , which serves as the corresponding weak skeleton hint c weak ; Concatenate the strong skeleton constraint c strong and the weak skeleton hint c weak as the generation condition of the video model:

[0155] c input = Concat(c strong , c weak );

[0156] Among them, Concat represents the concatenation of features.

[0157] This way of treating high-confidence and low-confidence skeleton points differently also improves the generation results in cases where the ground truth of human skeleton key points may not exist, such as occlusion, partial movement of the person out of the frame, and motion blur.

[0158] As Figure 4 、 Figure 5 shown, in Step 6, use approximate guidance to guide the generation of long videos using the reference key-frame set. When generating a video, it is necessary to generate a human action video according to the generation condition c input obtained in Step 5 and the reference key-frame set constructed in Step 2 . Replace the input skeleton sequence condition with the generation condition c input , and process the reference key-frame set into the form of reference images, that is, obtain the latent space that is truly referenced during the denoising process where wd = [s:e] is the window for generating the video segment, and s and e are the frame coordinates of the starting frame and the ending frame. The sampling process of denoising is formulated as:

[0159]

[0160] Among them, t is the current denoising step, represents the video segment of the window wd corresponding to the noise at the t-th step, represents the skeleton condition corresponding to the window wd, r represents the reference image, and rp s:e represents the reference skeleton sequence corresponding to the window wd, represents the stitching feature obtained in step 6; obtain an approximation by denoising in advance the video segment containing the key reference frame at the current denoising step:

[0161]

[0162] Among them, α t is a hyperparameter of the diffusion model, ∈ θ is a human action video generation model; after denoising the frames of the time window wd, the estimated value of the video segment is obtained Then from the estimated value extract one frame to obtain Finally, form the latent space Among them is the reference key frame condition used when generating the window wd:

[0163]

[0164] Among them, is the set of reference key frames, and the method for the mean μ θ and variance Σ t of the latent space obtained by video sampling is as follows:

[0165]

[0166] Among them, σ t is a hyperparameter of DDIM sampling, ∈ θ is a human action video generation model.

[0167] Step 7, use the sliding window strategy to achieve the generation of long videos. Step 7.1, define the long video L is the length of the long video, and then divide the long video into D overlapping segments:

[0168]

[0169] Among them, l d represents the starting frame position of the d-th segment, which is obtained by the recurrence formula:

[0170] l d+1 = l d + F - Δ;

[0171] Among them, F is the number of frames for one inference of the model. For a long video where L >> F, Δ is the size of the overlapping region;

[0172] Step 7.2: Denoise the D overlapping segments divided by the human action video model:

[0173]

[0174] Among them, G video is the sampling process of denoising in Step 6, denoted as:

[0175]

[0176] Among them, r represents the reference image, and rp s:e represents the corresponding reference skeleton sequence in the window, represents the stitching feature obtained in Step 6;

[0177] Obtain all the video segments contained in the entire video of the previous denoising step. During the generation of two adjacent video segments, a part of the latent space of the overlapping time region will be shared, and the denoising results of the front and back two video segments are averaged over these overlapping times. Due to the use of approximate guidance, the video segment where the reference key frame is located will be preferentially denoised.

[0178] Step 7.3: Iteratively execute for each video segment to gradually denoise the entire long video, and then iteratively execute Steps 7.1 - 7.2 for multi-step denoising to finally generate the entire human action video.

[0179] A human action video generation device, comprising:

[0180] A memory: used to store a computer program for implementing the human action video generation method;

[0181] A processor: used to implement the human action video generation method when executing the computer program.

[0182] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the human action video generation method.

[0183] Comparative analysis

[0184] Compared with the patent application "Long video generation method and device based on text input" (Publication No.: CN117793480A), the reference key frames used in the present invention are related to the video content and have nothing to do with the time points of the key frames. However, the video key frames generated in the first stage of the comparative document are only related to the time points of the key frames and have nothing to do with whether they can provide reference features. This results in that the key frames generated in the first stage of the comparative document cannot contain as many features as possible, and the inconsistent phenomena will occur when generating each video segment in the second stage for those features that are not included. The method of selecting key frames based on video content used in the present invention explicitly includes as many features as possible, avoiding the occurrence of this situation and solving the problem of lack of consistency in generating videos. In addition, the present invention uses an approximate guidance method to generate reference frames and other video frames simultaneously in a denoising stage, which is simpler and more efficient than the two-stage method of the compared patent.

[0185] Compared with the above patent application "Video generation method, system, storage medium and device driven by human body skeleton" (Publication No.: CN118609033A), the present invention does not separate the foreground and the background, and does not draw the generated human actions on the background through a mask. Instead, it generates simultaneously according to the foreground and the background in the reference key frames, and the implementation is simpler. In addition, the present invention uses a mask-based skeleton prediction network and confidence level classification control to make the generation result more robust.

[0186] The following table shows the comparison of the experimental results of the present invention and other methods for generating long videos on the UBC Fashion dataset. The experimental environment is the operating system Ubuntu 22.04, the programming language Python 3.9, the deep learning library Pytorch 2.0.1, and the CUDA version 12.1. The generated video size is 512×768:

[0187] Method LPIPS↓ FVD↓ GEN-L 0.060 65.9 FreeNoise 0.062 69.7 Ours (This Invention) 0.057 49.5

[0188] The experimental results in the table show that the present invention is superior to other methods in both the LPIPS index for evaluating the quality of a single frame and the FVD index for evaluating the quality of a video, indicating that the present invention has unique advantages in generating consistent high-quality long videos.

Claims

1. A method for generating a human motion video, characterized in that, It includes the following steps: Step 1: Obtain the skeleton sequence and confidence score of the target skeleton key points based on the skeleton estimation network, and calculate the similarity of the human skeletons in the content of different video frames according to the skeleton sequence; Step 2: Select reference key frames based on the skeleton similarity and construct a set of reference key frames; Step 3: Set a confidence threshold. Based on the confidence score obtained in Step 1 and the set confidence threshold, use the predicted occluded skeleton key points higher than the confidence threshold as the target to pre-train the skeleton prediction network; Step 4: Use the pre-trained skeleton prediction network to correct the coordinates of the skeleton key points with a confidence lower than the threshold; Step 5: Respectively extract and splice the features of the skeleton key points with a confidence score higher than the threshold obtained in Step 1 and all the skeleton key points corrected in Step 4 as the generation conditions of the video model; Step 6: Use the set of reference key frames constructed in Step 2 and the generation conditions in Step 5 to approximately guide the generation of video segments of the previous denoising step; Step 7: Use the sliding window strategy for Step 6 until all the video segments included in the entire video of the previous denoising step are generated; Loop through Step 7 until the entire human action video is generated.

2. The method according to claim 1, wherein The specific steps of Step 1 include the following steps: Step 1.1, obtain the skeleton sequence of the target through the skeleton estimation network, and obtain the two-dimensional coordinate values (x f,p , y f,p ) of the p-th skeleton key point in the f-th frame based on the skeleton sequence; Step 1.2, calculate the weighted center coordinates of all skeleton points within a single frame based on the two-dimensional coordinate values (x f,p , y f,p ) where P is the total number of skeleton points, and w p is the weight of the p-th skeleton point; Step 1.3, calculate the decentralized coordinates (x′ f,p , y′ f,p ) of each key point: Step 1.4: Calculate the skeleton similarity Similarity(i,j) between the i-th frame and the j-th frame:

3. The method according to claim 1, wherein The specific steps of Step 2 include the following steps: Step 2.1, initialize the set of reference key frames and the set of reference key frame candidates Record the reference image given by the user as the 0th frame, and define the maximum number of reference key frames m; Step 2.2, calculate the candidate set of reference key frames The similarity sum S between the middle frame j and the current set of reference key frames is calculated n,j as follows: Among them, Similarity(i,j) is the skeleton similarity between the i-th frame and the j-th frame; Step 2.3, find the frame k with the lowest similarity between frame j and the current set of reference key frames as the nth reference key frame: n ​ Step 2.4, store the newly obtained reference key frame k n into the reference key frame set as follows: Step 2.5, delete the newly obtained reference key frame k n from the reference key frame candidate set : Step 2.6, repeat Steps 2.2 to 2.5 until m reference key frames with the maximum number are obtained, and the final set of reference key frames is obtained. Among them, k is the frame coordinate of the reference key frame.

4. The method according to claim 1, wherein The specific steps of Step 3 include the following steps: Step 3.1, normalize the coordinates (x, y) of the skeleton key points to obtain (x norm , y norm ); Step 3.2: Set a confidence threshold τ. During training, occlude the skeleton key points with a confidence score score higher than the confidence threshold τ with a probability p, so that the skeleton prediction network can predict the true coordinates of the occluded skeleton points according to the positions and confidence scores score of other skeleton key points in the video; (x masked , y masked , score masked ) = f mask (x norm , y norm , score); where f mask is a random masking function that replaces the input values (x norm , y norm ) with random values (x masked , y masked ) from a normal distribution with probability p, while reducing the confidence score masked ; Step 3.3: Process the coordinates of the skeleton key points using Fourier embedding to obtain the position embedding of each skeleton key point. The formula for Fourier embedding is: φ(x) = [cos(xω0), sin(xω0),..., cos(xω B-1 ), sin(xω B-1 )]; Among them, B is the specified number of frequency bases, ω i = γ i / B , where γ controls the frequency band span; Fourier embeddings are calculated for x masked , y masked , score masked respectively, and then concatenated and sent into a multi-layer perceptron (MLP) for processing and dimensionality increase, finally obtaining the input skeleton key point features Among them, represents a splicing operation, is a skeleton embedding used to distinguish skeleton key points representing different body parts. F, P, and C are the number of frames, the number of skeleton key points, and the embedding dimension respectively; Step 3.4: Use skeleton attention to process all the skeleton key points within each frame, and use temporal attention to process the same skeleton key point on different frames. By alternately using skeleton attention and temporal attention to fuse and process the features h0 of all the skeleton key points, the features h0 of the skeleton key points are processed through N layers of Transformer blocks: h i = TemporalTransformer(SpatialTransformer(h i-1 )) for i = 1, …, N; Finally, the processed confidence-aware feature space h is obtained N ; At the same time, rotational position encoding is used in the temporal attention layer for the query q at position a a and the key k at position b b , and the resulting attention score is AttentionScore(q a ,k b ) = Re[(R a q a ) T (R b k b )]; Among them, is a complex-domain rotation matrix obtained from the hyperparameters of rotational position encoding; Step 3.5, input the feature space h N into a multi-layer perceptron (MLP) to obtain the predicted coordinates of each skeleton key point The loss function optimizes the predicted coordinates with the confidence score score of each skeleton key point predicted by the skeleton extraction network as the weight and the distance from the skeleton key point coordinates (x, y): Among them, 0 ≤ f ≤ F, 0 ≤ p ≤ P, w is the weight, and finally a mask-based skeleton prediction network is obtained.

5. The method according to claim 1, wherein The specific steps of Step 4 include the following steps: Step 4.1, during inference, obtain the single-frame skeleton key point coordinates (x, y) with a confidence score score through the skeleton estimation network, normalize the skeleton key point coordinates (x, y) to obtain the skeleton key point coordinates (x norm , y norm ); Step 4.2, according to the skeleton key point coordinates (x norm , y norm ), mask the skeleton key points with a confidence level lower than τ to obtain the coordinates of the masked skeleton key points (x masked , y masked , score masked ); Step 4.3, use Fourier embedding to process the coordinates (x masked , y masked , score masked ) of the occluded skeleton key points to obtain the position embedding of each skeleton key point, and then send it into a multi-layer perceptron (MLP) for dimensionality increase processing, and finally obtain the input skeleton key point features where φ represents the Fourier embedding, represents the concatenation operation, is the skeleton embedding, which is used to distinguish skeleton key points representing different body parts. F, P, and C are the number of frames, the number of skeleton key points, and the embedding dimension respectively; Step 4.4, using the pre-trained skeleton prediction network, take the obtained F×P skeleton key-point features h0 as tokens for feature processing, and finally obtain the processed confidence-aware feature space h N ; Step 4.5, input the feature space h N into a multi-layer perceptron (MLP) to obtain the predicted coordinates of each skeleton key point.

6. The method according to claim 1, characterized in that, The specific steps of Step 5 include the following steps: The part with a confidence higher than the threshold τ in the skeleton result that may contain noise extracted by the skeleton estimation network is used as a reliable feature and sent to the high-confidence skeleton encoder to obtain the corresponding strong skeleton constraint c strong ; In addition, the skeleton key points corrected in step 4 are used as low-confidence features and sent to the low-confidence skeleton encoder to obtain the confidence-aware feature space h N , which serves as the corresponding weak skeleton hint c weak ; The strong skeleton constraint c strong , the weak skeleton hint c weak are concatenated as the generation condition for the video model: c input = Concat(c strong , c weak ); Among them, Concat represents the splicing of features.

7. The method according to claim 1, wherein The specific steps of Step 6 include the following steps: Generate condition c input Instead of the input skeleton sequence condition, the reference key frame set will be Processed in the form of a reference image, that is, the latent space that is truly referenced is obtained during the denoising process Where wd = [s:e] is the window for generating the video segment, s and e are the frame coordinates of the starting frame and the ending frame, and the sampling process of denoising is formulated as: where t is the current denoising step, represents the video segment of window wd corresponding to the noise at the t-th step, represents the skeleton condition corresponding to window wd, r represents the reference image, rp s:e represents the reference skeleton sequence corresponding to window wd, represents the splicing feature obtained in step 6; an approximation is obtained by pre-denoising the video segment containing the key reference frame at the current denoising step: Among them, α t is a hyperparameter of the diffusion model, ∈ θ is a human action video generation model; after denoising the frames of the time window wd, the estimated value of the video segment is obtained Then, from the estimated value one frame is extracted to obtain Finally, a latent space is formed Among them is the reference key frame condition used when generating the window wd: Among them, is the set of reference key frames, and the mean μ of the latent space obtained by video sampling θ and the variance Σ t are as follows: Among them, σ t is a hyperparameter of DDIM sampling, ∈ θ is a model for generating human action videos.

8. The method according to claim 1, characterized in that, The specific steps of Step 7 include the following steps: Step 7.1, define a long video L is the length of the long video, and then divide the long video into D overlapping segments: where l d represents the starting frame position of the d-th segment, which is obtained by a recursive formula: l d+1 = l d + F - Δ; Among them, F is the number of frames for one inference of the model, and Δ is the size of the overlapping region; Step 7.2: Denoise the D overlapping segments divided by the human action video model: Obtain all video segments contained in the entire video of the previous denoising step; where G video is the sampling process of denoising, denoted as: Among them, r represents the reference image, and rp s:e represents the corresponding reference skeleton sequence in the window, represents the step splicing feature; Step 7.3: By iteratively executing on each video segment, gradually denoise the entire long video, and then iteratively execute Steps 7.1 to 7.2 for multi-step denoising to finally generate the entire human motion video.

9. A human motion video generation device, characterized in that, Including: A memory: for storing a computer program for implementing a human motion video generation method according to any one of Claims 1 to 8; A processor: for implementing a human motion video generation method according to any one of Claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of a human motion video generation method according to any one of Claims 1 to 8 are implemented.

Images