A method for generating multi-pose guided limb motion videos based on a diffusion model

Abstract

This invention provides a multi-pose guided limb movement video generation method based on a diffusion model, belonging to the field of AIGC video generation technology. It introduces HaMeR hand estimation and a lightweight hand pose guidance network to provide accurate 3D and spatiotemporal features of the hand, improving the quality of hand generation. It simplifies 3D information input through gradient color surface rendering based on the SMPler-X model and incorporates an adaptive pose gating module to achieve adaptive learning through multi-condition fusion, improving the stability and overall quality of human body generation. It adopts an end-to-end single UNet architecture and a lightweight pose guidance network to simplify model training and efficiently convey pose information. Finally, it constructs a distributed data processing pipeline based on the Ray framework to accelerate dataset preparation.

Description

Technical Field

[0001] This invention relates to the field of AIGC video generation technology, and in particular to a method for generating multi-pose guided limb movement videos based on a diffusion model. Background Technology

[0002] With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), video games, film and television special effects, and human-computer interaction, the demand for generating realistic and natural human body movements is increasing. Traditional methods for generating human movements, such as keyframe animation, motion capture, or physical simulation techniques, suffer from low efficiency, high cost, and unnatural generated movements.

[0003] In recent years, AI-generated content (AIGC) technology has made significant progress in the field of video generation. Among them, diffusion models, as a powerful generative model, generate high-quality image and video sequences through a progressive denoising process. With the rise of diffusion models, significant progress has been made in human image animation. This task aims to convert static reference images into dynamic video sequences that depict realistic and temporally coherent movements, following a sequence of driving poses.

[0004] Diffusion models, by simulating stochastic processes in data distribution, can generate diverse and realistic content, but they still face challenges when applied to human limb motion generation. The human body and hands have complex skeletal structures and joint degrees of freedom, requiring precise capture of subtle changes and coordination in limb movements for motion generation. The pose sequences used in existing training and inference are typically estimated from real video data using pose detection algorithms, and may therefore be contaminated by false detections, leading to unnatural distortions, limb penetration, or motion logic errors in the generated motion. An intuitive solution is to select high-quality pose sequences, but this does not fundamentally address the underlying problems. In hand motion generation, due to the numerous finger joints, limited range of motion, and dependence on context, coupled with the relatively small proportion of the hand in the input image, models struggle to learn sufficient information, easily resulting in missing details or jitter artifacts, making the generated hand movements lack realism and coherence. Furthermore, as mentioned above, the pose sequence is extracted using pose estimation methods such as DWPose. These methods are typically designed for static images, which inevitably introduces inter-frame jitter into the estimated pose sequence. This can lead to timing inconsistencies, abrupt transitions, or repetitive patterns in the motion when the diffusion model generates long-term motion sequences. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a multi-pose guided limb movement video generation method based on a diffusion model. The method aims to improve the stability of human body generation and the quality of human hand generation by using additional human hand and body condition information, adaptive posture gating module design, and lightweight posture guidance network.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] On one hand, the present invention provides a method for generating multi-pose guided limb movement videos based on a diffusion model, comprising the following steps:

[0008] Step 1: Collect and preprocess the target limb movement video data, and build a distributed data processing pipeline to clean and label the video dataset;

[0009] Step 2: Perform hand pose estimation based on the preprocessed video dataset, extract the 3D mesh and joint parameters of the hand, and generate spatiotemporal pose features as conditional representations through a lightweight hand pose guidance network;

[0010] Step 3: Perform human pose estimation based on the preprocessed video dataset, introduce multi-condition 3D information input, and fuse multiple pose conditions through an adaptive pose gating module and a lightweight pose guidance network;

[0011] Step 4: The conditional information obtained in Step 2 and Step 3 is injected into the end-to-end single UNet diffusion model through a cross-attention mechanism. During the denoising process, the model gradually recovers the video frame sequence from the noise and generates limb action videos guided by heterogeneous data.

[0012] Step 5: Based on the reconstruction loss function of the generated video and the original video, backpropagate to optimize the model parameters, and then verify and evaluate the model.

[0013] Furthermore, in step 1, a multi-machine distributed video processing pipeline based on the Ray framework is adopted. The pipeline integrates a variety of modular operators, including lens boundary detection operators, optical flow motion scoring operators, optical character recognition operators, video aesthetic quality assessment operators, and face and body pose detection operators. Through Ray's task scheduling and Actor model, cross-node parallel processing is achieved, and the video data is cleaned into high-quality samples. The ytdlp and wget tools are used to collect body movement video data from the network and integrate open source datasets to form a dedicated video dataset suitable for pose-driven body movement generation tasks.

[0014] Further, the specific method of step 2 is as follows: First, hand region detection and cropping are performed on the input video slices, and the hand region is located using a bounding box detection algorithm; then, the cropped hand image sequence is fed into the HaMeR model to predict the 3D mesh of the hand, the 24-dimensional joint rotation parameters, and the corresponding MANO model shape parameters, and finally rendered as a gesture mesh image sequence as additional pose condition input; then, a lightweight hand pose guidance network is introduced to generate spatiotemporal pose features. This network first encodes the original gesture pose heatmap using multi-layer convolution, and then introduces a temporal convolution module. The temporal convolution module is configured to perform convolution operations on the features of consecutive frames to fuse cross-frame dynamic information and obtain a stable and smooth motion trajectory expression; the extracted spatiotemporal pose features are mapped into a conditional representation aligned with the latent space of the diffusion model, which is used to guide the realistic and accurate generation of the hand during the video generation process.

[0015] Further, the specific method for human pose estimation in step 3 is as follows: 3D parameters of the human body are estimated based on the SMPLer-X model, including joint position, rotation, and shape parameters; then, gradient color rendering is used to render the surface, thereby simultaneously fusing 3D, depth, and continuous semantic information, simplifying the injection method of geometric information, and obtaining a gradient color rendering surface map sequence; the rendering process is represented as: surface coloring of the mesh output by the SMPLer-X model, where the color value C is calculated as C=f(D,S) based on the depth D and semantic label S, where f is the gradient function, and the gradient color rendering surface map sequence is used as dense geometric prior input to the subsequent backbone network.

[0016] Furthermore, in step 3, the specific method for fusing multiple pose conditions is as follows: First, construct joint input features; then, stitch together the two-dimensional keypoint heatmap sequence extracted by the DWPose model and the gradient color rendering surface map sequence rendered by the SMPler-X model in the channel dimension to form a shape of... The combined input, of which, Indicates batch size, and These represent the number of channels under two-dimensional and three-dimensional conditions, respectively. and These represent the height and width of the feature map, respectively.

[0017] Subsequently, features are extracted using a backbone network. The joint input is fed into a multi-condition backbone network (MultiCondBackbone), which adopts a grouped convolutional structure and configures the number of convolutional groups to be consistent with the number of modalities of the input conditions. This is to maintain the independence of different modal pose conditions in the channel dimension during downsampling, while sharing convolutional weights between groups to achieve preliminary feature extraction.

[0018] Subsequently, feature fusion is performed through an adaptive pose gating module. First, the output channels of the backbone network are compressed using two convolutional layers to generate spatial-channel joint attention features. Then, the sigmoid activation function maps these spatial-channel joint attention features to normalized gating weights. The gating weights are configured to have two channels, corresponding to two-dimensional pose conditions and three-dimensional geometric conditions, respectively. The dimension of the gating weights is... Finally, the gating weights are multiplied element-wise with the two reshaped features, and the results are summed to obtain the fused features; this process can be represented as: obtaining weighted two-dimensional features. Obtain weighted three-dimensional features ; Obtain the final fused pose features ;in, and Representing two-dimensional and three-dimensional features respectively. This represents the gating weights. Through the above mechanism, when a serious error occurs in the two-dimensional keypoints of a frame due to occlusion or detection failure, the adaptive attitude gating module automatically reduces the gating weights of the corresponding channel, making the model focus on reliable three-dimensional geometric conditions. Conversely, when there is a deviation in the three-dimensional mesh fitting, it automatically focuses on the two-dimensional keypoint features.

[0019] Furthermore, in step 3, the lightweight pose guidance network is used to inject the fused pose features into the diffusion model. Its specific structure includes: multiple cascaded lightweight processing blocks, each block comprising two convolutional kernels with a size of [missing information]. The lightweight processing block includes convolutional layers, SiLU activation functions, and optional downsampling layers; the number of output channels of the lightweight processing block is configured to match the channel dimensions of each layer in the subsequently connected diffusion model backbone network; a zero-convolution module, located at the end of the lightweight pose guidance network, consists of modules with initialized weights and biases of zero. The convolutional layers are used to generate the final multi-scale control signals and ensure that the model maintains the pre-trained distribution characteristics in the early stages of training. The output features of each layer are multiplied by a learnable scaling factor before being injected into the backbone network to adjust the control strength of the pose conditions on the generation process.

[0020] Furthermore, in step 4, the specific process of generating multi-pose conditional limb action videos using an end-to-end single UNet network architecture is as follows: During the training phase, Gaussian noise is added to the latent representation of the original video frames; the spatiotemporal pose features generated in step 2 and the fused pose features extracted in step 3 are used as conditional inputs and injected into the UNet network through a cross-attention mechanism; the UNet network predicts and removes the added noise; during the inference generation phase, starting from pure Gaussian noise, guided by the fused pose features, the UNet network performs multi-step iterative denoising to gradually recover a temporally coherent video frame sequence with actions aligned with conditions.

[0021] On the other hand, this application proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the multi-pose guided limb movement video generation method based on the diffusion model.

[0022] Thirdly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the multi-pose guided limb movement video generation method based on a diffusion model.

[0023] Fourthly, this application proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for generating multi-pose guided limb movement videos based on a diffusion model.

[0024] The beneficial effects of the above technical solutions are as follows: The multi-pose guided limb movement video generation method based on the diffusion model provided by this invention, based on the diffusion model, addresses the problems of poor hand generation quality and unstable generation and poor human body quality caused by pose estimation errors in existing pose-driven limb movement video generation methods, and proposes optimization methods: (1) Introducing HaMeR hand estimation and a lightweight hand pose guidance network to provide accurate 3D and spatiotemporal features of the hand and improve the quality of hand generation; (2) Simplifying the input of three-dimensional information through gradient color surface rendering based on the SMPler-X model, and adding an adaptive pose gating module to achieve adaptive learning with multi-condition fusion, thereby improving the stability and overall quality of human body generation; (3) Adopting an end-to-end single UNet architecture and a lightweight pose guidance network to simplify model training and efficiently convey pose information; (4) Constructing a distributed data processing pipeline based on the Ray framework to accelerate dataset preparation. This invention significantly reduces FID-VID and FVD scores on the TikTok dataset, and improves Hand-C, PSNR and SSIM metrics, outperforming baseline methods such as AnimateAnyone, AudCast and Champ. Attached Figure Description

[0025] Figure 1 This is a flowchart of the multi-pose guided limb movement video generation method based on a diffusion model provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of the data processing flow based on the Ray framework provided in Embodiment 1 of the present invention. Detailed Implementation

[0027] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0028] Example 1:

[0029] A multi-pose guided limb movement video generation method based on a diffusion model is proposed. This method addresses the issues of poor hand generation quality and unstable generation due to pose estimation errors, as well as poor body quality, in existing pose-driven limb movement video generation methods, by proposing an optimization approach. Figure 1 As shown, the method of this embodiment is described below.

[0030] Step 1: Collect and preprocess the target limb movement video data, and build a distributed data processing pipeline to clean and label the video dataset.

[0031] Figure 2 This is a flowchart of the data processing flow based on the Ray framework in this embodiment. It shows a fully automated distributed pipeline from the input of raw video to the final output of a high-quality multi-condition limb motion dataset, which is divided into three stages: stage 1 (video slicing and meta-analysis), stage 2 (deep content understanding and filtering), and stage 3 (multimodal fine pose extraction).

[0032] In this embodiment, ytdlp and wget tools were used to collect and download video data related to body movements from online platforms (such as bilibili, YouTube, etc.). Simultaneously, existing open-source datasets (such as HumanVid or similar body movement datasets) were integrated to obtain a raw dataset containing 21,000 video segments. The video data covers a variety of body movement scenarios, ensuring diversity and representativeness. The resolution of the acquired videos was ensured to be no less than 720p, and the length of each video segment was controlled within 10-120 seconds for subsequent processing.

[0033] This embodiment constructs a multi-machine distributed video processing pipeline based on the Ray framework, integrating modular components including an AutoShot operator for shot boundary detection, a Unimatch operator for optical flow motion scoring, an Optical Character Recognition (OCR) operator, a DoV video quality assessment operator, and a face and body pose detection operator. Parallel processing across nodes is achieved through Ray's task scheduling and Actor model. The specific process is as follows: First, metadata (meta) such as duration, resolution, and bitrate of the video is collected and initially screened; then, the aforementioned operators are used for cleaning, removing low-quality frames with blurred transitions, low lighting, or large areas of subtitle obstruction; finally, DWPose and SMPler-X are used for preliminary annotation. Compared to traditional single-machine processing, this pipeline improves data preparation efficiency by approximately 3 to 5 times.

[0034] Step 2: Perform hand pose estimation and feature extraction based on the preprocessed video dataset.

[0035] Step 2.1: Hand Pose Estimation and 3D Reconstruction. First, a bounding box detection algorithm (such as the YOLO series) is used to detect and crop the hand region from the input video slices. Then, the cropped hand image sequence is fed into the HaMeR model. This model, based on the MANO hand parametric framework, predicts the dense 3D mesh of the hand, 24-dimensional joint rotation parameters, and corresponding shape parameters. Finally, these parameters are rendered into a sequence of gesture mesh images containing depth and normal information, serving as additional strong geometric pose conditions. This process ensures the accuracy of the hand's 3D information and avoids the depth loss problem inherent in traditional 2D keypoint estimation.

[0036] Step 2.2: Generate Spatiotemporal Pose Features. A lightweight HandNet gesture guidance network is introduced. This network first encodes the original gesture heatmap using 3 to 5 layers of 2D convolutions (Conv2D) to extract local spatial features. Subsequently, a Temporal Convolutional Network (TCN) module is introduced to convolve the features of consecutive frames to fuse cross-frame dynamic information and obtain a smooth motion trajectory. Finally, the extracted spatiotemporal pose features are mapped to a conditional representation aligned with the latent space of the diffusion model, which is used to guide the realistic generation of hands during video generation. In this embodiment, the number of network parameters is controlled within 1M to avoid increasing the overall model burden.

[0037] Step 3: Integrate multi-condition 3D information for human posture estimation and adaptive control.

[0038] Step 3.1: Extraction of parametric geometric features of the human body.

[0039] Based on the SMPLer-X model, 3D human body parameters are estimated from video frame sequences to obtain joint position, rotation, and shape parameters. Subsequently, a gradient color rendering strategy is used to color the human body surface, where the color value C is calculated as C=f(D,S) (f is a gradient function) based on the depth D and semantic label S, thereby generating a gradient color human body control map sequence containing 3D structure, depth relationships, and continuous semantics, simplifying the injection of 3D geometric information.

[0040] Step 3.2: Adaptive gating and feature fusion.

[0041] A lightweight pose guide network (PoseGuider) is used to convert various pose conditions into control signals. This network includes multiple cascaded lightweight processing blocks and zero convolutional modules.

[0042] Specifically, the two-dimensional keypoint heatmap sequence extracted by DWPose and the gradient color three-dimensional human body control map sequence rendered by SMPler-X are first stitched together in the channel dimension to form a shape of... The input is a combined input. This input is fed into a multi-condition backbone network, where features are extracted while maintaining modality independence through grouped convolutions. The number of output channels is configured in this embodiment as follows: .

[0043] Subsequently, an adaptive pose gating module is used for fusion. This module first compresses the 256×2 channels using two convolutional layers (3×3 + 7×7) to generate a spatial-channel attention map, which is then activated by a sigmoid function to obtain a dimension of... The normalized gated weights are then multiplied element-wise with the reshaped features from both paths to achieve weighted fusion.

[0044] ;

[0045] ;

[0046] ;

[0047] in, and Representing two-dimensional and three-dimensional features respectively. This indicates the gating weight.

[0048] Adaptive Mechanism Explanation: When the confidence level of a 2D keypoint in a frame decreases due to occlusion or detection failure, the gating module automatically reduces the weight of the corresponding channel, making the model rely on reliable 3D rendering conditions. Conversely, when there is a deviation in the 3D mesh fitting (such as depth estimation error), the model automatically focuses on 2D keypoint features. This bidirectional complementary mechanism realizes the complementary advantages of heterogeneous features, ensuring the stability of the generated results.

[0049] The fused features are further processed through five lightweight blocks (each containing only two 3×3 convolutions + SiLU activation + optional downsampling), with output channels of 320, 320, 640, 1280, and 1280 respectively. The final multi-scale control signal is generated by a 1×1 convolution wrapped in a zero module. Each layer output is multiplied by a learnable scaling factor (initialized to 2.0) to match the feature amplitudes of each layer in the diffusion model UNet.

[0050] Zero-convolution modules are located at the output of each lightweight processing block and are used to map the pose features of the corresponding layer into control signals that match the backbone network of the diffusion model. In this embodiment, the zero-convolution modules preferably employ... The convolutional layers are implemented with their kernel weights and biases set to zero during initialization. Since the initial parameters are all zero, the zero-convolutional modules output near-zero control residuals in the early stages of training, thus avoiding abrupt changes in the original feature distribution of the pre-trained UNet backbone network and helping to maintain the generation stability of the diffusion model in the initial stage.

[0051] As training progresses, the parameters of the zero-convolutional modules gradually deviate from zero under backpropagation, enabling the multi-scale features output by the lightweight pose guidance network to be progressively injected into the corresponding layers of the diffusion model. Specifically, pose features at different resolutions are linearly mapped by the zero-convolutional modules to form control signals with the same feature dimensions as the UNet encoder, intermediate layers, or decoder, and participate in the feature update of the current layer through residual modulation. This design avoids the oscillation of generated results caused by excessively strong pose conditions in the early stages of training, and gradually enhances the constraint of key human structures, limb contours, and motion trends on the video generation process in the later stages of training, thereby improving the performance of the generated video in terms of motion tracking, structural consistency, and temporal smoothness.

[0052] To further capture temporal dynamics, a motion module can be optionally inserted after each processing block. However, in the optimal lightweight configuration of this embodiment, this option is turned off (use_motion_modules=False) to avoid memory overload, and sufficiently smooth motion trajectories can be obtained by relying solely on the temporal modeling capabilities of the convolution itself.

[0053] Through the aforementioned adaptive gating mechanism and lightweight multi-scale coding structure, this embodiment achieves robust fusion under conditions of uneven quality in two pose conditions, effectively solving the problem of generation collapse when pose is damaged by traditional fixed weight or simple splicing methods, while maintaining extremely high computational efficiency and extremely low parameter overhead.

[0054] Step 4: Generation process of end-to-end single UNet diffusion model.

[0055] An end-to-end single UNet architecture based on a variant of Stable Video Diffusion is adopted, avoiding the complexity of the classic ReferenceNet structure and the problem of temporal layers destroying spatial consistency. During training, the spatiotemporal pose features generated in step 2 and the fused pose features extracted in step 3 are injected into the intermediate and encoder layers of the UNet through a cross-attention mechanism and residual connections. During inference denoising, the model starts from pure Gaussian noise and, guided by the aforementioned multimodal geometric features, gradually recovers the resolution [resolution missing]. A video sequence of body movements with a length of 16 to 32 frames.

[0056] Step 5: Model optimization and validation.

[0057] Define a reconstruction loss function (such as MSE loss and LPIPS perceptual loss) between the generated video and the original reference video, and update the model parameters through the backpropagation algorithm.

[0058] To verify the effectiveness of the hand optimization module, 3D geometric prior, and adaptive gating module in this embodiment, detailed ablation experiments were conducted based on the TikTok dataset. Evaluation metrics included: FID-VID (Frechev Distance) and FVD (Frechev Distance) to measure the image quality and temporal coherence of the generated video; Hand-C (Hand Detection Confidence) to measure the anatomical plausibility of the hand structure in the generated video; and PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) to compare the pixel-level differences between the generated video and the reference video.

[0059] As shown in Table 1, Model A is the baseline model without optimization. After introducing HaMeR and HandNet (Model B), the Hand-C metric improved from 75.4 to 79.7, demonstrating the improvement in hand generation. Further introducing SMPler-X and the adaptive gating module (Model C, the complete method in this embodiment), the FVD significantly decreased from 505 to 476, and the SSIM improved to 0.864, demonstrating the improvement in overall temporal stability and structural integrity brought about by multi-condition adaptive fusion.

[0060] Table 1 Comparative experiments on the TikTok dataset

[0061] method FID-VID ↓ FVD ↓ Hand-C ↑ SSIM ↑ Model A (Baseline, DWPose only) 69.5 505 75.4 0.848 Model B (A+HaMeR&HandNet) 67.8 490 79.7 0.854 Model C(B+SMPL&Adaptive Gating) 65.8 476 79.9 0.864

[0062] Example 2:

[0063] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the multi-pose guided limb movement video generation method based on the diffusion model.

[0064] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the multi-pose guided limb movement video generation method based on the diffusion model as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0065] The processor is used to execute all or part of the steps in the diffusion model-based multi-pose guided limb motion video generation method described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0066] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the multi-pose guided limb movement video generation method based on the diffusion model described in the above embodiments.

[0067] Example 3:

[0068] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0069] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the diffusion model-based multi-pose guided limb movement video generation method described in the various embodiments of this application.

[0070] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory, etc.), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the diffusion model-based multi-pose guided limb movement video generation method described above.

[0071] Example 4:

[0072] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for generating multi-pose guided limb movement videos based on a diffusion model.

[0073] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the present invention.

Claims

1. A method for generating multi-pose guided limb motion video based on diffusion model, characterized in that: Includes the following steps: Step 1: Collect and preprocess the target limb movement video data, and build a distributed data processing pipeline to clean and label the video dataset; Step 2: Perform hand pose estimation based on the preprocessed video dataset, extract the 3D mesh and joint parameters of the hand, and generate spatiotemporal pose features as conditional representations through a lightweight hand pose guidance network; Step 3: Perform human pose estimation based on the preprocessed video dataset, introduce multi-condition 3D information input, and fuse multiple pose conditions through an adaptive pose gating module and a lightweight pose guidance network; Step 4: The conditional information obtained in Step 2 and Step 3 is injected into the end-to-end single UNet diffusion model through a cross-attention mechanism. During the denoising process, the model gradually recovers the video frame sequence from the noise and generates limb action videos guided by heterogeneous data. Step 5: Based on the reconstruction loss function of the generated video and the original video, backpropagate to optimize the model parameters, and then verify and evaluate the model.

2. The multi-pose guided limb motion video generation method based on a diffusion model according to claim 1, characterized in that: In step 1, a multi-machine distributed video processing pipeline based on the Ray framework is adopted. The pipeline integrates a variety of modular operators, including lens boundary detection operators, optical flow motion scoring operators, optical character recognition operators, video aesthetic quality assessment operators, and face and body pose detection operators. Through Ray's task scheduling and Actor model, cross-node parallel processing is achieved, and the video data is cleaned into high-quality samples. The ytdlp and wget tools are used to collect body movement video data from the network and integrate open source datasets to form a dedicated video dataset suitable for pose-driven body movement generation tasks.

3. The multi-pose guided limb motion video generation method based on a diffusion model according to claim 1, characterized in that: The specific method of step 2 is as follows: First, the hand region is detected and cropped in the input video slice, and the hand region is located using the bounding box detection algorithm; then the cropped hand image sequence is fed into the HaMeR model to predict the 3D mesh of the hand, the 24-dimensional joint rotation parameters and the corresponding MANO model shape parameters, and finally rendered into a gesture mesh image sequence as additional pose condition input. Subsequently, a lightweight hand pose guidance network is introduced to generate spatiotemporal pose features. This network first encodes the original hand pose heatmap using multi-layer convolutions, and then introduces a temporal convolution module. The temporal convolution module is configured to perform convolution operations on features of consecutive frames to fuse dynamic information across frames and obtain a stable and smooth motion trajectory representation. The extracted spatiotemporal pose features are mapped to a conditional representation aligned with the latent space of the diffusion model, which is used to guide the realistic and accurate generation of hands during video generation.

4. The multi-pose guided limb motion video generation method based on a diffusion model according to claim 3, characterized in that: The specific method for human pose estimation in step 3 is as follows: 3D parameters of the human body are estimated based on the SMPLer-X model, including joint position, rotation, and shape parameters; then, gradient color rendering is used to simultaneously fuse 3D, depth, and continuous semantic information, simplifying the injection of geometric information, and obtaining a gradient color rendering surface map sequence; the rendering process is represented as follows: the mesh output by the SMPLer-X model is surface-colored, where the color value C is calculated based on the depth D and semantic label S as C=f(D,S), where f is the gradient function, and the gradient color rendering surface map sequence is used as dense geometric prior input to the subsequent backbone network.

5. The method for generating multi-pose guided limb movement videos based on a diffusion model according to claim 4, characterized in that: In step 3, the specific method of fusing multiple posture conditions is: first, constructing a joint input feature; the two-dimensional key point heat map sequence extracted by the DWPose model and the gradient color rendering surface graph sequence rendered by the SMPLer-X model are spliced in the channel dimension to form a joint input with a shape of , wherein, represents a batch size, and respectively represent the channel numbers of two-dimensional and three-dimensional conditions, and respectively represent the height and width of the feature map; Subsequently, features are extracted using a backbone network. The joint input is fed into a multi-condition backbone network, which adopts a grouped convolutional structure. The number of convolutional groups is configured to be consistent with the number of modalities of the input conditions, so as to maintain the independence of different modal pose conditions in the channel dimension during downsampling, while sharing convolutional weights between groups to achieve preliminary feature extraction. After that, the feature fusion is performed by the adaptive pose gating module; first, the output channel of the backbone network is compressed and spatial-channel joint attention features are generated through two layers of convolution, and then the spatial-channel joint attention features are mapped into normalized gating weights through the Sigmoid activation function, the dimension of the gating weights is configured to contain two channels, which correspond to two-dimensional pose conditions and three-dimensional geometric conditions respectively, and the dimension of the gating weights is ; finally, the gating weights are multiplied with the two features after dimension reshaping element by element, and the results are added to obtain the fused features; the process is represented as: obtaining weighted two-dimensional features ; obtaining weighted three-dimensional features ; obtaining the final fused pose features ; wherein, and represent two-dimensional and three-dimensional features respectively, represent the gating weights; through the above mechanism, when the two-dimensional key points of a frame are seriously wrong due to occlusion or detection failure, the adaptive pose gating module automatically reduces the gating weight of the corresponding channel, so that the model focuses on the reliable three-dimensional geometric condition; on the contrary, when the three-dimensional grid fitting deviates, it automatically focuses on the two-dimensional key point features.

6. The method for generating multi-pose guided limb movement videos based on a diffusion model according to claim 5, characterized in that: In step 3, the lightweight pose guidance network is used to inject the fused pose features into the diffusion model. Its specific structure includes: multiple cascaded lightweight processing blocks, each block comprising two convolutional kernels of size [missing information]. The lightweight processing block includes convolutional layers, SiLU activation functions, and optional downsampling layers; the number of output channels of the lightweight processing block is configured to match the channel dimensions of each layer in the subsequently connected diffusion model backbone network; a zero-convolution module, located at the end of the lightweight pose guidance network, consists of modules with initialized weights and biases of zero. The convolutional layers are used to generate the final multi-scale control signals and ensure that the model maintains the pre-trained distribution characteristics in the early stages of training. The output features of each layer are multiplied by a learnable scaling factor before being injected into the backbone network to adjust the control strength of the pose conditions on the generation process.

7. The method for generating multi-pose guided limb movement videos based on a diffusion model according to claim 6, characterized in that: In step 4, the specific process of generating multi-pose conditional limb action videos using an end-to-end single UNet network architecture is as follows: During the training phase, Gaussian noise is added to the latent representation of the original video frames; the spatiotemporal pose features generated in step 2 and the fused pose features extracted in step 3 are used as conditional inputs and injected into the UNet network through a cross-attention mechanism; the UNet network predicts and removes the added noise; during the inference generation phase, starting from pure Gaussian noise, guided by the fused pose features, the UNet network performs multi-step iterative denoising to gradually recover a temporally coherent video frame sequence with actions aligned with conditions.

8. An electronic device, characterized in that: The method includes one or more processors and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the multi-pose guided limb movement video generation method based on the diffusion model as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: It stores executable instructions that, when executed, cause the processor to perform the multi-pose guided limb movement video generation method based on the diffusion model as described in any one of claims 1-7.

10. A computer program product, characterized in that: It includes a computer program or instructions that, when executed by a processor, implement the multi-pose guided limb movement video generation method based on the diffusion model as described in any one of claims 1-7.

Images