Method and system for generative video motion redrawing based on multi-modal pose control

Abstract

This invention relates to the field of video processing technology and discloses a generative video motion redrawing method and system based on multimodal pose control. The method involves: acquiring the original video frames of the input video and performing pose keypoint detection to obtain a pose keypoint sequence; mapping the pose keypoint sequence to a pose latent space to obtain a pose embedding vector, and extracting semantics from the text description to obtain a text embedding vector, while simultaneously extracting appearance features from a reference image to obtain a visual embedding vector; fusing these to obtain a multimodal control vector; performing multi-stage denoising processing on the multimodal control vector and a random noise image input diffusion model to obtain a generated video frame; and performing temporal constraint enhancement on the generated video frame to output a redrawn video frame. This invention improves the accuracy and robustness of pose keypoint extraction, effectively suppresses pixel abrupt changes and pose shifts between adjacent frames, and improves the temporal coherence and visual smoothness of the generated video.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a generative video motion redrawing method and system based on multimodal pose control. Background Technology

[0002] Currently, video motion redrawing and pose transfer technologies based on generative artificial intelligence mainly fall into three categories: 1. Traditional video redrawing and motion transfer technologies: These methods are mostly based on optical flow estimation, keyframe interpolation, and image fusion to redraw video content. They rely on accurate motion estimation results and are prone to artifacts, blurring, and misalignment in complex actions, occlusions, or multi-person scenes; 2. Video generation technologies based on generative adversarial networks (GANs): such as VideoGAN and StyleGAN-V, which improve the quality of single-frame generation through adversarial training, but have limited ability to model video temporal relationships and are prone to problems such as inter-frame flickering and style drift; 3. Video generation technologies based on diffusion models: such as Stable VideoDiffusion and WanVideo, which improve temporal consistency compared to GANs, but have insufficient ability to precisely control pose, especially in the synchronous transfer of multi-person actions and the control of complex limb movements.

[0003] The aforementioned existing technologies have the following technical problems in practical applications: 1. Insufficient pose transfer accuracy: In scenarios with multiple people or complex actions, existing methods have large pose alignment errors, and the joint positions of characters are prone to shift, resulting in low pose consistency; 2. Limited style and semantic control capabilities: It is difficult to simultaneously and finely control the character's pose, appearance style, and semantic content during the same generation process, resulting in low realism of the generated video; 3. Insufficient video temporal stability: Some methods do not introduce effective temporal consistency constraints, resulting in obvious jitter and flickering in the generated video; 4. Insufficient video temporal stability: Some methods do not introduce effective temporal consistency constraints, resulting in obvious jitter and flickering in the generated video. Summary of the Invention

[0004] The main objective of this invention is to provide a generative video motion redrawing method and system based on multimodal pose control. This invention improves the accuracy and robustness of pose key point extraction, effectively suppresses pixel abrupt changes and pose shifts between adjacent frames, and improves the temporal coherence and visual smoothness of the generated video.

[0005] To achieve the above objectives, this invention provides a generative video motion redrawing method based on multimodal pose control, comprising the following steps: The original video frames of the input video are acquired and pose key points are detected to obtain a pose key point sequence; The pose key point sequence is mapped to the pose latent space to obtain the pose embedding vector, and the semantics of the text description are extracted to obtain the text embedding vector. At the same time, the appearance features of the reference image are extracted to obtain the visual embedding vector. The pose embedding vector, the text embedding vector, and the visual embedding vector are fused to obtain a multimodal control vector; The multimodal control vector and the random noise image input diffusion model are subjected to multi-stage denoising processing to obtain generated video frames. The generated video frames are then subjected to temporal constraint enhancement to output redrawn video frames.

[0006] Optionally, in a first implementation of the first aspect of the present invention, obtaining the original video frames of the input video and performing pose key point detection to obtain a pose key point sequence includes: The original video frames of the input video are obtained, the resolution of the original video frames is normalized to obtain standard video frames, and the main body area of ​​the person and the background environment area in the standard video frames are separated to obtain the person area image. The image of the person's region is input into the OpenPose model to detect the positions of the main joints of the human body, and a first set of key points and a first confidence score are obtained; the image of the person's region is input into the HRNet model to detect high-precision key points including finger joints and toe joints, and a second set of key points and a second confidence score are obtained; the image of the person's region is input into the MediaPipe model to perform robust detection on occluded areas, and a third set of key points and a third confidence score are obtained. Based on the first confidence score, the second confidence score, and the third confidence score, a weighted average is performed on the first group of key points, the second group of key points, and the third group of key points to obtain the attitude key point sequence.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the pose keypoint sequence is mapped to the pose latent space to obtain a pose embedding vector, and semantic extraction is performed on the text description to obtain a text embedding vector. Simultaneously, appearance feature extraction is performed on the reference image to obtain a visual embedding vector, including: The pixel coordinates of the attitude key point sequence are normalized to obtain the normalized key point coordinates. The normalized keypoint coordinates are organized into structured attitude vectors, and the structured attitude vectors are linearly transformed and mapped to a preset dimension to obtain attitude embedding vectors. The text description is input into the BERT model for semantic extraction to obtain the text embedding vector, while the reference image is input into the ViT model for appearance feature extraction to obtain the visual embedding vector.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the text description is input into the BERT model for semantic extraction to obtain a text embedding vector, and the reference image is input into the ViT model for appearance feature extraction to obtain a visual embedding vector, including: The text description is segmented and converted into a word vector sequence. The word vector sequence is then input into the multi-layer Transformer encoder of the BERT model to capture contextual semantic relationships and extract text embedding vectors. The reference image is segmented into multiple image patch sequences and converted into image patch vectors. The image patch vectors are then input into the multi-layer Transformer encoder of the ViT model and the appearance style-related regions are weighted by the adaptive attention module to obtain the visual embedding vector.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the pose embedding vector, the text embedding vector, and the visual embedding vector are fused to obtain a multimodal control vector, including: The pose embedding vector, the text embedding vector, and the visual embedding vector are concatenated into a joint vector, and the pose weight coefficient, text weight coefficient, and visual weight coefficient are calculated based on the joint vector. Linear projection is performed on the joint vector to generate a query matrix, a key matrix, and a value matrix. An attention weight matrix is ​​calculated based on the query matrix and the key matrix. The attention weight matrix is ​​multiplied by the value matrix to obtain the attention weighted feature. The attention-weighted features are weighted and summed based on the posture weight coefficient, the text weight coefficient, and the visual weight coefficient, and then a multimodal control vector is obtained through linear transformation.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the pre-training process of the diffusion model includes: Collect a general video dataset covering different scenes and action types, as well as a multi-person action dataset, an occlusion scene dataset, and a stylized video dataset to obtain the training dataset; The training video frames of the training dataset are input into a conditionally controlled U-Net diffusion network. A diffusion training process of forward noise addition and backward noise reduction is adopted, and the pixel error between the predicted denoised frame and the original frame is calculated based on the MSE loss function for pre-training to obtain a pre-trained model. The parameters of the input encoding layer and the front downsampling block of the pre-trained model are frozen. Training video frames, pose key point annotations, text semantic labels and training reference images are input into the upper network. The model is fine-tuned based on a hybrid loss function of MSE pixel loss, pose consistency loss and semantic matching loss to obtain the diffusion model.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the multimodal control vector and the random noise image input diffusion model are subjected to multi-stage denoising processing to obtain a generated video frame, and the generated video frame is subjected to temporal constraint enhancement to output a redrawn video frame, including: The multimodal control vector is projected into a multi-channel feature map and concatenated with a random noise image. Then, it is input into the input coding layer of the diffusion model for convolution processing to obtain deep coding features. The multimodal control vector is mapped to a high-dimensional conditional vector and broadcast to expand it into a conditional feature map. The deep coding features are then multiplied element-wise at the bottleneck interaction layer of the diffusion model to obtain the modulated bottleneck features. The modulated bottleneck features are restored to feature map resolution through multiple upsampling blocks and combined with cross-layer connections. The multimodal control vector is then feature-decoded in the conditional fusion layer of the diffusion model to generate video frames. The feature maps of the generated video frames are extracted and temporal constraint enhancement is performed to output the redrawn video frames.

[0012] Optionally, in a seventh implementation of the first aspect of the present invention, the multimodal control vector is mapped to a high-dimensional conditional vector and broadcast to expand it into a conditional feature map. The deep encoded features are then multiplied element-wise at the bottleneck interaction layer of the diffusion model to obtain the modulated bottleneck features, including: The multimodal control vector is mapped to a high-dimensional conditional vector, and the high-dimensional conditional vector is copied and extended in the spatial dimension to the same spatial size and number of channels as the deep coding features, so as to obtain a conditional feature map with the same feature value at each spatial location. The conditional feature map and the deep coding feature are multiplied element-wise at their corresponding spatial locations and channel dimensions to obtain the modulated bottleneck feature.

[0013] Optionally, in the eighth implementation of the first aspect of the present invention, extracting the feature map of the generated video frame and performing temporal constraint enhancement to output the redrawn video frame includes: Extract the feature map of the generated video frame and calculate the inter-frame attention weighted feature. Then, fuse the feature map of the generated video frame with the inter-frame attention weighted feature to obtain the fused feature map. An attitude constraint vector is generated based on the attitude key point sequence. The attitude constraint vector is then expanded into a constraint feature map and multiplied element-wise with the fused feature map to output a redrawn video frame.

[0014] This invention also provides a generative video motion redrawing system based on multimodal pose control, comprising: The key point detection module is used to acquire the original video frames of the input video and perform pose key point detection to obtain a pose key point sequence. The mapping module is used to map the sequence of pose key points to the pose latent space to obtain the pose embedding vector, and to perform semantic extraction on the text description to obtain the text embedding vector. At the same time, it performs appearance feature extraction on the reference image to obtain the visual embedding vector. The fusion module is used to fuse the pose embedding vector, the text embedding vector, and the visual embedding vector to obtain a multimodal control vector; The output module is used to perform multi-stage denoising processing on the multimodal control vector and the random noise image input diffusion model to obtain the generated video frame, and to perform temporal constraint enhancement on the generated video frame to output the redrawn video frame.

[0015] In summary, this invention improves the accuracy and robustness of pose keypoint extraction by using three models—OpenPose, HRNet, and MediaPipe—in parallel detection and confidence-weighted fusion, particularly enhancing its adaptability in complex scenarios such as occlusion and multi-person scenarios. By unifying pose latent space encoding, BERT textual semantic features, and ViT visual reference features to the same dimensional space, and employing an adaptive weight prediction network to dynamically calculate the three-modal fusion coefficients combined with a Transformer attention mechanism to calculate intermodal correlations, it achieves coordinated and fine-grained control over pose structure, semantic content, and appearance style, ensuring that the generated results simultaneously meet multiple constraints of accurate action, semantic matching, and consistent appearance. By incorporating multimodal control vectors three times into the input encoding layer, bottleneck interaction layer, and conditional fusion layer of the conditionally controlled U-Net diffusion network in the diffusion model, the strength of conditional constraints during the generation process is continuously strengthened, avoiding the attenuation of control information during denoising. By constructing an inter-frame attention mechanism and combining it with element-wise modulation of pose trajectory smoothing constraint vectors, it effectively suppresses pixel abrupt changes and pose shifts between adjacent frames, improving the temporal coherence and visual smoothness of the generated video. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the steps of a generative video motion redrawing method based on multimodal pose control in one embodiment of the present invention; Figure 2 This is a system overall flowchart of the generative video motion redrawing method based on multimodal pose control in an embodiment of the present invention. Figure 3 This is a detailed technical implementation flowchart of the generative video motion redrawing method based on multimodal attitude control in this embodiment of the invention; Figure 4 This is a flowchart illustrating the technical architecture of the multimodal control fusion structure in this embodiment of the invention. Figure 5 This is a flowchart illustrating the video timing consistency enhancement mechanism in an embodiment of the present invention. Figure 6 This is a block diagram of the generative video motion redrawing system based on multimodal attitude control in an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Reference Figure 1 This embodiment provides a generative video motion redrawing method based on multimodal pose control, including the following steps: S1, acquire the original video frames of the input video and perform pose key point detection to obtain the pose key point sequence; S2, map the pose key point sequence to the pose latent space to obtain the pose embedding vector, extract the semantics of the text description to obtain the text embedding vector, and extract the appearance features of the reference image to obtain the visual embedding vector. S3, fuse the pose embedding vector, text embedding vector and visual embedding vector to obtain the multimodal control vector; S4 performs multi-stage denoising processing on the multimodal control vector and the random noise image input diffusion model to obtain the generated video frame, and performs temporal constraint enhancement on the generated video frame to output the redrawn video frame.

[0020] Figure 2This is a system flowchart of a generative video motion redrawing method based on multimodal pose control. Starting with the input video, the original video is standardized by a video preprocessing module. The preprocessed video data is divided into two parallel processing branches: one for extracting pose keypoints and coordinates, extracting keypoint information of the human skeleton using a multi-source pose detection model; and the other for generating semantic control and visual reference control, extracting textual semantic features and visual appearance features. The outputs of both branches are fed into a pose latent space encoding module, mapping the keypoint coordinates to a latent space representation. The encoded pose features are further processed in the pose latent space encoding module and then fed into a multimodal control fusion module to fuse pose, textual semantics, and visual reference control information. The fused multimodal control vectors are input to two parallel modules: a diffusion model generation module generates video frame content, and a temporal stability module ensures the continuity between frames. The outputs of both are fed into the diffusion model generation module for final video frame generation. The generated results are optimized by an alignment consistency enhancement module, and the final redrawn video is output.

[0021] In one example, the original video frames of the input video are obtained and pose keypoint detection is performed to obtain a pose keypoint sequence, including: The original video frames of the input video are obtained, the resolution of the original video frames is normalized to obtain standard video frames, and the main body area of ​​the person in the standard video frames is separated from the background environment area to obtain the person area image. The image of the human figure is input into the OpenPose model to detect the positions of the main joints of the human body, and the first set of key points and the first confidence score are obtained. The image of the human figure is input into the HRNet model to detect high-precision key points including finger joints and toe joints, and the second set of key points and the second confidence score are obtained. The image of the human figure is input into the MediaPipe model to perform robust detection on occluded areas, and the third set of key points and the third confidence score are obtained. The first, second, and third sets of keypoints are weighted and averaged based on the first, second, and third confidence scores to obtain the attitude keypoint sequence.

[0022] In this example, frame-by-frame parsing is performed on the input video, converting the original continuous video stream into frame-level image data, and each frame is resized uniformly. All video frames are scaled or cropped to a preset standard resolution, such as 640×480 pixels, to eliminate the uneven feature response caused by differences in the original resolution. Based on a semantic segmentation model or a person detection model, the video frames are divided into regions representing the main subject and the background environment. The person mask image is extracted and applied to the original video frames, and non-person regions are cropped or masked to obtain a foreground image containing only the person's body parts, which serves as the input image for pose recognition. The person region image is then input into three pose detection models with complementary characteristics to achieve joint extraction of multi-source features. The image of the person is input into the OpenPose model to obtain the main skeletal structure of the human body. Seventeen major joints, including the head, neck, shoulders, elbows, wrists, hips, knees, and ankles, are extracted, and the two-dimensional pixel coordinates and corresponding first confidence score of each keypoint are recorded. The image of the person is then input into the HRNet model to extract more fine-grained joint information. The HRNet model expands upon the basic keypoint set of OpenPose to 25 or more high-precision keypoints, specifically including the tips of the thumb, index, middle, and little fingers, and the big and little toes, to enhance the descriptive ability in high-motion-complexity scenes. A corresponding second confidence score is output to measure the model's recognition reliability in detailed regions. Supplementary detection of the person image is performed based on the MediaPipe model. For poor imaging conditions such as multiple people occluding, limb crossing, and image blurring, MediaPipe further enhances the robustness of pose recognition through lightweight skeleton modeling and occlusion recovery strategies, outputting a third set of keypoint locations and a third confidence score. For keypoints of the same category (common joints such as left shoulder and right knee, output by all three models), a confidence-weighted fusion operation is performed. For each category of keypoint, its coordinates and confidence values ​​in the three models are extracted. Let the coordinates output by the first model be P1 and the confidence value be C1, the coordinates output by the second model be P2 and the confidence value be C2, and the coordinates output by the third model be P3 and the confidence value be C3. Then the fused keypoint coordinates P1, P2, and C3 are... 融 The calculation method is: P 融 = (C1×P1+ C2×P2+ C3×P3) ÷ (C1+ C2+C3). Confidence is determined based on the model's output response value, the degree of occlusion of keypoints in the image, and the continuity of keypoints in adjacent frames. For example, keypoints without occlusion and with response values ​​higher than the threshold (default 0.7) have a confidence score of 0.8-1.0, while keypoints in occluded areas have a confidence score below 0.3. Multi-source results are weighted and fused. For multi-source detection results of the same type of keypoints in the same video frame, a "confidence-weighted average" strategy is used to calculate the final coordinates. The formula is: ,in , Output the confidence score of the keypoint for the i-th model. Here, n represents the keypoint coordinates output by the i-th model, and n is the number of models participating in the fusion (default n≥3). Low-confidence keypoint completion is performed. For keypoints with a confidence level below a set threshold (default 0.5) or missing keypoints, the pose information from adjacent frames (the previous 3 frames and the next 3 frames) is used to complete the keypoints through linear interpolation and skeletal structure constraints. For example, if the "wrist" keypoint is missing in a frame, the reasonable coordinates for completing the wrist in the current frame are calculated based on the motion trajectory trend of the wrist coordinates in adjacent frames, combined with the length ratio constraint of the arm bones. When a model fails to detect a keypoint, its confidence level is considered 0, and it does not contribute to the weighted fusion; the fusion result is biased towards the output of the model with higher confidence. The weighted fusion process is repeated for all joint types to obtain the pose keypoint sequence.

[0023] Figure 3 This is a detailed flowchart of the technical implementation of a generative video motion redrawing method based on multimodal pose control. Starting with the input video, video sequence extraction and segmentation are performed, decomposing the continuous video into independent frame sequences. The segmented video frames enter three parallel processing pathways: the first pathway performs image character segmentation, scene separation, and background extraction, separating the subject from the background environment; the second pathway performs pose keypoint extraction and multi-source fusion alignment, extracting and accurately aligning the character's skeletal keypoints through the collaborative work of multiple pose detection models such as OpenPose, HRNet, and MediaPipe; the third pathway performs semantic encoding generation, appearance / style feature extraction, and label vector generation, extracting high-level semantic information and appearance / style features from the video. The outputs of the three pathways are fed into the pose keypoint encoding module, converting discrete keypoint coordinates into continuous feature representations. The encoding results enter the pose latent space encoding module, mapping to a unified latent space dimension. The latent space representation is subsequently divided into three parallel control paths: first-order pose embedding control provides pose structure constraints, semantic feature control embedding provides textual semantic constraints, and multi-keyframe appearance style condition extraction provides visual appearance constraints. These three control signals are fed into a cross-modal fusion generation module to complete the deep fusion of multimodal information. The fused control vector simultaneously drives three parallel processing modules: a diffusion-based noise removal and reconstruction module executes the denoising generation process of the diffusion model; a temporal attention mechanism and inter-frame continuity constraint module ensure continuity between adjacent frames; and an adaptive inter-frame scheduling generation, hierarchical stability enhancement, and temporal jitter suppression module further optimize the temporal stability of the video.

[0024] In one example, the pose keypoint sequence is mapped to the pose latent space to obtain a pose embedding vector, and semantic extraction is performed on the text description to obtain a text embedding vector. Simultaneously, appearance features are extracted from the reference image to obtain a visual embedding vector, including: The pixel coordinates of the pose keypoint sequence are normalized to obtain the normalized keypoint coordinates. The normalized keypoint coordinates are organized into structured attitude vectors, and the structured attitude vectors are linearly transformed and mapped to a preset dimension to obtain attitude embedding vectors. The text description is input into the BERT model for semantic extraction to obtain the text embedding vector, while the reference image is input into the ViT model for appearance feature extraction to obtain the visual embedding vector.

[0025] In this example, image normalization is performed on the sequence of human pose keypoints. The pose keypoint sequence consists of several two-dimensional pixel coordinates, representing the location of each joint in the image space within the video frame. To eliminate the influence of different resolutions and human body shapes on the modeling process, the pixel coordinates of all keypoints are normalized by dividing the horizontal coordinate of each keypoint by the width of the input image and the vertical coordinate by the image height, compressing all coordinate values ​​into a normalized coordinate system within the range of [0,1]. The normalized coordinates of all keypoints are then organized into a structured pose vector. If n keypoints are detected per frame, each keypoint contains two coordinate components x... i With y i Then all key points are arranged sequentially into a one-dimensional vector of length 2n, i.e., P = [x1, y1, x2, y2, ..., x n , y nThe structured pose vector preserves the geometric topological order of the joints and has a standard input form that can be processed by neural networks. The structured vector is input into the fully connected neural network embedding module, where it is mapped to an intermediate dimension (e.g., 256-dimensional) through a linear transformation. Then, a non-linear activation function (e.g., GELU) is used to enhance its representational power. Following this, normalization (e.g., LayerNorm) is applied to suppress feature shift, and another linear transformation is used to compress or expand it to a unified preset dimension (e.g., 512-dimensional) to obtain the pose embedding vector. Simultaneously, the user-input text description is used as a second modality control source input into the natural language understanding model for semantic extraction. A pre-trained BERT language model is used to perform word segmentation on the original text sentence, forming a sequence of words. Each word is then mapped to a 768-dimensional word vector through a word embedding layer and input into a 12-layer Transformer encoder to model the contextual semantic dependencies between words layer by layer. The 768-dimensional global semantic vector corresponding to the [CLS] position is extracted from the BERT output; this vector represents a condensed semantic expression of the entire text sentence. To ensure dimensionality consistency with the pose embedding vector in subsequent fusion processes, the 768-dimensional text vector is linearly projected to 512 dimensions, forming a text embedding vector. To achieve controllability of appearance style, a user-provided reference image is input into the Transformer-based visual feature extraction model ViT. The input image is resized to a standard size (e.g., 224×224) and divided into fixed-size image blocks (e.g., 16×16). Each image block is linearly transformed to generate a 768-dimensional image block vector. All image block vector sequences are positionally encoded and then input into a multi-layer Transformer encoder to model the structural dependencies between image blocks and extract the overall image representation. An attention module is introduced into the output to enhance focus on appearance style-related regions in the image, such as clothing texture, hairstyle, and skin tone, while suppressing background interference. A 768-dimensional global visual feature representation is generated through weighted pooling, then compressed to 512 dimensions through a linear transformation to form a visual embedding vector, which serves as the input for the style control modality.

[0026] In one example, the text description is input into the BERT model for semantic extraction to obtain a text embedding vector, while the reference image is input into the ViT model for appearance feature extraction to obtain a visual embedding vector, including: The text description is segmented and converted into a word vector sequence. The word vector sequence is then input into the multi-layer Transformer encoder of the BERT model to capture the contextual semantic relationship and extract the text embedding vector. The reference image is segmented into multiple image patch sequences and converted into image patch vectors. The image patch vectors are then input into the multi-layer Transformer encoder of the ViT model and the appearance style-related regions are weighted by the adaptive attention module to obtain the visual embedding vector.

[0027] In this example, for the input natural language text description, a standard word segmenter is used to split the complete sentence into semantic units, converting the text into a sequence of tokens, each token representing a smallest recognizable semantic unit. The token sequence is then fed into a word embedding layer, mapping each token to a dense vector of fixed dimensions. 768 dimensions are chosen as the base dimension for each word vector, making the entire sentence a two-dimensional word vector matrix. The word vector sequence is then fed into the multi-layer Transformer encoder of the BERT model. The encoder consists of 12 stacked Transformer units, each layer containing a multi-head self-attention mechanism and a feedforward neural network structure. This mechanism captures the semantic relevance between the word and all other words at each position and aggregates its contextual information according to attention weights, iteratively strengthening the contextual association of each word's representation. Finally, the output vector at the BERT-specific [CLS] position is selected from all the output word representations as the overall semantic representation of the text sequence. Simultaneously, to extract appearance style information from the reference image, the input image is standardized to a fixed resolution (e.g., 224×224 pixels) and regularly divided into several 16×16 image blocks. Each image block is sequentially expanded and converted into a fixed-dimensional image block vector through linear mapping, set to 768 dimensions, forming an image block vector sequence. This image block vector sequence is added to the positional encoding and then input into the multi-layer Transformer encoder in the ViT model. Similar to the BERT structure, the Transformer layer includes a multi-head attention mechanism and a feedforward sub-layer, enabling it to model long-distance structural dependencies between image blocks and capture global features such as character pose layout, clothing texture, and boundary morphology. Based on this, an adaptive attention module is introduced to perform region-weighted processing on the image block vectors. By calculating the similarity score between each image block and appearance-related semantics, the response to regions containing character appearance elements (such as clothing, hairstyle, and skin tone) is enhanced, while the attention weight of background blocks is suppressed, completing the feature redistribution based on task requirements. By integrating all image patch vectors through attention-weighted pooling, a visual embedding vector representing the entire image is output, which has the ability to express style and identify the appearance of the target.

[0028] In one example, the pose embedding vector, text embedding vector, and visual embedding vector are fused to obtain a multimodal control vector, including: The pose embedding vector, text embedding vector, and visual embedding vector are concatenated into a joint vector, and the pose weight coefficient, text weight coefficient, and visual weight coefficient are calculated based on the joint vector. Linear projection is performed on the joint vector to generate a query matrix, a key matrix, and a value matrix. An attention weight matrix is ​​calculated based on the query matrix and the key matrix. The attention weight matrix is ​​then multiplied by the value matrix to obtain the attention-weighted features. The attention-weighted features are weighted and summed based on the pose weight coefficient, text weight coefficient, and visual weight coefficient, and then a multimodal control vector is obtained through linear transformation.

[0029] In this example, embedding vectors from different modalities are concatenated at a dimensionality level. The pose embedding vector output by the pose latent space encoding network, the text embedding vector extracted by the BERT model, and the visual embedding vector extracted by the ViT model are concatenated to form a joint feature vector of dimension 1536, where the first 512 dimensions are the pose vector, the middle 512 dimensions are the text vector, and the last 512 dimensions are the visual vector. This joint vector is then input into a weight prediction sub-network, which consists of a multilayer perceptron. A linear transformation compresses the 1536-dimensional vector into a 256-dimensional intermediate feature representation. This is then processed by the ReLU activation function and mapped back to 64 dimensions. Finally, a linear output layer generates three scalar outputs, corresponding to the initial weight logits values ​​for the pose, text, and visual modalities. Softmax normalization is performed on these three logits to obtain the pose weight coefficient, text weight coefficient, and visual weight coefficient, all of which sum to 1, dynamically reflecting the preference distribution of the input task among action control, semantic guidance, and style constraints. The 1536-dimensional joint vector is input into three sets of parameter-independent linear mapping layers to construct the three core matrices required for the Transformer-style attention module: query matrix Q, key matrix K, and value matrix V. Each mapping layer transforms the 1536-dimensional vector into a 512-dimensional space, making Q, K, and V all 3×512 matrices, with each row corresponding to the pose, text, and visual modal representations, respectively. The inner product of Q and K is calculated and divided by the square root of 512 as a scale normalization factor to obtain a 3×3 attention relevance matrix, where each element represents the semantic similarity between the query modality and the key modality. Row-wise softmax processing is applied to the similarity matrix to output the attention weight matrix, ensuring that when each modality is used as a query, the sum of its attention scores with all other modalities equals 1. Matrix multiplication (i.e., weighted summation) is performed on the attention weight matrix and the value matrix V to generate an attention-weighted modality feature matrix, still 3×512 in dimension, with each row representing the fused representation of a modality reconstructed under the guidance of attention from other modalities. The pose weight coefficient, text weight coefficient, and visual weight coefficient are multiplied by the corresponding three rows of vectors in the attention weighted feature matrix to construct a weighted fusion representation. This involves multiplying each row of attention features by its weight and then summing the results to generate a fused 512-dimensional multimodal weighted vector. This fused vector is then input into a fully connected linear transformation module to construct a 512×512 weight matrix W. This matrix is ​​used to perform a linear mapping operation on the fused vector, calculated as Y = W·X + b, where X is the 512-dimensional multimodal fusion vector obtained by weighted summation, W is the mapping weight, and b is the bias term.The linear mapping operation projects and adjusts the feature vectors after mixing different modes while maintaining consistency in input and output dimensions, making their feature distribution more closely match the input space of the downstream generation module. A nonlinear activation function is applied to Y, based on GELU (Gaussian Error Linear Unit) as the activation function, and a nonlinear transformation is performed element-wise on each output dimension. The output after linear mapping and GELU activation is the 512-dimensional multimodal control vector.

[0030] Figure 4 This describes the technical architecture of the multimodal control fusion structure. Starting from the multimodal control fusion structure module, it is then divided into three parallel embedding input paths: a pose embedding input module providing a 512-dimensional pose feature vector encoded from the pose latent space; an appearance reference embedding input module providing a 512-dimensional visual appearance feature vector extracted by the ViT model; and a semantic embedding input module providing a 512-dimensional textual semantic feature vector extracted by the BERT model. These three embedding vectors are input in parallel and then fed into the multimodal fusion encoding module, which is responsible for integrating control information from different modalities. The fusion encoding module calculates pose weight coefficients, text weight coefficients, and visual weight coefficients through a weight prediction network, and performs weighted fusion by calculating intermodal correlations through a Transformer attention mechanism. The preliminary fusion result enters three parallel validation processing branches: the left branch performs constraint validation of the person embedded by each model, evaluating the information retention of the fused vector and the pose embedding; the middle branch performs transitive interval weighted validation, checking whether the weight coefficients conform to the preset dynamic adjustment rules; and the right branch performs validation based on continuous external constraints, ensuring that the fusion result meets appearance style and semantic matching constraints. The outputs of the three verification branches are aggregated into the conditional control vector output module. The cosine similarity verification mechanism ensures that the fusion result simultaneously satisfies the posture structure constraint, semantic matching constraint, and appearance style constraint. If any constraint is not satisfied, the weight coefficients are adjusted in reverse and the fusion is repeated to finally generate a 512-dimensional multimodal control vector that passes all verifications.

[0031] In one example, the pre-training process of the diffusion model includes: Collect a general video dataset covering different scenes and action types, as well as a multi-person action dataset, an occlusion scene dataset, and a stylized video dataset to obtain the training dataset; The training video frames of the training dataset are input into the conditionally controlled U-Net diffusion network. The diffusion training process of forward noise addition and backward noise reduction is adopted, and the pixel error between the predicted denoised frame and the original frame is calculated based on the MSE loss function to pre-train the model and obtain the pre-trained model. The parameters of the input encoding layer and the front downsampling block of the pre-trained model are frozen. Training video frames, pose key point annotations, text semantic labels and training reference images are input into the upper network. The model is fine-tuned based on a hybrid loss function of MSE pixel loss, pose consistency loss and semantic matching loss to obtain the diffusion model.

[0032] In this example, a high-quality training dataset covering diverse scenes, action types, and appearance styles is constructed. This dataset comprises four complementary subsets: a general video dataset containing rich content including indoor and outdoor scenes, static and dynamic scenes, and human and non-human elements, covering typical human actions such as jumping, running, walking, and waving, used to support the training of the diffusion model's basic generation capabilities; a multi-person action dataset, focusing on complex scenes involving multi-target synchronous motion, limb interactions, and frequent occlusion events, ensuring the model's recognition and generation capabilities in multi-person posture control tasks; an occlusion scene dataset, specifically collecting or synthesizing video frames where foreground figures and occluders partially overlap, to improve the model's stable modeling ability in low-visibility areas; and a stylized video dataset encompassing visual materials with different aesthetic distributions, such as cartoon, oil painting, and realistic styles, enabling the model to handle visual style transfer and restoration tasks during generation. After merging these four subsets to form a complete training dataset, all video data is standardized, and equal-length video segments are extracted and stored frame by frame. Each segment contains 16 frames, with a uniform frame resolution of 512×512 pixels to adapt to the input requirements of the diffusion network. In the model pre-training phase, video frame sequences are input into the U-Net diffusion model, which possesses conditional control capabilities, and a standard forward denoising and backward denoising process is executed. The forward process progressively adds normally distributed random noise to real video frames, forming a noise perturbation sequence, allowing the model to learn the image generation trajectory. The backward process starts with the fully denoised image and progressively predicts the denoised image, enabling the model to restore image details at each step based on historical generation states. During training, mean squared error is used as the primary loss function. The pixel difference between each predicted denoised frame and the original real frame is measured pixel-by-pixel, and the total loss value is accumulated and used for backpropagation to update the learnable parameters of the diffusion model. This stage primarily trains the model's general image restoration and temporal modeling capabilities, without introducing pose, text, or visual control conditions. After training, the output model is the pre-trained model, possessing strong unconditional video generation capabilities. Conditional fine-tuning training is then performed based on the pre-trained model. During the fine-tuning phase, the weight parameters of the input encoding layer and the first two downsampling blocks of the pre-trained model are frozen to maintain its learned low-level feature extraction capability. Only the parameters of the bottleneck layer, upsampling decoding layer, conditional fusion layer, and control injection structure are updated. The structured pose keypoint sequence, natural language text semantic labels, and target person reference image corresponding to each frame in the training dataset are combined with the current training video frame to form a quadruple input into the model as joint training samples.In the loss function design, a hybrid loss function with three components is introduced: MSE pixel loss, which measures the pixel-level difference between the generated image and the target frame in overall visual reconstruction; pose consistency loss, which calculates the Euclidean distance deviation between the skeletal structure of the person in the generated frame and the labeled pose key points at each joint position to improve the consistency between the generated action and the target action; and semantic matching loss, which uses a pre-trained CLIP model to project the generated frame and text labels into the same semantic embedding space and calculates the cosine distance between their embedding vectors to measure image-text consistency. The weighted sum of the three components serves as the total loss function, controlling the dynamic balance between image fidelity, action accuracy, and semantic expression during training. During training, the validation set loss is continuously monitored, and the model version with the best validation metrics is saved as the final output, resulting in a conditionally controlled diffusion model with stable control over response and visual generation capabilities.

[0033] I. Model Architecture and Hierarchical Parameter Settings of the Diffusion Model: The architecture adopts a "conditionally controlled U-Net diffusion architecture", with core layers including an input coding layer, a multi-scale downsampling block, a bottleneck interaction layer, a multi-scale upsampling block, a conditional fusion layer, and an output decoding layer.

[0034] 1. The input coding layer receives a 512-dimensional multimodal control vector and a random noise image (the initial noise follows an N(0,1) distribution), embedding the control vector into the image feature space. The structure consists of one convolutional layer (3×3 kernel size, stride 1, padding=1) + a normalization layer (GroupNorm, 8 groups) + a GELU activation function; the number of input channels is 3 (image RGB) + 512 (channels after control vector projection) = 515, the number of output channels is 64, the number of convolutional kernels is 64, and the total number of parameters is approximately 64×(515×3×3 +1)≈292,000.

[0035] 2. Four multi-scale downsampling blocks. Their function is to progressively reduce the feature map resolution and extract high-level semantic and pose features. Each block contains two residual convolutional units (ResBlock) + one downsampling convolution (stride 2). Residual convolutional unit parameters: input channels are 64, 128, 256, and 512 respectively; output channels are 128, 256, 512, and 1024 respectively. Each ResBlock contains two 3×3 convolutional layers with a CBAM attention module inserted in between. Total parameters: Down1≈128×(64×3×3 +1)×2≈1.475 million, Down2≈256×(128×3×3 +1)×2≈5.901 million, Down3≈512×(256×3×3 +1)×2≈23.613 million, Down4≈1024×(512×3×3 +1)×2≈94.464 million.

[0036] 3. Bottleneck Interaction Layer. This layer integrates multimodal control information with deep features to enhance the constraints of pose and semantics. It consists of 2 residual convolutional units and 1 conditional interaction module (projecting the multimodal control vector to 1024 dimensions through a fully connected layer and multiplying it element-wise with the bottleneck features). The number of input / output channels is 1024. The parameters of the fully connected layer of the conditional interaction module are approximately 1024×1024 + 1024≈10.496 million. The overall bottleneck layer parameters are approximately 2×(1024×3×3 + 1)×1024 + 10.496 million≈19.968 million.

[0037] 4. Multi-scale upsampling blocks (4, denoted as Up1~Up4). The feature map resolution is gradually restored, mapping deep features to the pixel level. Each block contains one upsampled convolution (transposed convolution, stride 2) + two residual convolutional units (ResBlock) + cross-layer connection (concatenated with the output features of the corresponding downsampled block). The number of input channels are 1024, 512, 256, and 128, respectively. After upsampling, the number of channels are 512, 256, 128, and 64, respectively. The number of channels doubles after cross-layer concatenation. The total number of parameters is: Up1≈512×(1024×3×3 +1)×2≈94.464 million, Up2≈256×(512×3×3 +1)×2≈23.613 million, Up3≈128×(256×3×3 +1)×2≈5.901 million, Up4≈64×(128×3×3 +1)×2≈1.475 million.

[0038] 5. Conditional Fusion Layer. This layer incorporates multimodal control information a second time to correct pose and style deviations during the generation process; it consists of one convolutional layer (3×3) + a conditional attention module (which performs attention matching between the multimodal control vectors and the feature maps); 64 input channels, 32 output channels, convolutional kernel parameters approximately 32×(64×3×3 +1)≈184,000, conditional attention module parameters approximately 32×512 +512≈169,000, and total parameters approximately 353,000.

[0039] 6. Output Decoding Layer. Outputs the denoised video frame feature map to restore the RGB image; 1 convolutional layer (3×3, stride 1, padding=1) + Sigmoid activation function; 32 input channels, 3 output channels (RGB), convolutional kernel parameters 3×(32×3×3 +1)≈867, total parameters ≈867.

[0040] II. Pre-training requirements and pre-training process: This diffusion model requires pre-training. The pre-training process revolves around "general video generation pre-training + pose control fine-tuning," with the following specific steps: 1. Construction of pre-trained datasets. General dataset: Collect video datasets covering different scenes (indoor and outdoor), action types (dance, sports, daily behavior), and human characteristics (different ages, genders, clothing), with a total duration of ≥100 hours. Extract frame sequences and label pose key points and semantic labels. Specialized datasets: Construct multi-person action datasets (≥50 hours), occlusion scene datasets (≥30 hours), and stylized video datasets (≥20 hours, including cartoon, realistic, oil painting, and other styles) to enhance the generation capabilities for special scenes.

[0041] 2. Pre-training phase. The basic video generation capability and temporal consistency of the training model are maintained, without introducing pose control constraints; video frame sequences are input into the model in 16-frame segments, using the standard training process of the diffusion model (forward noise addition - backward noise reduction), with the loss function being MSE loss (pixel error between the predicted denoised frame and the original frame); batch size = 16, learning rate = 1e-4, AdamW optimizer is used, training epochs = 200, validation is performed every 20 epochs, and the model weights with the lowest loss on the validation set are saved; training hardware: 8 A100 GPUs, training time approximately 15 days.

[0042] 3. In the fine-tuning stage, the pre-trained model is adapted to the requirements of pose control, text semantic control, and visual appearance control; training process: The parameters of the model's bottom layer (input encoding layer, first two downsampling blocks) were frozen, and only the upper layers (last two downsampling blocks, bottleneck layer, upsampling block, conditional fusion layer) were trained. The input data consisted of a quadruple of "original video frame + pose keypoint annotation + text semantic label + visual reference image". The loss function used was a hybrid loss (MSE pixel loss + pose consistency loss + semantic matching loss). The pose consistency loss calculated the distance error between the keypoints of the generated frame and the labeled pose, and the semantic matching loss used the CLIP model to calculate the semantic similarity error between the generated frame and the text label. The training parameters were batch size = 8, learning rate = 5e-5, training epochs = 100, and validation was performed every 10 epochs, saving the model weights with the lowest overall loss. The training hardware consisted of 4 A100 GPUs, and the training time was approximately 7 days.

[0043] 4. Model Optimization and Deployment. The fine-tuned model is quantized (INT8 quantization) and pruned (redundant parameters are removed, pruning rate ≤30%) to ensure that the inference efficiency meets the requirements of practical applications.

[0044] In one example, a multimodal control vector is input to a random noise image diffusion model for multi-stage denoising to generate video frames. Temporal constraint enhancement is then applied to these generated video frames to output redrawn video frames, including: The multimodal control vector is projected into a multi-channel feature map and concatenated with a random noise image. Then, it is input into the input coding layer of the diffusion model for convolution processing to obtain deep coding features. The multimodal control vector is mapped to a high-dimensional conditional vector and broadcast to expand it into a conditional feature map. The conditional feature map is then multiplied element-wise with the deep encoded features at the bottleneck interaction layer of the diffusion model to obtain the modulated bottleneck features. The modulated bottleneck features are restored to feature map resolution through multiple upsampling blocks and combined with cross-layer connections. The multimodal control vectors are then decoded in the conditional fusion layer of the diffusion model to obtain the generated video frames. Extract feature maps from the generated video frames and perform temporal constraint enhancement to output redrawn video frames.

[0045] In this example, the multimodal control vector is used as a unified control signal input. At the initial stage of the model's encoding, i.e., at the input encoding layer, a linear projection operation is performed on the multimodal control vector, mapping it from a one-dimensional vector to a multi-channel conditional feature map with a specified number of channels (e.g., 512 channels) and spatial size (e.g., 512×512 pixels). The multi-channel conditional feature map is concatenated with the sampled initial random noise image along the channel dimension to form joint input features. These features are then processed by the convolutional module in the input encoding layer. The convolutional module contains multiple convolutional layers, normalization layers, and nonlinear activation functions, performing low-level semantic extraction and preliminary feature encoding on the concatenated input. The output deep encoded features serve as the initial representation of the downsampling path. In the deep bottleneck interaction layer of the diffusion model, to introduce multimodal control conditions to modulate the global semantic expression, the original control vector is mapped to a high-dimensional conditional vector through a fully connected layer. This high-dimensional vector is then expanded spatially into a conditional feature map with the same size as the bottleneck feature map through a broadcast operation. The conditional feature map and the deep encoded feature map generated by convolutional stacking of the downsampling path are multiplied element-wise to achieve a feature modulation mechanism based on conditional vectors. This ensures that the semantic representation output by the bottleneck layer is constrained by control signals globally, achieving consistent injection of style, action, and semantics. The modulated bottleneck feature map enters the upsampling decoding path of the diffusion model. The model restores the spatial resolution of the feature map step by step through multiple upsampling blocks. Each upsampling block includes transposed convolution or nearest neighbor upsampling operations, residual convolution structures, and channel attention mechanisms to finely restore image details. During the upsampling process, the model uses a cross-layer connection mechanism to concatenate or add the low-level features in the corresponding encoding path with the features of the current decoding layer, enhancing the accuracy of feature reconstruction and preventing information loss during upsampling. To improve the control consistency of the generated images, a multimodal control vector is introduced again in the final conditional fusion layer. This multimodal control vector is compressed into a vector adapted to the number of channels in the current feature map through linear transformation and then spatially broadcast. An attention mechanism is used to modulate the feature map at the end of decoding, allowing the control information to further correct deviations in action structure and appearance style before image output. Finally, the generated video frame image with RGB three channels is output through a convolutional output head and a sigmoid activation function. Temporal constraint enhancement is applied to the generated video frames. By extracting the intermediate feature map of the current frame at the end of decoding and combining it with feature maps from several frames before and after it, an inter-frame attention matrix is ​​calculated using a query-key-value (QKV) mechanism. The feature information from the preceding and following frames is weighted and injected into the current frame to form an attention-weighted feature map. This is then linearly fused with the original feature map of the current frame. Combined with the context scheduling vector calculated from inter-frame motion estimation, the continuity of the character's pose trajectory changes is further constrained, generating a redrawn video frame.

[0046] In one example, the multimodal control vector is mapped to a high-dimensional conditional vector and broadcast to expand it into a conditional feature map. This map is then element-wise multiplied with the deep encoded features at the bottleneck interaction layer of the diffusion model to obtain the modulated bottleneck features, including: The multimodal control vector is mapped to a high-dimensional conditional vector, and the high-dimensional conditional vector is copied and extended in the spatial dimension to the same spatial size and number of channels as the deep coding features, so as to obtain a conditional feature map with the same feature value at each spatial location. The conditional feature map and the deep coding feature are multiplied element-wise at their corresponding spatial locations and channel dimensions to obtain the modulated bottleneck feature.

[0047] In this example, the multimodal control vector is input into a high-dimensional mapping network for dimensionality upscaling of the conditional vector. The original multimodal control vector has 512 dimensions, containing a comprehensive representation encoded by three types of information: action structure, language description, and visual style. A set of fully connected neural networks maps the multimodal control vector to a higher-dimensional conditional vector, such as 1024 dimensions, providing sufficient capacity to modulate the high semantic information channels in the bottleneck feature layer. After mapping, the high-dimensional conditional vector is still a one-dimensional structure and lacks spatial distribution capability. Therefore, a spatial broadcasting expansion operation is performed to copy the high-dimensional conditional vector spatially, constructing a three-dimensional tensor with the same spatial size and channel dimension as the current deep-encoded feature map. Assuming the bottleneck feature map size is H×W×C, for example, 32×32×1024, the broadcasting operation will repeatedly generate the conditional value for each channel dimension at every position in H×W, ensuring that the resulting conditional feature map has a completely consistent channel vector representation at every spatial position; that is, each spatial point has the exact same semantic structure expression as the original control vector. The conditional feature map and the deep semantic feature map encoded by the model in the downsampling path are subjected to element-wise multiplication at the corresponding spatial locations and channel dimensions. That is, for each coordinate (x, y, c), the encoded feature value F(x, y, c) and the conditional feature value G(x, y, c) are multiplied to form the modulated bottleneck feature value H(x, y, c) = F(x, y, c) × G(x, y, c). The element-wise conditional modulation mechanism has a high degree of local consistency and global expression consistency. Since the conditional feature map is a constant copy in space, it can uniformly affect the semantic expression of specific channels throughout the entire space, ensuring that control information can be equally weighted to all spatial locations. Furthermore, since the encoded feature map centrally represents global action semantics and visual structural information at the bottleneck location, the element-wise modulation operation can effectively change the response intensity of the feature channels, thereby substantially guiding the action form, appearance style, and semantic direction of the generated result.

[0048] In one example, feature maps of the generated video frames are extracted and temporal constraint enhancement is performed, outputting redrawn video frames, including: Extract the feature map of the generated video frame and calculate the inter-frame attention weighted feature. Then, fuse the feature map of the generated video frame with the inter-frame attention weighted feature to obtain the fused feature map. An attitude constraint vector is generated based on the sequence of attitude key points. The attitude constraint vector is then expanded into a constraint feature map and multiplied element-wise with the fused feature map to output the redrawn video frame.

[0049] In this example, an intermediate feature map corresponding to the current video frame is extracted from the decoder, with a size of H×W×C, as the original feature representation of the current frame. Simultaneously, feature maps of generated video frames several frames before and after the current frame are loaded to construct a time window composed of context frame feature maps, including the previous 3 frames and the next 3 frames, totaling 6 frames, forming a time-series feature set. The current frame feature map is reshaped into a two-dimensional query matrix Q with a shape of (H×W)×C. The context frame feature maps are similarly flattened and concatenated to form a key matrix K and a value matrix V, with dimensions of (H×W×6)×C respectively. By calculating the inner product of Q and K and performing √C scale normalization, a similarity matrix between the query and the key is obtained. A Softmax operation is performed on the similarity matrix to obtain the inter-frame attention weight matrix. Each row of the inter-frame attention weight matrix represents the attention distribution of a spatial location in the current frame to all pixel locations in historical and future frames. Multiplying the inter-frame attention weight matrix by the value matrix V yields an inter-frame attention weighted feature map, representing the semantic fusion result of each location in the current frame within the reference time window. The attention-weighted feature map is restored to the same size as the original feature map, and a fusion coefficient α (e.g., 0.7) is set to control the fusion weights. The original feature map of the current frame and the attention-weighted feature map of the inter-frame are fused in a linear combination to obtain a fused feature map, which is used to suppress inter-frame flicker and improve temporal consistency. The set of two-dimensional keypoint coordinates corresponding to the current frame is extracted from the original pose keypoint sequence, and normalization and scaling are performed to construct a structured pose vector. This structured pose vector is then embedded into a pose latent space coding network to generate a pose constraint vector of dimension d (e.g., 512). The pose vector is expanded into a multi-channel tensor matching the fused feature map through a linear transformation. The pose vector is then copied and spatially broadcast to generate a pose constraint feature map of size H×W×C, which has consistent semantic structure information at all spatial locations. The constraint feature map and the fused feature map are multiplied element-wise, i.e., F is applied to the feature value at each spatial coordinate (x, y, c). 融 (x, y, c) × A 姿态(x, y, c) is used to modulate the spatial position and channel response of each element in the generated feature map using the attitude control signal. The modulated feature map is then fed into the decoding convolutional layer, and after pixel mapping and activation processing, the final redrawn video frame is output.

[0050] Specifically, this includes: Query-Key-Value (QKV) construction. (Based on the current frame) Feature map (Dimensions are H×W×C, where H and W are the feature map height and width, and C is the number of channels) as the query (Q), i.e. ; using feature maps from 6 frames excluding the current frame within the context window As key (K) and value (V), i.e. , Attention weight calculation. Transform the dimensions of Q and K, converting H×W×C to (HW)×C for easier matrix operations; calculate the similarity matrix between Q and K: (divided by) (Perform scale normalization to avoid gradient vanishing); perform Softmax normalization on the similarity matrix to obtain the inter-frame attention weight matrix. The weight matrix has dimensions (HW) × (6HW), where This represents the correlation strength between the current frame feature pixel p and the context frame feature pixel q. Attention feature fusion. Calculate the attention-weighted features: The dimension is (HW)×C; Convert the feature map back to H×W×C format and compare it with the original feature map of the current frame. Perform weighted fusion: ,in The fusion coefficient (default 0.7, which can be adaptively adjusted according to the motion differences between frames, such as when the motion differences between frames are large). Decrease, difference in hours (Improvement). Temporal consistency optimization output. The output after fusing attention features. constraint vectors generated by context scheduling strategy Element-wise multiplication is performed to obtain the final optimized feature map, which is then input into the subsequent decoding module to generate time-stable video frames, effectively suppressing flicker (inter-frame pixel abrupt changes) and jitter (inter-frame pose shift).

[0051] Figure 5This describes the processing flow of the video temporal consistency enhancement mechanism. It begins with the input of the original video frame. After passing through the video frame sequence input / output module, it enters the temporal context window construction stage. This stage organizes the current frame to be processed and its temporally adjacent frames into a context window, specifically through three parallel paths: the previous frame feature path reads the feature representation of the previous frame, the current frame feature path processes the current frame and extracts its features, and the next frame feature path reads the features of the next frame. The feature vectors from the three paths are fed into the inter-frame attention enhancement matrix module. This module calculates the correlation between the current frame and adjacent frames using the QKV attention mechanism, generating an inter-frame attention weight matrix. The attention weights are then input into the inter-frame weight initialization evaluation module for preliminary evaluation and standardization of the weight distribution. The evaluation results are sent to the intelligent difference evaluation module, which analyzes the degree of difference between adjacent frames. By calculating the L2 norm of the feature maps of the current frame and the previous frame, it determines the inter-frame motion amplitude and assesses whether there are excessive inter-frame jumps. Based on the difference assessment results, the temporal consistency optimization generation module optimizes and adjusts the generated video frames, adaptively adjusting the fusion coefficient according to the motion differences between frames: when the motion amplitude is large, the fusion coefficient is reduced to increase the dependence on the context frame to smooth large motions; when the motion amplitude is small, the fusion coefficient is increased to retain the original features of the current frame to avoid over-smoothing. Finally, the stability optimization module outputs a temporally smooth video frame sequence. This process, by introducing an inter-frame attention mechanism and a context scheduling strategy, combined with adaptive fusion coefficient adjustment, effectively solves the flickering and jitter problems in video generation, improving the temporal coherence and visual smoothness of the generated video.

[0052] Reference Figure 6 This embodiment provides a generative video motion redrawing system based on multimodal pose control, including: Key point detection module 1 is used to acquire the original video frames of the input video and perform pose key point detection to obtain a pose key point sequence; Mapping module 2 is used to map the sequence of pose key points to the pose latent space to obtain the pose embedding vector, and to extract the semantics of the text description to obtain the text embedding vector. At the same time, it extracts the appearance features of the reference image to obtain the visual embedding vector. Fusion module 3 is used to fuse the pose embedding vector, text embedding vector and visual embedding vector to obtain a multimodal control vector; Output module 4 is used to perform multi-stage denoising processing on the multimodal control vector and random noise image input diffusion model to obtain generated video frames, and to perform temporal constraint enhancement on the generated video frames to output redrawn video frames.

[0053] In this embodiment, the specific implementation of each unit in the above system embodiment is described in the above method embodiment, and will not be repeated here.

[0054] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0055] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A generative video motion redrawing method based on multimodal pose control, characterized in that, include: The original video frames of the input video are acquired and pose key points are detected to obtain a pose key point sequence; The pose key point sequence is mapped to the pose latent space to obtain the pose embedding vector, and the semantics of the text description are extracted to obtain the text embedding vector. At the same time, the appearance features of the reference image are extracted to obtain the visual embedding vector. The pose embedding vector, the text embedding vector, and the visual embedding vector are fused to obtain a multimodal control vector; The multimodal control vector and the random noise image input diffusion model are subjected to multi-stage denoising processing to obtain generated video frames. The generated video frames are then subjected to temporal constraint enhancement to output redrawn video frames.

2. The generative video motion redrawing method based on multimodal attitude control according to claim 1, characterized in that, The original video frames of the input video are acquired and pose keypoints are detected to obtain a pose keypoint sequence, including: The original video frames of the input video are obtained, the resolution of the original video frames is normalized to obtain standard video frames, and the main body area of ​​the person and the background environment area in the standard video frames are separated to obtain the person area image. The image of the person's region is input into the OpenPose model to detect the positions of the main joints of the human body, and a first set of key points and a first confidence score are obtained; the image of the person's region is input into the HRNet model to detect high-precision key points including finger joints and toe joints, and a second set of key points and a second confidence score are obtained; the image of the person's region is input into the MediaPipe model to perform robust detection on occluded areas, and a third set of key points and a third confidence score are obtained. Based on the first confidence score, the second confidence score, and the third confidence score, a weighted average is performed on the first group of key points, the second group of key points, and the third group of key points to obtain the attitude key point sequence.

3. The generative video motion redrawing method based on multimodal attitude control according to claim 1, characterized in that, The pose keypoint sequence is mapped to the pose latent space to obtain a pose embedding vector. Semantic extraction is performed on the text description to obtain a text embedding vector. Simultaneously, appearance features are extracted from the reference image to obtain a visual embedding vector, including: The pixel coordinates of the attitude key point sequence are normalized to obtain the normalized key point coordinates. The normalized keypoint coordinates are organized into structured attitude vectors, and the structured attitude vectors are linearly transformed and mapped to a preset dimension to obtain attitude embedding vectors. The text description is input into the BERT model for semantic extraction to obtain the text embedding vector, while the reference image is input into the ViT model for appearance feature extraction to obtain the visual embedding vector.

4. The generative video motion redrawing method based on multimodal attitude control according to claim 3, characterized in that, The text description is input into the BERT model for semantic extraction, yielding a text embedding vector. Simultaneously, the reference image is input into the ViT model for appearance feature extraction, yielding a visual embedding vector, including: The text description is segmented and converted into a word vector sequence. The word vector sequence is then input into the multi-layer Transformer encoder of the BERT model to capture contextual semantic relationships and extract text embedding vectors. The reference image is segmented into multiple image patch sequences and converted into image patch vectors. The image patch vectors are then input into the multi-layer Transformer encoder of the ViT model and the appearance style-related regions are weighted by the adaptive attention module to obtain the visual embedding vector.

5. The generative video motion redrawing method based on multimodal attitude control according to claim 1, characterized in that, The pose embedding vector, the text embedding vector, and the visual embedding vector are fused to obtain a multimodal control vector, including: The pose embedding vector, the text embedding vector, and the visual embedding vector are concatenated into a joint vector, and the pose weight coefficient, text weight coefficient, and visual weight coefficient are calculated based on the joint vector. Linear projection is performed on the joint vector to generate a query matrix, a key matrix, and a value matrix. An attention weight matrix is ​​calculated based on the query matrix and the key matrix. The attention weight matrix is ​​multiplied by the value matrix to obtain the attention weighted feature. The attention-weighted features are weighted and summed based on the posture weight coefficient, the text weight coefficient, and the visual weight coefficient, and then a multimodal control vector is obtained through linear transformation.

6. The generative video motion redrawing method based on multimodal attitude control according to claim 1, characterized in that, The pre-training process of the diffusion model includes: Collect a general video dataset covering different scenes and action types, as well as a multi-person action dataset, an occlusion scene dataset, and a stylized video dataset to obtain the training dataset; The training video frames of the training dataset are input into a conditionally controlled U-Net diffusion network. A diffusion training process of forward noise addition and backward noise reduction is adopted, and the pixel error between the predicted denoised frame and the original frame is calculated based on the MSE loss function for pre-training to obtain a pre-trained model. The parameters of the input encoding layer and the front downsampling block of the pre-trained model are frozen. Training video frames, pose key point annotations, text semantic labels and training reference images are input into the upper network. The model is fine-tuned based on a hybrid loss function of MSE pixel loss, pose consistency loss and semantic matching loss to obtain the diffusion model.

7. The generative video motion redrawing method based on multimodal attitude control according to claim 6, characterized in that, The multimodal control vector and the random noise image are input into a diffusion model for multi-stage denoising to obtain generated video frames. Temporal constraint enhancement is then applied to the generated video frames to output redrawn video frames, including: The multimodal control vector is projected into a multi-channel feature map and concatenated with a random noise image. Then, it is input into the input coding layer of the diffusion model for convolution processing to obtain deep coding features. The multimodal control vector is mapped to a high-dimensional conditional vector and broadcast to expand it into a conditional feature map. The deep coding features are then multiplied element-wise at the bottleneck interaction layer of the diffusion model to obtain the modulated bottleneck features. The modulated bottleneck features are restored to feature map resolution through multiple upsampling blocks and combined with cross-layer connections. The multimodal control vector is then feature-decoded in the conditional fusion layer of the diffusion model to generate video frames. The feature maps of the generated video frames are extracted and temporal constraint enhancement is performed to output the redrawn video frames.

8. The generative video motion redrawing method based on multimodal attitude control according to claim 7, characterized in that, The multimodal control vector is mapped to a high-dimensional conditional vector and broadcast to expand it into a conditional feature map. This map is then multiplied element-wise with the deep encoded features at the bottleneck interaction layer of the diffusion model to obtain the modulated bottleneck features, including: The multimodal control vector is mapped to a high-dimensional conditional vector, and the high-dimensional conditional vector is copied and extended in the spatial dimension to the same spatial size and number of channels as the deep coding features, so as to obtain a conditional feature map with the same feature value at each spatial location. The conditional feature map and the deep coding feature are multiplied element-wise at their corresponding spatial locations and channel dimensions to obtain the modulated bottleneck feature.

9. The generative video motion redrawing method based on multimodal attitude control according to claim 8, characterized in that, Extracting feature maps from the generated video frames and performing temporal constraint enhancement to output redrawn video frames includes: Extract the feature map of the generated video frame and calculate the inter-frame attention weighted feature. Then, fuse the feature map of the generated video frame with the inter-frame attention weighted feature to obtain the fused feature map. An attitude constraint vector is generated based on the attitude key point sequence. The attitude constraint vector is then expanded into a constraint feature map and multiplied element-wise with the fused feature map to output a redrawn video frame.

10. A generative video motion redrawing system based on multimodal pose control, characterized in that, The steps for implementing the generative video motion redrawing method based on multimodal pose control as described in any one of claims 1 to 9 include: The key point detection module is used to acquire the original video frames of the input video and perform pose key point detection to obtain a pose key point sequence. The mapping module is used to map the sequence of pose key points to the pose latent space to obtain the pose embedding vector, and to perform semantic extraction on the text description to obtain the text embedding vector. At the same time, it performs appearance feature extraction on the reference image to obtain the visual embedding vector. The fusion module is used to fuse the pose embedding vector, the text embedding vector, and the visual embedding vector to obtain a multimodal control vector; The output module is used to perform multi-stage denoising processing on the multimodal control vector and the random noise image input diffusion model to obtain the generated video frame, and to perform temporal constraint enhancement on the generated video frame to output the redrawn video frame.

Images