Multi-action human video generation method and device, and storage medium

Abstract

This application discloses a method, device, and storage medium for generating multi-action human body videos, comprising: inputting reference images and multi-action prompt text into a multimodal large language model; performing inference on the multi-action prompt text through the multimodal large language model to output a fine-grained action description sequence; generating intermediate image frames based on the reference images and the fine-grained action description sequence; extracting human skeleton key point data from the intermediate image frames to obtain a visual skeleton sequence; inputting the visual skeleton sequence, multi-action text prompts, and reference images into a video diffusion model; and integrating the visual skeleton sequence into the denoising generation process of the video diffusion model through a pose guidance adapter to output a coherent human body video matching the multi-action text prompts. This application achieves accurate, coherent, and efficient generation of complex multi-action videos by decoupling and fusing the chained inference of the multimodal language model, the intermediate representation of the skeleton sequence, and the pose guidance of the video diffusion model.

Description

Technical Field

[0001] This application relates to the fields of computer vision and generative artificial intelligence, and in particular to a method, device and storage medium for generating multi-action human videos. Background Technology

[0002] In the fields of computer vision and generative artificial intelligence, generating videos of human actions from static reference images and text descriptions has become an important research direction. Current mainstream methods typically rely on a single, end-to-end image-to-video diffusion model architecture, which attempts to simultaneously perform high-level semantic action planning and low-level pixel rendering within a single network. This coupled design poses a fundamental challenge to the model when handling multiple consecutive actions requiring complex temporal logic and causal relationships, often resulting in generated videos with missing actions, disordered sequences, or unnatural transitions, failing to faithfully reflect the complete intent in the text prompts.

[0003] To address these challenges, some improved methods attempt to break down complex multi-action text prompts into multiple sub-prompts and synthesize the video segment by segment using an iterative generation approach. However, this method introduces difficult-to-correct cumulative errors in temporal sequence. The independent generation of each video segment causes shifts in character appearance, lighting conditions, and background scenes between segments, disrupting the spatiotemporal consistency of the entire video. Furthermore, repeatedly calling the model for iteration significantly increases computational overhead, resulting in low generation efficiency and failing to meet the needs of practical applications.

[0004] Furthermore, while large-scale multimodal models exhibit powerful natural language understanding and reasoning capabilities, enabling them to parse action semantics, a significant modal gap exists between their output and the precise, structured visual guidance signals (such as human skeletal sequences) required to drive video generation. On the other hand, methods relying on predefined action libraries or requiring additional reference videos to obtain pose sequences, while providing precise guidance, sacrifice the flexibility and creativity driven by free text prompts, failing to meet users' needs for instantly generating diverse action videos based on abstract descriptions. Therefore, how to accurately translate high-level textual intent into continuous and accurate visual motion representations that can drive generative models, while ensuring the global consistency of the final video, remains a core unresolved technical challenge in this field.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a method, device, and storage medium for generating multi-action human body videos, aiming to solve the technical problems of disordered timing and poor consistency in the generation of multi-action videos by existing methods.

[0007] To achieve the above objectives, this application proposes a method for generating multi-action human body videos, the method comprising: Obtain reference images and multi-action prompt text; The reference image and the multi-action prompt text are input into a multimodal large language model. The multimodal large language model performs chain-like reasoning on the multi-action prompt text based on the reference image and preset motion prior knowledge, and outputs a structured fine-grained action description sequence. Based on the reference image and the fine-grained action description sequence, an intermediate image frame corresponding to each action description is generated; Extract human skeleton key point data from the intermediate image frame to form a key frame skeleton sequence; The keyframe skeleton sequence is subjected to temporal interpolation and smoothing to obtain a visual skeleton sequence; The visual skeleton sequence is loaded into the denoising generation process of the video diffusion model through a pose guidance adapter. Based on the denoising generation process, the reference image is integrated into the visual skeleton sequence to generate a coherent human motion video that matches the multi-action text prompt.

[0008] In one embodiment, the step of performing chain-like reasoning on the multi-action prompt text based on the reference image and preset motion prior knowledge using the multimodal large language model to output a structured, fine-grained sequence of action descriptions includes: The reference image is converted into a visual feature code, and the visual feature code is deployed as an embedded representation of the multimodal large language model; Obtain a comprehensive prompt, input the comprehensive prompt into the multimodal large language model, and activate the motion prior knowledge of the multimodal large language model; The multi-action prompt text is parsed and reasoned using the prior motion knowledge and the embedding representation to generate multiple intermediate descriptions corresponding to each action; The intermediate descriptions are organized in chronological order to form the fine-grained action description sequence.

[0009] In one embodiment, the step of generating intermediate image frames corresponding to each action description based on the reference image and the fine-grained action description sequence includes: The fine-grained action description sequence is split into multiple action descriptions, and the multiple action descriptions and the reference image are input into the image generation model; Using the image generation model, based on the appearance information of the person in the reference image and the action description, iterative denoising is performed in the latent space to generate intermediate image frames based on action posture.

[0010] In one embodiment, the step of generating intermediate image frames based on action pose by iteratively denoising in the latent space using the image generation model, based on the appearance information of the person in the reference image and the action description, includes: Based on the complexity and amplitude of the action description, the sampling step number or guidance scale of the iterative denoising is dynamically adjusted. Specifically, a lower sampling step number and guidance scale are set for actions with small amplitude, and a higher sampling step number and guidance scale are set for actions with large amplitude or complex posture. Iterative denoising is performed based on the adjusted sampling steps or guiding scale to generate intermediate image frames based on action pose.

[0011] In one embodiment, the step of extracting human skeletal key point data from the intermediate image frame to form a keyframe skeletal sequence includes: A pre-trained human pose estimation model is used to detect the two-dimensional pixel coordinates based on human skeletal key points in the intermediate image frames; The two-dimensional pixel coordinates are converted into normalized three-dimensional skeletal coordinates according to the camera model, and the three-dimensional skeletal coordinates are organized according to the standard human skeletal topology so that the three-dimensional skeletal coordinates extracted from the intermediate image frames corresponding to each action description are arranged in the action order to form a keyframe skeletal sequence.

[0012] In one embodiment, the step of performing temporal interpolation and smoothing on the keyframe skeleton sequence to obtain a visual skeleton sequence includes: Based on the time interval and coordinate data between adjacent keyframe bones, a linear interpolation algorithm is used to calculate the bone coordinates of intermediate transition frames, and an initial bone sequence with continuous time is generated based on the bone coordinates and the keyframe bones. The initial bone sequence is smoothed using a time-domain filtering algorithm to obtain an optimized smooth bone sequence. The smooth bone sequence is length-aligned and formatted to output the visual bone sequence.

[0013] In one embodiment, the step of loading the visual skeleton sequence into the denoising generation process of the video diffusion model via a pose guidance adapter, and then integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human motion video matching the multi-action text prompt, includes: At each time step in the denoising process, the visual skeleton sequence data corresponding to the current time step is encoded into a spatial conditional embedding; The spatial conditions are embedded into the weight offsets generated by the pose guidance adapter and fused with the noise prediction features in the denoising network to guide the alignment of the generated character pose with the visual skeleton sequence.

[0014] In one embodiment, the step of loading the visual skeleton sequence into the video diffusion model via a pose guidance adapter for denoising generation, and then integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human motion video matching the multi-action text prompt, includes: Calculate the motion velocity and acceleration of the joints in the intermediate transition frames between adjacent keyframe bones in the visual skeleton sequence, and determine whether there is abnormal motion based on the calculation results. Abnormal motion is an abnormal movement that exceeds the physiological limits of the human body. If the abnormal motion is determined to exist, the keyframe bone where the abnormal motion is located is determined in the visual skeleton sequence, and the bone coordinates of the keyframe bone are locally re-interpolated or smoothed to obtain the corrected visual skeleton sequence. Based on the corrected visual skeleton sequence, the visual skeleton sequence is reintegrated into the denoising generation process of the video diffusion model.

[0015] In addition, to achieve the above objectives, this application also proposes a multi-motion human body video generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-motion human body video generation method described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multi-action human body video generation method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: The technical solution of this application involves: acquiring a reference image and multi-action prompt text; inputting the reference image and the multi-action prompt text into a multimodal large language model; using the multimodal large language model to perform chain-like reasoning on the multi-action prompt text based on the reference image and preset motion prior knowledge, outputting a structured, fine-grained action description sequence; generating intermediate image frames corresponding to each action description based on the reference image and the fine-grained action description sequence; extracting human skeleton key point data from the intermediate image frames to form a keyframe skeleton sequence; performing temporal interpolation and smoothing on the keyframe skeleton sequence to obtain a visual skeleton sequence; loading the visual skeleton sequence into the denoising generation process of the video diffusion model through a pose guidance adapter; and integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human action video matching the multi-action text prompts.

[0018] This application decouples planning and rendering, and transforms text-based chain reasoning into a continuous skeleton sequence, achieving precise planning, natural transitions, and efficient generation of multiple actions, significantly improving the temporal logic and spatiotemporal consistency of the video. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the multi-action human body video generation method of this application; Figure 2 This is a detailed schematic diagram of step S20 in the first embodiment; Figure 3 This is a detailed schematic diagram of step S30 in the first embodiment; Figure 4 This is a detailed schematic diagram of step S40 in the first embodiment; Figure 5 This is a detailed schematic diagram of step S50 in the first embodiment; Figure 6 This is a detailed schematic diagram of step S60 in the first embodiment; Figure 7 A schematic diagram of the architecture for generating multi-motion human body videos; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the multi-action human body video generation method in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] In related technologies, image-to-video generation models are difficult to handle the temporal logic of multiple actions due to the coupling between high-level action planning and low-level pixel rendering. Furthermore, there is a modal gap between textual intent and visual guidance, resulting in chaotic and inconsistent video actions.

[0025] Based on the aforementioned deficiencies in related technologies, this application proposes a generalizable AI-generated image detection method. In this method, when faced with multiple problems such as inaccurate multi-action timing planning, lack of cross-modal guidance, and inconsistent generation, the method decouples and integrates the chain reasoning of the multimodal language model, the intermediate representation of the skeletal sequence, and the pose guidance of the video diffusion model, thereby achieving accurate, coherent, and efficient generation of complex multi-actions.

[0026] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0027] Based on this, embodiments of this application provide a method for generating multi-action human body videos, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multi-motion human body video generation method of this application. In this embodiment, the multi-motion human body video generation method includes steps S10 to S60: Step S10: Obtain the reference image and multi-action prompt text; In this embodiment, the initialization phase of the multi-action human video generation process involves the acquisition and preparation of raw data. First, a clear reference image needs to be received or retrieved, serving as the visual benchmark for all subsequent generation steps. The reference image should contain a complete frontal or side view of the target person. While there are no strict limitations on its resolution, lighting conditions, and background complexity, a clear outline and detailed appearance will facilitate the extraction of stable features by the model. In actual deployment, this reference image may originate from user uploads, database retrieval, or the output of a previous generation task. The system typically performs basic preprocessing operations on it, such as size normalization to a fixed pixel size and color space conversion, to ensure a consistent input format.

[0028] Acquired in parallel with the reference image is a natural language text describing multiple consecutive actions, i.e., multi-action prompt text. This text is freely input by the user or selected from a preset action template library. Its content must clearly express a sequence of actions containing temporal logic, such as "first walk five steps, then turn around and wave, and finally sit down." After receiving the prompt through the text input interface, the system performs basic syntax error checking and word segmentation, but preserves its original semantic integrity. This text and the reference image are logically related, jointly defining a generation task: based on the appearance of the person in the reference image, execute the action sequence described in the text. These two types of data are encapsulated into a structured input pair and passed to the subsequent parsing module as the starting point of the entire generation pipeline.

[0029] Step S20: Input the reference image and the multi-action prompt text into the multimodal large language model. The multimodal large language model performs chain-like reasoning on the multi-action prompt text based on the reference image and preset motion prior knowledge, and outputs a structured fine-grained action description sequence. In this embodiment, the process of translating high-level intent into executable action planning is led by a multimodal large language model. The model's input receives a reference image and multi-action cue text. The reference image is not processed directly as raw pixels, but is first converted into a series of visual tags by the model's visual encoder. These visual tags are embedded into the joint semantic space of the multimodal large language model as continuous contextual information, enabling the model to "see" the character's initial state. The multi-action cue text is then segmented and converted into a sequence of text tags.

[0030] The multimodal large language model's pre-defined motion prior knowledge is activated by specific prompting engineering strategies. A structured dialogue context is first constructed as the actual input, containing system role instructions, a few example scenarios, and the current query. The system instructions explicitly require the multimodal large language model to act as a human motion analysis expert; the example scenarios are presented in the form of "brief user prompts - detailed model decomposition," for example, breaking down "jumping" into five sub-steps: "crouching to build up momentum, pushing off the ground to jump, extending the body in the air, preparing to land, and landing with knee flexion to cushion the impact"; the current query is accompanied by the instruction "please reason step by step." This contextual learning approach allows the model to access its internal knowledge of human kinematics without additional training.

[0031] Chain-like reasoning unfolds within this context. The multimodal large language model first parses the overall semantics of the multi-action cue text, identifying the number, sequence, and possible causal relationships of the actions. Next, combining its understanding of the initial posture of the person in the reference image, the multimodal large language model performs a temporal decomposition of each macro-action. The reasoning process is progressive and causal; for example, for "picking up an object and then throwing it," the model infers that "bending over" is a prerequisite for "reaching out," "gripping" occurs after contact with the object, and "twisting the torso" is required to build up momentum before "swinging the arm." The final output is a strictly linear sequence, where each element is a complete sentence describing a brief point in time or phase of human posture change. All sentences together constitute a detailed action script that can be mapped to specific postures—a structured, fine-grained sequence of action descriptions.

[0032] Step S30: Based on the reference image and the fine-grained action description sequence, generate an intermediate image frame corresponding to each action description; In this embodiment, the process of generating a visual pose frame based on text action descriptions is a conditional image generation task. Input data includes a reference image serving as the source of identity and appearance, and a sequence of fine-grained action descriptions. Instead of processing the entire sequence at once, the system uses a cyclical or parallel approach to generate an independent intermediate image frame for each action description in the sequence.

[0033] The generation process relies on a pre-trained text-to-image diffusion model. For each target action description, the generator's conditional input consists of two parts: first, depth features from a reference image extracted by an image encoder, which define appearance attributes such as the person's identity, clothing texture, and hairstyle in the latent space; and second, text features of the current action description processed by a text encoder, which guide pose changes. The model iteratively denoises from a random Gaussian noise tensor within the latent space.

[0034] In each denoising iteration, the noise prediction network integrates the current latent noise variables, time step information, and the aforementioned biconditional embedding to predict the noise components that should be removed. The denoising process essentially involves, under guidance, gradually reconstructing random noise into clear latent features that meet the conditional constraints. The key technical point of the denoising process shown lies in the dynamic control of the generation process. The system adaptively adjusts two key generation parameters—the number of sampling steps and the classifier guidance scale—based on the complexity of the action description. For simple poses such as "standing and smiling," fewer sampling steps and a lower guidance scale can be used for rapid synthesis; for complex, high-amplitude movements such as "completing a backflip," more sampling steps and stronger text guidance are used to ensure the accuracy of the pose and the integrity of the limb details. After a set number of iterations, the denoised latent features are obtained, which are then converted back to pixel space by the decoder, outputting an intermediate image frame that matches the appearance of the person in the reference image but whose pose conforms to the current text description. This frame, as the visual representation of the current action description, will be used for the next step of geometric information extraction.

[0035] Step S40: Extract human skeleton key point data from the intermediate image frame to form a key frame skeleton sequence. In this embodiment, extracting structured skeletal data from the pose image is a key transformation step for achieving precise motion control. The input to this transformation step is a series of intermediate image frames generated above, with each frame corresponding to a key action pose. The processing is performed independently for each image frame, and the core of the process is to call a high-precision human pose estimation model, such as a detector based on a deep convolutional neural network or Transformer architecture.

[0036] The pose estimation model first performs human instance detection in the input image to locate the bounding box of the person. Subsequently, within the bounding box area, the model performs key point detection to identify the main anatomical joint points of the human body. A common configuration is to detect 17 key points, including the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles, etc. For each detected joint point, the model outputs its pixel coordinates in the two-dimensional coordinate system of the image and a confidence score. The system usually sets a confidence threshold to filter out mis-detected points with low confidence.

[0037] After obtaining the two-dimensional pixel coordinates, data normalization and structuring are required. First, the coordinate values are normalized according to the human body bounding box or the image size to eliminate the influence of image resolution differences. To obtain data more suitable for three-dimensional motion interpolation and spatial reasoning, the system usually uses pre-calibrated camera intrinsics or adopts a learning-based depth estimation module to lift these two-dimensional key points to normalized three-dimensional space coordinates. Whether using two-dimensional or three-dimensional representations, subsequently, the discrete key points need to be connected according to the standard bone topology to form a complete bone graph. This topology defines the parent-child connection relationships between joint points. For example, the wrist is connected to the elbow, and the elbow is connected to the shoulder. Finally, a bone data object is generated for each intermediate image frame, which contains the coordinate information of all joint points and their connection relationships. Arranging the bone data objects corresponding to all frames in the chronological order of the original action description sequence forms a discrete key frame bone sequence, which precisely records the bone space configurations of the entire action at several key moments.

[0038] Step S50, perform temporal interpolation and smoothing processing on the key frame bone sequence to obtain a visual bone sequence; In this embodiment, the process of converting the discrete bone sequence into a continuous motion signal involves interpolation and filtering techniques in digital signal processing. The input is the key frame bone sequence, where each bone data frame is marked with a time stamp indicating its position in the overall action timeline. However, the time intervals between these key frames are much larger than the frame intervals required for the final video, and directly using them to guide video generation will cause jerky motions.

[0039] Temporal interpolation is the primary operation to solve the insufficient frame rate. The system uses a linear interpolation algorithm to generate intermediate frames between two adjacent key frame bones. For the bone poses S_i and S_{i + 1} of two key frames at times t_i and t_{i + 1}, if we want to generate the bone pose S_t at an intermediate time t (t_i < t < t_{i + 1}), the three-dimensional coordinates of each corresponding joint point are calculated independently. The interpolation formula is: S_t = S_i + ( (t - t_i) / (t_{i + 1} - t_i) ) (S_{i+1} - S_i). By performing this operation between all adjacent keyframe pairs and determining the number of transition frames to be inserted according to the target video frame rate, an initial skeleton sequence that is temporally continuous but may contain minor jitter due to linear assumptions or keyframe estimation errors can be obtained.

[0040] To eliminate unnatural jitter and smooth motion trajectories, temporal filtering is required on the initial continuous sequence. Gaussian smoothing is a commonly used method. A one-dimensional Gaussian kernel is defined as a sliding window, with a weight distribution defined on the time axis. For each keypoint coordinate in each frame of the sequence, the coordinate values ​​within the preceding and following time windows are convolved with the weights of the Gaussian kernel to obtain the smoothed coordinate values. The variance parameter σ of the Gaussian kernel controls the smoothing intensity; a larger σ value results in a more significant smoothing effect but may excessively weaken fast motion details. After Gaussian filtering with appropriately set parameters, the motion trajectory of the skeletal sequence becomes smooth and natural, more in line with biomechanical motion principles. Finally, this smoothed skeletal sequence is truncated or padded to the exact same number of frames as the target video length and encapsulated in the tensor format required by the video diffusion model, outputting the final visual skeletal sequence, which is now a dense, smooth time-series signal.

[0041] Step S60: The visual skeleton sequence is loaded into the denoising generation process of the video diffusion model through the pose guidance adapter. Based on the denoising generation process, the reference image is integrated into the visual skeleton sequence to generate a coherent human motion video that matches the multi-action text prompt.

[0042] In this embodiment, the final video synthesis is accomplished using a large-scale pre-trained video diffusion model strongly constrained by the visual skeleton sequence. The input to this step includes three core elements: the visual skeleton sequence, the original multi-action text cue, and the initial reference image. These conditions constrain the generation process from three aspects: spatial pose, high-level semantics, and appearance details.

[0043] The pre-trained video diffusion model inherently possesses the ability to generate short videos based on text and reference images. The key improvement of this method lies in integrating a lightweight pose guidance adapter, typically implemented in the form of LoRA, into the video diffusion model. This pose guidance adapter is injected into the attention layer or feedforward network of the U-Net video diffusion model in a low-rank decomposition manner, fine-tuning or expanding only a very small number (e.g., 0.5%) of the model parameters, thereby endowing the model with the ability to understand and respond to skeletal sequence conditions.

[0044] During the denoising process, for each frame of the target video, the visual skeleton sequence data at the corresponding time step is mapped to a spatial conditional embedding by an independent encoder. At each layer of the denoising network, the current noise features, text embedding, reference image features, and this spatial skeleton embedding are shared as input. The pose-guided adapter generates a feature modulation vector or weight offset based on the skeleton embedding, which is used to adjust the intermediate features computed by the backbone network. This adjustment is fine-grained and spatially aligned, prompting the denoising network to tend to generate pixel content that matches the person's pose to the specified pose in the input skeleton sequence when predicting noise.

[0045] Meanwhile, the original text cross-attention mechanism of the video diffusion model ensures that the semantics of the generated content conforms to the overall action description, while the reference image provides appearance features through a path similar to a CLIP image encoder, ensuring consistency in the identity and clothing of the characters. The entire generation process begins with a noise tensor that conforms to the target video size, and through iterative denoising at multiple time steps, the noise is gradually transformed into a clear sequence of video frames. The entire generation process can be viewed... Figure 7 , Figure 7 This is a schematic diagram of the multi-motion human video generation architecture. Due to the strong guidance of skeletal conditions, the movements of the characters in the generated video strictly follow the trajectory defined by the visual skeletal sequence; due to the presence of text and reference image conditions, the semantics of the movements and the appearance of the characters are correctly preserved. The final output is a coherent, realistic multi-motion human video that highly matches the initial input instructions.

[0046] Furthermore, you can also view Figure 2 , Figure 2 This is a detailed schematic diagram of step S20 in the first embodiment, based on the shown... Figure 2 The step of using the multimodal large language model to perform chain-like reasoning on the multi-action prompt text based on the reference image and preset motion prior knowledge, and outputting a structured, fine-grained sequence of action descriptions, includes S21-24: Step S21: Convert the reference image into a visual feature code, and deploy the visual feature code as an embedding representation of the multimodal large language model; Step S22: Obtain the comprehensive prompt, input the comprehensive prompt into the multimodal large language model, and activate the motion prior knowledge of the multimodal large language model; Step S23: Parse and reason about the multi-action prompt text using the prior knowledge of motion and the embedded representation to generate multiple intermediate descriptions corresponding to each action; Step S24: Organize the intermediate descriptions into the fine-grained action description sequence in chronological order.

[0047] In this embodiment, the visual information of the reference image needs to be transformed into an internal representation that the multimodal large language model can understand and utilize. This process begins with preprocessing the reference image, such as scaling its resolution to a fixed size desired by the model's visual encoder and normalizing the pixel values. The preprocessed image is input into the model's visual encoder module, which is typically a pre-trained visual Transformer or convolutional neural network. The encoder performs hierarchical feature extraction on the image, outputting a dense visual feature map or a sequence of visual labels.

[0048] The visual feature encodings shown are not directly processed as text, but rather need to be deployed as a special embedding representation in the model's joint semantic space. The system maps the high-dimensional features of the visual encodings to a vector space aligned with the dimensions of the model's text word embedding space through a learnable projection layer. The mapped sequence of visual feature vectors is typically combined with an embedding of a special [image] tag, or directly prepended to subsequent text input sequences. In this way, visual information is transformed into a kind of "visual memory" that persists within the model's context window. This embedding representation allows the model to continuously access and reference the appearance of people, initial poses, and scene context information in images during subsequent text reasoning, laying the foundation for image-based chain-like reasoning.

[0049] Furthermore, activating the implicit motion prior knowledge within a multimodal large language model relies on a carefully constructed syntactic cue. This syntactic cue is not a simple user query, but rather a structured dialogue context designed to guide the model into a specific role and follow desired reasoning patterns in zero-shot or few-shot settings. The construction of the syntactic cue typically comprises three core parts, sequentially organized into a coherent cue input.

[0050] The first part consists of system instructions, explicitly defining the model's role in the current task, such as "You are a professional human motion analysis expert, skilled at breaking down complex motion descriptions into continuous, fine-grained posture change steps." The second part comprises a small number of contextual examples, presented in pairs of "User prompt: ... --> Expert decomposition: ...". Each example demonstrates how to break down a short motion instruction (such as "turn and sit down") into multiple logically consecutive intermediate posture descriptions (e.g., "1. Using your feet as a pivot, begin rotating your torso to the left; 2. Your upper body completes approximately a 90-degree turn, with your gaze following the movement; 3. Your center of gravity slowly lowers, and your hips move backward; 4. Your knees bend, lowering your body to seat height; 5. Your body stably contacts the seat, keeping your back straight"). These examples provide demonstrations of output format and level of detail. The third part is the actual query, containing the original multi-motion prompt text provided by the user, often accompanied by guiding phrases that trigger chain thinking, such as "Based on the reference image and motion knowledge, please reason step-by-step and decompose the following motion sequence:" or "Let's think step by step." The integrated prompt formed by combining the above three parts is input into the multimodal large language model. By recognizing system roles and drawing on the patterns of contextual examples, the knowledge modules related to human kinematics and temporal logical reasoning in its parameters are effectively activated, preparing to execute subsequent parsing tasks.

[0051] With both visual context and motion priors activated, the video diffusion model begins deep parsing and inference of the multi-action cue text. The inference process is essentially an autoregressive text generation by the model, but the generated content is constrained and guided by both visual embedding representations and structured cues. The video diffusion model first reads and understands the macroscopic sequence of the entire multi-action cue text, identifying temporal connectors such as "first," "then," "next," and "finally," thereby establishing a preliminary timeline between actions.

[0052] Subsequently, the video diffusion model, combined with its activated prior knowledge of motion, progressively breaks down each macroscopic action unit in the sequence. For example, for the action of "jumping," the video diffusion model infers from its knowledge base that it is a periodic action involving charging, takeoff, and landing phases. The key interaction occurs during the inference process, where the model continuously references visual contextual information as embedded representations. For instance, when the reference image shows a person wearing a long skirt, the intermediate descriptions generated by the model when decomposing "walking" might implicitly consider the skirt's movement or avoid generating extreme leg posture descriptions that are severely inconsistent with the physical constraints of a long skirt. This resolution is iterative and causal; the conclusion of a previous intermediate description may serve as a premise for a subsequent description. The model generates a set of multiple (e.g., 5-10) intermediate descriptions for each macroscopic action. Each intermediate description is a complete short sentence that characterizes a specific moment or a tiny sub-action during the execution of the action, such as "the right leg steps forward, heel first," "the body's center of gravity begins to smoothly transition from the left foot to the right foot," and "the left leg pushes off the ground, propelling the body forward." These intermediate descriptions together constitute a complete trajectory depiction of a macroscopic action.

[0053] The video diffusion model generates multiple intermediate descriptions for each macro-action, which are initially presented in groups at the output. These scattered groups are then reorganized and normalized over time to form a final, unified, coherent, structured, fine-grained sequence of action descriptions. The system first determines the approximate location blocks of each group of intermediate descriptions on the global timeline according to the order of appearance of the macro-actions in the original multi-action prompt text.

[0054] Next, within each description group corresponding to a macro-action, the intermediate descriptions generated by the model typically already have an inherent logical order, which the system directly preserves. Then, the transitions between action groups need to be handled. The system checks whether the last intermediate description of the previous action and the first intermediate description of the next action transition smoothly in posture. In some implementations, a simple logical rule or a quick check using the model can be introduced to ensure reasonable transitions; for example, the ending posture of "walking" (e.g., "alternating support with both feet, body upright") should naturally transition to the starting posture of "waving" (e.g., "keeping standing, right arm begins to rise from the side of the body"). Finally, all intermediate descriptions are concatenated into a linear list or sequence according to their global temporal order. Each item in this sequence is assigned a sequential index and may be accompanied by a relative timestamp or stage label. Before output, the system may perform minor formatting on the sequence, such as unifying the language style of the descriptions and ensuring syntactic completeness, but maintaining its core semantic content. Thus, this structured, fine-grained sequence of motion descriptions serves as the final form of the text domain motion chain, providing a precise, temporal text script for the next stage of visualization transformation.

[0055] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed schematic diagram of step S30 in the first embodiment, based on the shown... Figure 3 The step of generating intermediate image frames corresponding to each action description based on the reference image and the fine-grained action description sequence includes S31-32: Step S31: The fine-grained action description sequence is split into multiple action descriptions, and the multiple action descriptions and the reference image are input into the image generation model; Step S32: Using the image generation model, based on the appearance information of the person in the reference image and the action description, iterative denoising is performed in the latent space to generate intermediate image frames based on action posture.

[0056] In this embodiment, the fine-grained action description sequence is a text list containing multiple independent action description units. The primary operation is to parse the sequence, splitting it into discrete, independently processable action descriptions. Each action description is a complete natural language phrase, such as "left arm extended forward to a horizontal position" or "right foot takes a large step forward while shifting body weight forward." The splitting process is based on a preset delimiter or sequential index in the sequence, ensuring that each description unit is fully extracted without losing semantic information. After splitting, the system obtains an ordered set of action descriptions.

[0057] Subsequently, each individual action description needs to be paired with the original reference image to form a set of conditional input pairs. The image generation model processes multiple such input pairs. The reference image, as a common condition for all generation, is responsible for providing stable character identity and appearance features, defining an invariant visual anchor point in the latent space. Each specific action description, as a variable condition, is responsible for guiding differentiated pose changes during generation. The system feeds the reference image and the currently processed action description text into the encoder module of the image generation model through a data loading pipeline. The reference image undergoes a visual encoder to extract depth features, which are mapped to the model's latent space representation. The action description text is then converted into a series of text embedding vectors by a text encoder. These two types of conditional information are fused within the model, jointly forming a joint conditional context that guides the subsequent denoising process, preparing for the generation of pose images that strictly correspond to specific text descriptions.

[0058] After obtaining the conditional context for fusion, the image generation model performs its core iterative denoising process to synthesize intermediate image frames. This process takes place in the model's latent space, a lower-dimensional and more compact feature representation space than the original pixel space. Generation begins with a latent variable tensor filled with random Gaussian noise corresponding to the size of the target image. The noise tensor does not possess any meaningful visual information.

[0059] The denoising process consists of a series of discrete time steps. At each time step, the model's noise prediction network receives three key inputs: the noise latent variable at the current moment, the time step embedding representing the denoising process, and the conditional embedding formed by fusing features from the reference image and the action description text. The noise prediction network, typically a U-Net architecture, is tasked with predicting the "noise" components that should be removed from the current noise latent variable. The conditional embedding plays a crucial guiding role here; it influences the network's prediction direction through cross-attention mechanisms or feature concatenation fusion, ensuring that the predicted "signal" conforms to both the appearance constraints of the reference image and the pose constraints of the action description.

[0060] The network iteratively performs prediction and denoising operations. At each step, based on the predicted noise, the latent variables are updated via a specific sampler, moving them in the direction of the conditional data distribution. After dozens or even hundreds of such iterations, the initial random noise is gradually "sculpted" into a clear and meaningful latent feature representation. This final latent feature no longer contains noise but encodes information about a human image with a specific appearance and pose. Finally, this clean latent feature is fed into the model's decoder, which is responsible for reconstructing it from the latent space back to the high-resolution pixel space, outputting a complete intermediate image frame. The person in this frame maintains a highly consistent appearance with the reference image, while their body pose accurately corresponds to the input action description text, thus achieving a reliable conversion from text to specific visual pose.

[0061] In addition, based on the above Figure 2 The content of step S32 shown, which involves using the image generation model to iteratively denoise in the latent space based on the appearance information of the person in the reference image and the action description, to generate intermediate image frames based on action poses, includes S32-1 to S32-2: Step S32-1: Dynamically adjust the sampling step number or guiding scale of the iterative denoising according to the complexity and amplitude of the action description. Specifically, set a lower sampling step number and guiding scale for actions with small amplitude, and set a higher sampling step number and guiding scale for actions with large amplitude or complex posture. Step S32-2: Based on the adjusted sampling steps or guiding scale, perform iterative denoising to generate intermediate image frames based on action pose.

[0062] In this embodiment, to improve the balance between generation efficiency and quality, a dynamic adjustment mechanism for key parameters of iterative denoising is introduced. The core of this dynamic adjustment mechanism is to analyze each action description, assess its "complexity" and "motion amplitude," and adaptively configure the sampling steps and classifier guidance scale of the denoising process accordingly. Complexity assessment may be based on the lexical features of the action description text, the number of body parts involved, and the unconventionality of the posture. For example, "standing and smiling" involves subtle facial muscle changes and a standard standing posture, and is assessed as low complexity; while "performing a ballet fouetté turn" involves full-body coordination, high-speed rotation, and a specific dance posture, and is assessed as high complexity. Motion amplitude can be roughly estimated by analyzing the motion verbs in the description (such as "fine-tuning," "stretching," "jumping") or their semantic distance from the standard posture.

[0063] For poses assessed as having small amplitude of motion or low complexity, a lower sampling step count and a lower guiding scale are set. Fewer sampling steps mean faster generation speed because the number of denoising iterations is reduced. At the same time, a lower guiding scale weakens the constraint of text conditions on the generation process, which is suitable for poses that do not deviate much from the basic appearance (standing), and can avoid image artifacts or stiffness caused by overcorrection.

[0064] Conversely, for actions assessed as having large amplitude of motion or complex postures, a higher sampling step count and a higher guidance scale are set. More sampling steps provide the model with a more refined and progressive opportunity to remove noise and shape complex postures, contributing to the generation of accurate limb angles, spatial relationships, and details. Simultaneously, a higher guidance scale strengthens the control of the action description text over the generation process, forcing the model to more strictly follow the text instructions and ensuring the accuracy of generating large-amplitude or atypical postures. This dynamic adjustment strategy allows the system to avoid processing all actions with fixed, costly parameters, instead allocating resources according to task requirements, improving overall processing efficiency while ensuring keyframe quality.

[0065] After determining the optimized parameters for the current action description, iterative denoising is performed based on these adjusted sampling steps and guiding scale to generate the final intermediate image frame. The adjusted parameters are specifically configured in the denoising sampler and the conditional guiding module. If a deterministic sampler such as DDIM is used, reducing the number of sampling steps means selecting sparser points for skip sampling along the same trajectory from noise to a clear image, thus accelerating generation; increasing the number of steps means performing denser, progressive sampling, resulting in a smoother trajectory and more accurate generation.

[0066] Adjusting the guide scale directly affects the weights of conditional embeddings in the noisy prediction network. A higher guide scale amplifies the influence of conditional embeddings in cross-attention computation or feature modulation, increasing the generation process's sensitivity to the action description text. At each step of denoising, when calculating predicted noise, the network considers more "how to generate an image that matches the text description," making it more likely to overcome the bias of initial random noise and generate poses highly consistent with the description, even those with large amplitudes or high difficulty. A lower guide scale, on the other hand, makes the generation process more inclined to preserve the appearance prior and natural image statistical properties inherited from the reference image, suitable for generating poses that require only minor adjustments.

[0067] The complete iterative denoising loop is initiated using the configured parameters. At each time step, the noise prediction network makes predictions based on the currently adjusted guidance strength and conditional information. The sampler updates the latent variables according to the set step plan. The entire process runs within the adjusted parameter framework, ultimately outputting a denoised latent feature. This feature is then decoded to obtain the final generated intermediate image frame. Because the parameters are optimized for the current action description, the generated images achieve a targeted balance between quality and efficiency: simple poses are generated quickly, while complex poses require more computational resources to ensure accuracy. All action descriptions sequentially pass through this dynamically parameterized generation process, collectively producing a complete sequence of intermediate image frames.

[0068] Furthermore, you can also view Figure 4 , Figure 4 This is a detailed schematic diagram of step S40 in the first embodiment, based on the shown... Figure 4 The step of extracting human skeletal key point data from the intermediate image frame to form a keyframe skeletal sequence includes S41-42: Step S41: Using a pre-trained human pose estimation model, detect the two-dimensional pixel coordinates based on human skeletal key points in the intermediate image frame; Step S42: The two-dimensional pixel coordinates are converted into normalized three-dimensional skeletal coordinates according to the camera model, and the three-dimensional skeletal coordinates are organized according to the standard human skeletal topology so that the three-dimensional skeletal coordinates extracted from the intermediate image frames corresponding to each action description are arranged in the action order to form a keyframe skeletal sequence.

[0069] In this embodiment, the first step in extracting human skeletal keypoint data is to perform accurate human pose estimation on intermediate image frames. This process is accomplished using a pre-trained human pose estimation model, typically based on a deep convolutional neural network or a visual Transformer architecture, and trained on a large-scale dataset containing human annotations. The intermediate image frames, as input to the model, may first undergo preprocessing, such as size normalization, to adapt to the model's input requirements. The core task of the model is to detect the positions of the major anatomical joints of the human body in the image.

[0070] The model executes a two-stage detection process. First, it performs human instance detection, locating the bounding box of the person in the image to ensure subsequent keypoint detection focuses on the correct region. Then, within the bounding box region, the model performs keypoint detection. This is done by outputting a series of predefined points on the image, corresponding to standard human joints. A common setup includes 17 points, covering the head (nose, eyes, ears), torso (neck, shoulders, hips), and limbs (elbows, wrists, knees, ankles). For each detected keypoint, the model outputs its two-dimensional pixel coordinates in the original image coordinate system. These coordinates are typically represented as (x, y) pairs, where x represents the horizontal pixel position and y represents the vertical pixel position. Additionally, the model outputs a confidence score for each keypoint, reflecting the model's confidence in predicting the point's location. The system typically sets a confidence threshold to filter out low-confidence, potentially unreliable detection points due to occlusion, image blur, or extreme poses. Finally, for each intermediate image frame, this step produces a set of multiple two-dimensional pixel coordinates, each corresponding to a specific human joint.

[0071] After obtaining the two-dimensional pixel coordinates, they need to be transformed into a standardized data format more suitable for representing motion in three-dimensional space and subsequent temporal processing. Two-dimensional pixel coordinates only contain planar information and lack depth dimension, making their direct use in describing three-dimensional motion limited. Therefore, based on a pre-defined camera model or a learning-based depth estimation module, these two-dimensional coordinates are converted into normalized three-dimensional skeletal coordinates. The camera model provides the geometric constraints for backprojecting from the two-dimensional image plane to three-dimensional space. By assuming a standard human body size or using monocular depth estimation techniques, a relative depth value (z-coordinate) can be estimated for each two-dimensional keypoint, thus forming (x, y, z) three-dimensional coordinates. These three-dimensional coordinates are typically normalized, for example, by scaling with the center of the hip or the midpoint of the spine as the origin, or by scaling with a certain bone length as a reference, to eliminate scale differences caused by different human body shapes and image shooting distances, making the skeletal data more universal and comparable.

[0072] Subsequently, these discrete 3D joints need to be organized according to a standard human skeletal topology. This topology defines the connections between joints; for example, the left wrist connects to the left elbow, the left elbow to the left shoulder, and the left shoulder to the center of the torso. The organization process involves connecting the disordered set of coordinate points according to this predefined parent-child hierarchical relationship to form a complete skeletal diagram or skeleton. This skeleton not only contains the positions of the points but also the connection information between them, which can intuitively represent the human posture. After completing the above processing for all intermediate image frames, each frame corresponds to a structured 3D skeletal data object. Finally, the extracted and processed 3D skeletal data objects are arranged strictly according to the chronological order of the original fine-grained motion descriptions corresponding to the intermediate image frames. This ordered set of 3D skeletal data objects arranged in chronological order is the final keyframe skeletal sequence. This sequence precisely records the 3D posture configuration of the human body at each key time point throughout the entire multi-motion process in numerical form.

[0073] Furthermore, you can also view Figure 5 , Figure 5 This is a detailed schematic diagram of step S50 in the first embodiment, based on the shown... Figure 5 The step of performing temporal interpolation and smoothing on the keyframe skeleton sequence to obtain the visual skeleton sequence includes S51-53: Step S51: Based on the time interval and coordinate data between adjacent keyframe bones, a linear interpolation algorithm is used to calculate the bone coordinates of the intermediate transition frame, and an initial bone sequence with continuous time is generated based on the bone coordinates and the keyframe bones. Step S52: The initial bone sequence is smoothed using a time-domain filtering algorithm to obtain an optimized smooth bone sequence; Step S53: Perform length alignment and format regularization on the smooth bone sequence, and output the visual bone sequence.

[0074] In this embodiment, the keyframe skeleton sequence only describes the pose of the action sequence at several discrete key moments. By calculating and filling the poses between keyframes, a temporally continuous initial skeleton sequence is generated. This process relies on a temporal interpolation algorithm, with linear interpolation being widely used due to its simplicity and efficiency. The input is the keyframe skeleton sequence and the timestamp information corresponding to each keyframe.

[0075] The interpolation process is carried out independently between each pair of adjacent key frames. Suppose the timestamps of adjacent key frames are \(t_k\) and \(t_{k + 1}\) respectively, and the corresponding sets of 3D skeleton coordinates are \(S_k\) and \(S_{k + 1}\). It is necessary to insert \(N\) intermediate transition frames between these two time points at a higher frame rate required for the final video. For any time point \(t\) to be inserted (satisfying \(t_k < t < t_{k + 1}\)), the corresponding skeleton coordinates \(S_t\) are independently calculated for each 3D coordinate component (\(x\), \(y\), \(z\)) of each joint point through the linear interpolation formula. The specific calculation formula is: \(S_t = S_k+\frac{(t - t_k)}{(t_{k + 1}-t_k)} (S_{k + 1}-S_k)\). This means that the intermediate position of a certain joint point is considered to be the result of a linear movement at a time ratio between the positions of this joint point in the two adjacent key frames.

[0076] By performing the above calculations between all pairs of adjacent key frames and ensuring that the entire timeline is evenly covered, the system generates an initial skeleton sequence with a time resolution consistent with the target video frame rate. The skeleton frames in this initial skeleton sequence are continuous and equally spaced in time. However, since linear interpolation assumes that the joint moves in a uniform straight line between two key points, there may be differences from the circular motion of real human joints, and the estimation errors of the key frames themselves may also be transmitted during the interpolation process. Therefore, this initial skeleton sequence may contain slight kinks or unnatural jitters in the motion trajectory, appearing not smooth enough.

[0077] To make the motion more natural and smooth and conform to biomechanical characteristics, it is necessary to smooth the initial skeleton sequence. This step is achieved through a time-domain filtering algorithm, whose core idea is to use the information of adjacent frames to correct the data of the current frame, thereby suppressing high-frequency noise and unreasonable mutations. Gaussian smoothing filtering is a commonly used option.

[0078] Gaussian smoothing performs a convolution operation in the time dimension through a one-dimensional Gaussian kernel function. This Gaussian kernel is essentially a weight window, with the largest weight at the center of the window and symmetric attenuation towards both ends. For a certain coordinate component (such as the \(x\) coordinate) of a certain joint point in the \(t\)-th frame of the initial skeleton sequence, the corresponding coordinate values within a certain time window before and after it are taken, multiplied by the weights at the corresponding positions in the Gaussian kernel respectively, and then summed to obtain the new smoothed value of this coordinate component. This process is carried out independently for each coordinate component (\(x\), \(y\), \(z\)) of each joint point in each frame of the sequence.

[0079] The standard deviation parameter σ of the Gaussian kernel controls the intensity of smoothing. A larger σ value results in a wider time window for weighted averaging, leading to a more significant smoothing effect and effectively filtering out severe jitter. However, excessive smoothing can also weaken realistic details of fast motion. Therefore, a suitable σ value (e.g., corresponding to a time span of several frames) needs to be selected based on prior knowledge of the action (such as the range of motion speed) and visual requirements. After Gaussian filtering, the motion trajectories of the joints become smoother, eliminating abrupt transitions and minor jumps caused by linear interpolation or estimation errors. The resulting skeletal sequence visually approximates the continuity and smoothness of real human motion; this processed sequence is the optimized smoothed skeletal sequence.

[0080] The smoothed skeleton sequence may require further adjustments in terms of duration and data format to fully match the input specifications of the downstream video diffusion model. Step S53 is responsible for performing this length alignment and format normalization. First, length alignment is performed, as video diffusion models typically require the input conditional sequence (such as a skeleton sequence) to have a fixed duration T (i.e., number of frames). The system checks whether the current frame number of the smoothed skeleton sequence is equal to T.

[0081] If the number of frames in the sequence exceeds T, pruning is required. Pruning can be done by symmetrically removing redundant frames from the beginning and end of the sequence, or by downsampling based on temporal importance. If the number of frames in the sequence is less than T, padding is required. Padding can be done by using the last frame or the first frame of a repeating sequence, or by more precisely by extrapolating at the end or beginning of the sequence to generate additional frames, ensuring the total length reaches T. The goal is to ensure the sequence length strictly conforms to the model requirements without introducing significant motion abrupt changes.

[0082] Next, format normalization is performed. The video diffusion model expects the input skeletal conditions to be a tensor with a specific shape and numerical range. The system needs to convert the smooth skeletal sequence, typically a list or array containing frame number, joint number, and 3D coordinates, into the tensor format specified by the model. This conversion operation includes adjusting the dimensional order (e.g., from [frame number, joint number, 3] to [batch size, channel number, frame number, joint number, 3]), performing a final normalization of the coordinate values ​​(e.g., scaling to the [-1, 1] range), and converting to the data type required by the model's operating framework (e.g., float32). After all format conversions are completed, the output data structure is the final visual skeletal sequence. This sequence is a standardized, continuous, smooth, and uniformly formatted motion guidance signal that can be directly embedded as a spatial condition and input into the video diffusion model to precisely control the pose of the character in each frame of the generated video.

[0083] Furthermore, you can also view Figure 6 , Figure 6 This is a detailed schematic diagram of step S60 in the first embodiment, based on the shown... Figure 6 The steps of loading the visual skeleton sequence into the denoising generation process of the video diffusion model through a pose guidance adapter, and integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human motion video matching the multi-action text prompt, include S61-62: Step S61: At each time step of the denoising process, the visual skeleton sequence data corresponding to the current time step is encoded into a spatial conditional embedding. Step S62: The spatial conditions are embedded into the weight offset generated by the pose guidance adapter and fused with the noise prediction features in the denoising network to guide the alignment of the generated character pose with the visual skeleton sequence.

[0084] In this embodiment, during the iterative denoising process of the video diffusion model, the visual skeleton sequence, as a crucial spatial guiding condition, needs to be injected into the generator network in real time, frame by frame. The encoding and preparation of this conditional information are completed at each denoising time step. The visual skeleton sequence is a data stream containing the three-dimensional skeleton coordinates of T frames, where T corresponds to the frame number of the target video. When the denoising process reaches the generation of the t-th frame of the simulated video, the system needs to extract the skeleton data corresponding to the t-th frame time point from the visual skeleton sequence.

[0085] The extracted skeletal data of the current frame is a collection of three-dimensional coordinates of multiple joints. This data needs to be encoded into a representation that aligns with the feature space within the diffusion model. The system uses a separate skeletal encoder to accomplish this task, typically a small feedforward neural network or a lightweight Transformer. The skeletal encoder receives the skeletal coordinates of the current frame, may first flatten them into a feature vector, then perform a nonlinear transformation through several fully connected layers, ultimately outputting a dense vector of fixed dimensions, known as the spatial conditional embedding.

[0086] This spatial conditional embedding captures the geometric configuration information of the human pose at the current time point. The embedding's dimension is designed to match the key-value feature dimension of the cross-attention layer used to fuse textual conditions in the video diffusion model, or to be compatible with the input dimension of certain feature modulation layers. In this way, the spatial conditional embedding becomes a conditional signal representing specific pose requirements that can be seamlessly inserted into the model's existing computational flow, preparing it for the next step of influencing the generation process through an adapter.

[0087] The core of the key mechanism for embedding spatial conditions into the pose information and effectively integrating it into the noise prediction calculation of the video diffusion model is the pose guidance adapter. This adapter is integrated into specific network layers of the pre-trained video diffusion model, such as the attention module or feedforward network layer of U-Net, in the form of a low-rank factorization matrix. Low-rank adaptation technology allows the model behavior to be adjusted by training only a very small number of new parameters without destroying its original powerful generative capabilities.

[0088] After obtaining the spatial conditional embedding, the pose guidance adapter begins to function. The adapter internally contains trainable low-rank matrix pairs. The spatial conditional embedding first maps it from the original dimension to a low-rank space using a down-projection matrix. Then, this low-rank representation is passed through a non-linear activation function and mapped back to the same dimension as the original layer weight changes using an up-projection matrix. This series of operations ultimately generates a weight offset ΔW for a specific network layer (such as the query, key, and value projection matrices of a particular attention layer).

[0089] During the forward propagation of the noise prediction network, for layers integrating the adapter, their original weights W are combined with the weight offset ΔW generated by the adapter, forming the effective weights W' = W + ΔW for that layer at that time step. The weight offset ΔW is a function of the current spatial conditional embedding; therefore, the network parameters are fine-tuned at each time step according to the current pose requirements. When the latent noise features pass through this layer, their transformation process is guided by the pose information carried by ΔW. This guidance is a deep fusion at the feature level, causing the network to tend to output image features that match the pose specified by the spatial conditional embedding when predicting noise to be removed. By repeating this mechanism in multiple key layers throughout the denoising process, the pose of the person in the generated video is progressively and strongly constrained to align with the visual skeleton sequence, achieving precise pose control.

[0090] In addition, based on the above Figure 6 The content of step S62 shown is further refined. The step of loading the visual skeleton sequence into the denoising generation process of the video diffusion model through the pose guidance adapter, and integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human motion video matching the multi-action text prompt, includes S62-1 to S62-3: Step S62-1: Calculate the motion velocity and acceleration of the joints in the intermediate transition frames between adjacent keyframe bones in the visual skeleton sequence, and determine whether there is abnormal motion based on the calculation results. Abnormal motion is abnormal movement that exceeds the physiological limits of the human body. Step S62-2: If the abnormal motion is determined to exist, the keyframe bone where the abnormal motion is located is determined in the visual skeleton sequence, and the bone coordinates of the keyframe bone are locally re-interpolated or smoothed to obtain the corrected visual skeleton sequence. Step S62-3: Based on the corrected visual skeleton sequence, reintegrate the visual skeleton sequence into the denoising generation process of the video diffusion model.

[0091] In this embodiment, to ensure the physical plausibility of the visual skeleton sequence and the realism of the generated video, an abnormal motion detection mechanism for the skeleton sequence is introduced. This step is performed before the visual skeleton sequence is input into the video diffusion model or during its final verification. Abnormal motion is defined as movements that exceed the normal physiological limits of the human body, such as excessive joint rotation angles or limb movement speeds or accelerations that violate biomechanical principles.

[0092] The detection process begins by calculating the kinematic parameters of joints based on the 3D coordinate data of the joints in the visual skeleton sequence. The system iterates through each frame of the sequence, and for each joint, calculates its displacement vector from the current frame to the next frame. Combining this with the known inter-frame time intervals, the system calculates the average velocity of the joint within the current time interval. Furthermore, the instantaneous acceleration of the joint can be estimated based on the position data of three consecutive frames. These calculations are performed on all joints across all inter-frame intervals.

[0093] After obtaining velocity and acceleration data, the system compares them with a preset human motion physiology limit threshold library. This threshold library defines the reasonable velocity, acceleration, and angle ranges for each major joint in different motion directions. For example, the flexion-extension angular velocity of the knee joint and the abduction velocity of the shoulder joint have reasonable upper limits. If the system detects that the velocity or acceleration of a joint continuously exceeds its corresponding threshold within a certain time period, or if the posture change between adjacent keyframes causes the joint angle to exceed the reasonable anatomical range, the system determines that abnormal motion exists. The determination result will mark the time period in which the abnormality occurred and the specific joint involved.

[0094] Once abnormal motion is detected, a correction process is initiated to eliminate unreasonable data points and generate a smoother, more natural visual skeleton sequence. The system first locates the root cause of the problem based on the anomaly markers. Usually, the anomaly is caused by inaccurate estimation of the coordinates of one or more keyframe bones, or by unreasonable motion path planning between these keyframes during temporal interpolation.

[0095] The correction strategies are mainly divided into two types: local re-interpolation and smooth correction. If the anomaly is determined to be concentrated in a few frames of the transition segment between two adjacent keyframes, a local re-interpolation strategy may be adopted. Specifically, the system will temporarily exclude the start and end keyframes marked as abnormal transitions (or use more normal keyframes before and after them as new references), and between these two corrected reference frames, use different interpolation parameters (such as using spline interpolation instead of linear interpolation) or increase the density of intermediate transition frames to recalculate the skeletal coordinates of the transition segment.

[0096] If the anomaly manifests as widespread high-frequency jitter or small, unreasonable offsets throughout the sequence, a smoothing correction strategy is employed. This is equivalent to applying a stronger, more targeted temporal filter to the data of the anomalous joint within the anomalous time period, building upon the initial smoothing process. For example, a wider-window Gaussian smoothing can be applied to the joint's coordinate trajectory, or a Kalman filter based on a kinematic model can be used to force the trajectory to conform to the physical constraints of motion continuity and smoothness. The correction process is performed only on the data identified as anomalous, preserving as much of the reasonable motion information in the original sequence as possible. After correction, a new visual skeleton sequence is obtained, eliminating the identified anomalous motions.

[0097] Based on the generated corrected visual skeleton sequence, the core video generation process is restarted or continues. The corrected sequence, while maintaining the overall timing and macroscopic form of the original movements, has motion details that are more consistent with human dynamics, providing more reliable guidance for generating high-quality video.

[0098] The corrected visual skeleton sequence, along with the original multi-action text cues and reference images, is again provided as a conditional input set to the pre-trained video diffusion model. The entire denoising generation process, including the aforementioned process of fusing spatial conditional embeddings through a pose-guided adapter, will be re-executed based on this corrected, improved skeleton sequence.

[0099] If anomaly detection and correction are completed before final generation begins, the corrected sequence is used directly for a one-time generation. If the mechanism is designed as a feedback loop in the generation process—that is, after generating the initial result, analyzing its implicit motion trajectory and detecting anomalies—the system may need to locally adjust or regenerate relevant segments of the generated video latent representation based on the corrected skeletal sequence. Regardless of the integration method used, the ultimate goal is to ensure that the pose guidance signal driving video generation is physically reasonable and visually natural, thereby significantly reducing the probability of illogical phenomena such as distorted limbs and bizarre movements in the final generated video, and improving the overall realism and credibility of the generated results.

[0100] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multi-action human body video generation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0101] This application provides a multi-motion human body video generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-motion human body video generation method in the above embodiment 1.

[0102] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a multi-motion human video generation device suitable for implementing embodiments of this application. The multi-motion human video generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The multi-motion human video generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0103] like Figure 8As shown, the multi-motion human video generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multi-motion human video generation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the multi-motion human video generation device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show multi-motion human video generation devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0104] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0105] The multi-motion human video generation device provided in this application, employing the multi-motion human video generation method described in the above embodiments, can solve the technical problems of temporal disorder and poor consistency in the generation of multi-motion videos by existing methods. Compared with the prior art, the beneficial effects of the multi-motion human video generation device provided in this application are the same as those of the multi-motion human video generation method provided in the above embodiments, and other technical features in this multi-motion human video generation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0106] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0107] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0108] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multi-action human body video generation method in the above embodiments.

[0109] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0110] The aforementioned computer-readable storage medium may be included in the multi-motion human body video generation device; or it may exist independently and not be assembled into the multi-motion human body video generation device.

[0111] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the multi-action human body video generation device, enable the multi-action human body video generation device to implement the technical content of the multi-action human body video generation method embodiment shown above.

[0112] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0115] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multi-action human video generation method, which can solve the technical problems of disordered timing and poor consistency in the generation of multi-action videos by existing methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the multi-action human video generation method provided in the above embodiments, and will not be repeated here.

Claims

1. A method for generating multi-motion human body videos, characterized in that, The method for generating multi-motion human videos includes the following steps: Obtain reference images and multi-action prompt text; The reference image and the multi-action prompt text are input into a multimodal large language model, and a structured fine-grained action description sequence is obtained through the multimodal large language model; Based on the reference image and the fine-grained action description sequence, an intermediate image frame corresponding to each action description is generated; Extract human skeleton key point data from the intermediate image frame to form a key frame skeleton sequence; The keyframe skeleton sequence is subjected to temporal interpolation and smoothing to obtain a visual skeleton sequence; The visual skeleton sequence is loaded into the denoising generation process of the video diffusion model through a pose guidance adapter. Based on the denoising generation process, the reference image is integrated into the visual skeleton sequence to generate a coherent human motion video that matches the multi-action text prompt.

2. The multi-motion human body video generation method as described in claim 1, characterized in that, The step of performing chain-like reasoning on the multi-action prompt text based on the reference image and preset motion prior knowledge using the multimodal large language model to output a structured, fine-grained sequence of action descriptions includes: The reference image is converted into a visual feature code, and the visual feature code is deployed as an embedded representation of the multimodal large language model; Obtain a comprehensive prompt, input the comprehensive prompt into the multimodal large language model, and activate the motion prior knowledge of the multimodal large language model; The multi-action prompt text is parsed and reasoned using the prior motion knowledge and the embedding representation to generate multiple intermediate descriptions corresponding to each action; The intermediate descriptions are organized in chronological order to form the fine-grained action description sequence.

3. The multi-motion human body video generation method as described in claim 1, characterized in that, The step of generating intermediate image frames corresponding to each action description based on the reference image and the fine-grained action description sequence includes: The fine-grained action description sequence is split into multiple action descriptions, and the multiple action descriptions and the reference image are input into the image generation model; Using the image generation model, based on the appearance information of the person in the reference image and the action description, iterative denoising is performed in the latent space to generate intermediate image frames based on action posture.

4. The method for generating multi-motion human body video according to claim 3, characterized in that, The step of generating intermediate image frames based on the action pose by iteratively denoising in the latent space using the image generation model, based on the appearance information of the person in the reference image and the action description, includes: Based on the complexity and amplitude of the action description, the sampling step number or guidance scale of the iterative denoising is dynamically adjusted. Specifically, a lower sampling step number and guidance scale are set for actions with small amplitude, and a higher sampling step number and guidance scale are set for actions with large amplitude or complex posture. Iterative denoising is performed based on the adjusted sampling steps or guiding scale to generate intermediate image frames based on action pose.

5. The method for generating multi-motion human body video according to claim 1, characterized in that, The step of extracting human skeletal key point data from the intermediate image frame to form a keyframe skeletal sequence includes: A pre-trained human pose estimation model is used to detect the two-dimensional pixel coordinates based on human skeletal key points in the intermediate image frames; The two-dimensional pixel coordinates are converted into normalized three-dimensional skeletal coordinates according to the camera model, and the three-dimensional skeletal coordinates are organized according to the standard human skeletal topology so that the three-dimensional skeletal coordinates extracted from the intermediate image frames corresponding to each action description are arranged in the action order to form a keyframe skeletal sequence.

6. The method for generating multi-motion human body video according to claim 1, characterized in that, The step of performing temporal interpolation and smoothing on the keyframe skeleton sequence to obtain a visual skeleton sequence includes: Based on the time interval and coordinate data between adjacent keyframe bones, a linear interpolation algorithm is used to calculate the bone coordinates of intermediate transition frames, and an initial bone sequence with continuous time is generated based on the bone coordinates and the keyframe bones. The initial bone sequence is smoothed using a time-domain filtering algorithm to obtain an optimized smooth bone sequence. The smooth bone sequence is length-aligned and formatted to output the visual bone sequence.

7. The method for generating multi-motion human body video according to claim 1, characterized in that, The step of loading the visual skeleton sequence into the video diffusion model via a pose guidance adapter for denoising generation, and then integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human motion video matching the multi-action text prompt, includes: At each time step in the denoising process, the visual skeleton sequence data corresponding to the current time step is encoded into a spatial conditional embedding; The spatial conditions are embedded into the weight offsets generated by the pose guidance adapter and fused with the noise prediction features in the denoising network to guide the alignment of the generated character pose with the visual skeleton sequence.

8. The method for generating multi-motion human body video according to claim 1, characterized in that, The step of loading the visual skeleton sequence into the video diffusion model via a pose guidance adapter for denoising generation, and then integrating the reference image into the visual skeleton sequence based on the denoising generation process to generate a coherent human motion video matching the multi-action text prompt, includes: Calculate the motion velocity and acceleration of the joints in the intermediate transition frames between adjacent keyframe bones in the visual skeleton sequence, and determine whether there is abnormal motion based on the calculation results. Abnormal motion is an abnormal movement that exceeds the physiological limits of the human body. If the abnormal motion is determined to exist, the keyframe bone where the abnormal motion is located is determined in the visual skeleton sequence, and the bone coordinates of the keyframe bone are locally re-interpolated or smoothed to obtain the corrected visual skeleton sequence. Based on the corrected visual skeleton sequence, the visual skeleton sequence is reloaded into the denoising generation process of the video diffusion model.

9. A multi-motion human body video generation device, characterized in that, The multi-motion human video generation device stores a computer program, which, when executed by a processor, implements the multi-motion human video generation method according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the multi-action human video generation method according to any one of claims 1-8.

Images