Motion video generation method, model training method, and device

Abstract

This application relates to the interdisciplinary fields of computer vision, natural language processing, and generative models. It provides a method for generating motion videos, a model training method, and an apparatus to overcome the shortcomings of related text-to-motion synthesis techniques in terms of cross-modal semantic alignment accuracy and motion generation stability. First, natural language text and initial video data are input into the video branch of a two-branch joint generation model to obtain a first video latent variable. Then, the natural language text, initial video data, and the first video latent variable are input into the motion branch of the two-branch joint generation model to obtain a first fused latent variable output by the motion branch. Finally, a target video is generated based on the first video latent variable. Furthermore, a target motion sequence corresponding to the natural language text is generated based on the first fused latent variable. By fusing video features and motion features, the cross-modal semantic alignment accuracy is enhanced, enabling the generation of more accurate motion sequences.

Description

Technical Field

[0001] This application relates to the technical field of computer vision, natural language processing and generative models, and in particular to a motion video generation method, model training method and device. Background Technology

[0002] Text-to-motion generation technology is one of the core research directions in the interdisciplinary field of computer vision, natural language processing, and generative models. It aims to automatically generate semantically consistent and temporally coherent 3D human motion sequences based on natural language descriptions. It has irreplaceable application value in fields such as virtual reality, video game development, film and television special effects production, and digital human interaction. For example, in video game development, developers input natural language text, and the generative model generates motion sequences based on the natural language text.

[0003] Current mainstream text-to-motion synthesis technologies are mainly developed based on two technical paths: one is generative model methods that rely on text-motion pairing datasets, including end-to-end generation schemes based on autoregressive and diffusion models, which achieve synthesis by learning the mapping relationship between text descriptions and motion sequences; the other is mediation methods that utilize cross-modal alignment tools, such as building image mediation bridges based on vision-language pre-trained models like CLIP, or using large language models (LLMs) to parse complex text into motion primitives before sequence generation. These technologies have made some progress in simple action description scenarios (such as "walking" or "waving"), but the following key problems and shortcomings still exist in practical applications: Manually annotated text-motion pairing datasets generally suffer from small size, poor diversity, and simple descriptions. These limitations lead to insufficient generalization ability of generative models. Building image mediation bridges based on vision-language pre-trained models such as CLIP can easily lead to information loss; Large Language Models (LLMs) are prone to semantic bias when parsing complex text, resulting in broken action connections in motion sequences, incomplete semantic expression, and unstable generated motion sequences.

[0004] Therefore, text-to-motion synthesis technology in related technologies has shortcomings in terms of adaptability to complex scenes, cross-modal semantic alignment accuracy, and motion generation stability. Summary of the Invention

[0005] The purpose of this application is to provide a motion video generation method, model training method, and device to address the shortcomings of text-to-motion synthesis technology in related technologies in terms of adaptability to complex scenes, cross-modal semantic alignment accuracy, and motion generation stability.

[0006] In a first aspect, this application provides a method for generating motion videos, the method comprising: The natural language text and initial video data are input into the video branch of the two-branch joint generation model. The natural language text and initial video data are processed through the video branch to obtain the first video latent variable. The natural language text, the initial video data, and the first video latent variable are input into the motion branch of the dual-branch joint generation model to obtain the first fused latent variable output by the motion branch. Based on the first video latent variables, generate the target video corresponding to the natural language text; Based on the first fusion latent variable, a motion sequence corresponding to the natural language text is generated.

[0007] In one possible implementation, generating the motion sequence corresponding to the natural language text based on the first fusion latent variable includes: The first fusion latent variable is normalized by root mean square to obtain the first motion feature; The first motion feature is adjusted by a modulation signal generated based on a time step to obtain a second motion feature; The second motion feature is mapped to the third motion feature through linear projection; The third motion feature is decoded to obtain the motion sequence.

[0008] Secondly, this application provides a model training method, wherein the model is a dual-branch joint generation model, the model including a video branch and a motion branch, and the method includes: Obtain a cross-modal training dataset, which includes multiple sets of samples; each set of samples includes video-guided text, motion-guided text, motion sequence samples, and video clip samples. The video clip samples and video guidance text of the first group of samples are input into the video branch to obtain the second video latent variable obtained by the video branch processing the video clip samples and video guidance text of the first group of samples. The predicted video noise is then determined based on the second video latent variable. The motion sequence samples and motion-guided text of the first group of samples are input into the motion branch. The motion sequence samples of the first group of samples are processed by the motion branch to obtain the first motion latent variable and the motion text features obtained by processing the motion-guided text of the first group of samples. A second fused latent variable is obtained by fusing the first motion latent variable, the motion text feature, and the second video latent variable, so as to determine the predicted motion noise based on the second fused latent variable; The total loss value is determined based on the predicted video noise and predicted motion noise, and the model parameters of the dual-branch joint generation model are updated based on the total loss value.

[0009] In one possible implementation, the first set of samples further includes labeled video noise and labeled motion noise; determining the total loss value based on the predicted video noise and predicted motion noise includes: The predicted video noise is compared with the labeled video noise to determine the video loss value; The predicted motion noise is compared with the label motion noise to determine the motion loss value; The total loss value is determined based on the video loss value and the motion loss value.

[0010] In one possible implementation, processing the motion sequence samples of the first group of samples through the motion branch to obtain the first motion latent variable includes: The motion sequence samples are mapped to high-dimensional feature vectors through the convolutional network in the motion branch, and the discrete vectors corresponding to the high-dimensional feature vectors are determined. The discrete vector is downsampled to obtain the fourth motion feature; The label motion noise is injected into the fourth motion feature to obtain the first motion latent variable.

[0011] In one possible implementation, the step of fusing the first motion latent variable, the motion text feature, and the second video latent variable to obtain the second fused latent variable includes: Enhanced self-attention calculation is performed on the first motion latent variable and temporal information to obtain the fifth motion feature; the temporal information is a sinusoidal position code generated based on time steps. The motion text feature and the second video latent variable are concatenated to obtain the sixth motion feature; the motion text feature and the second video latent variable have the same data dimension. Perform cross-attention calculations on the fifth and sixth motion features to obtain the seventh motion feature; The seventh motion feature is adjusted by adjusting the modulation parameters generated based on the time step to obtain the eighth motion feature; The seventh motion feature and the eighth motion feature are fused by gating transformation to obtain the second fused latent variable.

[0012] In one possible implementation, the cross-modal training dataset is obtained in the following manner: The captured video is segmented to obtain multiple segmented video clips; The multiple segmented video segments are filtered to obtain multiple filtered video segments; For each filtered video segment, a multimodal model is used to process the filtered video segment to generate video guidance text and motion guidance text; a human 3D pose estimation model is used to extract motion sequences from the filtered video segment; the video guidance text, the motion guidance text, the motion sequences, and the filtered video segment are used as a cross-modal training dataset.

[0013] In one possible implementation, the segmentation process of the acquired video to obtain multiple segmented video segments includes: Starting from the initial video frame, if a camera change is detected within the preset video frame length, segmentation is performed at the position of the video frame with the camera change, resulting in segmented video segments and the video to be segmented. If no camera change is detected within the preset video frame length range, the video is segmented at the preset video frame length position to obtain the segmented video segment and the video to be segmented. After segmenting at the video frame position where the camera changes or at a preset video frame length position, the first video frame of the video to be segmented is taken as the starting video frame, and the process returns to the steps starting from the starting video frame.

[0014] In one possible implementation, the filtering process on the plurality of segmented video segments to obtain a plurality of filtered video segments includes: For each segmented video segment, the video quality of the segmented video segment is determined, the optical flow smoothness of the segmented video segment is determined based on the standard deviation of the optical flow field of the segmented video segment, and the text matching degree of the segmented video segment is determined by detecting the text information in the segmented video segment. Video clips that meet preset criteria in terms of video quality, optical flow smoothness, and text matching are used as multiple filtered video clips.

[0015] Thirdly, this application provides a motion video generation apparatus, the apparatus comprising: The video branch module is configured to input natural language text and initial video data into the video branch of the dual-branch joint generation model, and process the natural language text and initial video data through the video branch to obtain the first video latent variable; The motion branch module is configured to input the natural language text, the initial video data, and the first video latent variable into the motion branch of the dual-branch joint generation model to obtain the first fused latent variable output by the motion branch. The video branching module is further configured to generate a target video corresponding to the natural language text based on the first video latent variable; The motion branch module is further configured to generate a target motion sequence corresponding to the natural language text based on the first fusion latent variable.

[0016] In one possible implementation, the motion branch module is specifically configured to generate the motion sequence corresponding to the natural language text based on the first fusion latent variable, wherein the motion branch module is configured as follows: The first fusion latent variable is normalized by root mean square to obtain the first motion feature; The first motion feature is adjusted by a modulation signal generated based on a time step to obtain a second motion feature; The second motion feature is mapped to the third motion feature through linear projection; The third motion feature is decoded to obtain the motion sequence.

[0017] Fourthly, this application provides a model training apparatus, wherein the model is a dual-branch joint generation model, the model including a video branch and a motion branch, and the apparatus includes: The dataset acquisition module is configured to acquire a cross-modal training dataset, which includes multiple sets of samples; each set of samples includes video-guided text, motion-guided text, motion sequence samples, and video clip samples. The video processing module is configured to input video clip samples and video guidance text of the first group of samples into the video branch, obtain the second video latent variable obtained by the video branch processing the video clip samples and video guidance text of the first group of samples, and determine the predicted video noise based on the second video latent variable; The motion processing module is configured to input the motion sequence samples and motion-guided text of the first group of samples into the motion branch, and obtain the first motion latent variable obtained by the motion branch processing the motion sequence samples of the first group of samples, and the motion text features obtained by processing the motion-guided text of the first group of samples. The motion processing module is further configured to fuse the first motion latent variable, the motion text feature, and the second video latent variable to obtain a second fused latent variable, so as to determine the predicted motion noise based on the second fused latent variable; The model update module is configured to determine the total loss value based on the predicted video noise and the predicted motion noise, and update the model parameters of the dual-branch joint generation model based on the total loss value.

[0018] In one possible implementation, the first set of samples further includes labeled video noise and labeled motion noise; the model update module is specifically configured to determine the total loss value based on the predicted video noise and predicted motion noise, wherein: The predicted video noise is compared with the labeled video noise to determine the video loss value; The predicted motion noise is compared with the label motion noise to determine the motion loss value; The total loss value is determined based on the video loss value and the motion loss value.

[0019] In one possible implementation, the motion processing module is specifically configured to process the motion sequence samples of the first group of samples through the motion branch to obtain the first motion latent variable. The motion sequence samples are mapped to high-dimensional feature vectors through the convolutional network in the motion branch, and the discrete vectors corresponding to the high-dimensional feature vectors are determined. The discrete vector is downsampled to obtain the fourth motion feature; The label motion noise is injected into the fourth motion feature to obtain the first motion latent variable.

[0020] In one possible implementation, the second fused latent variable is obtained by fusing the first motion latent variable, the motion text feature, and the second video latent variable. Specifically, the motion processing module is configured as follows: Enhanced self-attention calculation is performed on the first motion latent variable and temporal information to obtain the fifth motion feature; the temporal information is a sinusoidal position code generated based on time steps. The motion text feature and the second video latent variable are concatenated to obtain the sixth motion feature; the motion text feature and the second video latent variable have the same data dimension. Perform cross-attention calculations on the fifth and sixth motion features to obtain the seventh motion feature; The seventh motion feature is adjusted by adjusting the modulation parameters generated based on the time step to obtain the eighth motion feature; The seventh motion feature and the eighth motion feature are fused by gating transformation to obtain the second fused latent variable.

[0021] In one possible implementation, the dataset acquisition module is configured to obtain the cross-modal training dataset in the following manner: The captured video is segmented to obtain multiple segmented video clips; The multiple segmented video segments are filtered to obtain multiple filtered video segments; For each filtered video segment, a multimodal model is used to process the filtered video segment to generate video guidance text and motion guidance text; a human 3D pose estimation model is used to extract motion sequences from the filtered video segment; the video guidance text, the motion guidance text, the motion sequences, and the filtered video segment are used as a cross-modal training dataset.

[0022] In one possible implementation, the process of segmenting the acquired video to obtain multiple segmented video clips, wherein the dataset acquisition module is specifically configured as follows: Starting from the initial video frame, if a camera change is detected within the preset video frame length, segmentation is performed at the position of the video frame with the camera change, resulting in segmented video segments and the video to be segmented. If no camera change is detected within the preset video frame length range, the video is segmented at the preset video frame length position to obtain the segmented video segment and the video to be segmented. After segmenting at the video frame position where the camera changes or at a preset video frame length position, the first video frame of the video to be segmented is taken as the starting video frame, and the process returns to the steps starting from the starting video frame.

[0023] In one possible implementation, the filtering process for the plurality of segmented video segments to obtain a plurality of filtered video segments, wherein the dataset acquisition module is specifically configured as follows: For each segmented video segment, the video quality of the segmented video segment is determined, the optical flow smoothness of the segmented video segment is determined based on the standard deviation of the optical flow field of the segmented video segment, and the text matching degree of the segmented video segment is determined by detecting the text information in the segmented video segment. Video clips that meet preset criteria in terms of video quality, optical flow smoothness, and text matching are used as multiple filtered video clips.

[0024] Fifthly, this application provides an electronic device, comprising: Processor and memory; The memory is used to store the processor-executable instructions; The processor is configured to execute the instructions to implement the motion video generation method as provided in any of the first aspects of this application and the model training method as provided in any of the second aspects.

[0025] In a sixth aspect, this application provides a computer-readable storage medium in which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the motion video generation method as described in any of the first aspects of this application and the model training method as described in any of the second aspects.

[0026] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor of an electronic device, enables the electronic device to perform the motion video generation method as described in any of the first aspects of this application and the model training method as described in any of the second aspects.

[0027] The technical solutions provided by the embodiments of this application have at least the following beneficial effects: This application's embodiments construct a cross-modal training dataset containing text, video, and motion to overcome the shortcomings of traditional text-motion binary datasets, such as small size, insufficient diversity, and lack of coverage of complex scenes. This provides the model with rich cross-modal information and improves the model's generalization ability. By fusing video features and motion features, the accuracy of cross-modal semantic alignment is enhanced, overcoming the ambiguity of text input. By introducing strong spatiotemporal priors from the video generation model, a precise association mechanism among text, video, and motion is established, ensuring that the generated motion sequence is highly consistent with the text description in both global semantics and local details. The spatiotemporal modeling capability of the video generation model is transformed into the performance advantage of the motion generation model, achieving a breakthrough improvement in the quality of motion generation under complex text input.

[0028] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Based on common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this application. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram illustrating an application scenario of a motion video generation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the overall process of a motion video generation method provided in an embodiment of this application; Figure 3 A flowchart illustrating step 204 provided in an embodiment of this application; Figure 4 A schematic diagram of the architecture of the dual-branch joint generation model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the overall process of a model training method provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the process of obtaining a cross-modal training dataset provided in an embodiment of this application; Figure 7 This is a flowchart illustrating step 601 provided in an embodiment of this application; Figure 8 A flowchart illustrating step 602 provided in an embodiment of this application; Figure 9 A schematic diagram of the process for obtaining the first motion latent variable provided in an embodiment of this application; Figure 10 A flowchart illustrating step 504 provided in an embodiment of this application; Figure 11 This is a flowchart illustrating step 505 provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a motion video generation device provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application; Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0032] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0034] The following explains the technical terms and technologies involved in this application: 1. VQ-VAE Motion Encoder: A motion sequence encoding module built on a vector quantization variational autoencoder. Its core function is to map high-dimensional raw 3D human motion sequences into low-dimensional discrete motion tokens. By discretizing the motion latent space through vector quantization, it effectively filters noise from the raw motion data, reduces motion jitter, and compresses motion features into a representation format consistent with the video token dimension. This provides a foundation for cross-modal fusion in a dual-branch framework. Its discretized representation characteristics also improve the stability and controllability of motion generation.

[0035] 2. DiT Architecture: This architecture uses Transformer to replace the convolutional layers in the traditional UNet diffusion model. It decomposes the video into cross-frame spatial patches and uses an attention mechanism to model global spatiotemporal relationships. When processing long video sequences and complex motion scenes, it outperforms UNet-based models in terms of temporal consistency and detail preservation.

[0036] Text-to-motion generation technology is one of the core research directions in the interdisciplinary field of computer vision, natural language processing and generative models. It aims to automatically generate semantically consistent and temporally coherent 3D human motion sequences based on natural language descriptions. It has irreplaceable application value in fields such as virtual reality, video game development, film and television special effects production, and digital human interaction.

[0037] Current mainstream text-to-motion synthesis technologies are mainly developed based on two technical paths: one is generative model methods that rely on text-motion pairing datasets, including end-to-end generation schemes based on autoregressive and diffusion models, which achieve synthesis by learning the mapping relationship between text descriptions and motion sequences; the other is mediation methods that utilize cross-modal alignment tools, such as building image mediation bridges based on vision-language pre-trained models like CLIP, or using large language models (LLMs) to parse complex text into motion primitives before sequence generation. These technologies have made some progress in simple action description scenarios (such as "walking" or "waving"), but the following key problems and shortcomings still exist in practical applications: Manually annotated text-motion pairing datasets generally suffer from problems such as small size, poor diversity, and simple descriptions. The limitations of the datasets lead to insufficient generalization ability of the models. Based on vision-language pre-trained models such as CLIP, motion sequences need to be rendered as static images to achieve text-motion alignment. However, the rendering process leads to the loss of action temporal information and a reduction in diversity, resulting in a large deviation between the generated motion and the semantics of the text. Although text parsing methods based on LLMs decompose complex text, they are limited by the scale of text-motion pairing data. The parsed motion primitives are difficult to accurately map into coherent actions, which easily leads to problems such as broken action connections and incomplete semantic expression, resulting in unstable generated motion sequences. When relying solely on text input as the generation condition, the ambiguity and vagueness of language itself lead to low controllability of motion generation. The same text description may generate vastly different motion sequences, and it is difficult to accurately capture the action details in the user's intent.

[0038] Therefore, text-to-motion synthesis technology in related technologies has shortcomings in terms of adaptability to complex scenes, cross-modal semantic alignment accuracy, and motion generation stability.

[0039] In view of this, this application provides a motion video generation method, a model training method, and an apparatus to address the shortcomings of text-to-motion synthesis technology in related technologies in terms of adaptability to complex scenes, cross-modal semantic alignment accuracy, and motion generation stability.

[0040] The inventive concept of this application can be summarized as follows: First, natural language text and initial video data are input into the video branch of the two-branch joint generation model to obtain the first video latent variable; then, natural language text, initial video data and the first video latent variable are input into the motion branch of the two-branch joint generation model to obtain the first fusion latent variable output by the motion branch; finally, based on the first fusion latent variable, a target motion sequence corresponding to the natural language text is generated.

[0041] In summary, the embodiments of this application enhance the accuracy of cross-modal semantic alignment by fusing video features and motion features, overcoming the ambiguity of text input. By introducing strong spatiotemporal priors from the video branch, a precise association mechanism is established between text, video, and motion, ensuring that the generated motion sequence is highly consistent with the text description in both global semantics and local details. The spatiotemporal modeling capability of the video branch is transformed into the performance advantage of the motion branch, enabling the generation of more accurate motion sequences under complex text input.

[0042] After introducing the main inventive concepts of the embodiments of this application, the following is a brief description of the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0043] For ease of understanding, the following detailed description of a motion video generation method provided in this application, with reference to the accompanying drawings, is provided: like Figure 1 The diagram illustrates an application scenario of a motion video generation method provided in this embodiment. The diagram includes: a network 10, a server 20, and a storage device 30. The server 20 acquires various types of data, such as natural language text, through the network. Using the method provided in this embodiment, it acquires the natural language text and inputs it into a dual-branch joint generation model to generate a target motion sequence.

[0044] The description in this application focuses on a single server only. However, those skilled in the art should understand that the illustrated network 10, server 20, and memory 30 are intended to illustrate the operation of the electronic devices, servers, and memory involved in the technical solutions of this application. The detailed description of a single server and memory is at least for ease of explanation and does not imply any limitation on the number, type, or location of servers. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application. Furthermore, although detailed descriptions are provided for ease of explanation... Figure 1 The diagram shows a bidirectional arrow from memory 30 to server 20, but those skilled in the art will understand that the sending and receiving of the aforementioned data also needs to be achieved through network 10.

[0045] It should be noted that the memory in the embodiments of this application can be, for example, a cache system, hard disk storage, memory storage, etc. Furthermore, the network security incident handling method proposed in this application is not only applicable to… Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application do not impose any limitations.

[0046] Based on the above description, this application provides a motion video generation method that can be used for generative intelligent transmission in intelligent transmission networks. The overall process is as follows: Figure 2 As shown, it includes the following: In step 201, the natural language text and initial video data are input into the video branch of the dual-branch joint generation model. The natural language text and initial video data are processed through the video branch to obtain the first video latent variable.

[0047] In step 202, the natural language text, initial video data, and the first video latent variable are input into the motion branch of the dual-branch joint generation model to obtain the first fused latent variable output by the motion branch.

[0048] In step 203, a target video corresponding to the natural language text is generated based on the first video latent variable.

[0049] In step 204, a target motion sequence corresponding to the natural language text is generated based on the first fusion latent variable.

[0050] It should be noted that the initial video data can be random Gaussian noise or video data containing a large amount of noise, and this initial video data can be pre-stored in memory. The two-branch joint generation model is the process of generating the target motion sequence based on natural language text, which is essentially a process of progressively denoising the initial video data.

[0051] For example, the inference module receives natural language text input and infers and generates a 3D human motion sequence that meets the requirements.

[0052] In one possible implementation, in step 204, a motion sequence corresponding to the natural language text is generated based on the first fusion latent variable, as detailed below. Figure 3 As shown, it includes the following steps: In step 301, the first fusion latent variable is subjected to root mean square normalization to obtain the first motion feature.

[0053] In step 302, the first motion feature is adjusted by the modulation signal generated based on the time step to obtain the second motion feature.

[0054] In step 303, the second motion feature is mapped to the third motion feature through linear projection.

[0055] In step 304, the third motion feature is decoded to obtain the motion sequence.

[0056] For example, firstly, use the following formula Output features of moving blocks (i.e., the first fusion latent variable) is normalized using root mean square to obtain the first motion feature. Then, based on the time step modulation signal The first motion feature is modulated using the following formula. Dynamically adjust the first motion feature (where, Scaling factor (as an offset factor), to obtain the second motion feature Subsequently, the modulated second motion feature is mapped to a dimension of [dimensional value missing] using linear projection. The motion latent represents (i.e., the third motion feature); finally, the third motion feature is decoded to obtain the motion sequence.

[0057] The above-mentioned motion video generation method enhances the accuracy of cross-modal semantic alignment by fusing video features and motion features, overcomes the ambiguity of text input, can efficiently process complex text input, accurately achieve cross-modal semantic alignment, and stably generate high-quality human motion sequences.

[0058] In one possible implementation, the architecture of the dual-branch joint generation model provided in this application embodiment is as follows: Figure 4 As shown, the model includes a video branch and a motion branch: The video branch is built based on the Wan2.1-I2V video generation model, adopting a DiT architecture and stream matching strategy. The motion branch includes a motion encoder, motion blocks, and a motion head. The motion encoder adopts a VQ-VAE architecture, and its core function is to encode the original 3D human motion sequence into discrete motion tokens (i.e., the second video latent variable) adapted for cross-modal fusion. The motion blocks correspond one-to-one with the video blocks in the video branch. Their core function is to achieve temporal modeling of the motion sequence through multi-step feature processing and to realize cross-modal fusion of video features and motion features. The data for motion blocks and video blocks can be set to 30. The core function of the motion head is to map the features output by the motion blocks to the motion sequence, generating an accurate motion sequence.

[0059] In one possible implementation, this application provides a model training method where the model is the aforementioned dual-branch joint generation model. This method can be used for generative intelligent transmission in intelligent transmission networks, and the overall process is as follows: Figure 5 As shown, it includes the following: In step 501, a cross-modal training dataset is obtained, which includes multiple sets of samples; each set of samples includes video-guided text, motion-guided text, motion sequence samples, and video clip samples.

[0060] In one possible implementation, the embodiments of this application obtain the cross-modal training dataset in the following manner, the specific process of which is as follows: Figure 6 As shown, it includes the following steps: In step 601, the acquired video is segmented to obtain multiple segmented video clips.

[0061] In step 602, multiple segmented video segments are filtered to obtain multiple filtered video segments.

[0062] In step 603, for each filtered video segment, a multimodal model is used to process the filtered video segment to generate video guidance text and motion guidance text; a human 3D pose estimation model is used to extract motion sequences from the filtered video segments; and the video guidance text, motion guidance text, motion sequences, and filtered video segments are used as a cross-modal training dataset.

[0063] Among them, video-oriented text describes the background environment, scene layout, character appearance and overall atmosphere of the video, focusing on visual semantics; motion-oriented text describes in detail the human body's movements, posture changes, limb trajectories and center of gravity shifts in the video, focusing on motion semantics.

[0064] For example, video data from both public and private sources (total duration no less than 50 hours, covering everyday actions, dance, sports, etc.) is collected, with a uniform resolution of 1024×768 and a frame rate of 30fps. Adaptive video segmentation is performed using a "frame threshold + RGB feature difference method" to obtain multiple segmented video clips. These segmented video clips are then subjected to multi-dimensional filtering based on "aesthetic quality + optical flow smoothness + OCR consistency" to obtain multiple filtered video clips. The filtered video clips are processed using a multimodal model (such as Qwen2.5-VL-72B) to generate dual-perspective text descriptions, including video-guided and motion-guided text, i.e., video-guided text. and motion-guided text The global motion SMPL parameters are extracted using a human 3D pose estimation model (such as the GVHMR model) and converted into motion sequences in the HumanML3D standard format. (T is the frame number, and the 263-dimensional array includes joint positions, rotations, and global displacements), ultimately forming aligned quadruple data. , as a cross-modal training dataset.

[0065] In one possible implementation, step 601 involves segmenting the acquired video to obtain multiple segmented video clips, as detailed below. Figure 7 As shown, it includes the following steps: In step 701, starting from the initial video frame, if a camera change is detected within the preset video frame length range, segmentation is performed at the video frame position where the camera change occurs, resulting in segmented video segments and the video to be segmented.

[0066] In step 702, if no camera change is detected within the preset video frame length range, the video is segmented at the preset video frame length position to obtain the segmented video segment and the video to be segmented.

[0067] In step 703, after segmenting at the video frame position where the camera changes or at the preset video frame length position, the first video frame of the video to be segmented is taken as the starting video frame, and the process returns to the steps starting from the starting video frame.

[0068] For example, in a preset video frame length of When the video clip length exceeds When a change in camera angle is detected, video segmentation is triggered. Simultaneously, a detection model is used to detect camera angle changes. When a change in camera angle is detected, video segmentation is triggered. For example, if the total length of the captured video is 1000 frames and the preset video frame length is 10 frames, starting from the first frame, if no camera angle change is detected within the 10-frame range, video segmentation is performed. After the first segmentation, a 10-frame video segment and a 990-frame video segment to be segmented are obtained. When a camera angle change is detected at the 5th frame, segmentation is performed at the 5th frame position. After the first segmentation, a 5-frame video segment and a 995-frame video segment to be segmented are obtained. Then, the first frame of the 995-frame video segment to be segmented is used as the starting video frame for the second segmentation (i.e., the 6th frame of the entire captured video), and so on, to complete the segmentation process of the captured video.

[0069] By performing the segmentation process described above on the acquired video, large video data is divided into smaller video data, thereby improving the model's processing efficiency.

[0070] In one possible implementation, step 602 involves filtering multiple segmented video segments to obtain multiple filtered video segments. The specific process is as follows: Figure 8 As shown, it includes the following steps: In step 801, for each segmented video segment, the video quality of the segmented video segment is determined, the optical flow smoothness of the segmented video segment is determined based on the standard deviation of the optical flow field of the segmented video segment, and the text matching degree of the segmented video segment is determined by detecting the text information in the segmented video segment.

[0071] In step 802, video segments that meet preset conditions in terms of video quality, optical flow smoothness, and text matching are selected as multiple filtered video segments.

[0072] For example, if the video quality is A, a pre-trained aesthetic evaluation model can be used to directly output a score, with a set threshold. The optical flow smoothness is F, and the standard deviation of the optical flow field in the video clip is calculated. and according to the formula (in Convert the empirical threshold into a standardized score and set the threshold. The text matching degree is 0. ORC detection technology is used to detect text information in the video. If the text content conflicts with the semantic meaning of actions in the video (e.g., a video of a "running" scene contains text related to "sitting"), then a value of O=0 is assigned; otherwise, a value of O=1 is assigned. A threshold is set. Finally, select those that simultaneously meet the following conditions. The video clips were used as multiple filtered video clips.

[0073] The above-mentioned thresholds can be adjusted as needed.

[0074] By performing the above-described filtering process on the collected videos, low-quality video clips were filtered out, ensuring the reliability and validity of the dataset samples.

[0075] In step 502, the video clip samples and video guidance text of the first group of samples are input into the video branch to obtain the second video latent variable by processing the video clip samples and video guidance text of the first group of samples. The predicted video noise is then determined based on the second video latent variable.

[0076] In one possible implementation, video branching can be carried out as follows: For example, receiving video-guided text , start frame With the end frame The video clip replica is encoded as a latent representation. The text is translated using the CLIP text encoder, which directs the video to text. Encoding as text encoding (L is the text length, D is the feature dimension); The DiT module contains N=30 video blocks, each video block adopts a spatiotemporal self-attention and cross-attention mechanism: spatiotemporal self-attention models the temporal dependency between video frames and the spatial dependency within frames, and cross-attention fuses text encoding and video latent features to achieve text-video semantic alignment and obtain the second video latent variable. .

[0077] The first set of samples can be any set of samples from the cross-modal training dataset.

[0078] In step 503, the motion sequence samples and motion-guided text of the first group of samples are input into the motion branch. The motion sequence samples of the first group of samples are processed by the motion branch to obtain the first motion latent variable and the motion text features obtained by processing the motion-guided text of the first group of samples.

[0079] In one possible implementation, the embodiments of this application obtain the first motion latent variable in the following manner, the specific process of which is as follows: Figure 9 As shown, it includes the following steps: In step 901, motion sequence samples are mapped into high-dimensional feature vectors using a convolutional network.

[0080] In step 902, the high-dimensional feature vector is matched with the preset codebook to obtain the discrete vector corresponding to the high-dimensional feature vector.

[0081] In step 903, the discrete vector is temporally downsampled to obtain the fourth motion feature.

[0082] In step 904, the label motion noise is injected into the fourth motion feature to obtain the first motion latent variable.

[0083] For example, firstly, motion sequence samples are processed through a 3-layer 1D convolutional network. Mapped to high-dimensional feature vectors Subsequently, based on the scale of codebook (in Vector quantization is performed (keeping the video feature dimensions consistent), i.e., using the formula... For each high-dimensional feature vector, match the most similar discrete vector in the codebook; simultaneously, perform temporal downsampling (downsampling factor). ), to obtain the final movement token (i.e., the fourth motion feature); injecting tag motion noise into the motion token. In this process, the first latent motion variable is obtained.

[0084] The convolutional network, codebook size, and downsampling factor can be adjusted as needed.

[0085] It should be noted that the embodiments of this application also ensure that the dimensions of the first motion latent variable and the second video latent variable are completely consistent through the alignment module, providing a basic condition for cross-modal fusion of the dual-branch framework.

[0086] In step 504, a second fused latent variable is obtained by fusing the first motion latent variable, the motion text feature, and the second video latent variable, so as to determine the predicted motion noise based on the second fused latent variable.

[0087] In one possible implementation, step 504 involves fusing the first motion latent variable, motion text features, and the second video latent variable to obtain the second fused latent variable. The specific process is as follows: Figure 10 As shown, it includes the following steps: In step 1001, enhanced self-attention calculation is performed on the first motion latent variable and the temporal information to obtain the fifth motion feature; the temporal information is a sinusoidal position code generated based on the time step.

[0088] In step 1002, the motion text feature and the second video latent variable are concatenated to obtain the sixth motion feature; the motion text feature and the second video latent variable have the same data dimension.

[0089] In step 1003, cross-attention calculation is performed on the fifth and sixth motion features to obtain the seventh motion feature.

[0090] In step 1004, the seventh motion feature is adjusted by the modulation parameters generated based on the time step to obtain the eighth motion feature.

[0091] In step 1005, the seventh motion feature and the eighth motion feature are fused through gating transformation to obtain the second fused latent variable.

[0092] For example, first, determine the time step. Generated sinusoidal position code (in , Indexed by feature dimensions, (For element-wise product); subsequently, the RoPE-enhanced self-attention layer injects temporal information into the first motion latent variable, thereby modeling the intrinsic temporal dependency of the motion sequence, using the following formula. Enhanced self-attention calculations are performed on the first motion latent variable and temporal information to obtain the fifth motion feature. This enhances the ability to model long sequences; next, it incorporates motion text features. (Motion-guided text) (obtained via CLIP encoding) and second video latent variables The features are then concatenated; then, the fifth and sixth motion features are fused through a cross-attention injection mechanism to obtain the seventh motion feature. ,Right now ( This involves concatenating text and video features (i.e., the sixth motion feature) to achieve multimodal semantic alignment; finally, based on time steps... To generate modulation parameters (shift, scale), first use... The seventh motion feature is dynamically adjusted to obtain the eighth motion feature. Furthermore, through gating conversion (in , The sigmoid function fuses the seventh motion feature with the modulated eighth motion feature to output the second fused latent variable.

[0093] The steps described above for obtaining the second fusion latent variable are mainly composed of Figure 4 The motion block execution in the model enhances cross-modal semantic alignment through the fusion of video features and motion features; it overcomes the ambiguity of text input by introducing strong spatiotemporal priors from the video branch to establish a precise association mechanism between text, video, and motion, ensuring that the generated motion sequence is highly consistent with the text description in both global semantics and local details; it transforms the spatiotemporal modeling capability of the video branch into the performance advantage of the motion branch, enabling the generation of more accurate motion sequences under complex text input.

[0094] In step 505, the total loss value is determined based on the predicted video noise and the predicted motion noise, and the model parameters of the dual-branch joint generation model are updated based on the total loss value.

[0095] In one possible implementation, the first set of samples further includes labeled video noise and labeled motion noise. In this embodiment, the total loss value is determined based on the predicted video noise and predicted motion noise, as detailed in the following process: Figure 11 As shown, it includes the following steps: In step 1101, the predicted video noise is compared with the labeled video noise to determine the video loss value.

[0096] In step 1102, the predicted motion noise is compared with the tag motion noise to determine the motion loss value.

[0097] In step 1103, the total loss value is determined based on the video loss value and the motion loss value.

[0098] For example, the embodiments of this application use the following joint loss function to train the two-branch joint generative model:

[0099] in, This represents the video loss value. This represents the motion loss value. For time-step dependent weight function, For time steps, As a latent variable for the third video, To predict video noise, For labeled video noise, To integrate latent variables, To predict motion noise, This is noise from the label's motion.

[0100] In one possible implementation, the model training environment is as follows: 8 NVIDIA H200 GPUs (≥80GB VRAM); training parameters: batch size 16, learning rate 5×10^(-5), weight decay 1×10^(-4), training epochs 200; the loss function is...

[0101] Both video loss and motion loss are MSE losses between the predicted and actual flow fields, with time step weights. A linearly increasing strategy is adopted to increase the loss weight in the later denoising stage.

[0102] In summary, this application's embodiments, by constructing a cross-modal training dataset containing text, video, and motion, overcome the shortcomings of traditional text-motion binary datasets, such as small size, insufficient diversity, and lack of coverage of complex scenes, providing the model with rich cross-modal information and improving its generalization ability. Through the fusion processing of video and motion features, the accuracy of cross-modal semantic alignment is enhanced, overcoming the ambiguity of text input. By introducing strong spatiotemporal priors from the video generation model, a precise association mechanism among text, video, and motion is established, ensuring that the generated motion sequence is highly consistent with the text description in both global semantics and local details. The spatiotemporal modeling capability of the video generation model is transformed into the performance advantage of the motion generation model, achieving a breakthrough improvement in motion generation quality under complex text input.

[0103] Based on the same inventive concept, embodiments of this application also provide a motion video generation device, such as... Figure 12 As shown, the device 1200 includes: The video branch module 1201 is configured to input natural language text and initial video data into the video branch of the dual-branch joint generation model, and process the natural language text and initial video data through the video branch to obtain the first video latent variable; The motion branch module 1202 is configured to input the natural language text, the initial video data and the first video latent variable into the motion branch of the dual-branch joint generation model to obtain the first fusion latent variable output by the motion branch. The video branching module is further configured to generate a target video corresponding to the natural language text based on the first video latent variable; The motion branch module is further configured to generate a target motion sequence corresponding to the natural language text based on the first fusion latent variable.

[0104] In one possible implementation, the motion branch module is specifically configured to generate the motion sequence corresponding to the natural language text based on the first fusion latent variable, wherein the motion branch module is configured as follows: The first fusion latent variable is normalized by root mean square to obtain the first motion feature; The first motion feature is adjusted by a modulation signal generated based on a time step to obtain a second motion feature; The second motion feature is mapped to the third motion feature through linear projection; The third motion feature is decoded to obtain the motion sequence.

[0105] Based on the same inventive concept, this application also provides a model training device, wherein the model is a dual-branch joint generation model, and the model includes a video branch and a motion branch, such as... Figure 13As shown, the device 1300 includes: The dataset acquisition module 1301 is configured to acquire a cross-modal training dataset, which includes multiple sets of samples; each set of samples includes video-guided text, motion-guided text, motion sequence samples, and video clip samples. The video processing module 1302 is configured to input video clip samples and video guidance text of the first group of samples into the video branch, obtain the second video latent variable obtained by the video branch processing the video clip samples and video guidance text of the first group of samples, and determine the predicted video noise based on the second video latent variable; The motion processing module 1303 is configured to input the motion sequence samples and motion-guided text of the first group of samples into the motion branch, and obtain the first motion latent variable obtained by the motion branch processing the motion sequence samples of the first group of samples, and the motion text features obtained by processing the motion-guided text of the first group of samples. The motion processing module 1303 is further configured to fuse the first motion latent variable, the motion text feature, and the second video latent variable to obtain a second fused latent variable, so as to determine the predicted motion noise based on the second fused latent variable; The model update module 1304 is configured to determine the total loss value based on the predicted video noise and the predicted motion noise, and update the model parameters of the dual-branch joint generation model based on the total loss value.

[0106] In one possible implementation, the first set of samples further includes labeled video noise and labeled motion noise; the model update module is specifically configured to determine the total loss value based on the predicted video noise and predicted motion noise, wherein: The predicted video noise is compared with the labeled video noise to determine the video loss value; The predicted motion noise is compared with the label motion noise to determine the motion loss value; The total loss value is determined based on the video loss value and the motion loss value.

[0107] In one possible implementation, the motion processing module is specifically configured to process the motion sequence samples of the first group of samples through the motion branch to obtain the first motion latent variable. The motion sequence samples are mapped to high-dimensional feature vectors through the convolutional network in the motion branch, and the discrete vectors corresponding to the high-dimensional feature vectors are determined. The discrete vector is downsampled to obtain the fourth motion feature; The label motion noise is injected into the fourth motion feature to obtain the first motion latent variable.

[0108] In one possible implementation, the second fused latent variable is obtained by fusing the first motion latent variable, the motion text feature, and the second video latent variable. Specifically, the motion processing module is configured as follows: Enhanced self-attention calculation is performed on the first motion latent variable and temporal information to obtain the fifth motion feature; the temporal information is a sinusoidal position code generated based on time steps. The motion text feature and the second video latent variable are concatenated to obtain the sixth motion feature; the motion text feature and the second video latent variable have the same data dimension. Perform cross-attention calculations on the fifth and sixth motion features to obtain the seventh motion feature; The seventh motion feature is adjusted by adjusting the modulation parameters generated based on the time step to obtain the eighth motion feature; The seventh motion feature and the eighth motion feature are fused by gating transformation to obtain the second fused latent variable.

[0109] In one possible implementation, the dataset acquisition module is configured to obtain the cross-modal training dataset in the following manner: The captured video is segmented to obtain multiple segmented video clips; The multiple segmented video segments are filtered to obtain multiple filtered video segments; For each filtered video segment, a multimodal model is used to process the filtered video segment to generate video guidance text and motion guidance text; a human 3D pose estimation model is used to extract motion sequences from the filtered video segment; the video guidance text, the motion guidance text, the motion sequences, and the filtered video segment are used as a cross-modal training dataset.

[0110] In one possible implementation, the process of segmenting the acquired video to obtain multiple segmented video clips, wherein the dataset acquisition module is specifically configured as follows: Starting from the initial video frame, if a camera change is detected within the preset video frame length, segmentation is performed at the position of the video frame with the camera change, resulting in segmented video segments and the video to be segmented. If no camera change is detected within the preset video frame length range, the video is segmented at the preset video frame length position to obtain the segmented video segment and the video to be segmented. After segmenting at the video frame position where the camera changes or at a preset video frame length position, the first video frame of the video to be segmented is taken as the starting video frame, and the process returns to the steps starting from the starting video frame.

[0111] In one possible implementation, the filtering process for the plurality of segmented video segments to obtain a plurality of filtered video segments, wherein the dataset acquisition module is specifically configured as follows: For each segmented video segment, the video quality of the segmented video segment is determined, the optical flow smoothness of the segmented video segment is determined based on the standard deviation of the optical flow field of the segmented video segment, and the text matching degree of the segmented video segment is determined by detecting the text information in the segmented video segment. Video clips that meet preset criteria in terms of video quality, optical flow smoothness, and text matching are used as multiple filtered video clips.

[0112] The following reference Figure 14 To describe an electronic device 130 according to this embodiment of the present application. Figure 14 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0113] like Figure 14 As shown, the electronic device 130 is presented in the form of a general electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0114] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0115] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0116] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0117] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] In an exemplary embodiment, this application also provides a computer-readable storage medium including instructions, such as a memory 132 including instructions, which can be executed by a processor 131 of an electronic device 130 to complete the above-described motion video generation method and model training method. Optionally, the computer-readable storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0119] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor 131, implements the motion video generation method and model training method provided in this application.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating motion video, characterized in that, The method includes: The natural language text and initial video data are input into the video branch of the two-branch joint generation model. The natural language text and initial video data are processed through the video branch to obtain the first video latent variable. The natural language text, the initial video data, and the first video latent variable are input into the motion branch of the dual-branch joint generation model to obtain the first fused latent variable output by the motion branch. Based on the first video latent variables, generate the target video corresponding to the natural language text; Based on the first fusion latent variable, a motion sequence corresponding to the natural language text is generated.

2. The method according to claim 1, characterized in that, The step of generating the motion sequence corresponding to the natural language text based on the first fusion latent variable includes: The first fusion latent variable is normalized by root mean square to obtain the first motion feature; The first motion feature is adjusted by a modulation signal generated based on a time step to obtain a second motion feature; The second motion feature is mapped to the third motion feature through linear projection; The third motion feature is decoded to obtain the motion sequence.

3. A model training method, characterized in that, The model is a dual-branch joint generation model, which includes a video branch and a motion branch. The method includes: Obtain a cross-modal training dataset, which includes multiple sets of samples; each set of samples includes video-guided text, motion-guided text, motion sequence samples, and video clip samples. The video clip samples and video guidance text of the first group of samples are input into the video branch to obtain the second video latent variable obtained by the video branch processing the video clip samples and video guidance text of the first group of samples. The predicted video noise is then determined based on the second video latent variable. The motion sequence samples and motion-guided text of the first group of samples are input into the motion branch. The motion sequence samples of the first group of samples are processed by the motion branch to obtain the first motion latent variable and the motion text features obtained by processing the motion-guided text of the first group of samples. A second fused latent variable is obtained by fusing the first motion latent variable, the motion text feature, and the second video latent variable, so as to determine the predicted motion noise based on the second fused latent variable; The total loss value is determined based on the predicted video noise and predicted motion noise, and the model parameters of the dual-branch joint generation model are updated based on the total loss value.

4. The method according to claim 3, characterized in that, The first set of samples also includes labeled video noise and labeled motion noise; the step of determining the total loss value based on the predicted video noise and predicted motion noise includes: The predicted video noise is compared with the labeled video noise to determine the video loss value; The predicted motion noise is compared with the label motion noise to determine the motion loss value; The total loss value is determined based on the video loss value and the motion loss value.

5. The method according to claim 4, characterized in that, The process of processing the motion sequence samples of the first group of samples through the motion branch to obtain the first motion latent variable includes: The motion sequence samples are mapped to high-dimensional feature vectors through the convolutional network in the motion branch, and the discrete vectors corresponding to the high-dimensional feature vectors are determined. The discrete vector is downsampled to obtain the fourth motion feature; The label motion noise is injected into the fourth motion feature to obtain the first motion latent variable.

6. The method according to claim 3, characterized in that, The process of fusing the first motion latent variable, the motion text feature, and the second video latent variable to obtain the second fused latent variable includes: Enhanced self-attention calculation is performed on the first motion latent variable and temporal information to obtain the fifth motion feature; the temporal information is a sinusoidal position code generated based on time steps. The motion text feature and the second video latent variable are concatenated to obtain the sixth motion feature; the motion text feature and the second video latent variable have the same data dimension. Perform cross-attention calculations on the fifth and sixth motion features to obtain the seventh motion feature; The seventh motion feature is adjusted by adjusting the modulation parameters generated based on the time step to obtain the eighth motion feature; The seventh motion feature and the eighth motion feature are fused by gating transformation to obtain the second fused latent variable.

7. The method according to claim 3, characterized in that, The cross-modal training dataset was obtained using the following method: The captured video is segmented to obtain multiple segmented video clips; The multiple segmented video segments are filtered to obtain multiple filtered video segments; For each filtered video segment, a multimodal model is used to process the filtered video segment to generate video guidance text and motion guidance text; a human 3D pose estimation model is used to extract motion sequences from the filtered video segment; the video guidance text, the motion guidance text, the motion sequences, and the filtered video segment are used as a cross-modal training dataset.

8. The method according to claim 7, characterized in that, The process of segmenting the acquired video yields multiple segmented video clips, including: Starting from the initial video frame, if a camera change is detected within the preset video frame length range, segmentation is performed at the position of the video frame with the camera change, resulting in segmented video segments and the video to be segmented. If no camera change is detected within the preset video frame length range, the video is segmented at the preset video frame length position to obtain the segmented video segment and the video to be segmented. After segmenting at the video frame position where the camera changes or at a preset video frame length position, the first video frame of the video to be segmented is taken as the starting video frame, and the process returns to the steps starting from the starting video frame.

9. The method according to claim 7, characterized in that, The filtering process of the multiple segmented video segments to obtain multiple filtered video segments includes: For each segmented video segment, the video quality of the segmented video segment is determined, the optical flow smoothness of the segmented video segment is determined based on the standard deviation of the optical flow field of the segmented video segment, and the text matching degree of the segmented video segment is determined by detecting the text information in the segmented video segment. Video clips that meet preset criteria in terms of video quality, optical flow smoothness, and text matching are used as multiple filtered video clips.

10. An electronic device, characterized in that, include: Processor and memory; The memory is used to store the processor-executable instructions; The processor is configured to execute the instructions to implement the motion video generation method as described in any one of claims 1-2 and the model training method as described in any one of claims 3-9.

Images