A video generation method and system based on cross-modal collaborative learning

Abstract

The application discloses a video generation method and system based on cross-modal collaborative learning, and belongs to the field of video generation and multi-modal data processing. Multi-modal input is received and multi-modal features are extracted; the multi-modal features are processed by an improved conditional diffusion model to generate a video sequence; the improved conditional diffusion model includes a camera perception attention module and a decoupled cross-attention module arranged side by side in each DiT block, both of which input video latent features obtained after processing by an image sequence latent variable, and output after weighting to replace the output of the cross-attention module in the original conditional diffusion model; a low-rank adaptive technology is used to fine-tune the improved conditional diffusion model, and a multi-stage curriculum learning strategy is introduced in the training process. Experimental results show that the application has excellent performance in multi-task video generation and can realize high-quality and fine scene subject and camera motion joint control.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video generation and multi-modal data processing, and particularly relates to a video generation method and system based on cross-modal collaborative learning. BACKGROUND

[0002] In recent years, video generation technology has made significant progress in content quality and generation efficiency. However, achieving synchronization and fine control of scene subjects and camera motion, especially supporting large-scale and arbitrary direction camera trajectories in complex scenes, remains a major challenge.

[0003] Existing methods usually rely on predefined camera trajectories, mainly relying on data sets such as RealEstate10K. However, the camera poses in these data sets are derived from SLAM estimation and subsequent post-processing, which inherently limits accuracy. Moreover, when directly inputting these trajectories, there is a clear modal gap with the text prompt, often causing the model to become confused or reducing its native generation capabilities. Predefined trajectories also have limitations in flexibility and cannot adapt to complex camera motion in arbitrary directions.

[0004] Although recent research has explored multi-modal camera control, it has only remained at the path control level and cannot achieve joint control of the camera and scene subject. When coordinating complex interactive scenes such as "make the camera rotate around the mechanical arm that is being lifted", the lack of ability becomes a problem. Overall, this field faces three major challenges: one is limited modalities, two is the inability to support large-scale and complex trajectories, and three is the lack of support for joint control. SUMMARY

[0005] To overcome the above shortcomings, the present application proposes a video generation method and system based on cross-modal collaborative learning, which can achieve joint fine control of scene subjects and camera motion.

[0006] The specific technical solutions adopted by the present application are as follows:

[0007] In a first aspect, the present application proposes a video generation method based on cross-modal collaborative learning, comprising the following steps:

[0008] (1) receiving multi-modal input and extracting multi-modal features, the multi-modal input including target images, text prompts, and camera motion trajectory information, the multi-modal features including image sequence latent variables and global visual features extracted based on target images, global semantic features extracted based on text prompts, and camera embeddings extracted based on camera motion trajectory information;

[0009] (2) generating the video sequence by processing the multi-modal features through the improved conditional diffusion model, each DiT block of the improved conditional diffusion model comprising a camera-aware attention module and a decoupled cross-attention module arranged side by side, both of the two attention modules inputting video latent features obtained after processing by image sequence latent variables;

[0010] And in the camera-aware attention module, the camera embedding dynamics are injected into the video latent features through a parallel cross-frame association path and an intra-frame control path, and the fusion of the generation results of the two paths is taken as the output of the camera-aware attention module; in the decoupled cross-attention module, a first cross-attention result of global visual features and video latent features is calculated separately, and a second cross-attention result of global semantic features and video latent features is calculated separately, and the fusion of the two cross-attention results is taken as the output of the decoupled cross-attention module;

[0011] The output results of the camera-aware attention module and the decoupled cross-attention module are weighted and replace the output of the cross-attention module in the original conditional diffusion model;

[0012] (3) using low-rank adaptive technology to fine-tune the improved conditional diffusion model, and introducing a multi-stage curriculum learning strategy in the training process to learn the scene subject motion, independent camera motion and joint control of the two in stages.

[0013] Further, the potential input of the DiT block in the improved conditional diffusion model is obtained by splicing noise latent variables and binary masks after image sequence latent variables, and the video latent features are generated after layer normalization and self-attention operation in the DiT block.

[0014] Further, the text prompt includes first description text for the scene subject part in the target image and second description text for the camera motion part.

[0015] Further, a trajectory extractor is used to obtain frame-level camera motion trajectory information containing camera intrinsic and extrinsic parameter matrices, and the trajectory extractor integrates trajectory extraction units for different modal inputs, including:

[0016] The first modal trajectory extraction unit uses a large language model to parse the second description text into discrete motion representation, calculates the extrinsic parameter matrix of each frame, and estimates the intrinsic parameter matrix according to the target image;

[0017] The second modal trajectory extraction unit extracts an initial trajectory from a reference video through a GOLMAP algorithm, refines the extrinsic parameter matrix through a multi-layer perception, and estimates the intrinsic parameter matrix according to the target image;

[0018] The third modal trajectory extraction unit directly reads the trajectory data and performs matrix format conversion.

[0019] Further, the camera embedding extraction process includes:

[0020] The camera center and pixel ray direction are calculated for each frame of the internal and external parameter matrix, and the internal and external parameter matrix is converted into Ray representation under the camera coordinate system;

[0021] The ray representation is projected into a camera embedding through a Transformer module.

[0022] Further, the camera-aware attention module calculation process includes:

[0023] The camera embedding is merged into time and space dimensions to obtain a cross-frame camera embedding; the video latent feature is reshaped into the same shape as the cross-frame camera embedding, the cross-attention of the two is calculated on the cross-frame association path, and the calculation result is restored to the original dimension as the first output result;

[0024] The camera embedding is merged into batch and time dimensions to obtain an intra-frame camera embedding; the video latent feature is reshaped into the same shape as the intra-frame camera embedding, the cross-attention of the two is calculated on the intra-frame control path, and the calculation result is restored to the original dimension as the second output result;

[0025] The key-value projection matrix parameters of the two paths are shared, the query projection matrix parameters are independent, and the weighted sum of the output results of the two paths is taken as the final output.

[0026] Further, the output results of the camera-aware attention module and the decoupled cross-attention module are weighted and then continue to perform layer normalization and feedforward network calculation processes in the DiT block as the output of the DiT block. The improved conditional diffusion model is obtained by stacking a series of DiT blocks.

[0027] Further, the multi-stage curriculum learning strategy includes the following stages:

[0028] Stage 1: Use a static camera dataset to train the model to learn the motion of the scene subject;

[0029] Stage 2: Learn the independent camera basic direction motion and the freedom upgrade motion in steps;

[0030] Stage 3: Use a joint control dataset to train the collaborative generation of scene subject and camera motion.

[0031] Further, the improved conditional diffusion model integrates an image encoder and an image decoder at both ends, the image encoder is used to extract image sequence latent variables from the target image and input the diffusion model, and the image decoder is used to decode the output of the diffusion model to generate a video frame sequence.

[0032] In a second aspect, the present application provides a video generation system based on cross-modal collaborative learning, which is used to implement the video generation method based on cross-modal collaborative learning.

[0033] The present application has the following beneficial effects:

[0034] The present application realizes breakthrough progress in the field of video generation through a cross-modal collaborative learning mechanism. Based on unified multi-modal input analysis, high-precision joint control of scene subjects and camera motion is realized through improved diffusion model structure, and the practicality and efficiency of complex scene generation are significantly improved by combining a lightweight training strategy.

[0035] Specifically, in the multi-modal feature extraction stage, a unified analysis framework compatible with target images, text prompts and camera motion trajectory information is used to realize cross-modal collaboration. In the diffusion generation mechanism, a camera perception attention module and a decoupled cross-attention module are used in parallel structure. The camera perception attention module ensures temporal coherence through a cross-frame association path, and the intra-frame control path realizes frame-by-frame pose accurate alignment, and realizes dynamic injection of camera control signals. The decoupled cross-attention module separates the influence of image features and text semantic features, avoids modal interference, and the weighted fusion of the outputs of the two replaces the original cross-attention module, which helps to realize fine-grained collaborative control of scene subject behavior and camera motion trajectory. Finally, at the training optimization level, the low-rank adaptive technology greatly reduces the computational burden, and combined with the systematic training framework of phased learning of scene subject motion, independent camera motion and joint control of the two, the generalization ability of the model to high-degree-of-freedom motion instructions is significantly improved. Experimental verification shows that the present application realizes breakthrough improvement in camera motion accuracy, generation quality and instruction compliance. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a framework structure diagram of a video generation method based on cross-modal collaborative learning of the present application;

[0037] Figure 2 is a flowchart of a video generation method based on cross-modal collaborative learning of the present application;

[0038] Figure 3 is a partial frame image schematic diagram of the present application in generating video in a complex scene;

[0039] Figure 4 is a structure diagram of a video generation system based on cross-modal collaborative learning of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described and illustrated with reference to the specific embodiments. The embodiments are only exemplary of the present disclosure and do not circumscribe the scope of the present application. The technical features of various embodiments of the present application can be combined with each other without conflict, as long as they are not mutually exclusive.

[0041] The accompanying drawings are merely schematic illustrations of the present application and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are only exemplary illustrations and do not necessarily include all steps. For example, some steps can be further decomposed, and some steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0043] As shown in Figure 1 The present application proposes a video generation method based on cross-modal collaborative learning. In the multi-modal feature extraction stage, a unified analysis framework compatible with target image, text prompt and camera motion trajectory information is used to realize cross-modal collaboration. In the DiT block, a camera perception attention module and a decoupled cross-attention module are used in parallel structure, and the weighted fusion output by the two modules replaces the original cross-attention module. At the training optimization level, the low-rank adaptive technology greatly reduces the computational burden, and combined with the systematic training framework of the scene subject motion, independent camera motion and their joint control in the stage learning, the generalization ability of the model to high degree of freedom motion instructions is significantly improved.

[0044] As shown in Figure 2 The method mainly includes the following steps:

[0045] S1, receiving multi-modal input and extracting multi-modal features

[0046] In this embodiment, the input target image I, text prompt and camera motion trajectory information are input, wherein the text prompt is divided into scene subject part description and camera motion description. The scene subject part description refers to the description of objects and motion states in the scene, for example, "shooting a panda eating bamboo in close-up", and the scene subject part description only controls the subject behavior in the target image. The camera motion description refers to the description of the camera motion trajectory, for example, "from 1 to 3 seconds, the camera moves slowly to the left". The two parts of the description together constitute the text prompt.

[0047] The target image I defines the scene base and the subject identity, and the processing process of the target image I includes:

[0048] Splice the input image I along the time axis with a blank frame to obtain an image sequence I c , which is of shape [t, h, w, 3], t represents the time dimension of the image, h and w represent the height and width of the time dimension of the image. 3 represents that the number of channels is 3; compressed into image sequence latent variable Z by image encoder (3D VAE) c , which is of shape [T, H, W, C], wherein, is the time dimension of the latent variable, which represents the downsampling of the original time dimension; H = h / S, W = w / S are the height and width of the latent variable, which represent the downsampling of the original spatial dimension, and C represents the number of channels; S is the downsampling factor.

[0049] The global visual feature F of the input image I is extracted by using the image encoder (CLIP) img ;

[0050] The global semantic feature F is extracted from the text prompt by using the text encoder (umT5) text ;

[0051] In addition, the above-mentioned camera motion trajectory information can be directly input, or can be extracted from the above-mentioned text prompt or a reference video, supporting multi-modal input, and using a trajectory extractor to process all modalities to generate the pose parameters of the camera, realizing accurate lens control. The trajectory extractor of the present application supports text prompt / video / explicit parameter input, outputs a frame-level extrinsic parameter sequence [R, t] (R is a rotation matrix, and t is a translation vector), and simultaneously estimates an intrinsic parameter K. Different input modalities are processed by a special processing scheme to generate camera pose parameters:

[0052] For the text mode, a large language model (LLM) is used to translate the natural language description of the text prompt into a discrete motion representation, and the spatial position p∈R 3 of each frame of the camera is calculated by mathematical calculation , is an identity matrix; assuming that the camera always points to the center of the target object, the rotation matrix R = LookAt(p t , o) is calculated, and the complete camera extrinsic parameter sequence C is generated by combining the translation and rotation matrices, and the intrinsic parameter matrix K ∈ R 3×3 is simultaneously estimated from the input image I, forming the camera pose parameters (extrinsic parameter matrix [R, t] + intrinsic parameter matrix K).

[0053] For the video mode, the GOLMAP algorithm is used to obtain an initial trajectory from a reference video, which is used as a preliminary estimate of the camera pose, and then the multi-layer perceptron (MLP) is used to refine the coarse trajectory, and the refined trajectory directly corresponds to the camera extrinsic parameter; the intrinsic parameter matrix is the same as above.

[0054] For explicit parameter modes, i.e., trajectory data is provided directly, only matrix format conversion is required.

[0055] The camera pose parameters are converted into a structured representation using a dedicated camera encoder module to obtain the camera embedding. The specific process is as follows: For each frame's extrinsic parameters [R,t] and intrinsic parameters K, the camera center O = -R is calculated. T t, Project the pixel (u, v) onto normalized camera coordinates, then rotate it to world coordinates to obtain the pixel ray direction d. u,v = RK -1 [u, v, 1] T Converted based on camera center and pixel ray direction Ray representation in coordinate system forms tensor P u,v =(d u,v , O×d u,v Tensor P, with shape [B, T, H', W', 6]. u,v The camera is projected into a P shape using the Transformer module, which is [B, T, H, W, C]. This embedding method ensures that the camera information is aligned with the video features, facilitating subsequent attention fusion.

[0056] The latent variable Z of the image sequence c The concatenated noise latent variable and binary mask are used as the latent input to the improved conditional diffusion model. The noise latent variable is a random tensor conforming to a standard normal distribution, with the same dimension as the latent variable in the image sequence. The binary mask is a 0-1 matrix identifying the conditional regions; the 0-value regions serve as the "canvas," guiding the model to modify only specified positions. The first frame contains all 1s, forcing the model to inherit the input image content, while subsequent frames allow the generation of new content. These three components are concatenated along the channel dimension as the latent input. The global visual feature F... img Global semantic features F text The camera-embedded P-fusion is used as a conditional control for the improved conditional diffusion model.

[0057] S2, Improved Conditional Diffusion Model

[0058] This invention, based on the Conditional Diffusion Transformer, adds a decoupled cross-attention module and a camera-aware attention module to each DiT block. The latent variable Z of the image sequence... c The latent noise variables and binary mask are concatenated and used as the latent input for the improved conditional diffusion model, which is the same as the traditional diffusion model and will not be described in detail here. The difference is that in each DiT block, the latent input is processed by layer normalization, self-attention, and layer normalization before being used as the input for the cross-attention module. This invention improves the cross-attention module.

[0059] (1) In the camera-aware attention module, the camera is embedded in the video latent feature space of the P dynamic DiT block through a parallel space-time modeling path, and motion instruction alignment is achieved. The specific implementation includes two feature processing paths:

[0060] Path 1: Convert camera embedding P to feature Z cam1 , which has a shape of [B, T x H x W, C]. The time dimension T is merged with other dimensions for calculation to achieve cross-frame association. The video latent feature Z is fused through cross-frame cross-attention. It should be noted that the video latent feature Z here is not the image sequence latent variable Z c , but the image sequence latent variable Z c after layer normalization, self-attention, and layer normalization operations, which is used to inject the camera-aware attention module and the decoupled cross-attention module. This feature comes from the target image I and is used to generate a video, called video latent feature Z, which has a shape of [B, T, H, W, C], and B is the batch size.

[0061] Flatten the video latent feature Z into Z flat1 , which has a shape of [B, T x H x W, C] to align the Z cam1 dimension. Take Z cam1 as the input of the query vector, and take Z flat as the input of the key vector K and the value vector V to calculate the cross-attention value Z fused1 , and reshape it into the first output result Z out1 = Reshape(Z fused1 ), which has a shape of [B, T, H, W, C].

[0062] Path 1 captures the inter-frame association to ensure the temporal coherence between consecutive frames and achieve smooth lens transitions.

[0063] Path 2: Convert camera embedding P to feature Z cam2 , which has a shape of [B x T, H x W, C]. The time dimension T is merged with the batch B, and the attention focuses on the internal information of each frame. The video latent feature Z is fused through intra-frame cross-attention. The video latent feature Z is described in Path 1; Flatten the video latent feature Z into Z flat2 , which has a shape of [B x T, H x W, C] to align the Z cam2 dimension. Take Z cam2 as the input of the query vector, and take Z flat2 as the input of the key vector K and the value vector V to calculate the cross-attention value Z fused2 , and reshape it into the second output result Z out2 = Reshape(Z fused2 ), which has a shape of [B, T, H, W, C].

[0064] The frame-by-frame accurate control of the camera pose is implemented through path 2.

[0065] The inputs for calculating the key vector and the value vector in path 2 are all from the video latent feature Z, and the projection matrix parameters for calculating K and V are shared, and the projection matrix parameters for calculating Q are independent.

[0066] The outputs of path 1 and path 2 are weighted and summed as the output CAA of the camera perception attention module. out .

[0067] (2) In the decoupled cross-attention module, the global visual feature F img and the global semantic feature F text are injected through the decoupled cross-attention mechanism, and the two features are calculated cross-attention with the video latent feature Z, and the internal fusion is used as the output. This mechanism separates the influence of image and text features on the DiT block, avoiding modal interference.

[0068] (3) The outputs of the camera perception attention module and the decoupled cross-attention module are added, and the camera motion control and content semantic description are fused to generate the fusion feature Z Cross . The fusion feature continues to perform layer normalization and feedforward network processing in the DiT block to obtain the output of the DiT block, which is used as the latent input of the next DiT block. The operation of the stacked DiT blocks is repeated, and the final result is output by the image decoder (3D VAE) to generate a video.

[0069] S3, introduce low-rank adaptive (LoRA) technology for full-model lightweight fine-tuning

[0070] In this embodiment, LoRA is integrated into the attention projection matrix and feedforward network layer of all DiT blocks to improve the convergence efficiency of the training process. Specifically, the injection method of LoRA is known in the art, and the principle will not be described again. By training only the newly added low-rank matrix parameters and freezing most of the pre-training parameters of the original model, the model can maintain high performance while reducing the computational burden, supporting efficient training under limited resources.

[0071] S4, introduce a multi-stage curriculum learning strategy

[0072] Existing camera control video generation models usually face two major challenges: first, it is difficult to simultaneously learn camera motion and object motion generation, and second, the model's ability to understand and execute complex camera trajectories is limited. To solve the above challenges, the present application proposes a multi-stage curriculum learning strategy.

[0073] In one specific implementation of the present application, the multi-stage curriculum learning strategy is as follows:

[0074] Phase one: the training data set of static cameras, so that the model focuses on learning the inherent motion of objects, such as object deformation, posture change, and learning the dynamic scene of the subject under the static camera. The training data set of the static camera is obtained by extracting the camera trajectory from the video and screening the static camera scene, which helps the model learn the independent motion characteristics of the scene subject.

[0075] Phase two: learning the independent control of the camera motion trajectory, including translation and rotation, training the model to understand the basic direction control and complex trajectory control of the camera.

[0076] In this phase, basic direction learning is first performed to master the camera motion of ten basic directions, including translation directions: left, right, up, down; diagonal directions: top left, top right, bottom left, bottom right; depth directions: forward, backward. This step uses 1000 DIV2K dataset images, each image is rendered in 10 motion directions, generating 10000 training videos to help the model establish the mapping relationship between motion direction and picture change.

[0077] Then upgrade the motion degrees of freedom, gradually perform arbitrary direction translation, rotation and variable length motion, such as acceleration and deceleration, which only needs to adjust the training data set.

[0078] Phase three: learning the joint control of scene subjects and camera motion, this phase gradually increases the learning difficulty, so that the model gradually adapts to the complexity of joint motion, and the RealCam-Vid dataset is used in this phase.

[0079] The multi-stage course learning strategy adopted by the application first masters the basic object motion in phase one, and then gradually learns the complex camera motion and object cooperative motion.

[0080] In order to evaluate the effect of the model of the application, comparative experiments are carried out with existing camera control models, Table 1 is a performance comparison of camera control indicators, quality and object control indicators under complex camera motion trajectory control, wherein TransErr is translation error, RoIErr is interest region error, CamMC is camera motion consistency error, FVD-VGPT is video generation quality, FVD-StyleGAN is video generation quality, and AIF is action instruction compliance degree.

[0081] Table 1 Experimental results of complex camera motion trajectory control

[0082]

[0083] Further compared with the existing mainstream models, the results are shown in Table 2, and the evaluation indexes include I2V score and video generation quality, and the I2V score is defined as the average of video-image subject consistency and video-image background consistency. It can be seen that the application achieves excellent performance.

[0084] Table 2 Experimental results of joint control

[0085]

[0086] Figure 3 An example in the embodiment is given, which shows part of the frame images in the generated video in a complex scene. The input text prompt is: a skier doing stunts in the air; the camera part is random motion. The input target image is the first frame image in Figure 2 It can be seen that the application realizes the synchronization, fine granularity and high degree of freedom control of the scene subject and the camera motion.

[0087] Based on the same inventive concept, as shown in Figure 4 The embodiment further proposes a video generation system based on cross-modal collaborative learning, which is used to implement the video generation method based on cross-modal collaborative learning described above, and the system comprises:

[0088] A multi-modal data input module is configured to receive multi-modal inputs and extract multi-modal features, wherein the multi-modal inputs include target images, text prompts and camera motion trajectory information, and the multi-modal features include image sequence latent variables and global visual features extracted based on the target images, global semantic features extracted based on the text prompts, and camera embeddings extracted based on the camera motion trajectory information.

[0089] An improved conditional diffusion model module is integrated and arranged in parallel with a camera perception attention module and a decoupled cross-attention module, and both attention modules are input with video latent features obtained by processing image sequence latent variables;

[0090] In the camera perception attention module, the camera embeddings are dynamically injected into the video latent features through a parallel cross-frame association path and an intra-frame control path, and the fusion of the generation results of the two paths is used as the output of the camera perception attention module; in the decoupled cross-attention module, the first cross-attention result of the global visual features and the video latent features is calculated separately, and the second cross-attention result of the global semantic features and the video latent features is calculated separately, and the fusion of the two cross-attention results is used as the output of the decoupled cross-attention module;

[0091] The output results of the camera perception attention module and the decoupled cross-attention module are weighted and used to replace the output of the cross-attention module in the original conditional diffusion model;

[0092] The light-weight fine-tuning module adopts a low-rank adaptive technology to perform light-weight fine-tuning on the improved conditional diffusion model;

[0093] The multi-stage training module performs a multi-stage curriculum learning strategy in a training stage to learn scene subject motion, independent camera motion and joint control of the two in stages.

[0094] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts are described in the method embodiment, and the implementation method of the remaining modules will not be repeated here. The system embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0095] The system embodiment of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiment can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability.

[0096] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but cannot be understood as limitations on the scope of the present application. For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A video generation method based on cross-modal collaborative learning, characterized in that, The method comprises the following steps: (1) receiving multi-modal input and extracting multi-modal features, the multi-modal input comprising a target image, a text prompt and camera motion trajectory information, the multi-modal features comprising image sequence latent variables and global visual features extracted based on the target image, global semantic features extracted based on the text prompt, and camera embeddings extracted based on the camera motion trajectory information; (2) processing the multi-modal features through an improved conditional diffusion model to generate a video sequence; each DiT block of the improved conditional diffusion model comprises a camera-aware attention module and a decoupled cross-attention module arranged side by side, and both the two attention modules input video latent features processed from the image sequence latent variables; and in the camera-aware attention module, the camera embeddings are dynamically injected into the video latent features through a parallel cross-frame association path and an intra-frame control path, and the fusion of the results generated by the two paths is taken as the output of the camera-aware attention module; in the decoupled cross-attention module, a first cross-attention result of the global visual features and the video latent features is calculated separately, and a second cross-attention result of the global semantic features and the video latent features is calculated separately, and the fusion of the two cross-attention results is taken as the output of the decoupled cross-attention module; the output results of the camera-aware attention module and the decoupled cross-attention module are weighted and replace the output of the cross-attention module in the original conditional diffusion model; (3) using a low-rank adaptive technology to fine-tune the improved conditional diffusion model, and introducing a multi-stage curriculum learning strategy in the training process to learn the scene subject motion, the independent camera motion and the joint control of the two in stages.

2. The cross-modal co-learning based video generation method according to claim 1, characterized in that, The potential input of the DiT block in the improved conditional diffusion model is obtained after the image sequence latent variables, the noise latent variables and the binary mask are spliced, and the video latent features are generated after the layer normalization and self-attention operation in the DiT block.

3. The cross-modal co-learning based video generation method of claim 1, wherein, The text prompt comprises a first description text for the scene subject part in the target image and a second description text for the camera motion part.

4. The cross-modal co-learning based video generation method of claim 3, wherein, The trajectory extractor is used to obtain frame-level camera motion trajectory information containing camera intrinsic and extrinsic parameter matrices, and the trajectory extractor integrates trajectory extraction units for different modal inputs, comprising: a first modal trajectory extraction unit which uses a large language model to parse the second description text into a discrete motion representation, calculates the extrinsic parameter matrix of each frame, and estimates the intrinsic parameter matrix according to the target image; a second modal trajectory extraction unit which extracts an initial trajectory from a reference video through a GOLMAP algorithm, refines the extrinsic parameter matrix through a multi-layer perceptron, and estimates the intrinsic parameter matrix according to the target image; a third modal trajectory extraction unit which directly reads the trajectory data and performs matrix format conversion.

5. The cross-modal co-learning based video generation method according to claim 1, characterized in that, The extraction process of the camera embeddings comprises: For each frame, the camera center and the pixel ray direction are calculated based on the intrinsic and extrinsic matrix, and the intrinsic and extrinsic matrix is converted into a ray representation in the coordinate system; projecting the ray representation into the camera embeddings through a Transformer module.

6. The cross-modal co-learning based video generation method according to claim 1, characterized in that, The calculation process of the camera-aware attention module comprises: merging the time and space dimensions of the camera embeddings to obtain cross-frame camera embeddings; reshaping the video latent features into the same shape as the cross-frame camera embeddings, calculating the cross-attention of the two on the cross-frame association path, and restoring the calculation result to the original dimension as the first output result; The camera embedding is merged with the batch and time dimensions to obtain an intra-frame camera embedding; the video latent feature is reshaped to have the same shape as the intra-frame camera embedding, cross-attention between the two is calculated on the intra-frame control path, and the calculation result is restored to the original dimension as the second output result; The key-value projection matrix parameters of the two paths are shared, the query projection matrix parameters are independent, and the output results of the two paths are weighted and summed as the final output.

7. The cross-modal co-learning based video generation method according to claim 1, characterized in that, The output results of the camera-aware attention module and the decoupled cross-attention module are weighted and then subjected to layer normalization and feedforward network calculation in the DiT block to obtain the output of the DiT block; The improved conditional diffusion model is obtained by stacking a series of DiT blocks.

8. The cross-modal co-learning based video generation method of claim 1, wherein, The multi-stage curriculum learning strategy includes the following stages: Stage 1: Using a static camera dataset, the model learns the motion of the scene subject; Stage 2: Step-by-step learning of the basic direction motion of the independent camera and the upgrade motion of the degree of freedom; Stage 3: Using a joint control dataset, the scene subject and camera motion are trained to generate a coordinated output.

9. The cross-modal co-learning based video generation method according to claim 1, characterized in that, The improved conditional diffusion model integrates an image encoder and an image decoder at both ends, the image encoder is used to extract image sequence latent variables from the target image and input them into the diffusion model, and the image decoder is used to decode the output of the diffusion model to generate a video frame sequence.

10. A cross-modal co-learning based video generation system for implementing the cross-modal co-learning based video generation method of claim 1, characterized in that, The system comprises: A multi-modal data input module for receiving multi-modal input and extracting multi-modal features, the multi-modal input including a target image, a text prompt, and camera motion trajectory information, and the multi-modal features including image sequence latent variables and global visual features extracted based on the target image, global semantic features extracted based on the text prompt, and camera embedding extracted based on the camera motion trajectory information; An improved conditional diffusion model module integrating a camera-aware attention module and a decoupled cross-attention module arranged side by side, both of which input video latent features processed from image sequence latent variables; In the camera-aware attention module, camera embedding is dynamically injected into the video latent features through a parallel cross-frame association path and an intra-frame control path, and the results of the two paths are fused as the output of the camera-aware attention module; in the decoupled cross-attention module, a first cross-attention result between the global visual features and the video latent features is calculated, and a second cross-attention result between the global semantic features and the video latent features is calculated, and the two cross-attention results are fused as the output of the decoupled cross-attention module; The output results of the camera-aware attention module and the decoupled cross-attention module are weighted and replace the output of the cross-attention module in the original conditional diffusion model; A lightweight fine-tuning module uses low-rank adaptive technology to fine-tune the improved conditional diffusion model; A multi-stage training module executes a multi-stage curriculum learning strategy in the training stage to learn the motion of the scene subject, the motion of the independent camera, and the joint control of the two. An improved conditional diffusion model is obtained by stacking a series of DiT blocks. The multi-stage curriculum learning strategy includes the following stages: Stage 1: Using a static camera dataset, the model learns the motion of the scene subject; Stage 2: Step-by-step learning of the basic direction motion of the independent camera and the upgrade motion of the degree of freedom; Stage 3: Using a joint control dataset, the scene subject and camera motion are trained to generate a coordinated output. The improved conditional diffusion model integrates an image encoder and an image decoder at both ends, the image encoder is used to extract image sequence latent variables from the target image and input them into the diffusion model, and the image decoder is used to decode the output of the diffusion model to generate a video frame sequence. The system comprises: A multi-modal data input module for receiving multi-modal input and extracting multi-modal features, the multi-modal input including a target image, a text prompt, and camera motion trajectory information, and the multi-modal features including image sequence latent variables and global visual features extracted based on the target image, global semantic features extracted based on the text prompt, and camera embedding extracted based on the camera motion trajectory information; An improved conditional diffusion model module integrating a camera-aware attention module and a decoupled cross-attention module arranged side by side, both of which input video latent features processed from image sequence latent variables; And in the camera-aware attention module, camera embedding is dynamically injected into the video latent features through a parallel cross-frame association path and an intra-frame control path, and the results of the two paths are fused as the output of the camera-aware attention module; in the decoupled cross-attention module, a first cross-attention result between the global visual features and the video latent features is calculated, and a second cross-attention result between the global semantic features and the video latent features is calculated, and the two cross-attention results are fused as the output of the decoupled cross-attention module; The output results of the camera-aware attention module and the decoupled cross-attention module are weighted and replace the output of the cross-attention module in the original conditional diffusion model; A lightweight fine-tuning module uses low-rank adaptive technology to fine-tune the improved conditional diffusion model; A multi-stage training module executes a multi-stage curriculum learning strategy in the training stage to learn the motion of the scene subject, the motion of the independent camera, and the joint control of the two.

Images