A method, system, storage medium, and computer equipment for multimodal video generation based on hand-drawn sketches.

Abstract

This invention discloses a multimodal video generation method, system, storage medium, and computer device based on hand-drawn sketches. The method includes the following steps: S1, constructing a text-sketch-video multimodal dataset; S2, encoding the information of the hand-drawn sketches to decompose them into low-frequency and high-frequency components, and then extracting features from the low-frequency and high-frequency components respectively to obtain sketch features composed of low-frequency and high-frequency features; S3, encoding the text description into text features through a text encoder, performing noise fusion processing on the sketch features, and fusing the text features and the noise-fused sketch features through a joint module to generate joint features; S4, inputting the joint features into a generative model containing spatiotemporal convolution and attention mechanisms to generate video frames with spatiotemporal consistency frame by frame. This invention effectively solves the consistency problem and insufficient responsiveness to customized needs in the video generation process.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method, system, storage medium, and computer device for generating multimodal videos based on hand-drawn sketches. Background Technology

[0002] In recent years, video generation technology has made significant progress in computer vision and generative modeling, especially in industries such as entertainment, education, and virtual reality. Existing video generation methods, such as those based on Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), while achieving breakthroughs in image generation, still face many challenges in video generation, such as insufficient spatiotemporal consistency, content diversity, and user customization control. Existing image generation techniques have been extended to the temporal dimension, but still struggle to achieve coherence between frames in a video, as corresponding frames lack adequate feature information.

[0003] Therefore, how to ensure the spatiotemporal consistency of video while enabling users to have fine-grained control over video content and improving the diversity and interpretability of generated content is a key issue that current video generation technology urgently needs to address. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the first objective of the present invention is to provide a multimodal video generation method based on hand-drawn sketches.

[0005] A second objective of this invention is to provide a multimodal video generation system based on hand-drawn sketches.

[0006] A third objective of this invention is to provide a storage medium.

[0007] A fourth objective of this invention is to provide a computer device.

[0008] The first objective of this invention is achieved through the following technical solution: This multimodal video generation method based on hand-drawn sketches includes the following steps:

[0009] S1. Construct a text-sketch-video multimodal dataset;

[0010] S2 、 The information of the hand-drawn sketch is encoded to decompose it into low-frequency and high-frequency components. Then, features are extracted from the low-frequency and high-frequency components respectively to obtain the sketch features composed of low-frequency and high-frequency features. ;

[0011] S3. The text description is encoded into text features by a text encoder, the sketch features are subjected to noise fusion processing, and the text features and the noise-fused sketch features are fused through a joint module to generate joint features.

[0012] S4. The joint feature input is used to generate video frames with spatiotemporal consistency by employing a generative model that includes spatiotemporal convolution and attention mechanisms.

[0013] The process of constructing the multimodal dataset in step S1 includes:

[0014] S11. Obtain sketches and their corresponding images from publicly available sketch-image datasets. These images are then extended into short videos using generative adversarial network techniques to form a sketch-image dataset.

[0015] S12. Extract keyframes from the publicly available video dataset, then use an edge detection algorithm to generate corresponding sketches, and combine this with manual corrections to construct a video-sketch dataset.

[0016] The edge detection algorithm is the Canny edge detection algorithm.

[0017] Step S2 includes the following specific steps:

[0018] S21. Encode the hand-drawn sketches to obtain a sketch sequence. Let N be a natural number. Decompose the sketch sequence into a frequency domain representation using a two-dimensional discrete Fourier transform:

[0019] ,

[0020] Where H and W are the height and width of the sketch image in the sketch sequence, respectively, u and v are two coordinates in the frequency domain, and S... i It is a two-dimensional grayscale image;

[0021] S22. Represent the frequency domain Divided into low-frequency components and high-frequency components ;

[0022] S23, Regarding low-frequency components and high-frequency components Perform two-dimensional discrete Fourier inverse transforms on each to obtain the low-frequency features. and high frequency characteristics The composition of the sketch features .

[0023] Step S3 includes the following specific steps:

[0024] S31. Encode the text description using a text encoder to generate text features that complement the sketch features;

[0025] S32. Generation and Sketch Features A uniform Gaussian noise matrix N, where , Control the amplitude of noise;

[0026] S33. Perform noise fusion between the Gaussian noise matrix and the sketch features according to preset weighting coefficients to obtain the fused sketch features. :

[0027]

[0028] in, These are the weighting coefficients;

[0029] S34. Text Features and Sketch Features By combining modules, joint features are generated.

[0030] The generative model employs a multi-layer network structure, with each layer comprising:

[0031] The spatial convolution module is used to extract the spatial structural features of the current frame image;

[0032] The temporal convolution module is used to extract dynamic change features between adjacent frames;

[0033] The attention mechanism module, including spatial attention and temporal attention, is used to enhance the feature representation of key regions within a frame and capture inter-frame dependencies.

[0034] The process of generating video frames in step S4 is as follows:

[0035] S41. Input the joint features into the generative model to perform spatial and temporal convolution operations on the joint features in order to extract preliminary spatial and dynamic information;

[0036] S42. An attention mechanism module is used to enhance the capture of important intra-frame regions and inter-frame dependencies;

[0037] S43. Through progressive upsampling and decoding operations, each frame is made consistent with other frames in both space and time, generating the final video frame.

[0038] The second objective of this invention is achieved through the following technical solution: a multimodal video generation system based on hand-drawn sketches, comprising:

[0039] The dataset building module is used to construct text-sketch-video multimodal datasets;

[0040] The sketch encoding and feature extraction module is used to decompose the sequence of hand-drawn sketches into low-frequency and high-frequency features through frequency domain transformation.

[0041] The multimodal feature fusion module is used to fuse text features with noise-enhanced sketch features to generate joint features;

[0042] Video generation module, used for by Using joint features as input, spatiotemporal convolution and attention mechanisms are employed to generate video frames with spatiotemporal consistency.

[0043] The third objective of this invention is achieved through the following technical solution: a storage medium storing a program, which, when executed by a processor, implements the multimodal video generation method based on hand-drawn sketches as described in the first objective.

[0044] The fourth objective of this invention is achieved through the following technical solution: a computer device, including a processor and a memory for storing processor-executable programs, wherein when the processor executes the program stored in the memory, it implements the multimodal video generation method based on hand-drawn sketches as described in the first objective.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] 1. This invention constructs a text-sketch-video multimodal dataset, using sketches to provide the initial outline for video generation, and combining text prompts to enhance the diversity and creativity of the generation, achieving fine-grained control over the structure, layout and dynamics of objects, and improving the interpretability and user customization capabilities of the generated content.

[0047] 2. This invention decomposes the sketch into low-frequency and high-frequency features through two-dimensional discrete Fourier transform, enabling the model to capture the overall structure of the object while preserving detailed information such as edge texture, thereby enhancing the structural fidelity and visual richness of the generated video.

[0048] 3. This invention introduces controllable Gaussian noise into the sketch features to improve the model's tolerance to poor or incomplete sketch quality, thus ensuring the stability of the generated video.

[0049] 4. This invention deeply integrates text features with enhanced sketch features to achieve alignment and complementarity of semantic and structural information, thereby improving the semantic consistency of generated content and user customization capabilities.

[0050] 5. This invention employs a joint modeling approach combining multi-layer spatiotemporal convolution and attention mechanisms, effectively improving the intra-frame spatial quality and inter-frame temporal coherence of the generated video, thus solving the consistency problem and insufficient responsiveness to customized needs in the video generation process.

[0051] 6. The multimodal dataset in this invention constructs a high-quality sketch-video pairing dataset by combining image generation and edge detection with manual correction, providing rich and high-quality training samples for multimodal video generation, and significantly improving the model's generalization ability and the diversity of generated content. Attached Figure Description

[0052] Figure 1 This is a flowchart of the multimodal video generation method based on hand-drawn sketches of the present invention.

[0053] Figure 2 This is an example diagram illustrating the construction of a multimodal dataset according to the present invention.

[0054] Figure 3 This is a schematic diagram illustrating the process of generating video using the multimodal video generation method based on hand-drawn sketches according to the present invention.

[0055] Figure 4 This is an example diagram of video generation using the multimodal video generation method based on hand-drawn sketches of the present invention. Detailed Implementation

[0056] The present invention is further described below through specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0057] like Figure 1 As shown, the multimodal video generation method based on hand-drawn sketches includes the following steps:

[0058] S1. Construct a text-sketch-video multimodal dataset;

[0059] The construction of the multimodal dataset in step S1 includes:

[0060] S11. Obtain sketches and their corresponding images from publicly available sketch-image datasets. These images are then extended into short videos using generative adversarial network techniques to form a sketch-image dataset.

[0061] S12. Keyframes are extracted from publicly available video datasets, and corresponding sketches are generated using an edge detection algorithm. These sketches are then combined with manual corrections to construct a video-sketch dataset. The edge detection algorithm used is the Canny edge detection algorithm. This method generates a large-scale, high-quality video-sketch dataset, enabling the model to learn the dynamic relationship between sketches and videos.

[0062] Specifically, this implementation constructs two different types of multimodal datasets, such as... Figure 2As shown, the dataset includes a sketch-image dataset consisting of single sketches and images, and a video-sketch dataset consisting of multiple videos and sketches. The sketch-image dataset has 8000 sketch-image pairs, while the video-sketch dataset has 20 video-sketch pairs. This covers a variety of scenes and objects, enriching the diversity and representativeness of the training data. Multimodal datasets are key to addressing the lack of diversity in generated content, enabling models to be trained in a wider range of scenarios, thereby improving the quality and content diversity of generated videos.

[0063] S2. Encode the information of the hand-drawn sketch into low-frequency and high-frequency components, and then extract features from the low-frequency and high-frequency components respectively to obtain the sketch features composed of low-frequency and high-frequency features. ;

[0064] Step S2 includes the following specific steps:

[0065] S21. Encode the hand-drawn sketches into a sketch sequence. N is a natural number, and each sketch It can be represented as a two-dimensional grayscale image with a size of Where H and W are the height and width of the image, respectively, the sketch sequence is decomposed into a frequency domain representation using a two-dimensional discrete Fourier transform:

[0066] ,

[0067] Where H and W are the height and width of the sketch image in the sketch sequence, respectively, u and v are two coordinates in the frequency domain, and S... i It is a two-dimensional grayscale image;

[0068] S22. Represent the frequency domain Divided into low-frequency components and high-frequency components Low-frequency components are mainly concentrated in the center of the frequency domain and contain the overall contour information of the image; while high-frequency components are located at the edge of the frequency domain and contain the detailed information of the image.

[0069] S23, Regarding low-frequency components and high-frequency components Perform two-dimensional discrete Fourier inverse transforms on each to obtain the low-frequency features. and high frequency characteristics The composition of the sketch features Among them, low-frequency characteristics It mainly contains the general outline information of the sketch, which is used by the model to understand the overall structure of the object; high-frequency features It mainly contains detailed information from the sketch, used to preserve the edge and texture features in the sketch.

[0070] S3. The text description is encoded into text features by a text encoder, the sketch features are subjected to noise fusion processing, and the text features and the noise-fused sketch features are fused through a joint module to generate joint features.

[0071] Step S3 includes the following specific steps:

[0072] S31. Encode the text description using a text encoder to generate text features that complement the sketch features;

[0073] S32. Generation and Sketch Features A uniform Gaussian noise matrix N, where , Control the amplitude of noise;

[0074] S33. Perform noise fusion between the Gaussian noise matrix and the sketch features according to preset weighting coefficients to obtain the fused sketch features. :

[0075]

[0076] in, These are the weighting coefficients;

[0077] Before fusing textual and sketch features, the sketch features are first subjected to noise fusion processing. The purpose of this is to enhance the robustness of the features even when the sketch quality is low or the sketch is incomplete, so that the model can still generate high-quality videos when processing imperfect sketches.

[0078] S34. Text Features and Sketch Features By combining modules, joint features are generated.

[0079] Specifically, this invention, by combining with text prompts, achieves effective control over the video generation process, allowing users to flexibly determine the object layout and dynamic characteristics of the video. Furthermore, the framework also considers the role of sequence sketches; if the input is a sequence sketch, multiple sketches will be used as intermediate features for the generation of subsequent video frames, thereby controlling the frame generation effect at different time points in the video.

[0080] Compared to traditional methods, which often don't prioritize sketches in video generation and rely primarily on text information, this invention places the highest weight on hand-drawn sketches to ensure their features effectively influence the video generation logic. Furthermore, the sketch sequence serves as a framework, directly impacting the entire video frame generation process. Therefore, by combining sketches and text prompts, users gain fine-grained control over video content, providing a highly personalized video generation experience with diverse and creative content.

[0081] S4. The joint feature input is used to generate video frames with spatiotemporal consistency by employing a generative model that includes spatiotemporal convolution and attention mechanisms.

[0082] The generative model employs a multi-layer network structure, with each layer comprising:

[0083] The spatial convolution module is used to extract the spatial structural features of the current frame image;

[0084] The temporal convolution module is used to extract dynamic change features between adjacent frames;

[0085] The attention mechanism module, including spatial attention and temporal attention, is used to enhance the feature representation of key regions within a frame and capture inter-frame dependencies.

[0086] like Figure 3 and 4 As shown, the process of generating video frames in step S4 is as follows:

[0087] S41. Input the joint features into the generative model to perform spatial and temporal convolution operations on the joint features in order to extract preliminary spatial and dynamic information;

[0088] S42. An attention mechanism module is used to enhance the capture of important intra-frame regions and inter-frame dependencies;

[0089] S43. Through progressive upsampling and decoding operations, each frame is made consistent with other frames in both space and time, generating the final video frame.

[0090] Specifically, the generative model is constructed by combining multi-layer spatiotemporal convolution with an attention mechanism. Spatiotemporal convolution is used to process the spatial and temporal information of each generated frame, while the attention mechanism is used to enhance the connection between frames during the generation process, thereby ensuring the consistency of the generated video frames in the spatial and temporal dimensions.

[0091] Spatial convolution is used to extract spatial features from each frame of an image, mainly information such as the shape, texture, and edges of objects. Given the feature representation of a particular frame... Spatial convolution operation can be represented as:

[0092]

[0093] in, The symbol represents the spatial convolution kernel, and * indicates the convolution operation. For bias.

[0094] Temporal convolution is used to handle the continuity between video frames, extract dynamic features between adjacent frames, and ensure that the generated video remains consistent in the temporal dimension.

[0095] make , and These represent the feature representations of adjacent frames, processed by temporal convolution kernels. Temporal convolution operations can be described as follows:

[0096] .

[0097] The generative model employs multi-layered spatiotemporal convolution and attention mechanisms, with each layer including spatial convolution, temporal convolution, spatial attention, and temporal attention. These modules are stacked layer by layer, progressively enhancing the spatial and temporal features during the generation process. First, spatial and temporal convolution operations are performed on the joint features of the input to extract preliminary spatial and dynamic information. Second, spatial and temporal attention mechanisms are added in the intermediate layers to enhance the model's ability to capture important intra-frame regions and inter-frame dependencies. Finally, through progressive upsampling and decoding, the final video frames are generated, ensuring that each frame is spatially and temporally consistent with other frames.

[0098] like Figure 4 As shown, this embodiment uses the example of "a motorcyclist riding on the road" to illustrate the specific video generation process as follows:

[0099] First, the input sketch sequence and text prompts (e.g., "a motorcyclist is riding on the road") are fed into a diffusible module (DiT initialization parameter module). This DiT initialization parameter module extracts the structural intent of the sketch and the semantic information of the text, transforming them into semantic features in the latent space (i.e., the "semantic initialization" feature blocks in the figure), which serve as the base tokens for the subsequent Transformer encoder. To ensure high consistency in appearance and subject features of the generated video, the system first generates a high-quality first frame based on the initial sketch and prompts. Subsequently, the generated first frame is fused with a pre-set noise map (i.e., "initialization control + noise") to extract control conditions that guide the generation of the entire video sequence. This first-frame-based control mechanism effectively anchors the global color, texture, and subject identity of the video.

[0100] Then, the tokens obtained from semantic initialization are fed into a DiT-based encoding layer. Unlike traditional Transformers, this encoding layer introduces an adaptive control mechanism (similar to AdaLN). Specifically:

[0101] The extracted control conditions are converted into different control signals (control parameter 1 and control parameter 2).

[0102] These control parameters are dynamically applied to the input and output of the multi-head attention module through scale (scaling / translation) operations:

[0103] Extraction control conditions These signals are converted into different control signals. Specifically, this is achieved through a multilayer perceptron (MLP). Mapped to include scaling factor Translation factor (Corresponding to control parameter 1) and gating factor (Parameter group corresponding to control parameter 2):

[0104]

[0105] First Scale Modulation (Feature Injection): These control parameters are dynamically applied to the input of the multi-head attention module. Input tokens undergo feature modulation via the Scale module.

[0106]

[0107] At this point, the attention mechanism not only handles spatial dependencies within the current frame (capturing relationships such as the relationship between the motorcyclist and the road), but also, combined with the aforementioned spatiotemporal convolution logic, processes the temporal continuity between frames. By introducing a control module with residual connections, the model can forcibly inject details and noise distribution information from the first frame without disrupting the original semantic feature flow, ensuring that the generated content both conforms to the text description and strictly follows the motion trajectory of the sketch.

[0108]

[0109] Second Scale Modulation and Residual Fusion: To avoid disrupting the original semantic feature stream, residual connections are used ( Figure 4 In The control module utilizes gating factors. The contribution of attention features to the main feature flow is dynamically determined and fused with the original input:

[0110]

[0111] This process ensures that the generated content both conforms to the text description and strictly follows the motion trajectory of the sketch.

[0112] Finally, after processing through multiple layers of "DiT-based coding layers," the latent representation, containing rich spatiotemporal dynamic information and fine-grained control features, is fed into the DiT Decoder. The decoding layer maps the features of the latent space back to the pixel space through progressive decoding and upsampling operations, ultimately outputting a continuous, smooth, and high-fidelity video frame sequence.

[0113] Thus, this invention cleverly solves the problem of insufficient control over hand-drawn sketches through a dual-branch design (a semantic token extraction branch and a first-frame control branch). It utilizes the first-frame image and noise to construct hard control conditions and dynamically modulates the multi-head attention mechanism through the scale parameter in the DiT network. This not only fully extracts the outline and detail information of the sketch but also enriches the semantics of the image using textual prompts (such as "motorcyclist" and "riding on the road"), ultimately achieving high-quality video generation that combines spatial consistency and temporal fluency in complex dynamic scenes. Furthermore, when combining sketch frequency domain processing with spatiotemporal convolution, this invention, on the one hand, uses frequency domain processing combined with noise during the initialization of video generation to enhance the diversity of the generated content and avoid overly concentrated content. The addition of noise gives each generation a certain degree of randomness, thereby increasing the diversity of the generated videos while ensuring consistency and making the content of the generated videos richer. On the other hand, it uses high-frequency and low-frequency features obtained from frequency domain processing of the sketch, combined with multi-layer spatiotemporal convolution and attention mechanisms, to fully model the sketch features. Frequency domain processing enables the model to have good representation capabilities for both high-frequency and low-frequency features of the sketch, while the combination of multi-layer spatiotemporal convolution and attention mechanism ensures the spatial consistency and temporal fluency of the generated video.

[0114] Therefore, compared to traditional processing methods, which typically disregard the multi-layered feature information of simple sketches, focusing only on the outline and neglecting the details, this invention addresses this issue. Traditional methods often fail to consider the multi-layered feature information of simple sketches, focusing instead on the outline and details. If the sketch is rudimentary, the outline information carries greater weight; however, if the sketch is detailed, more attention is paid to its details. Furthermore, by incorporating the spatiotemporal attention features of the sketch sequence, the model enhances its ability to capture important intra-frame regions and inter-frame dependencies.

[0115] A multimodal video generation system based on hand-drawn sketches includes:

[0116] The dataset building module is used to construct text-sketch-video multimodal datasets;

[0117] The sketch encoding and feature extraction module is used to decompose the sequence of hand-drawn sketches into low-frequency and high-frequency features through frequency domain transformation.

[0118] The multimodal feature fusion module is used to fuse text features with noise-enhanced sketch features to generate joint features;

[0119] The video generation module uses joint features as input and employs spatiotemporal convolution and attention mechanisms to generate video frames with spatiotemporal consistency.

[0120] The storage medium stores a program that, when executed by a processor, implements the aforementioned multimodal video generation method based on hand-drawn sketches.

[0121] A computer device includes a processor and a memory for storing processor-executable programs, wherein when the processor executes the program stored in the memory, it implements the above-described multimodal video generation method based on hand-drawn sketches.

[0122] Those skilled in the art will readily understand that the above description is merely an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal video generation method based on hand-drawn sketches, characterized in that, Includes the following steps: S1. Construct a text-sketch-video multimodal dataset; S2. Encode the information of the hand-drawn sketch into low-frequency and high-frequency components, and then extract features from the low-frequency and high-frequency components respectively to obtain the sketch features composed of low-frequency and high-frequency features. ; S3. The text description is encoded into text features by a text encoder, the sketch features are subjected to noise fusion processing, and the text features and the noise-fused sketch features are fused through a joint module to generate joint features. S4. Input the joint features into the generative model containing spatiotemporal convolution and attention mechanisms to generate video frames with spatiotemporal consistency frame by frame.

2. The multimodal video generation method based on hand-drawn sketches according to claim 1, characterized in that, The process of constructing the multimodal dataset in step S1 includes: S11. Obtain sketches and their corresponding images from publicly available sketch-image datasets. These images are then extended into short videos using generative adversarial network techniques to form a sketch-image dataset. S12. Extract keyframes from the publicly available video dataset, then use an edge detection algorithm to generate corresponding sketches, and combine this with manual corrections to construct a video-sketch dataset.

3. The multimodal video generation method based on hand-drawn sketches according to claim 2, characterized in that, The edge detection algorithm is the Canny edge detection algorithm.

4. The multimodal video generation method based on hand-drawn sketches according to claim 1, characterized in that, Step S2 includes the following specific steps: S21. Encode the hand-drawn sketches into a sketch sequence. Let N be a natural number. Decompose the sketch sequence into a frequency domain representation using a two-dimensional discrete Fourier transform: ， Where H and W are the height and width of the sketch image in the sketch sequence, respectively, u and v are two coordinates in the frequency domain, and S... i It is a two-dimensional grayscale image; S22. Represent the frequency domain Divided into low-frequency components and high-frequency components ; S23, Regarding low-frequency components and high-frequency components Perform two-dimensional discrete Fourier inverse transforms on each to obtain the low-frequency features. and high frequency characteristics The composition of the sketch features .

5. The multimodal video generation method based on hand-drawn sketches according to claim 1, characterized in that, Step S3 includes the following specific steps: S31. Encode the text description using a text encoder to generate text features that complement the sketch features; S32. Generation and Sketch Features A uniform Gaussian noise matrix N, where , Control the amplitude of noise; S33. Perform noise fusion between the Gaussian noise matrix and the sketch features according to preset weighting coefficients to obtain the fused sketch features. : in, These are the weighting coefficients; S34. Text Features and Sketch Features By combining modules, joint features are generated.

6. The multimodal video generation method based on hand-drawn sketches according to claim 1, characterized in that, The generative model employs a multi-layer network structure, with each layer comprising: The spatial convolution module is used to extract the spatial structural features of the current frame image; The temporal convolution module is used to extract dynamic change features between adjacent frames; The attention mechanism module, including spatial attention and temporal attention, is used to enhance the feature representation of key regions within a frame and capture inter-frame dependencies.

7. The multimodal video generation method based on hand-drawn sketches according to claim 5, characterized in that, The process of generating video frames in step S4 is as follows: S41. Input the joint features into the generative model to perform spatial and temporal convolution operations on the joint features in order to extract preliminary spatial and dynamic information; S42. An attention mechanism module is used to enhance the capture of important intra-frame regions and inter-frame dependencies; S43. Through progressive upsampling and decoding operations, each frame is made consistent with other frames in both space and time, generating the final video frame.

8. A multimodal video generation system based on hand-drawn sketches, characterized in that, include: The dataset building module is used to construct text-sketch-video multimodal datasets; The sketch encoding and feature extraction module is used to decompose the sequence of hand-drawn sketches into low-frequency and high-frequency features through frequency domain transformation. The multimodal feature fusion module is used to fuse text features with noise-enhanced sketch features to generate joint features; The video generation module uses joint features as input and employs spatiotemporal convolution and attention mechanisms to generate video frames with spatiotemporal consistency.

9. A storage medium, characterized in that, The system contains a program that, when executed by a processor, implements the multimodal video generation method based on hand-drawn sketches as described in any one of claims 1-7.

10. A computer device, comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the multimodal video generation method based on hand-drawn sketches as described in any one of claims 1-7.

Images