Video generation method and device based on multi-modal information fusion, equipment and medium

Abstract

The application relates to the fields of artificial intelligence and medical health, and discloses a video generation method based on multi-modal information fusion, which comprises the following steps: inputting image information and text information, generating image feature representation and text feature representation; injecting noise into the image feature representation to generate a noisy latent representation; inputting the noisy latent representation into a denoising model to generate an optimized latent representation; inputting the text feature representation into the denoising model, performing semantic fusion with the optimized latent representation, and generating a semantic latent representation; and inputting the semantic latent representation into a decoder to generate a video frame sequence and output the video frame sequence. Through multi-modal fusion of the image feature representation and the text feature representation, the application ensures that the generated video is consistent with the input content in terms of semantics; through noise injection and denoising optimization, the application enhances the diversity and detail quality of video generation; and through the decoder, the application generates a video frame sequence, thereby guaranteeing the visual coherence and natural transition of the video.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and medical health, and in particular to a video generation method, device, equipment and storage medium based on multimodal information fusion. Background Art

[0002] Image-guided video generation technology is an important area of ​​video generation, widely used in multimedia content production, virtual reality experiences, smart marketing, and other fields. In the medical field, this technology can be used in scenarios such as assisted diagnosis, surgical simulation, medical education, and health management. Generating videos using given image and text information as input can significantly reduce video production costs and improve content generation efficiency in medical scenarios. However, existing technologies still have many shortcomings in addressing the consistency, coherence, and semantic accuracy of video generation.

[0003] When generating medical videos, current technologies often struggle to ensure consistency in both spatial and temporal content. While some methods can generate medical video clips that semantically match the input image (e.g., a 3D reconstructed animation of a lesion), unnatural jumps or breaks occur in the transition between video frames, resulting in a less-than-smooth medical video. Furthermore, long-duration medical video generation tasks (e.g., surgical simulation videos or treatment process animations) are more prone to content distortion or inconsistency, especially when the simulation scene is complex or detailed, making it difficult for the model to effectively capture the logical connections between consecutive frames.

[0004] Existing video generation technologies have certain limitations in processing the integration of multimodal information (such as medical images, patient text information, etc.). Some methods generate medical videos through a staged optimization approach. However, because the optimization objectives of each stage are too similar, the distribution of model learning tends to be consistent, lacking the diversity and flexibility required to generate complex medical scenes, and the quality of the final generated video is limited. In addition, another type of method relies on external guidance information (such as lesion descriptions, diagnosis and treatment step text, or time series signals). Although it can improve the generation effect to a certain extent, its ability to integrate multimodal information is limited, making it difficult to achieve high-quality medical video generation in terms of vision, semantics, and timing.

[0005] Although some methods have improved generation performance through multimodal fusion and hierarchical modeling, these methods typically require a large amount of additional medical training data (such as lesion image annotation data and text description data) and complex modifications to the model architecture. Furthermore, multi-stage optimization or the introduction of external guidance signals significantly increases the computational overhead and hardware resource dependence of the training process. These factors limit the widespread application of video generation technology in the medical field, especially in resource-limited medical institutions.

[0006] In the medical field, the application potential of video generation technology has not yet been fully realized, especially in the areas of personalized medical video generation and batch production of medical teaching content, where there is still considerable room for improvement. For example, in medical image-assisted analysis, the image videos generated by existing technologies lack fine-grained control over the lesion area, and the generated medical animation content is not well matched to actual clinical needs. In addition, the high computational cost of existing methods further limits their potential for large-scale application in the medical field, making it difficult for medical institutions to effectively use this technology for daily teaching, surgical planning, and patient education. By improving video generation technology, while improving content consistency, detail accuracy, and generation efficiency, its practical application value in the medical field can be significantly expanded. Summary of the Invention

[0007] The main purpose of the present invention is to provide a video generation method, device, equipment and storage medium based on multimodal information fusion, aiming to solve the technical problem that the existing technology is difficult to generate semantically consistent and visually coherent video content through multimodal information fusion, resulting in insufficient content consistency and natural transition of the generated video.

[0008] To achieve the above object, the present invention provides a video generation method based on multimodal information fusion, comprising:

[0009] Obtain image information and generate image feature representation through image encoder;

[0010] Obtain text information and generate text feature representation through a text encoder;

[0011] injecting noise into the image feature representation to generate a noisy latent representation;

[0012] Inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0013] Inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0014] The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

[0015] Furthermore, to achieve the above-mentioned object, the present invention provides a video generation device based on multimodal information fusion, comprising:

[0016] Image encoding module, used to obtain image information and generate image feature representation through image encoder;

[0017] The text encoding module is used to obtain text information and generate text feature representation through the text encoder;

[0018] a noise injection module, configured to inject noise into the image feature representation to generate a noisy latent representation;

[0019] a denoising model module, configured to input the noisy latent representation into a denoising model to generate an optimized latent representation;

[0020] A semantic fusion module, configured to input the text feature representation into the denoising model and perform semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0021] The decoding module is used to input the semantic latent representation into a decoder, generate a video frame sequence through the decoder, and output it.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a video generation program based on multimodal information fusion stored in the memory and runnable on the processor. When the video generation program based on multimodal information fusion is executed by the processor, the steps of the video generation method based on multimodal information fusion as described above are implemented.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a video generation program based on multimodal information fusion is stored. When the video generation program based on multimodal information fusion is executed by a processor, the steps of the video generation method based on multimodal information fusion as described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and medical health, and discloses a video generation method based on multimodal information fusion, which generates image feature representation and text feature representation by inputting image information and text information; injects noise into the image feature representation to generate a noisy latent representation; inputs the noisy latent representation into a denoising model to generate an optimized latent representation; inputs the text feature representation into the denoising model, performs semantic fusion with the optimized latent representation to generate a semantic latent representation; inputs the semantic latent representation into a decoder to generate and output a video frame sequence. The present invention ensures that the generated video is semantically consistent with the input content through multimodal fusion of image feature representation and text feature representation; enhances the diversity and detail quality of video generation through noise injection and denoising optimization; generates a video frame sequence through a decoder to ensure the visual coherence and natural transition of the video. The semantic consistency, visual coherence and generation quality of the generated video are significantly improved, while reducing the complexity of the generation process, and is suitable for batch and efficient generation of video materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 Schematic diagram of an application environment of a video generation method based on multimodal information fusion in one embodiment of the present invention;

[0027] Figure 2 1 is a flow chart of an embodiment of a video generation method based on multimodal information fusion according to the present invention;

[0028] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a video generation device based on multimodal information fusion according to the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The video generation method based on multimodal information fusion provided by the embodiment of the present invention can be applied in the following fields: Figure 1 In an application environment, a user terminal communicates with a server terminal via a network. The server terminal can input image and text information through the user terminal to generate image feature representations and text feature representations; inject noise into the image feature representation to generate a noisy latent representation; input the noisy latent representation into a denoising model to generate an optimized latent representation; input the text feature representation into the denoising model and semantically fuse it with the optimized latent representation to generate a semantic latent representation; input the semantic latent representation into a decoder to generate and output a video frame sequence. The present invention ensures that the generated video is semantically consistent with the input content through multimodal fusion of image and text feature representations; enhances the diversity and detail quality of the generated video through noise injection and denoising optimization; and generates a video frame sequence through the decoder, ensuring visual coherence and natural transitions of the video. This significantly improves the semantic consistency, visual coherence, and generation quality of the generated video while reducing the complexity of the generation process, making it suitable for batch and efficient video material generation. The user terminal can include, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server terminal can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2This is a flow chart of an embodiment of a video generation method based on multimodal information fusion provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the video generation method based on multimodal information fusion proposed by the present invention includes the following steps:

[0035] S10, acquiring image information and generating image feature representation through an image encoder;

[0036] In this embodiment, acquiring image information is the starting point of the video generation process. This refers to obtaining the original image from an input device or a pre-stored dataset. This image information can be a single frame, the first frame of a dynamic image, or a set of images related to the generation target. The acquired image information must meet certain quality requirements, such as resolution and clarity, to ensure that the subsequent encoding process can effectively extract image features.

[0037] Read image files from databases or file systems, supporting common formats such as JPEG and PNG. Use image acquisition devices (such as cameras, mobile phones, or scanners) to acquire real-time images and convert them into standardized formats. Acquire remote image data (such as financial enterprise advertising images, user-uploaded images, etc.) through network interfaces and preprocess them to eliminate noise and compression artifacts.

[0038] An image encoder's task is to extract and represent features from the input image. Its primary function is to map high-dimensional pixel information into a low-dimensional latent space, extracting key image features such as color, texture, edges, and shape. The resulting image feature representation is a numerical intermediate representation that facilitates subsequent processing and the fusion of multimodal information.

[0039] The image encoder generates image feature representations using the following formula:

[0040] F I =E(X)

[0041] F I It is a generated image feature representation that contains the key features of the input image (such as color, texture, edge, and shape).

[0042] The image encoder E maps high-dimensional pixel information to a low-dimensional latent space and generates a numerical intermediate representation F. I , which is convenient for subsequent processing.

[0043] A convolutional neural network (CNN) is used as an image encoder to extract local features of the image through multi-layer convolution operations; an attention mechanism (such as a self-attention module) is added to the CNN structure to highlight important feature areas; image features are subjected to dimensionality reduction processing (such as a fully connected layer or pooling operation) to generate a latent representation vector of fixed dimension; and the image feature representation is output, usually as a fixed-size feature vector or feature map matrix.

[0044] Example: Animation of the disease diagnosis process can be used for patient education, helping patients intuitively understand the causes of the disease and the treatment process. By inputting an image of the lesion area (such as an X-ray or ultrasound image) and a text description of the disease, an easy-to-understand disease diagnosis animation can be generated. This can be achieved in the following ways:

[0045] Image input and feature extraction: Input the patient's chest X-ray and extract the lung features in the image, including the shape, size, and distribution of lung nodules.

[0046] Text input and feature extraction: Input text descriptions, such as "the formation mechanism and treatment plan of lung nodules"; extract key information from the text, including the causes of nodule formation, diagnostic methods, and recommended treatment steps.

[0047] Multimodal semantic fusion: Fusion of image and text features to generate semantic latent representations, ensuring that animation content contains scientific and educational significance.

[0048] Video generation and output: Decoding semantic latent representations to generate disease animations, showing the formation, development, and dynamic changes of treatment effects of lung nodules.

[0049] Help patients and their families understand their condition intuitively, and improve patients' understanding and cooperation with treatment plans; used for communication between doctors and patients, reducing communication barriers caused by professional terminology.

[0050] By acquiring image information and encoding feature representations, we can extract key visual content, providing the semantic and visual information foundation for subsequent video generation. This not only improves the quality and detail of the generated video content, but also ensures that the generated content accurately reflects the information of the input image, effectively reducing the potential for content deviation during the generation process.

[0051] S20, obtaining text information and generating text feature representation through a text encoder;

[0052] In this embodiment, acquiring text information is one of the key starting points of the multimodal fusion process. This text information can include natural language descriptions, keywords, or tags, and can be derived from user input, system-provided preset templates, or third-party data interfaces. The acquisition of text information requires preprocessing to ensure its grammatical and formatting validity.

[0053] Text source: Receive natural language text information from user input boxes, databases, or API interfaces.

[0054] Format standardization: Format the input text (such as removing special symbols, spaces, and punctuation marks) to generate clean text data.

[0055] Language detection: In multilingual scenarios, use language identification tools to annotate text data to select an appropriate text encoder.

[0056] The core task of a text encoder is to convert natural language text into numerical feature representations, extract the text's semantic content, and map it into a latent space. The text encoder performs word segmentation, context modeling, and semantic enhancement on the text data, ensuring that the generated feature representation fully captures the text's semantic information.

[0057] In this process, the feature extraction process of the text encoder can be expressed by the following formula:

[0058] F T =CLIP(Text)

[0059] Among them, Text represents the input natural language text data; CLIP (Text) represents the encoding of text data using the CLIP model; F T The generated text feature representation contains the global semantic information and contextual relationships of the text.

[0060] CLIP (Contrastive Language–Image Pretraining) is a multimodal model proposed by OpenAI that can simultaneously process text and image data. Its core goal is to jointly learn representations of text and images, mapping the two modalities into the same latent space, thereby achieving cross-modal feature alignment and mutual understanding. Its main application scenarios include image classification, image search, and multimodal generation (such as image description generation and video generation).

[0061] Perform word segmentation on the text data, breaking the sentence into a word sequence. For example, "financial product recommendation" is segmented into ["finance", "product", "recommendation"];

[0062] Use a pre-trained word vector model (such as Word2Vec or GloVe) to embed the word segmentation results and generate corresponding numerical representations;

[0063] Input the word embedding vector into the text encoder (such as the text encoding part of the CLIP model), capture the contextual relationship of the text through a multi-layer Transformer, and extract the initial text features;

[0064] Under the attention mechanism of the text encoder, the initial features are enhanced to highlight the semantic focus;

[0065] The final output is the encoded text feature representation F T .

[0066] Through the processing of the text encoder, natural language text can be accurately converted into numerical semantic representations. Compared with traditional labeling methods, text feature representation provides a deeper expression of contextual information, significantly improving the semantic accuracy and consistency of video generation. In financial scenarios, it can generate intelligent marketing videos for financial products and services that are more closely aligned with customer needs, reducing production costs and improving promotional efficiency.

[0067] S30, injecting noise into the image feature representation to generate a noisy latent representation;

[0068] In this embodiment, noise is added to the image feature representation to generate a noisy latent representation, enhancing the diversity of the generative model and providing a foundation for the subsequent denoising process. The noise injection process requires consideration of the noise type, intensity, and injection method to ensure that it is properly integrated with the image feature representation and that the noise injection does not destroy the core information of the image features.

[0069] In this process, the process of injecting noise can be expressed by the following formula:

[0070]

[0071] Among them, z t represents the noisy potential representation at time step t; z0 represents the initial image feature representation; ∈ t represents the random noise corresponding to the time step t; a t Represents the intensity weight of noise injection, which controls the fusion ratio of noise and image feature representation.

[0072] Generate random noise ∈ with the same dimension as the image feature representation using normal distribution t For example, assuming that the image features are represented as a 128-dimensional vector, a random noise vector of the same size is generated.

[0073] According to the time step t, the intensity weight a of the noise injection is calculated by a predefined function or a dynamic adjustment strategy t Generally, as the time step t increases, the noise weight gradually increases.

[0074] The generated random noise ∈ t The weighted superposition with the image feature representation z0 is performed according to the formula to obtain the noisy potential representation z of the current time step t t . The generated zt As the input of the subsequent denoising model, it is used to enhance the generative model's ability to recover from noise and the diverse expression of potential features.

[0075] Example: When generating a credit card marketing video, the system adds random noise (e.g., normally distributed noise with a mean of 0 and a standard deviation of 0.2) to the feature representation of a sample credit card image. This noisy latent representation serves as input to the generative model, enhancing the detail and richness of the generated video. The resulting advertising video showcases different credit card application scenarios.

[0076] By injecting noise into image feature representations, the model's generalization capabilities can be enhanced, improving the diversity and detail of generated videos. Furthermore, noise injection provides rich training data for the subsequent denoising process, helping the generative model remove irrelevant noise while retaining key feature information. In the financial sector, this can generate more realistic, natural, and diverse short video materials, reducing labor costs in the video generation process and improving efficiency and quality.

[0077] S40, inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0078] In this embodiment, a denoising model is used to process the noisy latent representation, reducing noise while preserving key image feature information, generating an optimized latent representation. The denoising model's role is to leverage its learned denoising capabilities to restore the core image features and further improve the quality of the latent representation, providing high-quality input for subsequent semantic fusion and decoding operations.

[0079] Use a pre-trained denoising model (such as a model based on the Unet structure) and load its parameters. The initialized model can recognize different levels of noise and perform targeted denoising.

[0080] The noisy latent representation z generated in step t In the input denoising model, the denoising model processes the input data step by step through specific network layers.

[0081] In the first few layers of the denoising model, the noise characteristics in the noisy latent representation are analyzed, and the core image features are extracted through attention mechanisms or convolution operations, while irrelevant noise components are eliminated.

[0082] The intermediate layer of the denoising model generates an intermediate denoising representation that has reduced most of the noise but still retains some potential feature information for subsequent optimization operations.

[0083] In subsequent layers of the model, the intermediate representation is further optimized by extracting global and local features to generate the final optimized latent representation.

[0084] In this process, dynamic fusion is the key operation, which is achieved through the following formula:

[0085]

[0086] in, represents the potential representation at time step t-1; g t Represents the dynamic weight, which is used to adjust the fusion ratio of noise features and image features; F I represents the image feature representation, extracted from the image encoder; Indicates the part that retains the noise characteristics; g t ·F I Represents the part that introduces image features.

[0087] Through this formula, the denoising model can effectively complete noise restoration and feature optimization in the potential representation.

[0088] Use a pre-trained denoising model (such as a diffusion model based on Unet) and load the trained parameters to ensure that the model has the ability to handle noise.

[0089] The noisy latent representation z t Input the denoising model as the initial input of the model. t Model the noise characteristics in; calculate the weight g according to the feature state of the current time step t-1 t , used to control the fusion ratio of noise and image features; dynamic fusion image features F I and noise features to generate intermediate representations.

[0090] Subsequent layers of the denoising model refine and optimize the intermediate representation, further enhancing the representational power of the latent representation by extracting global and local features. The final output of the denoising model serves as the optimized latent representation, providing input for subsequent semantic fusion.

[0091] Processing the noisy latent representation using a denoising model not only effectively reduces the noise component but also introduces image features through dynamic fusion, enhancing the detail and expressiveness of the latent representation. This step ensures the visual and semantic consistency of the generated video content, laying a high-quality foundation for subsequent multimodal semantic fusion and video decoding. In the financial sector, this can significantly enhance the detail of advertising and promotional videos, reduce the distortion rate of generated videos, and improve the efficiency and effectiveness of material production.

[0092] S50, inputting the text feature representation into the denoising model, and performing semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0093] In this embodiment, the text feature representation F T It is a numerical feature representation generated from the text encoder, which contains the semantic information of the input text. T The input denoising model is the basic operation for realizing multimodal information fusion. The denoising model needs to receive text features as input to complete the interactive fusion with the optimized latent representation.

[0094] Get the generated text feature representation F from the text encoder T 。 T Convert the data into a tensor format compatible with the denoising model input layer (e.g., normalization or dimension adjustment). Input it into a specific module of the denoising model (e.g., cross-modal attention layer) for subsequent calculations.

[0095] The optimized latent representation (F O ) is a high-quality feature representation generated from the image feature representation by the denoising model, which contains the main visual information of the image. Semantic fusion requires F O and text feature representation F T Combined to ensure that the generated semantic latent representation (F S ) is semantically consistent with the input text while preserving the image’s detail information.

[0096] Extract the optimized latent representation F O , and input to the fusion module of the denoising model. Inside the denoising model, the dynamic weight mechanism γ is used to balance F T and F O contribution ratio.

[0097] Initial fusion is done using the following formula:

[0098]

[0099] The preliminary fusion results Passed to the next layer for context-sensitive reinforcement processing.

[0100] Semantic association is the core task of semantic fusion. T and F O The multimodal semantic connections between the two ensure that the generated latent representations are aligned in the semantic space. The denoising model constructs semantic associations through the attention mechanism, ensuring that the text and image features remain consistent in multimodal fusion.

[0101] In the denoising model, the matching degree of text features and image features is calculated through the cross-modal attention layer; important features are enhanced based on the attention score, such as highlighting image details that are highly relevant to the text description; and a further optimized fusion representation is generated. Complete the construction of semantic association.

[0102] Semantic Latent Representation (F S ) is the final fusion result output by the denoising model, which contains both the semantic information of the text and the detailed features of the image, and serves as the direct input of the subsequent decoding stage. The output is the semantic latent representation F S The semantic latent representation is normalized into a decoder-compatible format (such as a specific dimension or tensor shape).

[0103] By inputting text feature representations into a denoising model and semantically fusing them with the optimized latent representation, we can effectively align the features of text and image in the semantic space and generate detailed and semantically consistent video footage. In the financial sector, this process provides reliable technical support for generating high-quality videos with precise marketing value while significantly reducing the cost of video production.

[0104] S60: Input the semantic latent representation into a decoder, and generate and output a video frame sequence through the decoder.

[0105] In this embodiment, the semantic latent representation is a multimodal feature generated through semantic fusion, containing information about images and text, and is used to guide video content generation. The semantic latent representation is passed as input to the decoder, which gradually generates video frames based on the latent representation.

[0106] Initialize the decoder: load the pre-trained decoder parameters to ensure that it can accept the semantic latent representation as input;

[0107] Input format processing: semantic latent representation F S Convert to the tensor format required by the decoder input layer (such as a specific dimension or layer structure);

[0108] Semantic information parsing: The initial layer of the decoder is S Parse the image and convert its semantic information into control signals for generating video frames.

[0109] The decoder gradually decodes the semantic latent representation through a multi-layer network, generating intermediate video frame data in the process of restoring the video content. This data is a preliminary representation of the video frame and contains the basic visual information for generating the video.

[0110] The decoder uses a multi-layer network structure (such as deconvolution layers or upsampling layers) to gradually convert the semantic latent representation into pixel data of video frames; during the decoding process, the generated features are dynamically adjusted through the attention mechanism to keep the video frame consistent with the semantic latent representation; the output intermediate video frame data is a preliminary representation of the video frame, which may have low resolution or insufficient details.

[0111] Based on the intermediate video frame data, the decoder further restores the details and generates a complete video frame sequence. This step ensures that the generated video frames meet the output requirements in terms of resolution, detail, and coherence.

[0112] The subsequent layers of the decoder enhance the details of the video frames through specific feature restoration mechanisms (such as residual blocks or self-attention modules); perform temporal modeling on the data of consecutive video frames to ensure natural transitions between frames; and output a complete video frame sequence containing video frame data at all time steps.

[0113] The generated complete video frame sequence needs to be output in a specified format for subsequent storage, transmission, or playback. The output video frames can be a frame sequence file (such as an image set) or a compressed video file (such as MP4 format).

[0114] The generated video frame data is organized into a frame sequence in chronological order; the frame sequence is converted into a specified video format (such as MP4, AVI, etc.) according to requirements; the generated video frame sequence is stored in a specified location or transmitted to the terminal device through a network interface.

[0115] Example: Surgical simulation videos are an important teaching and support tool in medical education and surgical planning. High-quality surgical simulation videos can be generated by inputting images of the surgical area (such as CT or MRI scans) and text descriptions of the surgical steps. This can be achieved by following the steps below:

[0116] Image input and feature extraction: Input the patient's surgical area CT image; use the image encoder to extract the key features of the CT image, including the boundaries of the anatomical structure and tissue density information.

[0117] Text input and feature extraction: Input text descriptions, such as "surgical steps for liver tumor resection"; use a text encoder to extract semantic features from the text, including tumor location, surgical instrument use, and resection steps.

[0118] Multimodal fusion of images and text: The extracted image and text features are multimodally fused to generate semantic latent representations, ensuring that the generated video content is consistent with the surgical description.

[0119] Video generation and output: The decoder decodes the semantic latent representation layer by layer to generate a simulated video of the surgical operation process, including the position of surgical instruments, changes in the resection area, and dynamic changes in the surgical environment.

[0120] Medical students can use the generated surgical simulation videos to become familiar with complex surgical steps; doctors can use videos to pre-plan surgeries, analyze key surgical steps, and reduce the risks of actual operations.

[0121] By feeding the semantic latent representation into a decoder and decoding it layer by layer, a video frame sequence that matches the input semantics can be generated. In the financial sector, this can quickly generate high-quality marketing video material, enabling financial institutions to more efficiently promote their products and services while reducing the time and human resources required for video production.

[0122] The present invention relates to the fields of artificial intelligence technology and medical health, and discloses a video generation method based on multimodal information fusion. The method comprises the following steps: inputting image information and text information to generate image feature representation and text feature representation; injecting noise into the image feature representation to generate a noisy latent representation; inputting the noisy latent representation into a denoising model to generate an optimized latent representation; inputting the text feature representation into the denoising model, performing semantic fusion with the optimized latent representation, and generating a semantic latent representation; inputting the semantic latent representation into a decoder to generate and output a video frame sequence. The present invention ensures that the generated video is semantically consistent with the input content through multimodal fusion of the image feature representation and the text feature representation; enhancing the diversity and detail quality of the generated video through noise injection and denoising optimization; and generating a video frame sequence through a decoder to ensure visual coherence and natural transitions of the video.

[0123] In one embodiment, the above S10 includes:

[0124] S101, acquiring multi-channel image data;

[0125] S102, compressing the multi-channel image data using an image encoder to generate a potential feature representation;

[0126] S103, extracting local texture features and edge information from the latent feature representation;

[0127] S104: performing normalization processing on the local texture features and edge information to generate a normalized image feature representation.

[0128] In this embodiment, multi-channel image data refers to image information containing multiple color or data channels (such as an RGB image or a depth image). Acquiring multi-channel image data is the starting point for image feature extraction. High-quality input data can be obtained in a variety of ways to ensure the accuracy of subsequent feature extraction.

[0129] Read image files from the file system (supporting formats such as JPEG and PNG); use cameras (such as HD cameras) to capture images in real time; obtain image data (such as advertising images or user-uploaded images) from remote servers through API interfaces.

[0130] Check that the image resolution meets preset requirements; remove noise and artifacts to optimize image quality; organize image channels into a standardized format (such as 3-channel RGB format or 4-channel RGBA format).

[0131] The image encoder's task is to reduce the dimensionality and extract features from high-dimensional, multi-channel image data, extracting key latent feature representations through deep learning models (such as convolutional neural networks). Latent feature representations are numerical, low-dimensional representations that retain the core information of the image and facilitate subsequent processing.

[0132] The acquired multi-channel image data is fed into an image encoder. A multi-layer convolutional neural network (CNN) is used to extract features from the image, capturing local features layer by layer. Pooling operations or fully connected layers are used to reduce high-dimensional image features to a fixed-size latent representation, preserving the image's global structure and important features.

[0133] The output potential feature representation Z=E(X), where X is the input multi-channel image data and E is the image encoder model.

[0134] Local texture features and edge information are important visual features of images, used to describe image details and shape information. Extracting these features can enhance the expressive power of image feature representation, making it more suitable for subsequent multimodal information fusion.

[0135] Use convolution kernels to detect local patterns in the image (such as color gradients and texture changes). Apply edge detection methods (such as the Sobel filter or the Canny operator) to capture edge features in the image. Separate the extracted local texture features from the edge information and store them as independent feature channels.

[0136] Normalization is the process of standardizing the eigenvalues ​​to make their numerical range consistent, which helps improve the stability and efficiency of subsequent model processing. Local texture features and edge information are standardized separately, scaling the eigenvalues ​​to the range of [0, 1].

[0137] The normalized local texture features and edge information are fused to generate the final normalized image feature representation. The output image feature representation is a fixed-dimensional numerical vector or matrix, which is convenient for subsequent processing.

[0138] This embodiment acquires multi-channel image data and uses an image encoder to extract local texture features and edge information, generating a normalized image feature representation that contains rich visual information. This improves the efficiency and accuracy of subsequent multimodal information fusion, providing a solid foundation for generating high-quality video content.

[0139] In one embodiment, the above S20 includes:

[0140] S201, obtaining natural language text data;

[0141] S202, performing word segmentation processing on the natural language text data to generate a vocabulary sequence;

[0142] S203, using a text encoder to perform feature extraction on the vocabulary sequence to generate initial text features;

[0143] S204: Based on the initial text features, a semantic enhancement operation is performed through a context modeling mechanism to generate an enhanced text feature representation.

[0144] In this embodiment, acquiring natural language text data is the starting point for text feature generation. Text data can be user-entered phrases, descriptions, titles, or content from a predefined dataset, typically in natural language. It is important to ensure that the acquired text is grammatically standardized and structurally complete.

[0145] The input sources can be: user input: receiving text entered by the user through the front-end interface, such as "fund income growth trend"; database: retrieving specific text content from the predefined copy database, such as advertising titles, slogans, etc.; network interface: obtaining relevant text information from external data sources (such as news websites, user comments) through API.

[0146] Remove redundant characters (such as special symbols and repeated spaces); standardize text formats to ensure uniform processing in multilingual scenarios.

[0147] Word segmentation is the process of dividing natural language text into words. Generating word sequences is the basis for further processing by the text encoder. The goal of word segmentation is to capture the basic semantic units of each word in the text.

[0148] The word segmentation method can be: rule-based: use separators (such as spaces) to split the text into words; model-based: use pre-trained word segmentation tools (such as BERT word segmenter) for word segmentation.

[0149] For example, given the input text "Fund income growth trend", the generated word sequence ["fund", "income", "growth", "trend"] can be used to standardize word formats (such as lowercase) and handle common abbreviations, spelling variations, and other issues.

[0150] The text encoder digitizes the word sequence after word segmentation, maps it to the latent semantic space, and generates initial text features. The initial text features are mainly used to represent the basic semantic information in the text.

[0151] Use a pre-trained word embedding model (such as Word2Vec or GloVe) to convert the word sequence into a numerical vector. Each word is mapped to a vector of fixed dimension, for example, "Fund" -> [0.1, 0.3, 0.5].

[0152] Input a word sequence to a text encoder (such as a Transformer model), which extracts the contextual relationships between words through a multi-layer neural network. Output is an initial text feature matrix representing the semantic relationship between each word.

[0153] The formula is:

[0154] Represents the initial text features; E represents the text encoder; W represents the vocabulary sequence.

[0155] The semantic enhancement operation further strengthens the semantic information based on the initial text features through context modeling mechanisms (such as self-attention mechanism). This process aims to capture the semantic relationships at the sentence level and the implicit meaning in the text.

[0156] The self-attention mechanism is used to capture long-range dependencies between words; it emphasizes word pairs with high semantic relevance in the text, such as "fund" and "income".

[0157] Use a multi-layer Transformer network to transform the initial text features Processing; output enhanced text feature representation F containing global semantic information T .

[0158] The formula is:

[0159] Attn represents the context modeling mechanism; F T Represents the enhanced text feature representation.

[0160] This embodiment generates text feature representations containing rich semantic information through word segmentation, feature extraction, and semantic enhancement of text information. This ensures the semantic integrity of the text content and provides high-quality text features for multimodal information fusion. In the financial sector, it can efficiently generate accurate text features for marketing video production, improving the semantic consistency and visual expressiveness of the videos.

[0161] In one embodiment, the above S30 includes:

[0162] S301, generating random noise;

[0163] S302, adjusting the amplitude of the random noise so that the numerical range of the random noise is consistent with the numerical range represented by the image feature;

[0164] S303: Superimpose the adjusted random noise on the image feature representation to generate the noisy latent representation.

[0165] In this embodiment, random noise is the core data used to perturb the image feature representation. Its purpose is to enhance the diversity of the underlying representation by adding randomness and to provide a training basis for subsequent denoising. The random noise generated must ensure that its distribution characteristics meet the requirements of the target application (e.g., Gaussian or uniform distribution).

[0166] Use normal distribution (Gaussian noise) to generate random noise, ensuring that the mean of the noise is 0. The variance can be set according to actual needs. If it is necessary to balance each feature channel, use multidimensional normal distribution to generate independent random noise.

[0167] Define the dimension of the noise, consistent with the shape of the image feature representation; use a random number generation function (such as NumPy's np.random.normal()) to generate a Gaussian noise matrix.

[0168] The generated random noise ∈ t It will be used as input for subsequent overlay operations.

[0169] In order to avoid excessive disturbance of the image feature representation by noise, the amplitude of the random noise needs to be adjusted so that its numerical range matches the image feature representation, thereby ensuring that the superimposed features are still usable.

[0170] Represent image features F I Perform statistics, calculate its mean and standard deviation, and determine its numerical range.

[0171] Adjust the noise amplitude using the following formula:

[0172]

[0173] ∈′ t represents the adjusted noise; σ ∈ represents the standard deviation of the original noise; Indicates the standard deviation of the image feature representation.

[0174] The generated random noise is standardized (zero mean, unit variance); according to the image feature table

[0175] The output noise after amplitude adjustment is rescaled to the range of values ​​shown. t For subsequent operations.

[0176] The noisy latent representation is generated by superimposing the adjusted random noise onto the image feature representation. This step aims to increase the randomness and diversity of the feature representation while preserving the core characteristics of the image.

[0177] The superposition is done using the following formula:

[0178]

[0179] z t represents the noisy latent representation; a t represents the noise intensity weight; F I Represents image feature representation; ∈′ t represents the adjusted random noise.

[0180] Dynamically adjust the noise weight a according to the current time step t t , the weight is small in the initial stage and gradually increases to enhance randomness. The adjusted noise is fused with the image feature representation according to the formula to output the final noisy potential representation z t .

[0181] This embodiment significantly improves the diversity of feature representations and the generalization capability of the generative model by injecting noise into the image feature representation. Adjusting the noise amplitude ensures the stability of the superposition process, avoids feature distortion, and provides a training foundation for subsequent denoising models.

[0182] In one embodiment, the above S40 includes:

[0183] S401, initializing the parameters of the denoising model, and inputting the noisy latent representation into the denoising model;

[0184] S402, in the denoising model, reducing the noise component in the noisy latent representation by a specific denoising layer according to the noise characteristics of the noisy latent representation to generate an intermediate denoised representation;

[0185] S403, extracting global features and local features from the intermediate denoised representation;

[0186] S404: Optimize the intermediate denoised representation based on the global features and the local features to generate a final optimized latent representation.

[0187] In this embodiment, initializing the denoising model is a prerequisite for ensuring the model can accurately process noisy latent representations. Initialization parameters include loading the model's pretrained weights and configuring dynamic adjustment parameters. After the noisy latent representation is input into the denoising model, the model begins performing denoising and feature optimization operations.

[0188] Load the pre-trained parameters of the denoising model (such as those based on the diffusion model or Unet architecture). Set the dynamic parameters required for the model to run (such as time step, learning rate, etc.).

[0189] The generated noisy latent representation z tConvert to an input format compatible with the denoising model (such as tensor dimension alignment); input to the initial layer of the denoising model to start feature processing.

[0190] The denoising model analyzes the noise distribution in the noisy latent representation and uses specific denoising layers (such as convolutional layers or deconvolutional layers) to gradually reduce the noise component to generate a preliminary restored intermediate denoised representation.

[0191] Use the initial layer of the denoising model to input z t Model the noise characteristics; dynamically adjust the denoising weights according to the time step t to adapt to the noise changes.

[0192] Apply specific denoising layers (such as deconvolution or residual blocks) to remove the noise component layer by layer; generate an intermediate denoised representation z t-1 .

[0193] The denoising formula is expressed as:

[0194] z t-1 =f denoise (z t )

[0195] Among them, f denoise represents the mapping function of the denoising layer; z t-1 represents the intermediate denoised representation.

[0196] The extraction of global features and local features helps to further optimize the denoising representation. Global features provide overall semantic information, while local features capture details and textures.

[0197] Use global pooling operations (such as average pooling or maximum pooling) to reduce the dimensionality of the intermediate denoised representation and extract the global feature representation; the generated global features contain the overall structural information of the image.

[0198] Apply convolution kernels to scan the intermediate denoising representation to capture local texture features and edge information; organize local features into feature maps for subsequent detail optimization.

[0199] Global feature F global and local features F local They are respectively used as inputs of the optimization module.

[0200] The intermediate denoised representation is refined and optimized using global and local features to generate the final optimized latent representation. Global features are used to maintain overall consistency, while local features are used to enhance detail representation.

[0201] The intermediate denoising representation is optimized by integrating global and local features:

[0202] F opt =α·F glocal +(1-α)·Flocal

[0203] F opt represents the optimized latent representation; α represents the dynamic weight of global and local features.

[0204] Adjust the consistency of the overall structure based on global features; use local features to enhance the performance of texture and edge details.

[0205] Finally, the optimized potential representation F is generated opt , used for subsequent multimodal semantic fusion.

[0206] This embodiment generates high-quality latent representations by feeding the noisy latent representation into a denoising model, combining noise removal with global and local feature optimization. This not only reduces noise interference but also preserves key image features, providing high-quality input for subsequent multimodal fusion and video generation.

[0207] In one embodiment, the above S50 includes:

[0208] S501, initializing the semantic fusion parameters of the denoising model;

[0209] S502, inputting the text feature representation into the denoising model, and establishing a semantic association between the text feature representation and the optimized latent representation to enhance the semantic information of the text feature representation;

[0210] S503 , in the denoising model, interacting the text feature representation and the optimized latent representation through a multimodal fusion layer to generate a fused semantic latent representation.

[0211] In this embodiment, semantic fusion parameters are key parameters in the denoising model that control the fusion of multimodal information. They are used to adjust the weight distribution, interaction rules, and semantic enhancement between text features and the optimized latent representation. Initializing these parameters ensures high stability and accuracy of the fusion process.

[0212] Load the predefined semantic fusion parameters in the denoising model, including the initial weight values ​​and fusion rules; ensure that the parameters are applicable to the input multimodal data format (such as text and image feature representation).

[0213] Dynamically adjust the initial weight distribution according to task requirements (such as the content and style of video generation); for example, set the weight of text features slightly higher than the optimized latent representation to highlight text semantics.

[0214] The output of the initialized semantic fusion module is used to process the features of subsequent input.

[0215] The establishment of semantic association is the core step of semantic fusion, which aims to align text feature representation and optimized latent representation through semantic matching and information exchange, thereby enhancing the semantic information of text features and making them more suitable for generating video content.

[0216] The text feature is represented by F T Input to the semantic fusion module of the denoising model; ensure that the input format is consistent with the model requirements (for example, the dimensions of the tensor are aligned).

[0217] In the denoising model, the cross-attention mechanism is used to calculate the semantic matching between text features and optimized latent representations; the optimized latent representation features related to text semantics are highlighted to make the text features more expressive.

[0218] Semantic enhancement formula is expressed as:

[0219]

[0220] represents the enhanced text feature representation; β represents the semantic enhancement weight; Attn(F T , F O ) represents the attention-weighted result of text features and optimized latent representation.

[0221] Output enhanced text feature representation for subsequent multimodal interaction.

[0222] The multimodal fusion layer completes the final fusion of text feature representation and optimized latent representation. The generated semantic latent representation integrates the semantic information of the text and the detailed features of the image, providing multimodal consistent input for video generation.

[0223] A multi-head attention mechanism is used to process text features and optimized potential representations in parallel; during the interaction process, fine-grained relationships between different modalities are captured.

[0224] The fused result is input into a nonlinear activation layer (such as ReLU) for further optimization; the final semantic potential representation F is output. S .

[0225] The fusion formula is expressed as:

[0226]

[0227] F S represents the semantic potential representation after fusion; γ represents the dynamic weight of multimodal fusion.

[0228] This embodiment generates semantically consistent and detail-rich latent representations by inputting text feature representations into a denoising model and semantically fusing them with the optimized latent representations. This ensures a high degree of semantic alignment between text and images, providing technical support for generating videos with both consistent content and rich detail.

[0229] In one embodiment, the above S60 includes:

[0230] S601, initializing the parameters of the decoder, and inputting the semantic latent representation into the decoder;

[0231] S602, in the decoder, decoding the semantic latent representation layer by layer according to the feature distribution of the semantic latent representation to generate intermediate video frame data;

[0232] S603: Serialize the intermediate video frame data to generate a video frame sequence and output the video frame sequence.

[0233] In this embodiment, decoder initialization is the starting point of the decoding process, which is used to load the necessary parameters and ensure that the semantic latent representation can be correctly processed. The initialization parameters include the decoder's pre-trained weights, network architecture configuration, etc.

[0234] Load decoder parameters (such as convolution kernel weights and attention mechanism parameters) from the pre-trained model; configure the decoder's operating environment, including the dimensions of the input data and the batch size.

[0235] Ensure that the semantic latent representation F of the input S The format is consistent with the decoder input layer; it is converted to a tensor format that meets the input requirements (such as a specific dimension or number of channels).

[0236] After initialization, the semantic latent representation enters the first layer of the decoder for feature parsing.

[0237] The decoder processes the semantic latent representation layer by layer to generate intermediate video frame data. This data is a preliminary representation of the video frame and contains the main visual information of the semantic latent representation.

[0238] The decoder uses deconvolution layers or upsampling layers to restore the high-dimensional features of semantic potential representation layer by layer; each layer increases the resolution and introduces detailed information.

[0239] According to the feature distribution of semantic latent representation, the decoding weight is adjusted to highlight the key features; the attention mechanism is used to enhance semantic consistency and ensure that the data of each frame matches the input text and image information.

[0240] Output intermediate video frame data F frameContains the basic content of a video frame, but may not be serialized or further optimized.

[0241] The formula is:

[0242] F frame =g decode (F S )

[0243] g decode represents the layer-by-layer mapping function of the decoder; F frame Represents intermediate video frame data.

[0244] The serialization process is to organize the intermediate video frame data into a time-ordered frame sequence and convert it into a standardized video format for output.

[0245] Arrange the intermediate video frame data in timeline order to form a frame sequence; ensure the continuity between frames and avoid video frame skipping due to frame sequence errors.

[0246] Compress the frame sequence into a video file according to the video coding standard (such as H.264 or H.265); if you need to output a frame image sequence, store it as a picture file (such as PNG, JPEG) according to the number.

[0247] The output video frame sequence can be directly used for playback, storage or transmission.

[0248] This embodiment effectively and naturally transforms the semantic latent representation into video content by inputting it into a decoder and decoding it layer by layer to generate a video frame sequence. The decoder's layer-by-layer structure ensures the clarity and semantic consistency of the generated video, and the serialization process ensures video coherence.

[0249] In one embodiment, a video generation device based on multimodal information fusion is provided, and the video generation device based on multimodal information fusion corresponds to the video generation method based on multimodal information fusion in the above embodiment. Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of a video generation device based on multimodal information fusion according to the present invention. These modules include an image encoding module 10, a text encoding module 20, a noise injection module 30, a denoising model module 40, a semantic fusion module 50, and a decoding module 60. Each functional module is described in detail below:

[0250] An image encoding module 10 is used to obtain image information and generate image feature representation through an image encoder;

[0251] A text encoding module 20 is used to obtain text information and generate text feature representation through a text encoder;

[0252] a noise injection module 30 for injecting noise into the image feature representation to generate a noisy latent representation;

[0253] a denoising model module 40 for inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0254] A semantic fusion module 50 is used to input the text feature representation into the denoising model and perform semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0255] The decoding module 60 is configured to input the semantic latent representation into a decoder, and generate and output a video frame sequence through the decoder.

[0256] In one embodiment, the image encoding module 10 is specifically configured to:

[0257] Acquire multi-channel image data;

[0258] Using an image encoder to compress the multi-channel image data to generate a potential feature representation;

[0259] extracting local texture features and edge information from the latent feature representation;

[0260] The local texture features and edge information are normalized to generate a normalized image feature representation.

[0261] In one embodiment, the text encoding module 20 is specifically configured to:

[0262] Obtain natural language text data;

[0263] Performing word segmentation processing on the natural language text data to generate a vocabulary sequence;

[0264] Using a text encoder to extract features from the vocabulary sequence to generate initial text features;

[0265] Based on the initial text features, a semantic enhancement operation is performed through a context modeling mechanism to generate an enhanced text feature representation.

[0266] In one embodiment, the noise injection module 30 is specifically configured to:

[0267] Generate random noise;

[0268] Adjusting the amplitude of the random noise so that the numerical range of the random noise is consistent with the numerical range represented by the image feature;

[0269] The adjusted random noise is superimposed on the image feature representation to generate the noisy latent representation.

[0270] In one embodiment, the denoising model module 40 is specifically configured to:

[0271] Initializing parameters of a denoising model and inputting the noisy latent representation into the denoising model;

[0272] In the denoising model, according to the noise characteristics of the noisy latent representation, a noise component in the noisy latent representation is reduced by a specific denoising layer to generate an intermediate denoised representation;

[0273] extracting global features and local features from the intermediate denoised representation;

[0274] The intermediate denoised representation is optimized based on the global features and the local features to generate a final optimized latent representation.

[0275] In one embodiment, the semantic fusion module 50 is specifically configured to:

[0276] Initializing semantic fusion parameters of the denoising model;

[0277] Inputting the text feature representation into the denoising model and establishing a semantic association between the text feature representation and the optimized latent representation to enhance the semantic information of the text feature representation;

[0278] In the denoising model, the text feature representation and the optimized latent representation are interacted through a multimodal fusion layer to generate a fused semantic latent representation.

[0279] In one embodiment, the decoding module 60 is specifically configured to:

[0280] Initializing parameters of a decoder and inputting the semantic latent representation into the decoder;

[0281] In the decoder, according to the feature distribution of the semantic latent representation, the semantic latent representation is decoded layer by layer to generate intermediate video frame data;

[0282] The intermediate video frame data is serialized to generate a video frame sequence and the video frame sequence is output.

[0283] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a video generation method based on multimodal information fusion.

[0284] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a video generation method based on multimodal information fusion.

[0285] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0286] Obtain image information and generate image feature representation through image encoder;

[0287] Obtain text information and generate text feature representation through a text encoder;

[0288] injecting noise into the image feature representation to generate a noisy latent representation;

[0289] Inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0290] Inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0291] The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

[0292] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0293] Obtain image information and generate image feature representation through image encoder;

[0294] Obtain text information and generate text feature representation through a text encoder;

[0295] injecting noise into the image feature representation to generate a noisy latent representation;

[0296] Inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0297] Inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0298] The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

[0299] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0300] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0301] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0302] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A video generation method based on multimodal information fusion, characterized in that: The following steps are involved: Obtain image information and generate image feature representation through image encoder; Obtain text information and generate text feature representation through a text encoder; injecting noise into the image feature representation to generate a noisy latent representation; Inputting the noisy latent representation into a denoising model to generate an optimized latent representation; Inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation; The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

2. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Obtain image information and generate image feature representation through the image encoder, including: Acquire multi-channel image data; Using an image encoder to compress the multi-channel image data to generate a potential feature representation; extracting local texture features and edge information from the latent feature representation; The local texture features and edge information are normalized to generate a normalized image feature representation.

3. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Obtain text information and generate text feature representation through a text encoder, including: Obtain natural language text data; Performing word segmentation processing on the natural language text data to generate a vocabulary sequence; Using a text encoder to extract features from the vocabulary sequence to generate initial text features; Based on the initial text features, a semantic enhancement operation is performed through a context modeling mechanism to generate an enhanced text feature representation.

4. The video generation method based on multimodal information fusion as claimed in claim 1, characterized in that: Injecting noise into the image feature representation to generate a noisy latent representation, including: Generate random noise; Adjusting the amplitude of the random noise so that the numerical range of the random noise is consistent with the numerical range represented by the image feature; The adjusted random noise is superimposed on the image feature representation to generate the noisy latent representation.

5. The video generation method based on multimodal information fusion according to claim 1, wherein: The noisy latent representation is input into the denoising model to generate an optimized latent representation, including: Initializing parameters of a denoising model and inputting the noisy latent representation into the denoising model; In the denoising model, according to the noise characteristics of the noisy latent representation, a noise component in the noisy latent representation is reduced by a specific denoising layer to generate an intermediate denoised representation; extracting global features and local features from the intermediate denoised representation; The intermediate denoised representation is optimized based on the global features and the local features to generate a final optimized latent representation.

6. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation, including: Initializing semantic fusion parameters of the denoising model; Inputting the text feature representation into the denoising model and establishing a semantic association between the text feature representation and the optimized latent representation to enhance the semantic information of the text feature representation; In the denoising model, the text feature representation and the optimized latent representation are interacted through a multimodal fusion layer to generate a fused semantic latent representation.

7. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Inputting the semantic latent representation into a decoder, generating and outputting a video frame sequence through the decoder, including: Initializing parameters of a decoder and inputting the semantic latent representation into the decoder; In the decoder, according to the feature distribution of the semantic latent representation, the semantic latent representation is decoded layer by layer to generate intermediate video frame data; The intermediate video frame data is serialized to generate a video frame sequence and the video frame sequence is output.

8. A video generation device based on multimodal information fusion, characterized in that: The video generation device based on multimodal information fusion includes: Image encoding module, used to obtain image information and generate image feature representation through image encoder; The text encoding module is used to obtain text information and generate text feature representation through a text encoder; a noise injection module, configured to inject noise into the image feature representation to generate a noisy latent representation; a denoising model module, configured to input the noisy latent representation into a denoising model to generate an optimized latent representation; A semantic fusion module, configured to input the text feature representation into the denoising model and perform semantic fusion with the optimized latent representation to generate a semantic latent representation; The decoding module is used to input the semantic latent representation into a decoder, generate a video frame sequence through the decoder, and output it.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a video generation program based on multimodal information fusion stored in the memory and capable of running on the processor. When the video generation program based on multimodal information fusion is executed by the processor, the steps of the video generation method based on multimodal information fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a video generation program based on multimodal information fusion, and when the video generation program based on multimodal information fusion is executed by the processor, the steps of the video generation method based on multimodal information fusion according to any one of claims 1 to 7 are implemented.

Images