Video generation method and device based on multi-modal information fusion, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of medical health, and discloses a video generation method based on multi-modal information fusion, which comprises the following steps: inputting image information and text information, and generating image feature representation and text feature representation; injecting noise into the image feature representation to generate a noisy potential representation; inputting the noisy potential representation into a denoising model to generate an optimized potential representation; inputting the text feature representation into a denoising model, and carrying out semantic fusion on the text feature representation and the optimized potential representation to generate a semantic potential representation; and inputting the semantic potential representation into a decoder, generating a video frame sequence and outputting the video frame sequence. According to the invention, through the multi-modal fusion of the image feature representation and the text feature representation, the generated video is ensured to be consistent with the input content semantically; through noise injection and denoising optimization, the diversity and detail quality of video generation are enhanced; and a video frame sequence is generated through the decoder, so that the visual coherence and natural transition of the video are ensured.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and medical health, and in particular to a video generation method, device, equipment and storage medium based on multimodal information fusion. Background Art

[0002] Image-guided video generation technology is an important direction in the field of video generation and is widely used in multimedia content production, virtual reality experience, smart marketing and other fields. In the medical field, this technology can be used in scenarios such as auxiliary diagnosis, surgical simulation, medical teaching and health management. Generating videos by using given images and text information as input conditions can significantly reduce the cost of video production and improve the efficiency of content generation in medical scenarios. However, the existing technology still has many shortcomings in solving the consistency, coherence and semantic accuracy of video generation.

[0003] When generating medical-related videos, current technologies usually have difficulty ensuring the consistency of video content in spatial and temporal dimensions. Although some methods can generate medical video clips that match the semantics of the input image (such as a 3D reconstruction animation of a lesion area), there are unnatural jumps or breaks in the transition between video frames, resulting in the generated medical video being not smooth enough. In addition, long-term medical video generation tasks (such as surgical simulation videos or treatment process animations) are more prone to content distortion or inconsistency, especially when the simulation scene is more complex or has more details, and the model is difficult to effectively capture the logical relationship between consecutive frames.

[0004] Existing video generation technologies have certain limitations in processing the fusion of multimodal information (such as medical images, patient text information, etc.). Some methods generate medical videos through staged optimization, but because the optimization objectives of each stage are too similar, the distribution of model learning tends to be consistent, lacking the diversity and flexibility required to generate complex medical scenes, and the quality of the final generated video is limited. In addition, another type of method relies on external guidance information (such as lesion descriptions, diagnosis and treatment step texts, or time series signals). Although it can improve the generation effect to a certain extent, its ability to integrate multimodal information is limited, and it is difficult to achieve high-quality medical video generation in terms of vision, semantics, and timing.

[0005] Although some methods have improved the generation effect through the design of multimodal fusion and hierarchical modeling, such methods usually require a large amount of additional medical training data (such as lesion image annotation data and text description data) and complex transformation of the model architecture. At the same time, multi-stage optimization or the introduction of external guidance signals significantly increases the computational overhead of the training process and the dependence on hardware resources. These factors limit the widespread application of video generation technology in the medical field, especially in medical institutions with limited resources.

[0006] In the medical field, the application potential of video generation technology has not yet been fully realized, especially in the personalized medical video generation and batch medical teaching content production. There is still a lot of room for improvement. For example, in medical image-assisted analysis, the image videos generated by existing technologies lack fine control over the lesion area, and the generated medical animation content does not match the actual clinical needs. In addition, the high computational cost of existing methods further limits the possibility of large-scale application in the medical field, making it difficult for medical institutions to efficiently use this technology for daily teaching, surgical planning and patient education. By improving video generation technology, while improving content consistency, detail accuracy and generation efficiency, its practical application value in the medical field can be significantly expanded. Summary of the invention

[0007] The main purpose of the present invention is to provide a video generation method, device, equipment and storage medium based on multimodal information fusion, aiming to solve the technical problem that the prior art is difficult to generate semantically consistent and visually coherent video content through multimodal information fusion, resulting in insufficient content consistency and natural transition of the generated video.

[0008] To achieve the above object, the present invention provides a video generation method based on multimodal information fusion, comprising:

[0009] Obtain image information and generate image feature representation through an image encoder;

[0010] Obtain text information and generate text feature representation through a text encoder;

[0011] injecting noise into the image feature representation to generate a noisy latent representation;

[0012] Inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0013] Inputting the text feature representation into the denoising model and semantically fusing it with the optimized latent representation to generate a semantic latent representation;

[0014] The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

[0015] Furthermore, to achieve the above object, the present invention provides a video generation device based on multimodal information fusion, comprising:

[0016] An image encoding module, used to obtain image information and generate image feature representation through an image encoder;

[0017] The text encoding module is used to obtain text information and generate text feature representation through a text encoder;

[0018] A noise injection module, used for injecting noise into the image feature representation to generate a noisy latent representation;

[0019] A denoising model module, configured to input the noisy latent representation into a denoising model to generate an optimized latent representation;

[0020] A semantic fusion module, used for inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0021] The decoding module is used to input the semantic latent representation into a decoder, and generate and output a video frame sequence through the decoder.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a video generation program based on multimodal information fusion stored in the memory and executable on the processor, wherein the video generation program based on multimodal information fusion, when executed by the processor, implements the steps of the video generation method based on multimodal information fusion as described above.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a video generation program based on multimodal information fusion is stored, and when the video generation program based on multimodal information fusion is executed by a processor, the steps of the video generation method based on multimodal information fusion as described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and medical health, and discloses a video generation method based on multimodal information fusion, which generates image feature representation and text feature representation by inputting image information and text information; injects noise into the image feature representation to generate a noisy potential representation; inputs the noisy potential representation into a denoising model to generate an optimized potential representation; inputs the text feature representation into the denoising model, performs semantic fusion with the optimized potential representation, and generates a semantic potential representation; inputs the semantic potential representation into a decoder to generate and output a video frame sequence. The present invention ensures that the generated video is semantically consistent with the input content through multimodal fusion of image feature representation and text feature representation; enhances the diversity and detail quality of video generation through noise injection and denoising optimization; generates a video frame sequence through a decoder to ensure the visual coherence and natural transition of the video. The semantic consistency, visual coherence and generation quality of the generated video are significantly improved, while the complexity of the generation process is reduced, and it is suitable for batch and efficient generation of video materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of a video generation method based on multimodal information fusion in an embodiment of the present invention;

[0027] Figure 2 It is a flow chart of an embodiment of a video generation method based on multimodal information fusion of the present invention;

[0028] Figure 3 It is a functional module diagram of a preferred embodiment of a video generation device based on multimodal information fusion of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0030] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0032] The video generation method based on multimodal information fusion provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can input image information and text information through the user terminal to generate image feature representation and text feature representation; inject noise into the image feature representation to generate a noisy potential representation; input the noisy potential representation into a denoising model to generate an optimized potential representation; input the text feature representation into the denoising model, and semantically fuse it with the optimized potential representation to generate a semantic potential representation; input the semantic potential representation into a decoder to generate a video frame sequence and output it. The present invention ensures that the generated video is semantically consistent with the input content by multimodal fusion of image feature representation and text feature representation; enhances the diversity and detail quality of video generation by injecting noise and denoising optimization; generates a video frame sequence by a decoder to ensure the visual coherence and natural transition of the video. The semantic consistency, visual coherence and generation quality of the generated video are significantly improved, while reducing the complexity of the generation process, and is suitable for batch and efficient video material generation. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2This is a flow chart of an embodiment of a video generation method based on multimodal information fusion provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the video generation method based on multimodal information fusion proposed by the present invention includes the following steps:

[0035] S10, acquiring image information and generating image feature representation through an image encoder;

[0036] In this embodiment, obtaining image information is the starting point of the video generation process, which refers to obtaining the original image from an input device or a pre-stored data set. The image information can be a single-frame image, the first frame of a dynamic image, or a group of pictures related to the generation target. The acquired image information needs to meet certain quality requirements, such as resolution and clarity, to ensure that the subsequent encoding process can effectively extract image features.

[0037] Read image files from databases or file systems, supporting common formats such as JPEG, PNG, etc.; use image acquisition devices (such as cameras, mobile phones or scanners) to obtain real-time images and convert them into standardized formats; obtain remote image data (such as financial enterprise advertising images, user uploaded pictures, etc.) through network interfaces, and perform preprocessing to eliminate noise and compression artifacts.

[0038] The task of an image encoder is to extract and represent the features of the input image information. Its main function is to map high-dimensional pixel information to a low-dimensional latent space and extract key features of the image, such as color, texture, edge, and shape. The generated image feature representation is a numerical intermediate representation that facilitates subsequent processing and fusion of multimodal information.

[0039] The image encoder generates image feature representation through the following formula:

[0040] F I =E(X)

[0041] F I It is a generated image feature representation that contains the key features of the input image (such as color, texture, edge, and shape, etc.).

[0042] The image encoder E maps high-dimensional pixel information to a low-dimensional latent space and generates a numerical intermediate representation F. I , which is convenient for subsequent processing.

[0043] Convolutional neural network (CNN) is used as an image encoder to extract local features of the image through multi-layer convolution operations; an attention mechanism (such as self-attention module) is added to the CNN structure to highlight important feature areas; image features are reduced in dimension (such as fully connected layer or pooling operation) to generate a latent representation vector of fixed dimension; image feature representation is output, usually as a fixed-size feature vector or feature map matrix.

[0044] Example description: Disease diagnosis process animation can be used for patient education to help patients intuitively understand the cause of the disease and the treatment process. By inputting images of the lesion area (such as X-rays or ultrasound images) and disease description text, an easy-to-understand disease diagnosis animation can be generated. This can be achieved in the following ways:

[0045] Image input and feature extraction: Input the patient's chest X-ray and extract the lung features in the image, including the shape, size, and distribution of lung nodules.

[0046] Text input and feature extraction: Input text description, such as "the formation mechanism and treatment plan of lung nodules"; extract key information from the text, including the causes of nodule formation, diagnostic methods, and recommended treatment steps.

[0047] Multimodal semantic fusion: Fusion of image features and text features to generate semantic latent representations, ensuring that the animation content contains scientific and educational significance.

[0048] Video generation and output: Decoding semantic latent representations to generate disease animations, showing the formation, development, and dynamic changes of treatment effects of lung nodules.

[0049] Help patients and their families understand their condition intuitively, and improve patients' understanding and cooperation with treatment plans; used for communication between doctors and patients, reducing communication barriers caused by professional terminology.

[0050] By acquiring image information and encoding feature representations, key visual content can be extracted, providing a basis for semantic and visual information for subsequent video generation. This not only improves the quality and detail of the generated video content, but also ensures that the generated content can accurately reflect the information of the input image, effectively reducing possible content deviations during the generation process.

[0051] S20, obtaining text information and generating text feature representation through a text encoder;

[0052] In this embodiment, obtaining text information is one of the important starting points of the multimodal fusion process. The text information may include natural language text descriptions, keywords or tags, which may come from user input, preset templates provided by the system or third-party data interfaces. The acquisition of text information requires preprocessing to ensure the validity of its syntax and format.

[0053] Text source: Receive natural language text information from user input boxes, databases, or API interfaces.

[0054] Format standardization: Format the input text (such as removing special symbols, spaces, and punctuation marks) to generate clean text data.

[0055] Language detection: In multilingual scenarios, use language identification tools to annotate text data in order to select an appropriate text encoder.

[0056] The core task of a text encoder is to convert natural language text into numerical feature representations, extract the semantic content of the text and map it to a latent space. The text encoder performs word segmentation, context modeling and semantic enhancement on text data to ensure that the generated feature representation can fully express the semantic information of the text.

[0057] In this process, the feature extraction process of the text encoder can be expressed by the following formula:

[0058] F T =CLIP(Text)

[0059] Where Text represents the input natural language text data; CLIP(Text) represents encoding the text data using the CLIP model; F T The generated text feature representation contains the global semantic information and contextual relationship of the text.

[0060] CLIP (Contrastive Language–Image Pretraining) is a multimodal model proposed by OpenAI that can process text and image data simultaneously. Its core goal is to map the data of two different modalities into the same latent space by jointly learning the representation of text and image, thereby achieving cross-modal feature alignment and mutual understanding. Its main application scenarios include image classification, image search, multimodal generation (such as image description generation, video generation), etc.

[0061] Perform word segmentation on the text data Text and break the sentence into word sequences. For example, "Financial product recommendation" is segmented into ["finance", "product", "recommendation"];

[0062] Use a pre-trained word vector model (such as Word2Vec or GloVe) to embed the word segmentation results and generate corresponding numerical representations;

[0063] Input the vocabulary embedding vector into the text encoder (such as the text encoding part of the CLIP model), capture the contextual relationship of the text through a multi-layer Transformer, and extract the initial text features;

[0064] Under the attention mechanism of the text encoder, the initial features are enhanced to highlight the semantic focus;

[0065] The final output encoded text feature representation F T .

[0066] Through the processing of the text encoder, natural language text can be accurately converted into numerical semantic representation. Compared with the traditional labeling method, text feature representation provides a deep expression of contextual information, significantly improving the semantic accuracy and consistency of video generation. In financial scenarios, it can generate smart marketing videos that are closer to customer needs for financial products and services, reducing production costs and improving promotion efficiency.

[0067] S30, injecting noise into the image feature representation to generate a noisy latent representation;

[0068] In this embodiment, by adding noise to the image feature representation, a noisy latent representation is generated to enhance the diversity of the generative model and provide a basis for the subsequent denoising process. The process of injecting noise needs to consider the type, intensity and injection method of the noise so that it can be reasonably integrated with the image feature representation to ensure that the injection of noise does not destroy the core information of the image features.

[0069] In this process, the process of injecting noise can be expressed by the following formula:

[0070]

[0071] Among them, z t represents the noisy potential representation at time step t; z0 represents the initial image feature representation; ∈ t represents the random noise corresponding to the time step t; a t Represents the intensity weight of noise injection, which controls the fusion ratio of noise and image feature representation.

[0072] Use normal distribution to generate random noise ∈ with the same dimension as the image feature representation t For example, assuming that the image features are represented as a 128-dimensional vector, a random noise vector of the same size is generated.

[0073] According to the time step t, the intensity weight a of the noise injection is calculated by a predefined function or a dynamic adjustment strategy t Generally, as the time step t increases, the noise weight gradually increases.

[0074] The generated random noise ∈ t The image feature representation z0 is weighted and superimposed according to the formula to obtain the noisy potential representation z of the current time step t. t . The generated zt As the input of the subsequent denoising model, it is used to enhance the generative model's ability to recover from noise and the diverse expression of potential features.

[0075] Example description: When generating a credit card marketing video, the system adds random noise (such as normal distribution noise with a mean of 0 and a standard deviation of 0.2) to the feature representation of the credit card sample image. The noisy latent representation is used as the input of the generative model to enhance the detail richness of the video generation. The final generated advertising video shows different scenarios of credit card applications.

[0076] By injecting noise into the image feature representation, the generalization ability of the model can be enhanced, and the diversity and detail expression of the generated video can be improved. At the same time, noise injection provides rich training data for the subsequent denoising process, which helps the generation model to remove irrelevant noise while retaining key feature information. In the financial field, more realistic, natural and diverse short video materials can be generated, reducing the labor cost in the video generation process and improving efficiency and quality.

[0077] S40, inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0078] In this embodiment, the noisy latent representation is processed by a denoising model to reduce the noise component and retain the key image feature information to generate an optimized latent representation. The role of the denoising model is to restore the core content of the image features through the learned denoising ability, and further improve the quality of the latent representation, providing high-quality input for subsequent semantic fusion and decoding operations.

[0079] Use a pre-trained denoising model (such as a model based on the Unet structure) and load its parameters. The initialized model can recognize different levels of noise and perform targeted denoising.

[0080] The noisy latent representation z generated in step t In the input denoising model, the denoising model processes the input data step by step through specific network layers.

[0081] In the first few layers of the denoising model, the noise characteristics in the noisy latent representation are analyzed, and the core image features are extracted through attention mechanisms or convolution operations, while irrelevant noise components are removed.

[0082] The intermediate layer of the denoising model generates an intermediate denoising representation, which has reduced most of the noise but still retains certain potential feature information for subsequent optimization operations.

[0083] In subsequent layers of the model, the intermediate representation is further optimized by extracting global and local features to generate the final optimized latent representation.

[0084] In this process, dynamic fusion is the key operation, which is implemented through the following formula:

[0085]

[0086] in, represents the potential representation at time step t-1; g t Represents the dynamic weight, which is used to adjust the fusion ratio of noise features and image features; F I represents the image feature representation, extracted from the image encoder; Indicates the part that retains the noise characteristics; g t ·F I Represents the part that introduces image features.

[0087] Through this formula, the denoising model can effectively complete noise restoration and feature optimization in the potential representation.

[0088] Use a pre-trained denoising model (such as a diffusion model based on Unet) and load the trained parameters to ensure that the model has the ability to handle noise.

[0089] The noisy latent representation z t Input the denoising model as the initial input of the model. t The noise characteristics in are modeled; the weight g is calculated according to the characteristic state of the current time step t-1 t , used to control the fusion ratio of noise and image features; dynamic fusion image features F I and noise features to generate an intermediate representation.

[0090] The subsequent layers of the denoising model refine and optimize the intermediate representation, and further improve the representation capability of the latent representation by extracting global and local features. The final output of the denoising model is used as the optimized latent representation to provide input for subsequent semantic fusion.

[0091] By processing the noisy latent representation through the denoising model, not only can the noise component be effectively reduced, but also the image features can be introduced through dynamic fusion to enhance the details and expressiveness of the latent representation. This step ensures the visual and semantic consistency of the generated video content, laying a high-quality foundation for subsequent multimodal semantic fusion and video decoding. In the financial field, it can significantly improve the detail performance of advertising videos and promotional videos, reduce the distortion rate of generated videos, and improve the efficiency and effectiveness of material production.

[0092] S50, inputting the text feature representation into the denoising model, and performing semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0093] In this embodiment, the text feature representation F T It is a numerical feature representation generated from the text encoder, which contains the semantic information of the input text. T Inputting the denoising model is the basic operation to realize multimodal information fusion. The denoising model needs to receive text features as input to complete the interactive fusion with the optimized latent representation.

[0094] Get the generated text feature representation F from the text encoder T . T Convert to a tensor format compatible with the denoising model input layer (such as normalization or dimension adjustment). Input to a specific module of the denoising model (such as the cross-modal attention layer) to participate in subsequent calculations.

[0095] The optimized latent representation (F O ) is a high-quality feature representation generated from the image feature representation by the denoising model, which contains the main visual information of the image. Semantic fusion requires F O and text feature representation F T Combined to ensure that the generated semantic latent representation (F S ) is semantically consistent with the input text while preserving the details of the image.

[0096] Extract the optimized latent representation F O , and input to the fusion module of the denoising model. Inside the denoising model, the dynamic weight mechanism γ is used to balance F T and F O contribution ratio.

[0097] Initial fusion is done using the following formula:

[0098]

[0099] The preliminary fusion results Passed to the next layer for contextual reinforcement processing.

[0100] Semantic association is the core task of semantic fusion. T and F O The multimodal semantic connection between them ensures that the generated latent representation can be aligned in the semantic space. The denoising model builds semantic associations through the attention mechanism, making the text and image features consistent in multimodal fusion.

[0101] In the denoising model, the matching degree of text features and image features is calculated through the cross-modal attention layer; important features are strengthened according to the attention score, such as highlighting image details that are highly related to the text description; and a further optimized fusion representation is generated. Complete the construction of semantic associations.

[0102] Semantic latent representation (F S ) is the final fusion result output by the denoising model, which contains both the semantic information of the text and the detailed features of the image, and serves as the direct input for the subsequent decoding stage. The output is the semantic latent representation F S The semantic latent representation is normalized into a decoder-compatible format (such as a specific dimension or tensor shape).

[0103] By inputting the text feature representation into the denoising model and semantically fusing it with the optimized latent representation, not only can the features of text and image in the semantic space be effectively aligned, but also detailed and semantically consistent video materials can be generated. In the financial field, this process provides reliable technical support for generating high-quality videos with precise marketing value, while significantly reducing the cost of material production.

[0104] S60, inputting the semantic latent representation into a decoder, generating a video frame sequence through the decoder and outputting the sequence.

[0105] In this embodiment, the semantic latent representation is a multimodal feature generated by semantic fusion, which contains information of images and texts and is used to guide the generation of video content. The semantic latent representation is passed as input to the decoder, which gradually generates video frames based on the latent representation.

[0106] Initialize the decoder: load the pre-trained decoder parameters to ensure that it can accept the semantic latent representation as input;

[0107] Input format processing: transform the semantic latent representation F S Convert to the tensor format required by the decoder input layer (such as a specific dimension or layer structure);

[0108] Semantic information parsing: The initial layer of the decoder is F S Parse and convert the semantic information into control signals for generating video frames.

[0109] The decoder gradually decodes the semantic latent representation through a multi-layer network, generating intermediate video frame data in the process of restoring the video content. These data are the preliminary representation of the video frame and contain the basic visual information for generating the video.

[0110] The decoder uses a multi-layer network structure (such as deconvolution layers or upsampling layers) to gradually convert the semantic latent representation into pixel data of video frames; during the decoding process, the generated features are dynamically adjusted through the attention mechanism to make the video frames consistent with the semantic latent representation; the output intermediate video frame data is a preliminary representation of the video frame, which may have low resolution or insufficient details.

[0111] Based on the intermediate video frame data, the decoder further restores the details and generates a complete video frame sequence. This step ensures that the generated video frames meet the output requirements in terms of resolution, details and coherence.

[0112] The subsequent layers of the decoder enhance the details of the video frames through specific feature restoration mechanisms (such as residual blocks or self-attention modules); perform temporal modeling on the data of consecutive video frames to ensure natural transitions between frames; and output a complete video frame sequence containing video frame data at all time steps.

[0113] The generated complete video frame sequence needs to be output in a specified format for subsequent storage, transmission or playback. The output video frames can be a frame sequence file (such as an image set) or a compressed video file (such as MP4 format).

[0114] The generated video frame data is organized into a frame sequence in chronological order; the frame sequence is converted into a specified video format (such as MP4, AVI, etc.) according to requirements; the generated video frame sequence is stored in a specified location or transmitted to the terminal device through a network interface.

[0115] Example description: In medical education and surgery planning, surgical simulation videos are an important teaching and auxiliary tool. High-quality surgical simulation videos can be generated by inputting images of the surgical area (such as CT or MRI scan results) and text descriptions of the surgical steps. This can be achieved by following the steps below:

[0116] Image input and feature extraction: Input the patient's surgical area CT image; use the image encoder to extract the key features of the CT image, including the boundaries of the anatomical structure and tissue density information.

[0117] Text input and feature extraction: Input text description, such as "surgical steps for liver tumor resection"; use the text encoder to extract semantic features in the text, including tumor location, surgical instrument use, and resection steps.

[0118] Multimodal fusion of images and text: The extracted image and text features are multimodally fused to generate semantic latent representations to ensure that the generated video content is consistent with the surgical description.

[0119] Video generation and output: The decoder decodes the semantic latent representation layer by layer to generate a simulated video of the surgical operation process, including the position of the surgical instruments, the changes in the resection area, and the dynamic changes of the surgical environment.

[0120] Medical students can use the generated surgical simulation videos to become familiar with complex surgical procedures; doctors can use videos to pre-plan surgeries, analyze key surgical steps, and reduce the risks of actual operations.

[0121] By inputting the semantic latent representation into the decoder and decoding it layer by layer, a video frame sequence that conforms to the input semantics can be generated. In the financial field, high-quality marketing video materials can be quickly generated, enabling financial institutions to promote products and services more efficiently while reducing the time cost and human resource investment in video production.

[0122] The present invention relates to the fields of artificial intelligence technology and medical health, and discloses a video generation method based on multimodal information fusion, which generates image feature representation and text feature representation by inputting image information and text information; injects noise into the image feature representation to generate a noisy potential representation; inputs the noisy potential representation into a denoising model to generate an optimized potential representation; inputs the text feature representation into the denoising model, performs semantic fusion with the optimized potential representation, generates a semantic potential representation; inputs the semantic potential representation into a decoder, generates a video frame sequence and outputs it. The present invention ensures that the generated video is semantically consistent with the input content by multimodal fusion of the image feature representation and the text feature representation; enhances the diversity and detail quality of video generation by injecting noise and denoising optimization; generates a video frame sequence by a decoder to ensure the visual coherence and natural transition of the video.

[0123] In one embodiment, the above S10 includes:

[0124] S101, acquiring multi-channel image data;

[0125] S102, using an image encoder to compress the multi-channel image data to generate a potential feature representation;

[0126] S103, extracting local texture features and edge information from the latent feature representation;

[0127] S104, normalizing the local texture features and edge information to generate a normalized image feature representation.

[0128] In this embodiment, multi-channel image data refers to image information containing multiple color or data channels (such as RGB images or depth images). Acquiring multi-channel image data is the starting point for image feature extraction. High-quality input data can be obtained in a variety of ways to ensure the accuracy of subsequent feature extraction.

[0129] Read image files from the file system (supporting formats such as JPEG and PNG); use video equipment (such as high-definition cameras) to capture images in real time; obtain image data (such as advertising images or user-uploaded images) from a remote server through an API interface.

[0130] Check if the image resolution meets the preset requirements; remove noise and artifacts to optimize image quality. Organize image channels into a standardized format (such as 3-channel RGB format or 4-channel RGBA format).

[0131] The task of the image encoder is to reduce the dimension and extract features of high-dimensional multi-channel image data, and extract key latent feature representations through deep learning models (such as convolutional neural networks). The latent feature representation is a numerical low-dimensional representation that retains the core information of the image and facilitates subsequent processing.

[0132] The acquired multi-channel image data is input into the image encoder. A multi-layer convolutional neural network (CNN) is used to extract features from the image and capture local features layer by layer. The high-dimensional image features are reduced to a potential representation of a fixed size through pooling operations or fully connected layers; the global structure and important features of the image are retained.

[0133] The output potential feature representation Z=E(X), where X is the input multi-channel image data and E is the image encoder model.

[0134] Local texture features and edge information are important visual features of images, which are used to describe the details and shape information of images. Extracting these features can enhance the expressive power of image feature representation, making it more suitable for subsequent multimodal information fusion.

[0135] Use convolution kernels to detect local patterns in the image (such as color gradients, texture changes). Apply edge detection methods (such as Sobel filter or Canny operator) to capture edge features in the image. Separate the extracted local texture features from the edge information and store them as independent feature channels.

[0136] Normalization is to standardize the eigenvalues ​​to make their numerical range consistent, which helps to improve the stability and efficiency of subsequent model processing. Local texture features and edge information are standardized separately, and the eigenvalues ​​are scaled to the range of [0,1].

[0137] The normalized local texture features and edge information are fused to generate the final normalized image feature representation. The output image feature representation is a numerical vector or matrix of fixed dimension, which is convenient for subsequent processing.

[0138] This embodiment can generate a normalized image feature representation containing rich visual information by acquiring multi-channel image data and using an image encoder to extract local texture features and edge information, thereby improving the efficiency and accuracy of subsequent multimodal information fusion and providing a solid foundation for generating high-quality video content.

[0139] In one embodiment, the above S20 includes:

[0140] S201, obtaining natural language text data;

[0141] S202, performing word segmentation processing on the natural language text data to generate a vocabulary sequence;

[0142] S203, using a text encoder to perform feature extraction on the vocabulary sequence to generate initial text features;

[0143] S204: Based on the initial text features, a semantic enhancement operation is performed through a context modeling mechanism to generate an enhanced text feature representation.

[0144] In this embodiment, obtaining natural language text data is the starting point for generating text features. The text data can be phrases, descriptions, titles, or content from a predefined data set input by a user, usually in the form of natural language. It is necessary to ensure that the acquired text has standardized grammar and complete structure.

[0145] The input sources can be: user input: receiving text entered by the user through the front-end interface, such as "fund return growth trend"; database: retrieving specific text content from the predefined copy database, such as advertising titles, slogans, etc.; network interface: obtaining relevant text information from external data sources (such as news websites, user comments) through API.

[0146] Remove redundant characters (such as special symbols and repeated spaces); standardize text formats to ensure uniform processing in multilingual scenarios.

[0147] Word segmentation is the process of dividing natural language text into words. Generating word sequences is the basis for further processing by the text encoder. The goal of word segmentation is to capture the basic semantic units of each word in the text.

[0148] The word segmentation method can be: rule-based: use separators (such as spaces) to segment the text into words; model-based: use pre-trained word segmentation tools (such as BERT word segmenter) for word segmentation.

[0149] For example, input the text "fund income growth trend"; generate the word sequence ["fund", "income", "growth", "trend"]. The word format can be unified (such as lowercase); and common abbreviations, spelling variations, etc. can be handled.

[0150] The text encoder digitizes the word sequence after word segmentation, maps it to the latent semantic space, and generates initial text features. The initial text features are mainly used to represent the basic semantic information in the text.

[0151] Use pre-trained word embedding models (such as Word2Vec, GloVe) to convert word sequences into numerical vectors. Each word is mapped to a vector of fixed dimension, for example, "Fund" -> [0.1, 0.3, 0.5].

[0152] Input the word sequence to the text encoder (such as the Transformer model), extract the contextual relationship between words through a multi-layer neural network, and output the initial text feature matrix, which represents the semantic relationship of each word.

[0153] The formula means:

[0154] represents the initial text features; E represents the text encoder; W represents the vocabulary sequence.

[0155] The semantic enhancement operation further strengthens the semantic information based on the initial text features through context modeling mechanisms such as self-attention mechanism. This process aims to capture the semantic relations at the sentence level as well as the implicit meaning in the text.

[0156] Capture long-range dependencies between words through the self-attention mechanism; emphasize word pairs with high semantic relevance in the text, such as "fund" and "income".

[0157] Use a multi-layer Transformer network to transform the initial text features Processing; output enhanced text feature representation F containing global semantic information T .

[0158] The formula means:

[0159] Attn represents the context modeling mechanism; F T Represents the enhanced text feature representation.

[0160] This embodiment can generate text feature representations containing rich semantic information through word segmentation, feature extraction, and semantic enhancement of text information. This ensures the semantic expression integrity of the text content and provides high-quality text features for multimodal information fusion. In the financial field, it can efficiently generate accurate text features for the generation of marketing videos, improving the semantic consistency and visual expression of the videos.

[0161] In one embodiment, the above S30 includes:

[0162] S301, generating random noise;

[0163] S302, adjusting the amplitude of the random noise so that the numerical range of the random noise is consistent with the numerical range represented by the image feature;

[0164] S303: superimpose the adjusted random noise with the image feature representation to generate the noisy latent representation.

[0165] In this embodiment, random noise is the core data for perturbing the image feature representation, the purpose of which is to enhance the diversity of potential representations by adding randomness and to provide a training basis for subsequent denoising. Generating random noise requires ensuring that its distribution characteristics meet the requirements of the target application (such as Gaussian distribution or uniform distribution).

[0166] Use normal distribution (Gaussian noise) to generate random noise, ensure that the mean of the noise is 0, and the variance can be set according to actual needs; if it is necessary to balance each feature channel, use multidimensional normal distribution to generate independent random noise.

[0167] Define the dimension of the noise, consistent with the shape of the image feature representation; use a random number generation function (such as NumPy's np.random.normal()) to generate a Gaussian noise matrix.

[0168] The generated random noise ∈ t It will be used as input for subsequent overlay operations.

[0169] In order to avoid excessive disturbance of the image feature representation by noise, the amplitude of the random noise needs to be adjusted so that its numerical range matches the image feature representation, thereby ensuring that the superimposed features are still usable.

[0170] The image feature representation F I Perform statistics, calculate the mean and standard deviation, and determine the numerical range.

[0171] The noise amplitude is adjusted using the following formula:

[0172]

[0173] ∈′ t represents the adjusted noise; σ ∈ represents the standard deviation of the original noise; Indicates the standard deviation of the image feature representation.

[0174] The generated random noise is standardized (zero mean, unit variance); according to the image feature table

[0175] The output amplitude adjusted noise ∈′ t For subsequent operations.

[0176] The noisy latent representation is generated by superimposing the adjusted random noise with the image feature representation. This step aims to increase the randomness and diversity of the feature representation while retaining the core features of the image.

[0177] The superposition is done using the following formula:

[0178]

[0179] z t represents the noisy latent representation; a t represents the noise intensity weight; F I Represents image feature representation; ∈′ t represents the adjusted random noise.

[0180] Dynamically adjust the noise weight a according to the current time step t t , the weight is small in the initial stage and gradually increases to enhance randomness. The adjusted noise is fused with the image feature representation according to the formula to output the final noisy potential representation z t .

[0181] This embodiment can significantly improve the diversity of feature representation and the generalization ability of the generated model by injecting noise into the image feature representation. Adjusting the noise amplitude ensures the stability of the superposition process, avoids feature distortion, and provides a training basis for the subsequent denoising model.

[0182] In one embodiment, the above S40 includes:

[0183] S401, initializing the parameters of the denoising model, and inputting the noisy latent representation into the denoising model;

[0184] S402, in the denoising model, according to the noise characteristics of the noisy latent representation, reducing the noise component in the noisy latent representation through a specific denoising layer to generate an intermediate denoised representation;

[0185] S403, extracting global features and local features in the intermediate denoising representation;

[0186] S404: Optimize the intermediate denoised representation based on the global features and the local features to generate a final optimized latent representation.

[0187] In this embodiment, the initialization of the denoising model is a prerequisite for ensuring that the model can accurately process the noisy latent representation. The initialization parameters include loading the pre-trained weights of the model, configuring dynamic adjustment parameters, etc. After the noisy latent representation is input into the denoising model, the model starts to perform denoising and feature optimization operations.

[0188] Load the pre-trained parameters of the denoising model (such as based on the diffusion model or Unet architecture). Set the dynamic parameters required for the model to run (such as time step, learning rate, etc.).

[0189] The generated noisy latent representation z tConvert to an input format compatible with the denoising model (such as tensor dimension alignment); input to the initial layer of the denoising model to start feature processing.

[0190] The denoising model analyzes the noise distribution in the noisy latent representation and uses specific denoising layers (such as convolutional layers or deconvolutional layers) to gradually reduce the noise component and generate a preliminary restored intermediate denoised representation.

[0191] Using the initial layer of the denoising model to denoise the input z t Model the noise characteristics; dynamically adjust the denoising weights according to the time step t to adapt to the noise changes.

[0192] Apply specific denoising layers (such as deconvolution or residual blocks) to remove the noise component layer by layer; generate an intermediate denoised representation z t-1 .

[0193] The denoising formula is expressed as:

[0194] z t-1 =f denoise (z t )

[0195] Among them, f denoise represents the mapping function of the denoising layer; z t-1 represents the intermediate denoised representation.

[0196] The extraction of global features and local features helps to further optimize the denoising representation. Global features provide overall semantic information, while local features capture details and textures.

[0197] Use global pooling operations (such as average pooling or maximum pooling) to reduce the dimension of the intermediate denoising representation and extract the global feature representation; the generated global features contain the overall structural information of the image.

[0198] Apply convolution kernels to scan the intermediate denoising representation to capture local texture features and edge information; organize local features into feature maps for subsequent detail optimization.

[0199] Global feature F global and local features F local They are used as inputs of the optimization module respectively.

[0200] The intermediate denoising representation is refined and optimized using global features and local features to generate the final optimized latent representation. Global features are used to maintain overall consistency, while local features are used to enhance detail representation.

[0201] The intermediate denoising representation is optimized by combining global and local features:

[0202] F opt =α·F glocal +(1-α)·Flocal

[0203] F opt represents the optimized potential representation; α represents the dynamic weights of global and local features.

[0204] Adjust the consistency of the overall structure based on global features; use local features to enhance the performance of texture and edge details.

[0205] Finally, the optimized potential representation F is generated opt , used for subsequent multimodal semantic fusion.

[0206] This embodiment can generate high-quality latent representations by inputting the noisy latent representation into the denoising model, combining noise removal and global / local feature optimization. This not only reduces the interference of noise, but also retains the key features of the image, providing high-quality input for subsequent multimodal fusion and video generation.

[0207] In one embodiment, the above S50 includes:

[0208] S501, initializing the semantic fusion parameters of the denoising model;

[0209] S502, inputting the text feature representation into the denoising model, and establishing a semantic association with the optimized latent representation to enhance the semantic information of the text feature representation;

[0210] S503, in the denoising model, the text feature representation and the optimized latent representation are interacted through a multimodal fusion layer to generate a fused semantic latent representation.

[0211] In this embodiment, the semantic fusion parameters are key parameters for controlling the multimodal information fusion process in the denoising model, and are used to adjust the weight distribution, interaction rules, and semantic enhancement methods between text features and optimized potential representations. Initializing these parameters ensures that the fusion process has high stability and accuracy.

[0212] Load the predefined semantic fusion parameters in the denoising model, including the initial weights and fusion rules; ensure that the parameters are applicable to the input multimodal data format (such as text and image feature representation).

[0213] According to the task requirements (such as the content and style of video generation), the initial weight assignment is dynamically adjusted; for example, the weight of text features is set slightly higher than the optimized latent representation to highlight the text semantics.

[0214] The output initialized semantic fusion module is used to process the features of subsequent input.

[0215] The establishment of semantic association is the core step of semantic fusion, which aims to align the text feature representation and the optimized latent representation through semantic matching and information exchange, thereby enhancing the semantic information of text features and making them more suitable for generating video content.

[0216] The text feature is represented by F T Input to the semantic fusion module of the denoising model; ensure that the input format is consistent with the model requirements (such as the dimension alignment of the tensor).

[0217] In the denoising model, the cross-attention mechanism is used to calculate the semantic matching degree between text features and optimized latent representations; the optimized latent representation features related to the text semantics are highlighted to make the text features more expressive.

[0218] The semantic enhancement formula is expressed as:

[0219]

[0220] represents the enhanced text feature representation; β represents the semantic enhancement weight; Attn(F T , F O ) represents the attention weighted result of text features and optimized latent representation.

[0221] Output enhanced text feature representation for subsequent multimodal interaction.

[0222] The multimodal fusion layer completes the final fusion of the text feature representation and the optimized latent representation. The generated semantic latent representation integrates the semantic information of the text and the detailed features of the image, providing multimodal consistent input for video generation.

[0223] Use the Multi-Head Attention mechanism to process text features and optimized potential representations in parallel; during the interaction process, capture the fine-grained relationship between different modalities.

[0224] The fused result is input into a nonlinear activation layer (such as ReLU) for further optimization; the final semantic potential representation F is output S .

[0225] The fusion formula is expressed as:

[0226]

[0227] F S represents the semantic potential representation after fusion; γ represents the dynamic weight of multimodal fusion.

[0228] This embodiment can generate semantically consistent and detailed semantic latent representations by inputting text feature representations into the denoising model and semantically fusing them with the optimized latent representations. This ensures that text and images are highly aligned in semantics, providing technical support for generating videos with both consistent content and excellent detail expression.

[0229] In one embodiment, the above S60 includes:

[0230] S601, initializing parameters of a decoder, and inputting the semantic latent representation into the decoder;

[0231] S602, in the decoder, according to the feature distribution of the semantic latent representation, decoding the semantic latent representation layer by layer to generate intermediate video frame data;

[0232] S603, serialize the intermediate video frame data to generate a video frame sequence and output the video frame sequence.

[0233] In this embodiment, the initialization of the decoder is the starting point of the decoding process, which is used to load the necessary parameters and ensure that the semantic potential representation can be processed correctly. The initialization parameters include the pre-trained weights of the decoder, the network architecture configuration, etc.

[0234] Load decoder parameters (such as convolution kernel weights, attention mechanism parameters) from the pre-trained model; configure the decoder's operating environment, including the dimensions of the input data and the batch size.

[0235] Ensure that the semantic latent representation F of the input S The format is consistent with the decoder input layer; converted to a tensor format that meets the input requirements (such as a specific dimension or number of channels).

[0236] After initialization, the semantic latent representation enters the first layer of the decoder for feature parsing.

[0237] The decoder generates intermediate video frame data by processing the semantic latent representation layer by layer. These data are the preliminary representation of the video frame and contain the main visual information of the semantic latent representation.

[0238] The decoder uses deconvolution layers or upsampling layers to restore the high-dimensional features of the semantic potential representation layer by layer; each layer increases the resolution and introduces detailed information.

[0239] According to the feature distribution of semantic latent representation, the decoding weights are adjusted to highlight the key features; the attention mechanism is used to enhance semantic consistency and ensure that the data of each frame matches the input text and image information.

[0240] Output intermediate video frame data F frameContains the basic content of a video frame, but may not have been serialized or further optimized.

[0241] The formula means:

[0242] F frame =g decode (F S )

[0243] g decode represents the layer-by-layer mapping function of the decoder; F frame Represents intermediate video frame data.

[0244] The serialization process organizes the intermediate video frame data into a time-ordered frame sequence and converts it into a standardized video format for output.

[0245] Arrange the intermediate video frame data in timeline order to form a frame sequence; ensure the continuity between frames and avoid video frame skipping due to frame sequence errors.

[0246] Compress the frame sequence into a video file according to the video coding standard (such as H.264 or H.265); if you need to output a frame image sequence, store it as a picture file (such as PNG, JPEG) according to the number.

[0247] The output video frame sequence can be directly used for playback, storage or transmission.

[0248] This embodiment can effectively realize the natural transformation of semantic latent representation into video content by inputting semantic latent representation into the decoder and decoding layer by layer to generate a video frame sequence. The layer-by-layer structure of the decoder ensures the clarity and semantic consistency of the generated video, and the serialization process ensures the coherence of the video.

[0249] In one embodiment, a video generation device based on multimodal information fusion is provided, and the video generation device based on multimodal information fusion corresponds to the video generation method based on multimodal information fusion in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a video generation device based on multimodal information fusion of the present invention. Image encoding module 10, text encoding module 20, noise injection module 30, denoising model module 40, semantic fusion module 50 and decoding module 60. The functional modules are described in detail as follows:

[0250] An image encoding module 10, used to obtain image information and generate image feature representation through an image encoder;

[0251] A text encoding module 20, used to obtain text information and generate text feature representation through a text encoder;

[0252] A noise injection module 30, used to inject noise into the image feature representation to generate a noisy latent representation;

[0253] A denoising model module 40, configured to input the noisy latent representation into a denoising model to generate an optimized latent representation;

[0254] A semantic fusion module 50, configured to input the text feature representation into the denoising model and perform semantic fusion with the optimized latent representation to generate a semantic latent representation;

[0255] The decoding module 60 is used to input the semantic latent representation into a decoder, and generate and output a video frame sequence through the decoder.

[0256] In one embodiment, the image encoding module 10 is specifically configured to:

[0257] Get multi-channel image data;

[0258] Using an image encoder to compress the multi-channel image data to generate a potential feature representation;

[0259] extracting local texture features and edge information from the latent feature representation;

[0260] The local texture features and edge information are normalized to generate a normalized image feature representation.

[0261] In one embodiment, the text encoding module 20 is specifically used for:

[0262] Obtain natural language text data;

[0263] Performing word segmentation processing on the natural language text data to generate a vocabulary sequence;

[0264] Using a text encoder to extract features from the vocabulary sequence to generate initial text features;

[0265] Based on the initial text features, a semantic enhancement operation is performed through a context modeling mechanism to generate an enhanced text feature representation.

[0266] In one embodiment, the noise injection module 30 is specifically configured to:

[0267] Generate random noise;

[0268] Adjusting the amplitude of the random noise so that the numerical range of the random noise is consistent with the numerical range represented by the image feature;

[0269] The adjusted random noise is superimposed on the image feature representation to generate the noisy latent representation.

[0270] In one embodiment, the denoising model module 40 is specifically used to:

[0271] Initializing parameters of a denoising model and inputting the noisy latent representation into the denoising model;

[0272] In the denoising model, according to the noise characteristics of the noisy latent representation, a noise component in the noisy latent representation is reduced by a specific denoising layer to generate an intermediate denoised representation;

[0273] extracting global features and local features in the intermediate denoised representation;

[0274] The intermediate denoised representation is optimized based on the global features and the local features to generate a final optimized latent representation.

[0275] In one embodiment, the semantic fusion module 50 is specifically used for:

[0276] Initializing semantic fusion parameters of the denoising model;

[0277] Inputting the text feature representation into the denoising model and establishing a semantic association with the optimized latent representation to enhance the semantic information of the text feature representation;

[0278] In the denoising model, the text feature representation and the optimized latent representation are interacted through a multimodal fusion layer to generate a fused semantic latent representation.

[0279] In one embodiment, the decoding module 60 is specifically configured to:

[0280] Initializing parameters of a decoder and inputting the semantic latent representation into the decoder;

[0281] In the decoder, according to the feature distribution of the semantic latent representation, the semantic latent representation is decoded layer by layer to generate intermediate video frame data;

[0282] The intermediate video frame data is serialized to generate a video frame sequence and the video frame sequence is output.

[0283] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a video generation method based on multimodal information fusion are realized.

[0284] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a video generation method based on multimodal information fusion

[0285] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0286] Obtain image information and generate image feature representation through an image encoder;

[0287] Obtain text information and generate text feature representation through a text encoder;

[0288] injecting noise into the image feature representation to generate a noisy latent representation;

[0289] Inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0290] Inputting the text feature representation into the denoising model and semantically fusing it with the optimized latent representation to generate a semantic latent representation;

[0291] The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

[0292] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0293] Obtain image information and generate image feature representation through an image encoder;

[0294] Obtain text information and generate text feature representation through a text encoder;

[0295] injecting noise into the image feature representation to generate a noisy latent representation;

[0296] Inputting the noisy latent representation into a denoising model to generate an optimized latent representation;

[0297] Inputting the text feature representation into the denoising model and semantically fusing it with the optimized latent representation to generate a semantic latent representation;

[0298] The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

[0299] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0300] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0301] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0302] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A video generation method based on multimodal information fusion, characterized in that: The following steps are involved: Obtain image information and generate image feature representation through an image encoder; Obtain text information and generate text feature representation through a text encoder; injecting noise into the image feature representation to generate a noisy latent representation; Inputting the noisy latent representation into a denoising model to generate an optimized latent representation; Inputting the text feature representation into the denoising model and semantically fusing it with the optimized latent representation to generate a semantic latent representation; The semantic latent representation is input into a decoder, and a video frame sequence is generated and output by the decoder.

2. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Obtain image information and generate image feature representation through the image encoder, including: Get multi-channel image data; Using an image encoder to compress the multi-channel image data to generate a potential feature representation; extracting local texture features and edge information from the latent feature representation; The local texture features and edge information are normalized to generate a normalized image feature representation.

3. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Get text information and generate text feature representation through the text encoder, including: Obtain natural language text data; Performing word segmentation processing on the natural language text data to generate a vocabulary sequence; Using a text encoder to extract features from the vocabulary sequence to generate initial text features; Based on the initial text features, a semantic enhancement operation is performed through a context modeling mechanism to generate an enhanced text feature representation.

4. The video generation method based on multimodal information fusion as claimed in claim 1, characterized in that: Injecting noise into the image feature representation to generate a noisy latent representation, including: Generate random noise; Adjusting the amplitude of the random noise so that the numerical range of the random noise is consistent with the numerical range represented by the image feature; The adjusted random noise is superimposed on the image feature representation to generate the noisy latent representation.

5. The video generation method based on multimodal information fusion according to claim 1, characterized in that: The noisy latent representation is input into a denoising model to generate an optimized latent representation, including: Initializing parameters of a denoising model and inputting the noisy latent representation into the denoising model; In the denoising model, according to the noise characteristics of the noisy latent representation, a noise component in the noisy latent representation is reduced by a specific denoising layer to generate an intermediate denoised representation; extracting global features and local features in the intermediate denoised representation; The intermediate denoised representation is optimized based on the global features and the local features to generate a final optimized latent representation.

6. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Inputting the text feature representation into the denoising model and semantically fusing it with the optimized latent representation to generate a semantic latent representation, including: Initializing semantic fusion parameters of the denoising model; Inputting the text feature representation into the denoising model and establishing a semantic association with the optimized latent representation to enhance the semantic information of the text feature representation; In the denoising model, the text feature representation and the optimized latent representation are interacted through a multimodal fusion layer to generate a fused semantic latent representation.

7. The video generation method based on multimodal information fusion according to claim 1, characterized in that: Inputting the semantic latent representation into a decoder, generating and outputting a video frame sequence through the decoder, including: Initializing parameters of a decoder and inputting the semantic latent representation into the decoder; In the decoder, according to the feature distribution of the semantic latent representation, the semantic latent representation is decoded layer by layer to generate intermediate video frame data; The intermediate video frame data is serialized to generate a video frame sequence and the video frame sequence is output.

8. A video generation device based on multimodal information fusion, characterized in that: The video generation device based on multimodal information fusion includes: An image encoding module, used to obtain image information and generate image feature representation through an image encoder; The text encoding module is used to obtain text information and generate text feature representation through a text encoder; A noise injection module, used for injecting noise into the image feature representation to generate a noisy latent representation; A denoising model module, configured to input the noisy latent representation into a denoising model to generate an optimized latent representation; A semantic fusion module, used for inputting the text feature representation into the denoising model and performing semantic fusion with the optimized latent representation to generate a semantic latent representation; The decoding module is used to input the semantic latent representation into a decoder, and generate and output a video frame sequence through the decoder.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a video generation program based on multimodal information fusion stored in the memory and executable on the processor. When the video generation program based on multimodal information fusion is executed by the processor, the steps of the video generation method based on multimodal information fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a video generation program based on multimodal information fusion, and when the video generation program based on multimodal information fusion is executed by the processor, the steps of the video generation method based on multimodal information fusion as described in any one of claims 1-7 are implemented.

Images