Video generation method, video generation device, equipment and computer storage medium

Abstract

The invention provides a video generation method, a video generation device, video generation equipment and a computer storage medium. The video generation method comprises the following steps: acquiring input data of a plurality of modes; extracting a single-mode feature of the input data of each mode; mapping the plurality of single-modal features to a potential representation space of the same dimension to obtain a plurality of potential modal features; fusing the plurality of potential modal features to obtain a fused modal feature; and generating a target video by using the fusion modal features. Through the video generation method, the multi-modal features are uniformly processed by utilizing the potential representation space mapped to the same dimension, an effective fusion mechanism is established between multi-modal input, efficient complementation and joint representation between multi-modal information are improved, and the accuracy of generating the target video is facilitated.

Description

Technical Field

[0001] The present application relates to the technical field of video generation, and in particular to a video generation method, a video generation apparatus, a video generation device and a computer storage medium. Background Art

[0002] Most existing video generation technologies rely on single-modal input, such as images or text descriptions, and it is difficult to generate highly consistent and complex video content. At the same time, there are large differences between multimodal information. How to effectively fuse various modal information to generate coherent, natural, and highly relevant videos to the input modality is an important challenge in current video generation technology. In recent years, deep learning technology has made significant progress in the generation field, especially video generation models based on diffusion models. However, existing methods still have difficulties in multimodal fusion and poor fusion effects when processing multimodal inputs, resulting in poor generated video effects. Summary of the invention

[0003] In order to solve the above technical problems, the present application proposes a video generation method, a video generation device, a video generation equipment and a computer storage medium.

[0004] In order to solve the above technical problems, the present application proposes a video generation method, which includes:

[0005] Get input data for several modalities;

[0006] Extract unimodal features of each modal input data;

[0007] Mapping several unimodal features to a latent representation space of the same dimension to obtain several latent modal features;

[0008] fusing the plurality of latent modal features to obtain a fused modal feature;

[0009] The target video is generated by utilizing the fused modality features.

[0010] The fusion of the plurality of potential modal features to obtain the fused modal features includes:

[0011] Using a cross-modal attention mechanism to extract correlation information of the multiple latent modal features;

[0012] The plurality of potential modal features are fused according to the correlation information to obtain the fused modal feature.

[0013] The step of fusing the plurality of potential modal features according to the correlation information to obtain the fused modal feature includes:

[0014] Determining a video scene based on the input data of the plurality of modalities;

[0015] Acquire a dynamic weight according to the video scene and the plurality of unimodal features;

[0016] The plurality of potential modal features are fused according to the correlation information and the dynamic weight to obtain the fused modal feature.

[0017] Wherein, the video generation method further includes:

[0018] Extract the time series information of each single modal feature;

[0019] Acquire a time series feature of the single modal feature based on the time series information;

[0020] Aligning the time series information of the plurality of unimodal features according to the time series features of the plurality of unimodal features;

[0021] The step of generating a target video by utilizing the fused modality features includes:

[0022] The target video is generated by utilizing the fused modal features and the aligned time series information.

[0023] The aligning of the time series information of the plurality of unimodal features according to the time series features of the plurality of unimodal features comprises:

[0024] Obtaining similarities between the temporal features of the plurality of unimodal features and features of a currently generated video frame;

[0025] The time series information of the plurality of unimodal features is aligned in a manner of minimizing similarity.

[0026] Wherein, the generating the target video by using the fused modal features and the aligned time series information includes:

[0027] At each generation step, the fused modal features and the aligned time series information are input into a diffusion model to generate a video frame at each generation step;

[0028] All video frames of the generated step lengths are combined into the target video.

[0029] Wherein, the step of generating a target video by utilizing the fused modality features includes:

[0030] Generate an original video using the fused modality features;

[0031] The resolution of each video frame of the original video is improved by a super-resolution module, and the target video is generated according to the video frames after the resolution is improved.

[0032] In order to solve the above technical problems, the present application also proposes a video generation device, which includes: an input module, an extraction module, a mapping module, a fusion module, and a generation module; wherein:

[0033] The input module is used to obtain input data of several modes;

[0034] The extraction module is used to extract the single modal features of each modal input data;

[0035] The mapping module is used to map a plurality of unimodal features to a latent representation space of the same dimension to obtain a plurality of latent modal features;

[0036] The fusion module is used to fuse the plurality of latent modal features to obtain a fused modal feature;

[0037] The generating module is used to generate a target video using the fused modality features.

[0038] To solve the above technical problems, the present application also proposes a video generating device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the video generating method as described above.

[0039] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above video generation method.

[0040] Compared with the prior art, the beneficial effects of the present application are: the video generation device obtains input data of several modes; extracts the unimodal features of each modal input data; maps several unimodal features to the potential representation space of the same dimension to obtain several latent modal features; fuses the several latent modal features to obtain fused modal features; and generates the target video using the fused modal features. Through the above-mentioned video generation method, the multimodal features are uniformly processed using the latent representation space mapped to the same dimension, an effective fusion mechanism is established between the multimodal inputs, and the efficient complementarity and joint representation between the multimodal information are improved, which is conducive to the accuracy of generating the target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] in:

[0043] Figure 1 It is a flowchart of an embodiment of a video generation method provided by the present application;

[0044] Figure 2 yes Figure 1 The specific flow chart of step S14 of the video generation method shown;

[0045] Figure 3 is a flow chart of another embodiment of the video generation method provided by the present application;

[0046] Figure 4 It is a flowchart of another embodiment of the video generation method provided by the present application;

[0047] Figure 5 It is a structural schematic diagram of an embodiment of a video generating device provided by the present application;

[0048] Figure 6 It is a structural schematic diagram of an embodiment of a video generating device provided by the present application;

[0049] Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] The video generation method of the present application aims to solve the problem of difficulty in multimodal fusion in the prior art: the information of multimodal input often has differences in time steps and feature spaces, and it is difficult for the prior art to establish an effective fusion and alignment mechanism between multimodal inputs, resulting in the generation result being biased towards one modality and ignoring other modalities.

[0053] The present application relates to the field of video generation, and in particular to a video generation model based on a diffusion model. Specifically, it relates to a controllable video generation method based on multi-modal inputs such as text, audio, and images, which is applicable to the fields of video production, virtual reality, game development, etc.

[0054] The video generation method of the present application mainly accepts one or more modal information inputs such as text, audio, and images during training and inference, and solves problems such as difficult multimodal alignment and poor temporal consistency through multimodal fusion and adaptive alignment modules to generate high-quality videos that are coherent, natural, and highly relevant to the input modality.

[0055] Please refer to Figure 1 , Figure 1 It is a flowchart of an embodiment of a video generation method provided by the present application.

[0056] The camera-based video generation method of the present application is applied to a video generation device, wherein the video generation device of the present application can be a server, a terminal device, or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the video generation device, such as various units, subunits, modules, and submodules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0057] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0058] like Figure 1 As shown, the specific steps are as follows:

[0059] Step S11: Obtain input data of several modes.

[0060] In an embodiment of the present application, the video generation device obtains user input, or extracts multimodal input data from a storage medium, including but not limited to: text, audio, image, posture, trajectory and other input data.

[0061] Step S12: extracting the unimodal features of each modal input data.

[0062] In the embodiment of the present application, the video generation device extracts single-modal features according to the data characteristics of each modal data, and the extraction method includes but is not limited to the following feature extraction process:

[0063] Text input: Text modality is a common input method in video generation. The generation of specific scenes, actions or plots is guided by text descriptions. The present invention uses a text processing model based on a pre-trained language model (such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-training Transformer)) to encode the input text into a potential vector representation. First, the input text is segmented and the embedded representation of each word is obtained through the pre-trained language model. By stacking these word embeddings and combining position encoding, the entire sentence is converted into a high-dimensional potential representation. The self-attention mechanism in the Transformer architecture is used to model the text context to ensure that the model can capture the semantic relationships and important information in the sentence. This vector will serve as guiding information in the subsequent generation process and provide semantic drive.

[0064] Audio input: The audio modality is used to provide action or rhythm information in video generation, and plays an important role in applications such as music videos and dance generation. This application uses an audio feature extraction model (such as wav2vec (unsupervised pre-trained speech model)) to extract relevant features from the input audio. First, audio feature extraction is performed, and the audio signal is converted into a spectrogram or feature vector through a convolutional network or a self-supervised learning model to capture the rhythm, frequency and amplitude changes in the audio. Secondly, for timing modeling, one-dimensional convolution or RNN (Recurrent Neural Network) models (such as LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit, gated recurrent network)) can be used to model the timing characteristics of the audio to ensure that the temporal dynamics of the audio features can be accurately captured and reflected in the video generation.

[0065] Image input: The image modality provides visual reference for video generation and helps generate video frames with high visual consistency. This application uses convolutional neural networks or visual language models (such as ResNet, CLIP (Contrastive Language-Image Pre-Training, multi-modal pre-training neural network)) to extract features from the input image.

[0066] Posture and trajectory input: Posture and trajectory data are often used to generate action-related video scenes, such as dance, motion capture, etc. This application extracts human key points and motion trajectory information from the input data through a posture estimation model (such as DWPose). First, the posture features are extracted to identify the joint positions and corresponding motion trajectories of the human body from the input posture data, and generate time series data of 2D or 3D key points. Secondly, the motion information is modeled, and the extracted posture information is modeled in time series to ensure that the motion trajectory of the human body can be naturally presented in the generated video.

[0067] Step S13: Mapping a plurality of unimodal features to a latent representation space of the same dimension to obtain a plurality of latent modal features.

[0068] Step S14: Fuse several latent modal features to obtain fused modal features.

[0069] In the embodiments of the present application, multimodal inputs are usually heterogeneous, with different information expression forms and time characteristics. The present application uses a multimodal fusion mechanism to fuse the feature vectors of each modality in a unified latent space, thereby achieving efficient complementarity and joint representation of information. The mechanism is designed based on a diffusion model and a cross-modal attention mechanism.

[0070] Specifically, the video generating device maps the plurality of unimodal features extracted in step S12 to a latent representation space of the same dimension, wherein the same dimension may specifically be 1024 dimensions, or other dimensions, which are not specifically limited herein.

[0071] The video generation device fuses multimodal features of the same latent representation space to obtain fused modal features, wherein the fusion method includes but is not limited to: splicing, embedding and other feature fusion methods.

[0072] Please refer to Figure 2 , Figure 2 yes Figure 1 The specific flow chart of step S14 of the video generating method is shown.

[0073] like Figure 2 As shown, the specific steps are as follows:

[0074] Step S141: Use the cross-modal attention mechanism to extract the correlation information of several potential modal features.

[0075] In the embodiment of the present application, in order to achieve unified processing of multimodal information, the feature vectors of each modality are first embedded into a joint latent space. This latent space has sufficient expressive power to capture the interactive information between different modalities. The specific design includes cross-modal feature conversion and modal attention mechanism.

[0076] Among them, cross-modal feature conversion is to map the feature vectors of different modalities to the latent representation space of the same dimension through the projection layer, so as to ensure that the features of each modality can interact effectively; the modal attention mechanism is to integrate and weight the multimodal information in the latent space through the cross-modal attention mechanism. The features of different modalities influence each other through the attention mechanism, so that the model can focus on the most important modal information at the current moment.

[0077] Step S142: Fusing several latent modal features according to the correlation information to obtain a fused modal feature.

[0078] In the embodiment of the present application, in the process of generating a video, the importance of different modes will change dynamically with time or scene changes. The present application uses a modality adaptive weight adjustment strategy to adjust the weight distribution of each mode in the generation process in real time.

[0079] First, the video generation device calculates the modal weights. At each time step, the model dynamically calculates the weight of each modality based on the input data and the current generation state. The weight is calculated through the weight matrix in the self-attention mechanism, reflecting the importance of different modalities. Secondly, the weights are dynamically adjusted. During the video generation process, as the audio rhythm, text content, or visual features change, the system automatically adjusts the weights of each modality. For example, in dance generation, the audio modality may dominate the action rhythm of the video; while in static scene descriptions, the weight of the text modality may be greater.

[0080] Specifically, the video generation device determines the video scene through the input data of all modes. For example, the text data describes: "Generate a beautiful dance video." It is analyzed that the video scene currently required to be generated is a dance scene. In the dance scene, the weight of the audio modality feature is higher than the features of other modes.

[0081] Step S15: Generate a target video using the fused modal features.

[0082] In the embodiment of the present application, the video generation device generates the final target video according to the fused modal features. Since the fused modal features have already performed multimodal fusion on each modal data, the fused modal features obtained can well reflect the correlation of each modal feature, so that the generated target video can accurately reflect the information of each modal data at the same time, thereby improving the accuracy of video generation.

[0083] In the present application, a video generation device obtains input data of several modes; extracts unimodal features of each modal input data; maps several unimodal features to a latent representation space of the same dimension to obtain several latent modal features; fuses the several latent modal features to obtain fused modal features; and generates a target video using the fused modal features. Through the above-mentioned video generation method, multimodal features are uniformly processed using a latent representation space mapped to the same dimension, an effective fusion mechanism is established between multimodal inputs, and efficient complementarity and joint representation between multimodal information are improved, which is conducive to the accuracy of generating a target video.

[0084] Another video generation method of the present application aims to solve the problem of poor temporal consistency in the prior art: when processing time series information, the existing methods have weak coherence between the generated video frames and weak temporal correlation with the input modality, especially in long-term video generation tasks, which are prone to temporal inconsistency.

[0085] Please refer to Figure 3 , Figure 3 It is a flowchart of another embodiment of the video generation method provided by the present application.

[0086] like Figure 3 As shown, the specific steps are as follows:

[0087] Step S21: extracting the time series information of each single modal feature.

[0088] In the embodiment of the present application, it is very important to ensure the temporal consistency between the modalities during the video generation process. The present application solves the problem of inconsistent time steps in multimodal input through an adaptive modality alignment mechanism, ensuring that the generated video is highly consistent with the temporal characteristics of the input modality.

[0089] Step S22: Acquire the time series features of the unimodal features based on the time series information.

[0090] In the embodiments of the present application, different modalities have their own independent temporal characteristics. For example, an audio signal has high-frequency changes, while a text description may remain unchanged for a long period of time. The present application models the temporal characteristics of different modalities through a temporal attention mechanism.

[0091] First, we extract temporal features and use the temporal attention mechanism to extract the temporal information of each modality, especially capturing the key changes of the modality at different time steps. For example, the rhythm change points in the audio or the key descriptive words in the text are identified as important temporal features.

[0092] Step S23: aligning the time series information of the plurality of unimodal features according to the time series features of the plurality of unimodal features.

[0093] In an embodiment of the present application, after extracting the timing features of each modal data in step 22, the video generation device further aligns the timing information. By aligning the timing features of each modality, the model can generate video content that is highly consistent in time with the multimodal input.

[0094] The specific alignment process is to minimize the error by calculating the similarity between the temporal features of the input modality and the features of the currently generated video frame.

[0095] Step S24: Generate a target video using the fused modal features and the aligned time series information.

[0096] In the embodiment of the present application, the diffusion model is used as the generation framework of the present application. The step-by-step generation process is used to ensure the high quality and natural coherence of the video generation, and clear video frames are generated through step-by-step denoising.

[0097] The diffusion model gradually generates video frames from noise through a reverse diffusion process. First, the noise is initialized. The generation process starts with a random noise video frame, gradually removes the noise, and generates visual content. Secondly, step-by-step generation is performed. At each generation step, the model generates video frames through multimodal fusion and modality alignment mechanisms, and ensures that the guidance information of each modality is correctly fused.

[0098] Furthermore, the video generation method of the present application can also integrate a super-resolution module for post-processing of the video. For details, please refer to Figure 4 , Figure 4 It is a flowchart of another embodiment of the video generation method provided by the present application.

[0099] like Figure 4 As shown, the specific steps are as follows:

[0100] Step S31: Generate the original video using the fused modal features.

[0101] Step S32: improving the resolution of each video frame of the original video through a super-resolution module, and generating the target video according to the video frames after the resolution is improved.

[0102] In the embodiment of the present application, the video super-resolution is to improve the clarity of the generated video. The present application integrates a super-resolution module based on a convolutional neural network (such as ESRGAN (Enhanced Super-Resolution Generative Adversarial Network)). The generated low-resolution video frame is restored in detail and the resolution is improved through the module to ensure that the final output video has high definition and visual quality.

[0103] The present application proposes a controllable video generation method for multimodal input. Specifically, based on the diffusion model, one or more modal information such as text, audio, image, posture, trajectory, etc. is accepted as input during training and inference. First, the information of different modalities is converted into a unified potential representation through a specific representation encoding module; secondly, the information of different modalities is fused into a unified latent space through a cross-modal attention mechanism to ensure the effective transmission and interaction of each modal feature, so that the system can obtain correlation information from multiple modal inputs; then, in order to ensure the temporal consistency of the generated video frame with the input modality, the temporal features of different modalities are processed through the time series attention mechanism to ensure that the video is closely aligned with the input modality in each time step. Finally, based on the step-by-step generation process of the diffusion model, each frame of video is generated starting from noise until a clear and natural output is obtained; in addition, in order to further improve the video quality, a video super-resolution module can be integrated to improve the resolution and detail performance of the video.

[0104] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0105] To implement the above video generation method, this application also proposes a video generation device, please refer to Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of a video generating device provided by the present application.

[0106] The video generating device 500 of this embodiment includes: an input module 51 , an extraction module 52 , a mapping module 53 , a fusion module 54 , and a generating module 55 .

[0107] The input module 51 is used to obtain input data of several modes.

[0108] The extraction module 52 is used to extract the single modal features of each modal input data.

[0109] The mapping module 53 is used to map a plurality of unimodal features to a latent representation space of the same dimension to obtain a plurality of latent modal features.

[0110] The fusion module 54 is used to fuse the multiple latent modal features to obtain a fused modal feature.

[0111] The generating module 55 is used to generate a target video using the fused modality features.

[0112] In order to implement the above-mentioned video generation method, the present application also proposes a video generation device, for details, please refer to Figure 6 , Figure 6 It is a structural schematic diagram of an embodiment of a video generating device provided by the present application.

[0113] The video generating device 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0114] The processor 41 , the memory 42 , and the input / output device 43 are respectively connected to the bus 44 . The memory 42 stores program data. The processor 41 is used to execute the program data to implement the video generation method described in the above embodiment.

[0115] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0116] This application also provides a computer storage medium, please continue to refer to Figure 7 , Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the video generation method of the above embodiment.

[0117] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0118] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video generation method, characterized in that: The video generation method comprises: Get input data for several modalities; Extract unimodal features of each modal input data; Mapping several unimodal features to a latent representation space of the same dimension to obtain several latent modal features; fusing the plurality of latent modal features to obtain a fused modal feature; The target video is generated by utilizing the fused modality features.

2. The video generation method according to claim 1, characterized in that: The step of fusing the plurality of potential modal features to obtain a fused modal feature includes: Using a cross-modal attention mechanism to extract correlation information of the multiple latent modal features; The plurality of potential modal features are fused according to the correlation information to obtain the fused modal feature.

3. The video generation method according to claim 2, characterized in that: The fusing the plurality of potential modal features according to the correlation information to obtain the fused modal feature includes: Determining a video scene based on the input data of the plurality of modalities; Acquire a dynamic weight according to the video scene and the plurality of unimodal features; The plurality of potential modal features are fused according to the correlation information and the dynamic weight to obtain the fused modal feature.

4. The video generation method according to claim 1, characterized in that: The video generation method further includes: Extract the time series information of each single modal feature; Acquire a time series feature of the single modal feature based on the time series information; Aligning the time series information of the plurality of unimodal features according to the time series features of the plurality of unimodal features; The step of generating a target video by utilizing the fused modality features includes: The target video is generated by utilizing the fused modal features and the aligned time series information.

5. The video generation method according to claim 4, characterized in that: The aligning the time series information of the plurality of unimodal features according to the time series features of the plurality of unimodal features comprises: Obtaining similarities between the temporal features of the plurality of unimodal features and features of a currently generated video frame; The time series information of the plurality of unimodal features is aligned in a manner of minimizing similarity.

6. The video generation method according to claim 4, characterized in that: The step of generating the target video by utilizing the fused modality features and the aligned time series information includes: At each generation step, the fused modal features and the aligned time series information are input into a diffusion model to generate a video frame at each generation step; All video frames of the generated step lengths are combined into the target video.

7. The video generation method according to claim 1 or 6, characterized in that: The step of generating a target video by utilizing the fused modality features includes: Generate an original video using the fused modality features; The resolution of each video frame of the original video is improved by a super-resolution module, and the target video is generated according to the video frames after the resolution is improved.

8. A video generating device, characterized in that: The video generation device comprises: an input module, an extraction module, a mapping module, a fusion module, and a generation module; wherein, The input module is used to obtain input data of several modes; The extraction module is used to extract the single modal features of each modal input data; The mapping module is used to map a plurality of unimodal features to a latent representation space of the same dimension to obtain a plurality of latent modal features; The fusion module is used to fuse the plurality of latent modal features to obtain a fused modal feature; The generating module is used to generate a target video using the fused modality features.

9. A video generating device, characterized in that: The video generating device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the video generation method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the video generation method according to any one of claims 1 to 7.

Images