Method and system for generating high-time-sequence video based on combined guidance of text and audio

Abstract

The invention discloses a high-time-sequence video generation method and system based on text and audio combined guidance, and relates to computer vision processing and artificial intelligence, in a training stage, audio extraction is performed on video data, a video-audio-text triple is constructed and coded respectively, audio sequence-level coding is performed to capture time sequence features, a U-Net structure is utilized, and video-audio-text triple coding is performed. Noise is gradually added to potential representation of a video as time steps deepen, and a text signal and an audio signal are fused into a video generation process through serial video-text cross attention and video-audio time sequence attention. Learning the mapping relation between the multi-modal signal and video generation and the influence of the time sequence of music on video generation; in the non-training stage, the text and the audio serve as condition input, denoising is carried out step by step from initial complete noise, text and audio information is injected through serial video-text cross attention and video-audio time sequence attention in the U-Net, and a high-time-sequence video is generated.

Description

Technical Field

[0001] The present application relates to the fields of computer vision processing and artificial intelligence technology, and in particular to a method and system for generating highly temporal videos based on joint guidance of text and audio. Background Art

[0002] In recent years, with the explosive growth of video content demand, video generation tasks have gradually become an important research direction in the field of information technology. Traditional video creation usually relies on professional shooting and editing, which takes a lot of time and effort. However, with the rise of social media and short video platforms, users' demand for personalized, instant and high-quality video content continues to grow. How to enable ordinary users to quickly generate videos that meet specific themes or emotions without complex editing processes has become a major challenge. For this reason, automated video generation technology has emerged.

[0003] The core of the video generation task is to generate high-quality video content through algorithms, and its applications cover multiple fields from social entertainment to commercial marketing. On social media, users can use video generation technology to easily create attractive content to share life fragments, express emotions or convey opinions. For enterprises, video generation technology can play an important role in marketing promotion, helping to increase user engagement and market communication effects by generating video content that matches the brand image. In addition, in the field of education and training, video generation technology can help lecturers and students quickly produce teaching videos, enrich the presentation of educational content, and thus improve learning effects.

[0004] The development of video generation technology has gone through several stages. Initially, video generation mainly relied on traditional computer graphics methods to generate simple animation sequences by defining rules and parameters. With the rise of deep learning, neural network-based video generation methods have gradually become mainstream. These methods initially focused on how to generate videos from random noise or single images, and learned the spatiotemporal characteristics of videos through convolutional neural networks and recurrent neural networks. However, the videos generated by such methods often lack the guidance of semantic information, making it difficult to generate content that meets specific needs.

[0005] In recent years, with the development of large-scale pre-trained models, especially the breakthrough progress made by diffusion models in the field of image generation, researchers have gradually turned their attention to the task of video generation based on text prompts. Compared with the complex visual modeling relied on by traditional video generation methods and the rigid demand for a large amount of supervised data to learn video dynamic and temporal information, the video generation technology based on text prompts introduces natural language descriptions as control signals, so that the model can directly generate videos based on the user's text input. By understanding and parsing the semantic information in the text, the model can generate dynamic video content that conforms to the description. This method has obvious advantages over traditional video generation methods. First, text prompts can contain rich high-level semantic information, making the generated content more diverse and in line with user intentions; second, the generated video not only matches the text information in visual performance, but also reflects the dynamic changes in the text in time sequence; third, the text-generated video method enables users to customize video content through simple text descriptions, lowering the threshold for creation.

[0006] With the continuous evolution of video generation technology and the increasing maturity of text-guided video generation technology, researchers have begun to explore the application of other information in text-generated videos, including the introduction of audio signals. Audio can provide rich emotional and rhythmic information, providing a new control dimension for video generation tasks. Combining text and audio signals can generate emotional and rhythmic videos. However, combining text and audio signals for video generation faces great challenges: first, the text and audio features are quite different, and there may be modal conflicts between the two; in addition, it is difficult to achieve a balance between semantic consistency and temporal consistency in the generated video.

[0007] To address the above difficulties, some researchers proposed a video generation framework based on audio-text multimodal fusion to coordinate the complementary information of audio and text signals; other researchers are committed to exploring the joint modeling of text and audio to enhance consistency.

[0008] like Figure 1 As shown, in one of the existing technologies, "Multimodal fusion method for automatic generation of music videos", a Music2Video framework is proposed, which guides the generation of music videos with consistent content by fusing audio and text modalities.

[0009] First, the audio and text are encoded into a shared representation space, and the CLIP model is used to provide multimodal joint input conditions. Second, based on the rhythm and changes of the music, the music is dynamically segmented into segments of different lengths, and visual theme switching points are set during the generation of each segment to match the scene of the video with the mood and rhythm changes of the music. Third, inter-frame consistency optimization is set to maintain temporal consistency during the video generation process, so that the generated music video transitions naturally between different segments.

[0010] like Figure 2 As shown, in the second prior art "text-audio guided video generation", a TAgVM framework is proposed to generate video content with strong consistency through joint modeling of text and audio.

[0011] First, text and audio are used as joint input conditions to provide multimodal guidance on video content. Then, 3DVQ-GAN is used to compress video data, and an autoregressive model is used to generate basic scene outlines that match the text description. Based on the generated preliminary scenes, an audio-guided diffusion model is introduced to refine the dynamic details of the generated video. Finally, a temporal self-attention layer is added to the generation process to enhance cross-frame consistency and enhance the control effect of text and audio signals on video dynamic changes.

[0012] In the prior art, “Automatic generation of music videos based on fused audio and text” and “Text-audio guided video generation”:

[0013] 1. The processing method of audio features is relatively simple;

[0014] 2. Text plays a larger role in control.

[0015] Due to the above technical problems, the prior art has the following disadvantages:

[0016] 1. It is difficult to fully utilize the timing information in the audio signal;

[0017] 2. The video generation effect is more dependent on text, while audio features are difficult to be well utilized. Summary of the invention

[0018] The embodiments of the present application provide a method and system for generating highly temporal videos based on joint guidance of text and audio, which is used to generate videos with text, audio and other information as input conditions through the joint guidance of multimodal signals.

[0019] The present application embodiment provides a method for generating highly temporal video based on joint guidance of text and audio, which is performed in the training phase:

[0020] Each video V in the training set i Use the audio extractor to extract the corresponding audio A i , obtain the video-audio pair (V i ,A i );

[0021] For any video-audio pair (V i ,A i )'s video i , use a multimodal large model to train video V i Generate text description Ti , to obtain the video-audio-text triplet (V i ,A i ,T i );

[0022] For video-audio-text triples (V i ,A i ,T i ) to process the audio into a sequence-level audio feature v in vector form A , for the text T i Segment the words and represent them as text features v in vector form T ;as well as,

[0023] For video-audio-text triples (V i ,A i ,T i ) in the video V i Processing is performed to obtain a processed video tensor x, and the video tensor x is mapped to a potential representation z0 to obtain a vectorized input triple (z0, v A ,v T );

[0024] Generate the corresponding proportional coefficient according to the current time step t And according to the proportionality factor Generate random Gaussian noise ∈ with the same shape as the potential representation z, according to the random Gaussian noise ∈, the scale factor The latent representation z is obtained by noise-based latent representation z t ;

[0025] Noisy latent representation z t , audio features v A , text features v T Input U-Net to fuse the noisy latent representation z using the video-text cross-attention module at each layer of U-Net t and text feature v T , and use the video-audio temporal attention module to interactively noise the latent representation z t and audio feature v A ;

[0026] Calculate the loss to perform training;

[0027] Used for high temporal video generation after training.

[0028] Optionally, for any video-audio pair (V i ,A i )'s video i , use a multimodal large model to train video V iGenerate text description T i include:

[0029] For any video-audio pair (V i ,A i )'s video i , use the LLaVA large model to analyze the video V i Generate text description T i .

[0030] Optionally, for a video-audio-text triplet (V i ,A i ,T i ) to process the audio into a sequence-level audio feature v in vector form A include:

[0031] For video-audio-text triples (V i ,A i ,T i ) in audio A i Resample, convert to mono and normalize, and use the audio encoder to encode the audio into a vector-based sequence-level audio feature v according to the preset number of compressed video frames. A , where the sequence-level audio feature v A Extract as follows:

[0032]

[0033]

[0034] Among them, A i is the segmented audio sequence, is the kth segment in the audio sequence, f is the preset number of compressed video frames, and Audio Encoder(x) is the audio encoder.

[0035] Optionally, generate the corresponding scaling factor according to the current time step t include:

[0036]

[0037] α t =1-β t

[0038]

[0039] Among them, β t is the preset linear schedule, α t is the intermediate coefficient;

[0040] Noisy latent representation z t satisfy:

[0041]

[0042] in, is the proportionality coefficient, It decreases as the time step t increases.

[0043] Optionally, noisy the latent representation z t , audio features v A , text features v T Input U-Net to fuse the noisy latent representation z using the video-text cross-attention module at each layer of U-Net t and text feature v T include:

[0044] The following calculations are used in each layer of U-Net:

[0045] z t ′=Reshape(z t )

[0046]

[0047]

[0048] in, is the potential representation after calculating the video-text cross attention update, z t ′ is the potential representation after dimension change;

[0049] Interactively noisy latent representation z using video-audio temporal attention module t and audio feature v A include:

[0050] v A′ =Reshape(v A )

[0051]

[0052]

[0053] Among them, v A′ It is the audio feature aligned with the video spatial dimension. is the latent representation after computing the video-audio temporal attention.

[0054] Optionally, calculate the loss to perform training including:

[0055] Calculate the noise mean square error loss function

[0056] Use the optimizer to update the parameters during backpropagation;

[0057] Update time step t = t + 1, and repeat generating the corresponding proportional coefficient according to the current time step t Training is performed until t reaches a total of time steps T.

[0058] Optionally, calculate the noise mean square error loss function satisfy:

[0059]

[0060] Where E is the mean square error loss function, is the predicted noise.

[0061] An embodiment of the present application also proposes a high-temporal video generation system based on text-audio joint guidance, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the high-temporal video generation method based on text-audio joint guidance as described above are implemented.

[0062] The method of the embodiment of the present application generates a video by jointly guiding multimodal signals with text, audio and other information as input conditions.

[0063] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0065] Figure 1 It is a schematic diagram of the Music2Video framework of the prior art;

[0066] Figure 2 It is a schematic diagram of the TAgVM framework of the prior art;

[0067] Figure 3 This is a basic flow diagram of a method for generating highly temporal video based on joint guidance of text and audio according to an embodiment of the present application;

[0068] Figure 4 The flowchart of the high-temporal video generation method based on the joint guidance of text and audio according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0070] The abbreviations and key terms that appear in this embodiment are defined as follows:

[0071] LLaVA: Large Language and Vision Assistant is a pre-trained multimodal model that can generate text descriptions for videos.

[0072] CLAP: Contrastive Language-Audio Pretraining is a contrastive language-audio pretraining model that can encode audio;

[0073] T5-Small: Text to Text Transfer Transformer Small is a pre-trained large language model that can encode text;

[0074] VAE: Variational Auto-Encoders, a generative model whose encoder maps data to a latent space and the decoder reconstructs data from the latent space, suitable for image or video generation;

[0075] Net: A deep learning network architecture that can learn how to remove Gaussian noise and can effectively handle image generation tasks;

[0076] Reshape: A method used to adjust the dimensions of an array when the dimensions need to match;

[0077] Repeat: A method used to copy array elements when the array dimension needs to be extended;

[0078] CrossAttention: A method for enhancing the model's understanding of the relationship between different input sequences by allowing the model to focus on key information of one sequence while processing another sequence;

[0079] MSE: Mean Square Error, a common indicator for measuring the difference between the model prediction value and the actual value;

[0080] Adam: Adaptive Moment Estimation, an optimizer;

[0081] BP: Back Propagation, a method used for training.

[0082] The present application embodiment provides a method for generating a highly temporal video based on joint guidance of text and audio. Figure 3 , Figure 4 As shown, during the training phase:

[0083] In step S101, each video V in the training set i Use the audio extractor to extract the corresponding audio A i , obtain the video-audio pair (V i ,A i ).

[0084] In step S102, for any video-audio pair (V i ,A i )'s video i , use a multimodal large model to train video V i Generate text description T i , to obtain the video-audio-text triplet (V i ,A i ,T i In some examples, for any video-audio pair (V i ,A i )'s video i , use a multimodal large model to train video V i Generate text description T i include:

[0085] For any video-audio pair (V i ,A i )'s video i , use the LLaVA large model to analyze the video V i Generate text description T i .

[0086] In step S103, the video-audio-text triplet (V i ,A i ,T i ) to process the audio into a sequence-level audio feature v in vector form A , for the text T i Segment the words and represent them as text features v in vector form T .

[0087] In some embodiments, the video-audio-text triple (Vi ,A i ,T i ) to process the audio into a sequence-level audio feature v in vector form A include:

[0088] For video-audio-text triples (V i ,A i ,T i ) in audio A i Resample, convert to mono and normalize, and use the audio encoder to encode the audio into a vector-based sequence-level audio feature v according to the preset number of compressed video frames. A , where the sequence-level audio feature v A Extract as follows:

[0089]

[0090]

[0091] Among them, A i is the segmented audio sequence, is the kth segment in the audio sequence, f is the preset number of compressed video frames, and Audio Encoder (x) is the audio encoder. In the specific example, the audio encoder uses CLAP, the text encoder uses T5-Small, and the preset number of compressed video frames f is 60.

[0092] In step S104, the video-audio-text triplet (V i ,A i ,T i ) in the video V i Processing is performed to obtain a processed video tensor x, and the video tensor x is mapped to a potential representation z0 to obtain a vectorized input triple (z0, v A ,v T ).

[0093] Specifically for the video-audio-text triple (V i ,A i ,T i ) in the video V i Perform frame extraction, resolution unification, and pixel value standardization to obtain the processed video tensor x, which is input into the VAE encoder to map the video data to the potential representation z0. In this example,

[0094] In step S105, the time step t is initialized to 0.

[0095] In step S106, the corresponding proportional coefficient is generated according to the current time step t And according to the proportionality factor Generate random Gaussian noise ∈ with the same shape as the potential representation z, according to the random Gaussian noise ∈, the scale factor The latent representation z is obtained by noise-based latent representation z t .

[0096] In step S107, the noise potential representation z t , audio features v A , text features v T Input U-Net to fuse the noisy latent representation z using the video-text cross-attention module at each layer of U-Net t and text feature v T , and use the video-audio temporal attention module to interactively noise the latent representation z t and audio feature v A .

[0097] In step S108, the loss is calculated to perform training.

[0098] In step S109, after training, it is used for high temporal sequence video generation.

[0099] The method of the embodiment of the present application generates a video by jointly guiding multimodal signals with text, audio and other information as input conditions.

[0100] In some embodiments, the corresponding scaling factor is generated according to the current time step t include:

[0101]

[0102] α t =1-β t

[0103]

[0104] Among them, β t is the preset linear schedule, α t is the intermediate coefficient;

[0105] Noisy latent representation z t satisfy:

[0106]

[0107] in, is the proportionality coefficient, As the time step t increases, it decreases. ∈ is a Gaussian noise randomly sampled from a normal distribution, zt is the noised potential representation corresponding to time step t. In this embodiment, β min =0.0001, β max =0.02,

[0108] In some embodiments, the noise potential representation z t , audio features v A , text features v T Input U-Net to fuse the noisy latent representation z using the video-text cross-attention module at each layer of U-Net t and text feature v T include:

[0109] The following calculations are used in each layer of U-Net:

[0110] z t ′=Reshape(z t )

[0111]

[0112]

[0113] in, is the potential representation after calculating the video-text cross attention update, z t ′ is the potential representation after dimension change;

[0114] Interactively noisy latent representation z using video-audio temporal attention module t and audio feature v A include:

[0115] v A′ =Reshape(v A )

[0116]

[0117]

[0118] Among them, v A′ It is the audio feature aligned with the video spatial dimension. is the latent representation after computing the video-audio temporal attention.

[0119] In a specific example,

[0120] In some embodiments, calculating the loss to perform training includes:

[0121] Calculate the noise mean square error loss function

[0122] Use the optimizer to update the parameters during backpropagation;

[0123] Update time step t = t + 1, and repeat generating the corresponding proportional coefficient according to the current time step t The training is performed until t reaches the total time step T, for example, the time step T can be set to 1000.

[0124] In some embodiments, the noise mean square error loss function is calculated satisfy:

[0125]

[0126] Where E is the mean square error loss function, is the predicted noise. In the specific example,

[0127] In the non-training case, when generating a video: for the existing audio A i Resample, convert to mono and normalize, and use the audio encoder to encode the audio into a vector-based sequence-level audio feature v according to the preset number of compressed video frames. A ; For text T i Word segmentation, using a text encoder to represent the text as a text feature v in vector form T ; Get the vectorized audio-text input pair (v A ,v T );

[0128] The extraction of sequence-level audio features is as follows:

[0129]

[0130]

[0131]

[0132] fps is the frame rate of the generated video, L is the audio duration, and f′ is the preset number of compressed video frames.

[0133] The audio encoder uses CLAP, and the text encoder uses T5-Small.

[0134] Initialize the time step t to the preset time step T and execute the steps of this method.

[0135] like Figure 4As shown in the figure, the text and audio are encoded, where the text is encoded using the T5-Small text encoder, and the audio is encoded using the CLAP audio encoder according to the number of video frames to obtain sequence-level encoding, and the text features and audio features are obtained; during the training process, the VAE encoder is used to encode the video for training, and a potential representation of the video is initialized and denoised according to the time step; then the potential representation, the text vector, and the audio vector are input into U-Net together. In each layer of U-Net, the video potential representation first calculates the video-text cross attention with the text features, and then calculates the video-audio temporal attention with the audio features; finally, the model predicts a noise and calculates the loss function with the actual noise, and uses the Adam optimizer to update the weights during the back propagation process, and repeats the training for multiple rounds until the preset time step is reached. During the non-training process, a pure noise potential variable is initialized, and the text features and audio features are input into U-Net together, and the video-text cross attention and video-audio temporal attention are calculated respectively; then, the model gradually predicts the noise and denoises the potential variable; finally, a completely denoised potential variable is obtained, and the VAE decoder is used to decode the potential variable, and the video screen and audio are combined to obtain the video output.

[0136] The embodiment of the present application uses a high temporal video generation method based on text-audio joint guidance. The method generates video with text, audio and other information as input conditions through the joint guidance of multimodal signals. In the training stage, audio is extracted from the video data, and an accurate text description is generated using the multimodal large model LLaVA. Video-audio-text triples are constructed and encoded separately, where the audio sequence level is encoded to capture the temporal features, and then the U-Net structure is used to gradually add noise to the potential representation of the video with time steps, and the text signal and audio signal are fused into the video generation process through serial video-text cross attention and video-audio temporal attention to learn the mapping relationship between multimodal signals and video generation and the influence of the temporal nature of music on video generation; in the non-training stage, text and audio are used as conditional inputs, and denoised step by step from the initial complete noise, and the serial video-text cross attention and video-audio temporal attention in U-Net are used to inject text and audio information to generate a high temporal video.

[0137] An embodiment of the present application also proposes a high-temporal video generation system based on text-audio joint guidance, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the high-temporal video generation method based on text-audio joint guidance as described above are implemented.

[0138] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0139] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0141] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A high temporal sequence video generation method based on text and audio joint guidance, characterized in that: During the training phase, execute: Each video V in the training set i Use the audio extractor to extract the corresponding audio A i , obtain the video-audio pair (V i ,A i ); For any video-audio pair (V i ,A i )'s video i , use a multimodal large model to train video V i Generate text description T i , to obtain the video-audio-text triplet (V i ,A i ,T i ); For video-audio-text triples (V i ,A i ,T i ) to process the audio into a sequence-level audio feature v in vector form A , for the text T i Segment the words and represent them as text features v in vector form T ;as well as, For video-audio-text triples (V i ,A i ,T i ) in the video V i Processing is performed to obtain a processed video tensor x, and the video tensor x is mapped to a potential representation z0 to obtain a vectorized input triple (z0, v A ,v T ); Generate the corresponding proportional coefficient according to the current time step t And according to the proportionality factor Generate random Gaussian noise ∈ with the same shape as the potential representation z, according to the random Gaussian noise ∈, the scale factor The latent representation z is obtained by noise-based latent representation z t ; Noisy latent representation z t , audio features v A , text features v T Input U-Net to fuse the noisy latent representation z using the video-text cross-attention module at each layer of U-Net t and text feature v T , and use the video-audio temporal attention module to interactively noise the latent representation z t and audio feature v A ; Calculate the loss to perform training; Used for high temporal video generation after training.

2. The method for generating high temporal sequence video based on joint guidance of text and audio as claimed in claim 1, characterized in that: For any video-audio pair (V i ,A i )'s video i , use a multimodal large model to train video V i Generate text description T i include: For any video-audio pair (V i ,A i )'s video i , use the LLaVA large model to analyze the video V i Generate text description T i .

3. The high temporal sequence video generation method based on text and audio joint guidance as claimed in claim 1 is characterized in that: For video-audio-text triples (V i ,A i ,T i ) to process the audio into a sequence-level audio feature v in vector form A include: For video-audio-text triples (V i ,A i ,T i ) in audio A i Resample, convert to mono and normalize, and use the audio encoder to encode the audio into a vector-based sequence-level audio feature v according to the preset number of compressed video frames. A , where the sequence-level audio feature v A Extract as follows: Among them, A i is the segmented audio sequence, is the kth segment in the audio sequence, f is the preset number of compressed video frames, and Audio Encoder(x) is the audio encoder.

4. The method for generating high temporal sequence video based on joint guidance of text and audio as claimed in claim 3, characterized in that: Generate the corresponding proportional coefficient according to the current time step t include: α t ＝1-β t Among them, β t is the preset linear schedule, α t is the intermediate coefficient; Noisy latent representation z t satisfy: in, is the proportionality coefficient, It decreases as the time step t increases.

5. The method for generating high temporal sequence video based on joint guidance of text and audio as claimed in claim 4, characterized in that: Noisy latent representation z t , audio features v A , text features v T Input U-Net to fuse the noisy latent representation z using the video-text cross-attention module at each layer of U-Net t and text feature v T include: The following calculations are used in each layer of U-Net: z t ′ =Reshape(z t ) Among them, z t ′ is the potential representation after dimension change, It is the potential representation after calculating the video-text cross attention update; Interactively noisy latent representation z using video-audio temporal attention module t and audio feature v A include: v A′ =Reshape(v A ) Among them, v A′ It is the audio feature aligned with the video spatial dimension. is the latent representation after computing the video-audio temporal attention.

6. The method for generating high temporal sequence video based on joint guidance of text and audio as claimed in claim 5, characterized in that: Calculating the loss to perform training involves: Calculate the noise mean square error loss function Use the optimizer to update the parameters during backpropagation; Update time step t = t + 1, and repeat generating the corresponding proportional coefficient according to the current time step t Training is performed until t reaches a total of time steps T.

7. The method for generating high temporal sequence video based on joint guidance of text and audio as claimed in claim 6, characterized in that: Calculate the noise mean square error loss function satisfy: Where E is the mean square error loss function, is the predicted noise.

8. A high temporal video generation system based on text and audio joint guidance, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method for generating high temporal sequence video based on joint guidance of text and audio are implemented as described in any one of claims 1 to 7.

Images