Text-to-video generation method, device, electronic device, and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for generating video from text, which relates to the field of computer technology. Since a spatial attention module, a temporal attention module and a cross-modal attention module are set to respectively calculate spatial attention feature information, temporal attention feature information and cross-modal attention feature information of a text sequence in parallel, parallel calculation of different attention mechanisms can reduce the amount of communication and significantly improve the efficiency of communication. When forming each frame of the picture, the spatial attention module, the temporal attention module and the cross-modal attention module are used to control the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information according to the change rate of the influence of the spatial attention module, the temporal attention module and the cross-modal attention module on the generated video, so that different attention feature information can be broadcast using broadcast mechanisms of different frequencies, thereby reducing computing resources and time and improving the efficiency of generating videos.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, and storage medium for generating video from text. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, tools like Wenshengtu and Wenshengvideo have become essential tools for creative professionals. For example, the Wenshengtu Stable Diffusion model is an image generation model based on the diffusion process, capable of generating high-quality, high-resolution images. It simulates the diffusion process, gradually transforming a noisy image into a target image. This model boasts strong stability and controllability, enabling it to generate images with diverse effects and visual appeal. Wenshengvideo's Open-Sora v has achieved a quantum leap in text-to-video capabilities. Users simply input a text description, and the model transforms it into a vivid and realistic video. This instantaneous conversion from text to visuals has undoubtedly injected new vitality and possibilities into the content creation field. By converting text descriptions into visual content, these tools significantly lower the barrier to entry and improve efficiency. However, due to hardware and video memory requirements, generating a video can take significantly longer. The larger the pixels and the higher the frame rate, the longer the video takes. Summary of the Invention

[0003] The present application provides a method, device, electronic device and storage medium for generating video from text, in order to at least solve the problem in the related art that when generating video from text, time, space and cross-modal attention must be alternately executed for each frame, and each alternating execution requires recalculating the attention weight, integrating local and global features in the spatial dimension, sorting out the temporal dependencies of the sequence in the temporal dimension, and fusing and interacting information across modalities, involving a large number of matrix operations and data processing, consuming a large amount of computing resources and time, and causing a technical problem of greatly increased time consumption.

[0004] This application provides a method for generating a video from text, including:

[0005] Forming a text sequence based on text information of a pre-generated video;

[0006] Setting a spatial attention module, a temporal attention module and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information and cross-modal attention feature information for the text sequence in parallel;

[0007] Obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the influence change rate;

[0008] The text sequence is decoded based on the virtual twin method to generate multiple frames, and each frame is processed according to the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information to output a video file.

[0009] This application also provides a text-generated video device, comprising:

[0010] A text conversion module, used to form a text sequence based on the text information of the pre-generated video;

[0011] An attention feature calculation module is used to set a spatial attention module, a temporal attention module, and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information, and cross-modal attention feature information for the text sequence in parallel;

[0012] a broadcast frequency control module, configured to obtain a rate of change of the influence of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and control the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the rate of change of the influence;

[0013] A picture processing module is used to decode the text sequence based on the virtual twin method to generate multiple frames, and process each frame according to the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information to output a video file.

[0014] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for generating a video from text when executing the computer program:

[0015] Forming a text sequence based on text information of a pre-generated video;

[0016] Setting a spatial attention module, a temporal attention module and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information and cross-modal attention feature information for the text sequence in parallel;

[0017] Obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the influence change rate;

[0018] The text sequence is decoded based on the virtual twin method to generate multiple frames, and each frame is processed according to the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information to output a video file.

[0019] The present application also provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating a video from text are implemented:

[0020] Forming a text sequence based on text information of a pre-generated video;

[0021] Setting a spatial attention module, a temporal attention module and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information and cross-modal attention feature information for the text sequence in parallel;

[0022] Obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the influence change rate;

[0023] The text sequence is decoded based on the virtual twin method to generate multiple frames, and each frame is processed according to the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information to output a video file.

[0024] Through the present application, since a spatial attention module, a temporal attention module and a cross-modal attention module are set to calculate the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information of the text sequence in parallel, the parallel calculation of different attention mechanisms can reduce the communication volume and significantly improve the efficiency of communication. When forming each frame of the picture, the spatial attention module, the temporal attention module and the cross-modal attention module are used to control the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information according to the change rate of the influence of the generated video. It is possible to broadcast different attention feature information using different frequency broadcast mechanisms, thereby reducing computing resources and time and improving the efficiency of generating videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 It is the logic diagram of the existing DiT module;

[0027] Figure 2 This is a diagram of the application environment of the method for generating video from text in one embodiment of the present application;

[0028] Figure 3 This is a flowchart of a method for generating a video from text in one embodiment of the present application;

[0029] Figure 4 This is a logic diagram of a method for generating video from text in one embodiment of the present application;

[0030] Figure 5 This is a structural block diagram of a device for generating a video from text in one embodiment of the present application;

[0031] Figure 6 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0033] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0034] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] As mentioned in the background, most text-based video models use the Diffusion Transformer (DiT) architecture. DiT (Diffusion Transformer) is a diffusion model combined with the Transformer architecture for image and video generation tasks. It efficiently captures dependencies in data and produces high-quality results. The core idea of ​​DiT is to gradually add noise to the data by simulating a diffusion process, and then learn to reverse this process to construct the desired data samples from the noise. For example, the text-based video model open-sora is based on the high-quality open-source text-based image model PixArt-α. It introduces a temporal attention layer to extend this model to video data. Specifically, the entire architecture consists of a pre-trained VAE, a text encoder, and an STDiT (Spatial Temporal Diffusion Transformer) model that utilizes a spatial-temporal attention mechanism. When predicting a video, the model randomly generates noise, which is then removed through multiple rounds of DiT, ultimately producing a video that is relevant to the input text.

[0036] In the Wensheng video model, the most time-consuming module is the DiT module, which accounts for about 90% of the total time. Taking the Opensora model as an example, the structure of each layer of its DiT is as shown in the attached figure. Figure 1 As shown in the figure. It superimposes a one-dimensional temporal attention module on a two-dimensional spatial attention module in a serial manner to model temporal relationships. After the temporal attention module, the cross-modal attention module is used to align the semantics of the text. The DiT module of the Openora model superimposes a one-dimensional temporal attention module on a two-dimensional spatial attention module in a serial manner. This module has a significant performance bottleneck during actual operation. The main part of its time consumption is spent on 28 alternating executions of temporal, spatial, and cross-modal attention. For each alternating execution, the model needs to recalculate the attention weights, integrate local and global features in the spatial dimension, sort out the temporal dependencies of the sequence in the temporal dimension, and fuse and interact information across modalities. Such frequent alternating calculations involve a large number of matrix operations and data processing, which consumes a lot of computing resources and time. The time consumed by this process accounts for 90% of the total time consumption, seriously affecting the operating efficiency and real-time performance of the model.

[0037] The text-to-video method provided in this application can be applied to Figure 2In the application environment shown, terminal 102 and server 104 communicate via a network. Terminal 102 can input pre-generated video text information into server 104, which then processes the pre-generated video text information to generate and output a video file. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0038] like Figure 3 As shown, an embodiment of the present application provides a method for generating a video from text, comprising the following steps:

[0039] Step S1, forming a text sequence according to the text information of the pre-generated video;

[0040] Step S2: setting a spatial attention module, a temporal attention module, and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information, and cross-modal attention feature information for the text sequence in parallel;

[0041] Step S3, obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the influence change rate;

[0042] Step S4: decode the text sequence based on the virtual twin method to generate multiple frames, process each frame according to the broadcast frequency of the spatial attention feature information, temporal attention feature information, and cross-modal attention feature information, and output a video file.

[0043] This application exploits the different characteristics of three attention mechanisms: spatial attention is the most variable, involving high-frequency elements such as edges and textures; temporal attention exhibits medium-frequency variations related to motion and dynamics in the video; and cross-modal attention is the most stable, connecting text with video content, similar to a low-frequency signal reflecting the text's semantics. They are then broadcast at different frequencies. For the spatial attention mechanism, we broadcast it once every two times; for the temporal and cross-modal attention mechanisms, because they are more stable, we broadcast it once every four or eight times. This not only improves the performance of the DiT module, but also results in negligible loss in the quality of the generated content.

[0044] Experiments have shown that the attention differences at different time steps show a U-shaped pattern, with significant changes in the first and last 15% of steps, while the middle 70% of steps are very stable with little difference.

[0045] Second, within the stable middle segment, there are differences between attention types: spatial attention varies the most, involving high-frequency elements such as edges and textures; temporal attention exhibits medium-frequency variations related to motion and dynamics in the video; and cross-modal attention is the most stable, linking text to video content, similar to a low-frequency signal reflecting the semantics of the text.

[0046] Furthermore, we utilize Dynamic Sequence Parallelism to parallelize the computation of different attention mechanisms. Since the computation of each attention mechanism's sequence dimension is independent of each other, for example, when computing a spatial transformer block, the computation is independent of the temporal dimension, and data can be split between machines without affecting the results. To leverage this advantage, we propose dynamically switching the dimension of sequence parallelism. This eliminates the need for complex communication within the attention module, requiring only a single dynamic switch between different computation stages, significantly reducing communication overhead. Leveraging this multi-dimensional computational capability, our approach can be generalized to all multi-dimensional transformers.

[0047] In this embodiment, obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information in descending order of the influence change rate includes:

[0048] Analyze the impact of spatial attention module, temporal attention module and cross-modal attention module on the rate of change of generated videos;

[0049] The influence weights of the spatial attention module, temporal attention module, and cross-modal attention module are set according to the influence change rate;

[0050] According to the influence weights of the spatial attention module, the temporal attention module and the cross-modal attention module, the broadcast frequencies of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information at each time step are set respectively.

[0051] In this embodiment, analyzing the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video; setting the influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module according to the influence change rate includes:

[0052] Obtain the image information changes of the spatial attention module, the temporal attention module, and the cross-modal attention module in two adjacent frames when generating a video, and calculate the impact change rate on the generated video based on the image information changes of the two adjacent frames;

[0053] Counting the number of consecutive frames when the influence change rate corresponding to the spatial attention module, the temporal attention module, and the cross-modal attention module is greater than a first threshold;

[0054] The influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module are set according to the number of consecutive picture frames when the influence change rate is greater than the first threshold, and the value of the influence weight is positively correlated with the number of consecutive picture frames when the influence change rate is greater than the first threshold.

[0055] Among them, a broadcast controller can be added, which is responsible for controlling the broadcast frequency of each attention module.

[0056] It is understandable that setting the broadcast frequency of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information also includes:

[0057] Identify the complexity of video content based on pre-generated text information of the video;

[0058] Adjust the broadcast frequency of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information according to the complexity of the video content;

[0059] During the video formation process, the computing resource load rate is obtained;

[0060] The broadcast frequency of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information is adjusted according to the computing resource load rate.

[0061] An algorithm that dynamically adjusts broadcast frequency based on video content complexity and model load makes the acceleration strategy more intelligent and flexible. For example, when processing videos with complex scenes, the broadcast frequency of the cross-modal attention module is automatically increased, while in simple scenes, the frequency is reduced, further improving efficiency while ensuring quality.

[0062] In this embodiment, the text sequence is decoded based on the virtual twin method to generate multiple frames, and each frame is processed according to the broadcast frequency of the spatial attention feature information, temporal attention feature information, and cross-modal attention feature information. The output video file includes:

[0063] Obtain a multi-frame sequence of video generated by decoding the text sequence, and generate each frame corresponding to a time step;

[0064] Obtaining a broadcast frequency value of spatial attention feature information, obtaining a broadcast frequency value of temporal attention feature information, obtaining a broadcast frequency value of cross-modal attention feature information, and forming a broadcast frequency control matrix based on the values ​​of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information;

[0065] At each time step, the time step for broadcasting spatial attention feature information, temporal attention feature information, and cross-modal attention feature information is controlled according to the broadcast frequency control matrix;

[0066] The picture at each time step is processed according to the broadcast's spatial attention feature information, temporal attention feature information, and cross-modal attention feature information, and a video file is formed and output in the order of a multi-frame picture sequence of the generated video.

[0067] The FID error of the video generated by the video generation model after optimization by the present invention is calculated frame by frame compared with the video generated before optimization. The FID is around 0.005, which shows that the quality of the generated video is guaranteed while achieving acceleration, which is an important advantage and key consideration of this solution.

[0068] In this embodiment, at each time step, the time step for controlling the broadcasting of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information according to the broadcast frequency control matrix includes:

[0069] Let the current time step be X t , obtaining a multi-frame sequence according to the time sequence of the multi-frame images of the generated video;

[0070] Get the broadcast frequency control matrix [A, B, C], where A is the broadcast frequency value of the spatial attention feature information, B is the broadcast frequency value of the temporal attention feature information, and C is the broadcast frequency value of the cross-modal attention feature information;

[0071] In response to the time sequence of the multiple frames of the generated video being in reverse order, the time step of starting the broadcast is set to X according to the broadcast frequency control matrix [A, B, C], respectively, for the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information. t-nA 、X t-nB 、X t-nC , n is an integer;

[0072] In response to the time sequence of the multiple frames of the generated video being in positive order, the time step of starting the broadcast is set to X according to the broadcast frequency control matrix [A, B, C], respectively, for the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information. t+nA 、X t+nB 、X t+nC , n is an integer;

[0073] According to the time sequence of the multi-frame pictures of the generated video, the spatial attention feature information is controlled from the current time step X t Sequentially propagate to the picture corresponding to the next A-1 time step, controlling the temporal attention feature information from the current time step Xt Sequentially propagate to the screen corresponding to the next B-1 time step, controlling the cross-modal attention feature information from the current time step X t It propagates sequentially to the frame corresponding to the subsequent C-1th time step.

[0074] Specifically, according to the time sequence of the multi-frame pictures of the generated video, the spatial attention feature information is controlled from the current time step X t Sequentially propagate to the picture corresponding to the next A-1 time step, controlling the temporal attention feature information from the current time step X t Sequentially propagate to the screen corresponding to the next B-1 time step, controlling the cross-modal attention feature information from the current time step X t The images corresponding to the subsequent C-1 time step are sequentially propagated as follows:

[0075] In response to the time sequence of the multiple frames of the generated video being in reverse order, the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information are controlled to be respectively from the current time step X t Propagate sequentially to time step X t-A-1 、X t-B-1 、X t-C-1 ;

[0076] In response to the time sequence of the multiple frames of the generated video being in positive order, the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information are controlled to be respectively from the current time step X t Propagate sequentially to time step X t+A-1 、X t+B-1 、X t+C-1 .

[0077] Among them, attention broadcasting is used to reduce unnecessary attention calculations. In the middle part, attention shows slight differences. This study broadcasts the attention output of a diffusion step to several subsequent steps, thereby significantly reducing the computational cost, as shown in the attached figure. Figure 4 As shown in the figure, dynamic switching requires only a single All2All operation to switch between parallel dimensions, ensuring the smooth execution of subsequent computations. This approach reduces communication traffic by over 75% compared to previous sequential parallel operations, significantly improving communication efficiency.

[0078] When the broadcast controller manages the broadcast frequency control matrix [A, B, C], the pseudo code of the broadcast controller is as follows:

[0079] class DiTBroadcastManager(Config):

[0080] def __init__(

[0081] self,

[0082] spatial_broadcast: bool = True,

[0083] spatial_threshold: list = [15,31],

[0084] spatial_range: int = 2, #For spatial_attn, the broadcast frequency is 2

[0085] temporal_broadcast: bool = True,

[0086] temporal_threshold: list = [15,31],

[0087] temporal_range: int = 4, #For temporal_attn, the broadcast frequency is 4

[0088] cross_broadcast: bool = True,

[0089] cross_threshold: list = [15,31],

[0090] cross_range: int = 4, #For cross_attn, the broadcast frequency is 4

[0091] mlp_broadcast : bool = False

[0092] ).

[0093] like Figure 4 As shown, the broadcast frequency control matrix [A, B, C] = [2, 4, 4], the function implemented by the above pseudo code is: Assume that the current time step is X t , the time order of the multi-frame pictures of the generated video is in reverse order. When the spatial attention module adopts a broadcast mechanism with a broadcast frequency of 2, then X t-1 It is by X t For the temporal attention module and the cross-modal attention module, a broadcast mechanism with a broadcast frequency of 4 is adopted, then X t-1 、X t-2 、X t-3 By X t The smaller the change in attention, the greater the broadcast frequency.

[0094] In this embodiment, processing the image at each time step according to the broadcasted spatial attention feature information, temporal attention feature information, and cross-modal attention feature information includes:

[0095] Determine whether there is broadcast spatial attention feature information, temporal attention feature information, or cross-modal attention feature information at the current time step;

[0096] In response to the presence of broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information at the current time step, obtaining the broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information;

[0097] In response to the absence of broadcast spatial attention feature information, temporal attention feature information or cross-modal attention feature information at the current time step, the spatial attention feature information, temporal attention feature information or cross-modal attention feature information is recalculated through the spatial attention module, the temporal attention module and the cross-modal attention module, and the recalculated spatial attention feature information, temporal attention feature information or cross-modal attention feature information is used in the picture at the current time step and broadcast.

[0098] Specifically, in response to the existence of broadcast spatial attention feature information at the current time step, the value of the spatial attention feature information at the previous time step is taken; otherwise, the spatial attention feature information is recalculated and the recalculated spatial attention feature information is used in the picture at the current time step and broadcast;

[0099] In response to the existence of time attention feature information to be broadcasted at the current time step, the value of the time attention feature information of the previous time step is taken; otherwise, the time attention feature information is recalculated and the recalculated time attention feature information is used in the picture of the current time step and broadcasted;

[0100] In response to the existence of cross-modal attention feature information to be broadcast at the current time step, the value of the cross-modal attention feature information of the previous time step is taken; otherwise, the cross-modal attention feature information is recalculated and the recalculated cross-modal attention feature information is used in the picture of the current time step and broadcast.

[0101] In this embodiment, when broadcasting spatial attention feature information, temporal attention feature information, and cross-modal attention feature information, it also includes:

[0102] Obtain the spatial attention feature information, temporal attention feature information, and cross-modal attention feature information broadcast at the current time step;

[0103] The spatial attention feature information, temporal attention feature information, and cross-modal attention feature information to be broadcast are set to a grouped broadcast mode or a layered broadcast mode.

[0104] Among them, in addition to the single broadcast mechanism, attempts are made to combine different types of broadcast mechanisms, such as packet broadcast mode and layered broadcast mode, to tap greater acceleration potential.

[0105] The actual application pseudo code is as follows:

[0106] def forward(

[0107] self,

[0108] x,

[0109] y,

[0110] t,

[0111] mask=None,# text mask

[0112] x_mask=None,# temporal mask

[0113] t0=None,# t with timestamp=0

[0114] T=None,# number of frames

[0115] S=None,# number of pixel patches

[0116] timestep=None,

[0117] all_timesteps=None, ):

[0119] B, N, C = x.shape

[0120] shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = (

[0121] self.scale_shift_table[None] + t.reshape(B, 6, -1)

[0122] .chunk(6, dim=1)

[0123] if x_mask is not None:

[0124] shift_msa_zero, scale_msa_zero, gate_msa_zero, shift_mlp_zero, scale_mlp_zero, gate_mlp_zero = (

[0125] self.scale_shift_table[None] + t0.reshape(B, 6, -1)

[0126] ).chunk(6, dim=1)

[0127] if enable_broadcast():

[0128] if self.temporal:

[0129] broadcast_attn, self.attn_count = if_broadcast_temporal(int(timestep[0]), self.attn_count)

[0130] else:

[0131] broadcast_attn, self.attn_count = if_broadcast_spatial(int(timestep[0]), self.attn_count)

[0132] if enable_broadcast() and broadcast_attn:

[0133] x_m_s = self.last_attn

[0134] else:

[0135] x_m = t2i_modulate(self.norm1(x), shift_msa, scale_msa)#torch.Size([2, 50400, 1152])

[0136] if x_mask is not None:

[0137] x_m_zero = t2i_modulate(self.norm1(x), shift_msa_zero, scale_msa_zero)

[0138] x_m = self.t_mask_select(x_mask, x_m, x_m_zero, T, S)

[0139] # attention

[0140] if self.temporal:

[0141] if self.parallel_manager.sp_size>1:

[0142] x_m, S, T = self.dynamic_switch(x_m, S, T, to_spatial_shard=True)

[0143] x_m = rearrange(x_m, "B (T S) C ->(B S) T C", T=T, S=S)

[0144] x_m = self.attn(x_m)

[0145] x_m = rearrange(x_m, "(B S) T C ->B (T S) C", T=T, S=S)

[0146] if self.parallel_manager.sp_size>1:

[0147] x_m, S, T = self.dynamic_switch(x_m, S, T, to_spatial_shard=False)

[0148] else:

[0149] x_m = rearrange(x_m, "B (T S) C ->(B T) S C", T=T, S=S)

[0150] x_m = self.attn(x_m)

[0151] x_m = rearrange(x_m, "(B T) S C ->B (T S) C", T=T, S=S)

[0152] x_m_s = gate_msa * x_m

[0153] if x_mask is not None:

[0154] x_m_s_zero = gate_msa_zero * x_m

[0155] x_m_s = self.t_mask_select(x_mask, x_m_s, x_m_s_zero, T, S)

[0156] if enable_broadcast():

[0157] self.last_attn = x_m_s

[0158] # residual

[0159] x = x + self.drop_path(x_m_s)

[0160] # cross attention

[0161] if enable_broadcast():

[0162] broadcast_cross, self.cross_count = if_broadcast_cross(int(timestep[0]), self.cross_count)

[0163] if enable_broadcast() and broadcast_cross:

[0164] x = x + self.last_cross

[0165] else:

[0166] x_cross = self.cross_attn(x, y, mask)

[0167] if enable_broadcast():

[0168] self.last_cross = x_cross

[0169] x = x + x_cross。

[0170] The main function of the above code is to determine whether enable_broadcast (enable broadcast) is True at the current time step. This judgment is performed before the calculation of the spatial attention module, temporal attention module, and cross-modal attention module. If it is True, the value of the previous time step is directly taken. If it is False, an attention calculation is performed and the attention feature information to be broadcast is updated.

[0171] By modifying the frequency controller module, you can modify the broadcast frequency of temporal attention, spatial attention, and cross-modal attention in the video generation model. For different video models, you can fine-tune the broadcast frequency of the attention mechanism to achieve optimal video generation efficiency and quality.

[0172] Furthermore, the present invention uses DSP technology, a highly efficient sequential parallel computation method for implementing multidimensional Transformer models, to improve communication performance between different attention modules. DSP is a method for dynamically switching the dimensions of parallel computation based on the current computation phase, leveraging the potential of multidimensional attention. This dynamic dimension switching allows for sequential parallel computation with minimal communication overhead compared to applying traditional single-dimensional parallel computation to multidimensional models.

[0173] In this solution, the spatial attention module, the temporal attention module, and the cross-modal attention module are three independent modules that can be applied to DSP technology. Through DSP technology, these three modules can be executed in parallel. Tests have found that after using DSP technology, the performance of our video generation can be improved by about 40%.

[0174] In this embodiment, forming a text sequence according to text information of a pre-generated video includes:

[0175] Obtaining text information of a pre-generated video, and encoding the text information to form a text code;

[0176] De-noise the text encoding noise to form a text sequence.

[0177] In the above-mentioned method of generating video from text, since the spatial attention module, the temporal attention module and the cross-modal attention module are set to calculate the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information of the text sequence in parallel, the parallel calculation of different attention mechanisms can reduce the communication volume and significantly improve the efficiency of communication. When forming each frame of the picture, the spatial attention module, the temporal attention module and the cross-modal attention module are used to control the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information according to the change rate of the influence of the generated video. It is possible to realize the broadcasting of different attention feature information using different frequency broadcasting mechanisms, thereby reducing computing resources and time and improving the efficiency of generating videos.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0179] In one embodiment, Figure 5 As shown, a text-generated video device 10 is provided, comprising: a text conversion module 1, an attention feature calculation module 2, a broadcast frequency control module 3, and a picture processing module 4.

[0180] The text conversion module 1 is used to form a text sequence according to the text information of the pre-generated video.

[0181] The attention feature calculation module 2 is used to set the spatial attention module, the temporal attention module and the cross-modal attention module to calculate the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information of the text sequence in parallel.

[0182] The broadcast frequency control module 3 is used to obtain the influence change rate of the spatial attention module, the temporal attention module and the cross-modal attention module on the generated video, and control the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information according to the influence change rate.

[0183] The picture processing module 4 is used to decode the text sequence based on the virtual twin method to generate multiple frames, and process each frame according to the broadcast frequency of the spatial attention feature information, temporal attention feature information and cross-modal attention feature information, and output a video file.

[0184] In this embodiment, obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information in descending order of the influence change rate includes:

[0185] Analyze the impact of spatial attention module, temporal attention module and cross-modal attention module on the rate of change of generated videos;

[0186] The influence weights of the spatial attention module, temporal attention module, and cross-modal attention module are set according to the influence change rate;

[0187] According to the influence weights of the spatial attention module, the temporal attention module and the cross-modal attention module, the broadcast frequencies of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information at each time step are set respectively.

[0188] In this embodiment, analyzing the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video; setting the influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module according to the influence change rate includes:

[0189] Obtain the image information changes of the spatial attention module, the temporal attention module, and the cross-modal attention module in two adjacent frames when generating a video, and calculate the impact change rate on the generated video based on the image information changes of the two adjacent frames;

[0190] Counting the number of consecutive frames when the influence change rate corresponding to the spatial attention module, the temporal attention module, and the cross-modal attention module is greater than a first threshold;

[0191] The influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module are set according to the number of consecutive picture frames when the influence change rate is greater than the first threshold, and the value of the influence weight is positively correlated with the number of consecutive picture frames when the influence change rate is greater than the first threshold.

[0192] In this embodiment, the text sequence is decoded based on the virtual twin method to generate multiple frames, and each frame is processed according to the broadcast frequency of the spatial attention feature information, temporal attention feature information, and cross-modal attention feature information. The output video file includes:

[0193] Obtain a multi-frame sequence of video generated by decoding the text sequence, and generate each frame corresponding to a time step;

[0194] Obtaining a broadcast frequency value of spatial attention feature information, obtaining a broadcast frequency value of temporal attention feature information, obtaining a broadcast frequency value of cross-modal attention feature information, and forming a broadcast frequency control matrix based on the values ​​of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information;

[0195] At each time step, the time step for broadcasting spatial attention feature information, temporal attention feature information, and cross-modal attention feature information is controlled according to the broadcast frequency control matrix;

[0196] The picture at each time step is processed according to the broadcast's spatial attention feature information, temporal attention feature information, and cross-modal attention feature information, and a video file is formed and output in the order of a multi-frame picture sequence of the generated video.

[0197] In this embodiment, at each time step, the time step for controlling the broadcasting of spatial attention feature information, temporal attention feature information, and cross-modal attention feature information according to the broadcast frequency control matrix includes:

[0198] Let the current time step be X t , obtaining a multi-frame sequence according to the time sequence of the multi-frame images of the generated video;

[0199] Get the broadcast frequency control matrix [A, B, C], where A is the broadcast frequency value of the spatial attention feature information, B is the broadcast frequency value of the temporal attention feature information, and C is the broadcast frequency value of the cross-modal attention feature information;

[0200] In response to the time sequence of the multiple frames of the generated video being in reverse order, the time step of starting the broadcast is set to X according to the broadcast frequency control matrix [A, B, C], respectively, for the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information. t-nA 、X t-nB 、X t-nC , n is an integer;

[0201] In response to the time sequence of the multiple frames of the generated video being in positive order, the time step of starting the broadcast is set to X according to the broadcast frequency control matrix [A, B, C], respectively, for the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information. t+nA 、X t+nB 、X t+nC , n is an integer;

[0202] According to the time sequence of the multi-frame pictures of the generated video, the spatial attention feature information is controlled from the current time step X t Sequentially propagate to the picture corresponding to the next A-1 time step, controlling the temporal attention feature information from the current time step X tSequentially propagate to the screen corresponding to the next B-1 time step, controlling the cross-modal attention feature information from the current time step X t It propagates sequentially to the frame corresponding to the subsequent C-1th time step.

[0203] In this embodiment, processing the image at each time step according to the broadcasted spatial attention feature information, temporal attention feature information, and cross-modal attention feature information includes:

[0204] Determine whether there is broadcast spatial attention feature information, temporal attention feature information, or cross-modal attention feature information at the current time step;

[0205] In response to the presence of broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information at the current time step, obtaining the broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information;

[0206] In response to the absence of broadcast spatial attention feature information, temporal attention feature information or cross-modal attention feature information at the current time step, the spatial attention feature information, temporal attention feature information or cross-modal attention feature information is recalculated through the spatial attention module, the temporal attention module and the cross-modal attention module, and the recalculated spatial attention feature information, temporal attention feature information or cross-modal attention feature information is used in the picture at the current time step and broadcast.

[0207] In this embodiment, forming a text sequence according to text information of a pre-generated video includes:

[0208] Obtaining text information of a pre-generated video, and encoding the text information to form a text code;

[0209] De-noise the text encoding noise to form a text sequence.

[0210] In the above-mentioned text-generated video device, since the spatial attention module, the temporal attention module and the cross-modal attention module are set to calculate the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information of the text sequence in parallel, the parallel calculation of different attention mechanisms can reduce the communication volume and significantly improve the efficiency of communication. When forming each frame of the picture, the spatial attention module, the temporal attention module and the cross-modal attention module are used to control the broadcast frequency of the spatial attention feature information, the temporal attention feature information and the cross-modal attention feature information according to the change rate of the influence of the generated video. It is possible to realize the broadcasting of different attention feature information using different frequency broadcasting mechanisms, thereby reducing computing resources and time and improving the efficiency of generating videos.

[0211] For the description of the features in the embodiment corresponding to the text-generated video device, please refer to the relevant description of the embodiment corresponding to the text-generated video method, and no further details will be given here.

[0212] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the text-to-video method.

[0213] In one embodiment, the electronic device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store text-generated video data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a text-generated video method is implemented.

[0214] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned text-to-video method embodiments when run.

[0215] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0216] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned text-to-video method embodiments are implemented.

[0217] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned text-to-video method embodiments.

[0218] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0219] The above is a detailed introduction to the text-generated video method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A method for generating video from text, characterized in that: include: Forming a text sequence based on text information of a pre-generated video; Setting a spatial attention module, a temporal attention module and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information and cross-modal attention feature information for the text sequence in parallel; Obtaining the influence change rate of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the influence change rate; The text sequence is decoded based on a virtual twin method to generate multiple frames, and each frame is processed according to the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information to output a video file, including: obtaining a multi-frame sequence of a video generated by decoding the text sequence, and corresponding each generated frame to a time step; Obtain the value of the broadcast frequency of the spatial attention feature information, obtain the value of the broadcast frequency of the temporal attention feature information, obtain the value of the broadcast frequency of the cross-modal attention feature information, and form a broadcast frequency control matrix according to the values ​​of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information; at each time step, control the time step for broadcasting the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the broadcast frequency control matrix; process the picture of each time step according to the broadcast spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information, and form and output a video file in the order of a multi-frame picture sequence of the generated video.

2. The method for generating video from text according to claim 1, wherein: The obtaining of the influence change rates of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and controlling the broadcast frequencies of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the influence change rates includes: Analyzing the rate of change of influence of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video; Setting influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module according to the influence change rate; The broadcast frequencies of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information at each time step are set respectively according to the influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module.

3. The method for generating video from text according to claim 2, wherein: The analyzing the rate of change of the impact of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video; Setting the influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module according to the influence change rate includes: Obtaining a change in picture information between two adjacent frames when the spatial attention module, the temporal attention module, and the cross-modal attention module generate a video, and calculating a rate of change of influence on the generated video based on the change in picture information between the two adjacent frames; Counting the number of consecutive picture frames when the influence change rate corresponding to the spatial attention module, the temporal attention module, and the cross-modal attention module is greater than a first threshold respectively; The influence weights of the spatial attention module, the temporal attention module, and the cross-modal attention module are set according to the number of continuous picture frames when the influence change rate is greater than the first threshold, and the value of the influence weight is positively correlated with the number of continuous picture frames when the influence change rate is greater than the first threshold.

4. The method for generating video from text according to claim 1, wherein: The time step of controlling the broadcasting of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the broadcast frequency control matrix at each time step includes: Let the current time step be X t , obtaining a multi-frame sequence according to the time sequence of the multi-frame images of the generated video; Obtain the broadcast frequency control matrix [A, B, C], where A is the value of the broadcast frequency of the spatial attention feature information, B is the value of the broadcast frequency of the temporal attention feature information, and C is the value of the broadcast frequency of the cross-modal attention feature information; In response to the time sequence of the multiple frames of the generated video being in reverse order, the time step for starting the broadcast of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information is set to X according to the broadcast frequency control matrix [A, B, C]. t-nA 、X t-nB 、X t-nC , n is an integer; In response to the time sequence of the multiple frames of the generated video being in positive order, the time step for starting the broadcast of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information is set to X according to the broadcast frequency control matrix [A, B, C]. t+nA 、X t+nB 、X t+nC , n is an integer; According to the time sequence of the multi-frame pictures of the generated video, the spatial attention feature information is controlled from the current time step X t Sequentially propagate to the picture corresponding to the subsequent A-1 time step, controlling the time attention feature information from the current time step X t Sequentially propagate to the screen corresponding to the subsequent B-1 time step, controlling the cross-modal attention feature information from the current time step X t It propagates sequentially to the frame corresponding to the subsequent C-1th time step.

5. The method for generating video from text according to claim 4, wherein: The processing of the picture at each time step according to the broadcasted spatial attention feature information, temporal attention feature information, and cross-modal attention feature information includes: Determining whether there is broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information at the current time step; In response to the presence of the broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information at the current time step, obtaining the broadcasted spatial attention feature information, temporal attention feature information, or cross-modal attention feature information; In response to the absence of the broadcasted spatial attention feature information, temporal attention feature information or cross-modal attention feature information at the current time step, the spatial attention feature information, temporal attention feature information or cross-modal attention feature information is recalculated by the spatial attention module, the temporal attention module and the cross-modal attention module, and the recalculated spatial attention feature information, temporal attention feature information or cross-modal attention feature information is used in the picture at the current time step and broadcast.

6. The method for generating video from text according to claim 1, wherein: The forming of a text sequence according to the text information of the pre-generated video includes: Obtaining text information of a pre-generated video, and encoding the text information to form a text code; Noise in the text encoding is denoised to form a text sequence.

7. A text-generated video device, characterized in that: include: A text conversion module, used to form a text sequence based on the text information of the pre-generated video; An attention feature calculation module is used to set a spatial attention module, a temporal attention module, and a cross-modal attention module to respectively calculate spatial attention feature information, temporal attention feature information, and cross-modal attention feature information for the text sequence in parallel; a broadcast frequency control module, configured to obtain a rate of change of the influence of the spatial attention module, the temporal attention module, and the cross-modal attention module on the generated video, and control the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the rate of change of the influence; A picture processing module is used to decode the text sequence based on the virtual twin method to generate multiple frames, and process each frame according to the broadcast frequency of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information to output a video file, including: obtaining a multi-frame picture sequence generated by decoding the text sequence, and corresponding each generated frame to a time step; Obtain the value of the broadcast frequency of the spatial attention feature information, obtain the value of the broadcast frequency of the temporal attention feature information, obtain the value of the broadcast frequency of the cross-modal attention feature information, and form a broadcast frequency control matrix according to the values ​​of the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information; at each time step, control the time step for broadcasting the spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information according to the broadcast frequency control matrix; process the picture of each time step according to the broadcast spatial attention feature information, the temporal attention feature information, and the cross-modal attention feature information, and form and output a video file in the order of a multi-frame picture sequence of the generated video.

8. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for generating video from text as claimed in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for generating video from text are implemented as described in any one of claims 1 to 6.

Images