A video generation method based on text generation based on feature decoupling enhancement
By introducing visual feature decoupling networks and cross-frame and time attention modules into text-generated video technology, the problem of insufficient fidelity and continuity of video generation in the prior art is solved, and higher quality and smooth video generation are achieved.
Patent Information
- Application Number
- CN202411132278.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing text-generating video technology has shortcomings in the fidelity and continuity of generated videos, and the calculation cost is high and the generated videos are not smooth enough.
Adopt text-generating video methods based on feature decoupling enhancement, including visual feature decoupling network, cross-frame attention module and temporal attention module. By decoupling and layering video features and strengthening inter-frame correlation, improve video details quality and smoothness. sex.
Higher quality video details generation and video coherence are achieved, significantly improving the fidelity and fluency of generated videos and reducing computing costs.
Smart Images

Figure CN118658106B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a text-to-video generation method based on feature decoupling enhancement, and belongs to the technical field of text-to-video generation. Background Art
[0002] Text-generated video is an innovative direction in computer vision and artificial intelligence, and is mainly used in content creation, film and television production, advertising industry, education and training and other fields. By automatically converting text descriptions into corresponding video content, this technology significantly reduces the time and cost of content creation, while improving production efficiency and accessibility. Text-generated video technology relies on advanced deep learning models and algorithms, such as generative adversarial networks (GANs) and natural language processing (NLP), to understand text content and generate visual representations. Therefore, text-generated video has become a research hotspot in the field of computer vision and artificial intelligence, and is an important part of modern digital media and intelligent content generation systems.
[0003] With the advancement of technology and changes in demand, users not only seek static visual presentations, but are also increasingly inclined to create dynamic, continuous video content from text descriptions to achieve richer and more vivid information delivery. However, there are also some challenges in text-to-video generation. For example, the computational cost is too high and the fidelity and continuity of the generated video are insufficient.
[0004] In recent years, the rapid development of generative artificial intelligence technology in the field of computer vision has attracted great attention from academia and industry. In particular, with the rise of diffusion models, the process of generating realistic images based on text descriptions has become more efficient and accurate. This innovation not only inspires people's artistic creation, but also enables AI enthusiasts and professional designers to create visual content with unprecedented speed and creativity, greatly improving production efficiency. With the advancement of technology and the continuous expansion of creative needs, people's focus has gradually shifted from single image generation to a more complex and dynamic form of visual expression - video. This shift not only marks the broadening of the scope of technology application, but also reflects the market's desire for richer and more vivid visual content.
[0005] Early text-to-video tasks were usually supervised training on large-scale text-video pair datasets. In addition to the need to adjust complex hyperparameters, a large number of labeled text videos are usually required to improve the generation quality. Recently, researchers have worked hard to transfer the successful experience of text-to-image (T2I) to the more complex text-to-video (T2V) generation task. There are mainly two methods: One-shot and Zero-shot. The One-shot method performs well in learning the continuity of a specified video, but its generalization ability is not as strong as Zero-shot. Both methods still have certain defects in the continuity and fidelity of the generated video. In view of this, it is believed that the key to optimizing video generation lies in two aspects: the continuity of the generated content and the fidelity of the generated quality. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a text-to-video generation method based on feature decoupling enhancement, which can improve the quality of generated video details and ensure the fluency of generated video;
[0007] In order to achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions:
[0008] A method for generating video from text based on feature decoupling enhancement, comprising:
[0009] Inputting user text into a pre-trained text-generated video network model to obtain a video that matches the user text description;
[0010] Among them, the text generation video network model includes a visual feature decoupling network, a cross-frame attention module and a temporal attention module. The visual feature decoupling network is used to decouple and layer video features, the cross-frame attention module is used to strengthen the association between three consecutive frames of decoupled and layered video features, and the temporal attention module is used to strengthen the association between the global frames of hidden layer features output by cross-frame attention.
[0011] Optionally, the training of the text-to-video network model includes:
[0012] Get the training video and the corresponding training text;
[0013] After VAE encoding the training video, Gaussian noise is added to obtain noise features;
[0014] The noise features and training text are input into the initial text generation video network model for training to obtain a trained text generation video network model.
[0015] Optionally, the step of inputting the noise features and the training text into the initial text generation video network model for training to obtain a trained text generation video network model includes:
[0016] Input the noise features into the visual feature decoupling network to obtain a feature decoupling set;
[0017] The feature decoupling set and the training text are input into the downsampling of the Unet network to calculate the product similarity and form a multi-scale fusion;
[0018] Input the multi-scale fusion into the cross-frame attention module and output the cross-frame attention weight;
[0019] Input the cross-frame attention weights into the temporal attention module and output the time dimension weight vector;
[0020] Multiply the time dimension weight vector by the time dimension weight vector to get the output feature ;
[0021] Until the output characteristics It can generate videos corresponding to the training text and obtain a trained text-generated video network model.
[0022] Optionally, the step of inputting the feature decoupling set and the training text into the downsampling of the Unet network to perform product similarity calculation to form a multi-scale fusion includes:
[0023] The first element is extracted from the decoupled feature set, and a product attention calculation is performed with the potential feature. The calculation result is input into the first downsampling layer of the Unet network, and the output of the first downsampling layer is used as the input of the next product attention calculation. At the same time, the second element in the set is extracted, and this process is continued. The result is fed to the next downsampling layer until all elements in the set are taken out and applied, thus forming a multi-scale fusion.
[0024] Optionally, the process formula for forming multi-scale fusion is expressed as: ;
[0025] in, Indicates Downsampling is about to add hidden layer features, Represents the output of the previous layer, as the input of the current layer, and takes features and Perform matrix multiplication and add it to itself to get , Represents a collection of features.
[0026] Optionally, the multi-scale fusion is input into the cross-frame attention module to obtain a matrix representation of the cross-frame attention weights:
[0027] ;
[0028] ;
[0029] in, represents the current frame as the query vector, represents the key vector, represents a value vector to capture dynamic changes across frames, Represents the entire The dimensions of the matrix are Indicates a query, Indicates the key, represents the value matrix, Indicates the current frame features, Represents the features of the previous frame based on the current frame, Represents the features of the first two frames based on the current frame, represents the matrix transpose, A matrix representing the attention weights across frames, is an activation function used to increase nonlinear representation.
[0030] Optionally, the step of inputting the cross-frame attention weights into the temporal attention module to obtain a temporal dimension weight vector includes:
[0031] Concatenate the channel dimensions of each frame and the width and height of the feature map to form a Large feature map;
[0032] Set the frame dimensions As the channel dimension C, The feature map of each frame of the large feature map is squeezed and excited;
[0033] Through average pooling of the input The feature map of each frame of the large feature map is transformed from a dimension of The input features are compressed into A feature vector of size , where represents the width of the feature map, represents the height of the eigenvector;
[0034] The time dimension weight vector is obtained by performing function calculation on the compressed feature vector through the excitation part.
[0035] Optionally, the calculation process formula of the time dimension weight vector includes:
[0036] ;
[0037] in, represents the feature vector, Represents the output of the previous layer, subscript represents the frame dimension, Represents the feature map Locations, Indicates that the input Perform a squeeze operation. In the excitation part, add a full connection and activation function to excite the feature vector to obtain a distribution weight between 0 and 1. The process is as follows:
[0038] ;
[0039] in, represents a fully connected matrix, Indicates a squeeze operation, which converts a size of The feature map is squeezed into 1x1 size, and we get , According to Calculate the weight distribution, is the Sigmoid activation function, represents the temporal attention weight vector, represents the first input after encoding and adding noise and passing through the decoupling network.
[0040] Optionally, the output feature The expression is:
[0041] ;
[0042] in, express No. Frame features, Indicates the amplification operation, which is the output of the previous layer Dot Product .
[0043] Optionally, the visual feature decoupling network includes convolutions of four different channels, which are 320, 320, 640, and 1280 respectively.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] The present invention uses a feature decoupling network to decouple the noise map of the video into a set, and fuses it into the Unet network for predicting noise, thereby achieving higher quality video detail generation. The present invention can make full use of the decoupled features and achieve in-depth attention to video details through multi-scale feature representation, effectively improving the quality of generated video details;
[0046] The present invention designs a combination of cross-frame attention and temporal attention in the model to enhance the attention to different positions of the current frame and its previous two frames, as well as the overall correlation of the global frames, thereby ensuring the coherence and fluency of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is a model framework diagram of an improved embodiment of the present invention;
[0048] Figure 2 It is a structural diagram of an embodiment of the visual feature decoupling network of the present invention;
[0049] Figure 3 is a structural diagram of an embodiment of the cross-frame attention module of the present invention;
[0050] Figure 4 It is a structural diagram of an embodiment of the time attention map of the present invention;
[0051] Figure 5 This is a qualitative result comparison experiment of the present invention. Figure 1 ;
[0052] Figure 6 This is a qualitative result comparison experiment of the present invention. Figure 2 ;
[0053] Figure 7 This is a qualitative result comparison experiment of the present invention. Figure 3 ;
[0054] Figure 8 This is a comparison diagram of the cross-frame attention ablation experiment of the present invention;
[0055] Fig. 9 It is a comparison diagram of the characteristic decoupling ablation experiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.
[0057] In view of the problems of the prior art, the present invention proposes a method for generating videos from text based on feature decoupling enhancement based on a pre-trained text-to-image model. The video noise map is decoupled and layered, and added to the Unet network for predicting noise, making the generated video details more realistic. In addition, in order to ensure the continuity of the video, a module combining cross-frame attention and temporal attention is proposed, which can effectively enhance the association between frames.
[0058] Experimental results show that the method proposed in this invention can generate smoother and more realistic videos than other existing methods, as follows:
[0059] like Figure 1 As shown, a method for generating a video from text based on feature decoupling enhancement is disclosed, comprising:
[0060] Inputting user text into a pre-trained text-generated video network model to obtain a video that matches the user text description;
[0061] Among them, the text generation video network model includes a visual feature decoupling network, a cross-frame attention module and a temporal attention module. The visual feature decoupling network is used to decouple and layer video features, the cross-frame attention module is used to strengthen the association between three consecutive frames of decoupled and layered video features, and the temporal attention module is used to strengthen the association between the global frames of hidden layer features output by cross-frame attention.
[0062] In this embodiment, the training of the text-to-video network model includes:
[0063] Get the training video and the corresponding training text;
[0064] After VAE encoding the training video, Gaussian noise is added to obtain noise features;
[0065] The noise features and training text are input into the initial text generation video network model for training to obtain a trained text generation video network model.
[0066] like Figure 1 As shown in the figure, the training of the text generation video network model is further introduced. First, the video is encoded by VAE and Gaussian noise is added to obtain noise features. These features are fed into the visual feature decoupling network to obtain a feature decoupling set.
[0067] Each element in the set is added to the downsampling of the Unet network, and the product similarity is calculated to form a multi-scale fusion.
[0068] In this embodiment, in order to overcome the problem of insufficient detail representation when generating a single frame image and maintain the coherence between frames, the structure of the video feature decoupling network (VFDN) is as follows: Figure 2 As shown: The architecture of this network is designed to process the noise mapping encoded by the variational auto-encoder VAE (variational auto-encoder) and optimize the richness and hierarchy of feature representation.
[0069] Specifically, the dimension of the encoded noise feature tensor is BFCHW, where C represents the number of channels. It is equally divided into p1, p2, …, pn according to the channel dimension, where n is set to 4. Each feature vector undergoes convolution, activation, and pooling operations of different scales to obtain different levels of feature representation.
[0070] Specifically, the number of channels of p1 becomes 320 after a single layer of convolution, and p2 also obtains 320-channel features after two layers of convolution. For p3 and p4, they are expanded to 640 and 1280 channels respectively through double-layer convolution. This series of operations constructs a decoupled feature set containing information at different levels, and each element in the set stores the hierarchical details of the noise map.
[0071] In the four downsampling stages of the Unet network, elements are extracted from the disentangled feature set one by one and applied.
[0072] First, the first element is extracted from the set and multiplied with the latent features to expand the representation of the features, and the result is fed into the first downsampling layer of Unet. Subsequently, the output of the first downsampling layer is used as the input for the next multiplication attention calculation, and the second element in the set is extracted, and the process continues and the result is fed into the next downsampling layer. This cycle continues until all elements in the set are taken out and applied. The process can be expressed as follows:
[0073] ;
[0074] in, Indicates Downsampling is about to add hidden layer features, Represents the output of the previous layer, as the input of the current layer, and takes features and Perform matrix multiplication and add it to itself to get , Represents a collection of features.
[0075] The advantage of this method is that it can make full use of decoupled features and achieve deep attention to video details through multi-scale feature representation. Figure 3 As shown, the multi-scale fusion is input into the cross-frame attention module, and the calculation method of outputting the cross-frame attention weight includes:
[0076] ;
[0077] ;
[0078] in, represents the current frame as the query vector, represents the key vector, represents a value vector to capture dynamic changes across frames, Represents the entire The dimensions of the matrix are Indicates a query, Indicates the key, represents the value matrix, Indicates the current frame features, Represents the features of the previous frame based on the current frame, Represents the features of the first two frames based on the current frame, represents the matrix transpose, A matrix representing the attention weights across frames, is an activation function used to increase nonlinear representation.
[0079] By introducing the cross-frame attention module in the model, the model can understand the dynamic content in the video sequence more carefully, not only limited to the continuous changes in the same position, but also including the interactions between different spatial positions.
[0080] like Figure 4 As shown in Figure 2, the cross-frame attention weights are input into the temporal attention module, and the temporal dimension weight vector is output; the temporal dimension weight vector is multiplied by the temporal dimension weight vector to obtain the output feature , including:
[0081] The channel dimension of each frame and the width and height of the feature map are spliced together to form a large H×W feature map. The frame dimension F is regarded as the channel dimension C. For the feature map of each frame, squeeze and excitation are performed. The purpose of the squeeze operation is to compress the global space of time so as to perform feature learning in the time dimension, and then the compressed feature vector is calculated by the excitation part to obtain the time weight vector.
[0082] Specifically, in the squeezing part, the input feature map is transformed from a dimension of The input feature Feature is compressed into Vector of size Z , H.W. and F Represents height, width and number of frames respectively. This operation can be achieved by average pooling, and the calculation formula is as follows:
[0083] ;
[0084] In the incentive part, in order to reflect the importance of each time dimension and allocate the time dimension weight coefficient, a full connection and activation function are added to the vector Perform incentive operations to obtain a distribution weight between 0 and 1. The calculation process is as follows:
[0085] ;
[0086] in, represents a fully connected matrix, represents the feature vector, Indicates a squeeze operation, which converts a size of The feature map is squeezed into 1x1 size, and we get , According to Calculate the weight distribution, is the Sigmoid activation function, represents the temporal attention weight vector, Z represents the first input after encoding and adding noise and passing through the decoupling network. Among them, is the fully connected layer, Represents the time attention weight vector. Finally, the time dimension weight vector With input features X Multiply and get the output : ;
[0087] in, express No. Frame features, Indicates the amplification operation, which is the output of the previous layer Dot Product .
[0088] like Figure 5-Figure 9 As shown in the figure, in order to fully verify the effectiveness of the decoupled network in enhancing video details and the improvement of cross-frame attention continuity, three groups of experiments were conducted using the same baseline method, with four in each group. The experiments used the same prompt words, which are marked above the pictures.
[0089] The first set of experiments: Overall, Tune-A-video (baseline method) has serious flickering, frame skipping, forward and backward movement of the car, and overall incoherence. There is a deviation in the motion trajectory of the first three videos and the last video. The coast of the first video is not obvious in the Tune-A-Video method, and the generated ground is very smooth, which does not conform to the actual beach. In the method of the present invention, the coast is sparkling and the road surface is also folded layers of sand.
[0090] In the second video, the trees in Tune-A-Video appear too suddenly and without continuity, and the ground is smooth without the texture of snow. However, the scenery in the method of the present invention is always continuous, with snow gullies on the ground and snow-capped mountains in the distance.
[0091] The carton cartoon style of the third video of the present invention is not as good as the Tune-A-Video baseline method.
[0092] The fourth video, generated by Tune-A-Video, is not natural enough. The colors are too bright. The four groups of videos of the present invention all have good smoothness. In terms of scenes, except for the third carton cartoon style, the method of the present invention is closer to natural scenes.
[0093] The second set of experiments: In the first video, the Mickey Mouse generated by Tune-A-Video showed a third color, yellow, while the Mickey Mouse of the present invention only has black and white, which is consistent with the public's perception of Mickey Mouse. In the second video, the Spider-Man generated by Tune-A-Video has uncoordinated limbs and no shadow on the ground. The lines on Spider-Man's chest do not have the details of the method of the present invention. In the third video, the female face generated by Tune-A-Video is always in profile, while the method of the present invention shows the entire face. In the fourth video, the green mask generated by the present invention is more shiny than that generated by Tune-A-Video. And in the final stage, the skateboard at the bottom of the video generated by Tune-A-Video changed color, while the method of the present invention can well maintain the shape of the skateboard. Overall, Tune-A-Video still has frame skipping.
[0094] The third group of experiments: The panda and raccoon generated by Tune-A-Video in the first and third videos still show human legs, which creates a sense of disharmony with the body. Although the feet of the method of the present invention have defects, the overall appearance makes people know that they are pandas and raccoons at a glance. The shoulders generated by Tune-A-Video in the second video have multiple double images. The arms of Iron Man generated by Tune-A-Video in the fourth video are all biased to the left, without the sense of balance of surfing. In the video generated by the method of the present invention, the left and right hands are on both sides, which is more realistic.
[0095] Quantitative indicators In order to quantify the comparison results, the present invention uses FVD and CLIP scores as indicators to measure the degree of alignment between video and text. In order to ensure the fairness of the experiment, FVD uniformly uses (Inflated 3D ConvNet) as the feature extraction network of the video. CLIP score selects the VITB-32 model for feature extraction. In the specific experiment, the present invention conducted comparative experiments on five methods: Cogvideo, MagicVideo, Text2LIVE, Make-A-Video and Tune-A-Video. 25 video samples generated by each method were selected as comparison data. And the comparison object is converted into the corresponding video frame.
[0096] The experimental results are shown in Table 1. The CLIP score obtained by the method of the present invention is 0.3052, which is the best. Since Text2LIVE uses the strategy of replacing the background instead of directly generating the video, Make-A-Video uses the super-resolution framework to improve the quality of the generated video, so the FVD score is relatively high. In addition, the present invention also analyzes the pixels of the video generated by each method and whether pre-training is performed.
[0097] In the research and application of generative models, it is crucial to effectively evaluate the quality of generated results. This is not only related to the direction of model optimization, but also an important basis for comparing the performance of different models. For the two tasks of text-to-image and text-to-video, the selection of evaluation indicators is particularly critical because they directly affect our understanding and evaluation of the model's generation capabilities.
[0098] The task of generating images from text aims to generate corresponding images based on a given text description. In this task, Fréchet Inception Distance (FID) and Inception Score (IS) are two widely used evaluation indicators. They quantify the quality and diversity of generated images from different perspectives. The feature extraction network uniformly uses the Inception-v3 network pre-trained on Image-net.
[0099] The calculation process of FID can be expressed as:
[0100] ;
[0101] in, , Represent the feature mean of the real sample and the feature mean of the generated sample, respectively. Represents the covariance matrix of two types, , A lower FID indicates that the generated image is closer to the real image. The calculation process can be expressed as:
[0102] ;
[0103] in, Represents the generated image, is the predicted label, represents the KL divergence of two distributions, namely the true distribution and the test distribution, A larger value indicates a higher diversity of generated images. Similarly, the text-to-video task requires the model to generate dynamic video content based on text descriptions. This task is more complex because in addition to image quality and diversity, it also needs to consider coherence and logic in the temporal dimension.
[0104] Frechet Video Distance (FVD) and Clip score are the two main metrics for evaluating the quality of text generation videos, which evaluate them from the visual and textual consistency aspects of the video respectively.
[0105] The calculation formula of FVD is very similar to FID (Fréchet Inception Distance) in structure. They are both based on the distribution difference between generated data and real data in a certain feature space. The difference lies in the feature extraction network. This application uses I3D (Inflated 3D ConvNet) as the feature extraction network for videos.
[0106] The CLIP score is usually determined by calculating the similarity between the generated video frames and the corresponding text descriptions. Specifically, the video frames and texts are encoded separately using the CLIP model, and then the cosine similarity between the two encoded vectors is calculated.
[0107] The calculation formula of cosine similarity is as follows:
[0108] ;
[0109] in, The encoding vector representing the number of video frames, represents the encoded vector of the text description, represents the dot product of vectors, represents the dot product of vectors, and represents the norm of vectors (that is, their length).
[0110] Experimental results show that the method of the present invention is superior to the baseline method Tune-A-Video in generating video quality.
[0111] Table 1: Comparison of quantitative results of different methods
[0112]
[0113] Ablation experiment:
[0114] In order to fully demonstrate the effectiveness of the method of the present invention, the present invention conducts an ablation experiment of cross-frame attention and visual feature decoupling. Specifically, the present invention compares 10 consecutive frames of the generated video, respectively without adding cross-frame attention, stacking the first frame and the previous frame, and stacking the first two frames as the KV in the attention. The experimental results are as follows: Figure 8 , the text prompt used is "ajeep car is moving on the beach". It can be seen that two cars appear in the second frame without using cross-frame attention, and the coherence between the first frame and the third frame is poor. In the experiment using the stacked previous frame and the first frame, multiple jumps occur between frames. For example, the second frame, the third frame and the last two frames, while using the stacked first two frames of attention, the frames are basically coherent. This shows the effectiveness of the present invention using cross-two-frame attention.
[0115] The present invention also conducted an ablation experiment on visual feature decoupling, and the experimental results are as follows: Fig. 9 As shown. 10 consecutive frames are also selected. Compared with the tiger in the first video, the tiger Tune-A-Video has too dark fur color and the stripes on its body are not as delicate as the tiger generated by the method of the present invention. The appearance of the rabbit Tune-A-Video rabbit generated in the second and third videos, especially the face, has an irregular alternation of black and white, while the rabbit generated by the present invention has a fixed appearance. The puppy in the fourth video has more vivid eyes.
[0116] In summary, the present invention uses a feature decoupling network to decouple the noise map of the video into a set, and fuses it into the Unet network for predicting noise, thereby achieving higher quality video detail generation. The present invention can make full use of the decoupled features and achieve deep attention to video details through multi-scale feature representation, effectively improving the quality of generated video details;
[0117] The present invention designs a combination of cross-frame attention and temporal attention in the model to enhance the attention to different positions of the current frame and its previous two frames, as well as the overall correlation of the global frames, thereby ensuring the coherence and fluency of the generated video.
[0118] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for generating video from text based on feature decoupling enhancement, characterized in that: include: Inputting user text into a pre-trained text-generated video network model to obtain a video that matches the user text description; The text generation video network model includes a visual feature decoupling network, a cross-frame attention module and a temporal attention module. The visual feature decoupling network is used to decouple and layer video features. The cross-frame attention module is used to increase the association of three consecutive frames of decoupled and layered video features. The temporal attention module is used to increase the association of the global frame of the hidden layer features output by the cross-frame attention. The training of the text-to-video network model includes: Get the training video and the corresponding training text; After VAE encoding the training video, Gaussian noise is added to obtain noise features; Inputting the noise features and the training text into the initial text generation video network model for training, thereby obtaining a trained text generation video network model; The step of inputting the noise features and the training text into the initial text generation video network model for training to obtain a trained text generation video network model comprises: Input the noise features into the visual feature decoupling network to obtain a feature decoupling set; The feature decoupling set and the training text are input into the downsampling of the Unet network to calculate the product similarity and form a multi-scale fusion; Input the multi-scale fusion into the cross-frame attention module and output the cross-frame attention weight; Input the cross-frame attention weights into the temporal attention module and output the time dimension weight vector; Multiply the time dimension weight vector by the time dimension weight vector to get the output feature ; Until the output characteristics It can generate videos corresponding to the training texts and obtain a trained text-to-video network model; The multi-scale fusion is input into the cross-frame attention module, and the matrix representation of the cross-frame attention weight is obtained as follows: ; ; in, represents the current frame as the query vector, represents the key vector, represents a value vector to capture dynamic changes across frames, Represents the entire The dimensions of the matrix are Indicates a query, Indicates the key, represents the value matrix, Indicates the current frame features, Represents the features of the previous frame based on the current frame, Represents the features of the first two frames based on the current frame, represents the matrix transpose, A matrix representing the attention weights across frames, is an activation function used to increase nonlinear representation; The cross-frame attention weights are input into the temporal attention module to obtain a time dimension weight vector, including: Concatenate the channel dimensions of each frame and the width and height of the feature map to form a Large feature map; Set the frame dimensions As the channel dimension C, The feature map of each frame of the large feature map is squeezed and excited; Through average pooling of the input The feature map of each frame of the large feature map is transformed from a dimension of The input features are compressed into A feature vector of size , where represents the width of the feature map, represents the height of the eigenvector; The time dimension weight vector is obtained by performing function calculation on the compressed feature vector through the excitation part; The calculation process formula of the time dimension weight vector includes: ; in, represents the feature vector, Represents the output of the previous layer, subscript represents the frame dimension, Represents the feature map Locations, Indicates that the input Perform a squeeze operation. In the excitation part, add a full connection and activation function to excite the feature vector to obtain a distribution weight between 0 and 1. The process is as follows: ; in, represents a fully connected matrix, Indicates a squeeze operation, which converts a size of The feature map is squeezed into 1x1 size, and we get , Indicates based on Calculate the weight distribution, is the Sigmoid activation function, represents the temporal attention weight vector, represents the first input after encoding and adding noise and passing through the decoupling network.
2. The method for generating video from text based on feature decoupling enhancement according to claim 1, characterized in that: The feature decoupling set and the training text are input into the downsampling of the Unet network to perform product similarity calculation to form a multi-scale fusion, including: The first element is extracted from the decoupled feature set, and a product attention calculation is performed with the potential feature. The calculation result is input into the first downsampling layer of the Unet network, and the output of the first downsampling layer is used as the input of the next product attention calculation. At the same time, the second element in the set is extracted, and this process is continued. The result is fed to the next downsampling layer until all elements in the set are taken out and applied, thus forming a multi-scale fusion.
3. The method for generating video from text based on feature decoupling enhancement according to claim 2 is characterized in that: The process formula for forming multi-scale fusion is expressed as: ; in, Indicates Downsampling is about to add hidden layer features, Represents the output of the previous layer, as the input of the current layer, and takes features and Perform matrix multiplication and add it to itself to get , Represents a collection of features.
4. The method for generating video from text based on feature decoupling enhancement according to claim 1, characterized in that:The output characteristics The expression is: ; in, express No. Frame features, Indicates the amplification operation, which is the output of the previous layer Dot Product .
5. The method for generating video from text based on feature decoupling enhancement according to claim 1, characterized in that: The visual feature decoupling network includes convolutions of four different channels, which are 320, 320, 640, and 1280 respectively.
