Method for generating video from text with controllable position based on diffusion model

Abstract

The application discloses a position-controllable text-to-video method based on a diffusion model, and the method specifically comprises the following steps: obtaining an arbitrary training data set; designing an overall framework of a text-to-video model, using a space-time position perception layer to enable the text-to-video model to have positioning perception ability and time sequence perception ability; defining an optimization target, training the text-to-video model on the given training data set, and continuously iterating; after the text-to-video model is well trained, performing reasoning to generate a corresponding video. The position information and the time sequence information are inserted into the text-to-video model in a plug-and-play manner, and the position-controllable text-to-video task is efficiently realized. The application can generate a position-controllable time sequence continuous video, and can have good effects on numerous open source text-to-image models.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a position-controllable text-based video method based on a diffusion model. Background Technology

[0002] Video generation is a challenging task in computer vision and artificial intelligence, aiming to automatically create visually appealing and content-rich video sequences. This task is significant in many applications, such as automated content creation, video effects, and virtual reality. With technological advancements, video generation methods have evolved from simple rule-based techniques to complex deep learning-driven approaches.

[0003] Before deep learning became mainstream, video generation primarily relied on rule-based systems and simple templates. These methods typically involved predefined scenes, characters, and actions, and synthesized videos according to a set of rules. While these methods could produce relatively consistent results under specific conditions, they lacked flexibility and generalization ability, making them ill-suited for complex or unknown video generation tasks.

[0004] With the rapid development of deep learning technology, neural network-based video generation methods have emerged, including Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs), and more recently, the Transformer architecture. These methods can learn the underlying distribution of massive amounts of video data and then generate new video sequences, achieving significant improvements in both content and quality.

[0005] The following is a brief summary of the problems with existing video generation methods.

[0006] Generative Adversarial Networks (GANs): By training a generator network to generate videos and a discriminator network to distinguish between generated and real videos, GANs can produce high-quality video content. However, GANs are very difficult to train and are prone to pattern collapse, resulting in a lack of diversity in the generated videos.

[0007] Recurrent Neural Networks (RNNs): Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) are particularly used in video generation, leveraging their temporal processing capabilities to generate coherent video frames. However, RNNs encounter vanishing or exploding gradient problems when processing long sequences, and their image quality is generally inferior to that of GRUs.

[0008] Transformer: In recent years, Transformer-based models have shown promise in video generation tasks due to their efficient parallel processing capabilities and powerful long-range dependency capture ability. However, these models often have a large number of parameters, requiring significant computational resources and data for training.

[0009] In summary, despite significant progress in video generation using deep learning methods, several challenges remain: 1) Realism and coherence: Generated videos often struggle to maintain both high realism and temporal coherence, especially in complex scenes and over long periods. 2) Model complexity and resource requirements: Current high-performance video generation models typically have a large number of parameters, demanding substantial computational resources and data, thus limiting their scalability in practical applications. 3) Diversity and creativity: Generated videos may be overly conservative in style and content, lacking diversity and novelty, which is closely related to the model's training data. Summary of the Invention

[0010] The purpose of this invention is to solve the problems existing in the prior art and to provide a location-controllable text-based video method based on a diffusion model.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A location-controllable text-based video method based on a diffusion model includes the following steps:

[0013] S1. Obtain a training dataset for training the text-based video model. The training dataset consists of target video segments with the same resolution, and each target video segment corresponds to a text description and a location information.

[0014] S2. Construct a text-based video model, which consists of a feature encoder, a position-aware UNet network, and a feature decoder. The encoder and decoder of the UNet network have a symmetrical structure. The encoder of the UNet network includes several stages of downsampling, and the decoder of the UNet network includes several stages of upsampling. The downsampled feature maps at each level are fused with the upsampled feature maps at each level through residual connections.

[0015] The downsampling process has the same number of stages as the upsampling process. Each stage of the upsampling or downsampling process is implemented by several spatiotemporal position awareness layers that are cascaded in sequence. The spatiotemporal position awareness layer includes a self-attention layer, a localization attention layer with token selection, a cross attention layer, a temporal attention layer, and a dynamic gating network with learnable layer embedding.

[0016] S3. Train the text-based video model on the training dataset. When training the text-based video model, only update the parameters of the localization attention layer, the temporal attention layer, and the dynamic gating network in the spatiotemporal location awareness layer.

[0017] In each iteration of the Wensheng video model training process, the target video segment, text description, and location information are input into the Wensheng video model. The noisy target video segment is mapped into a latent representation by a pre-trained feature encoder. The latent representation is input into the UNet network for downsampling and upsampling to obtain the sampled features, which are then input into a pre-trained feature decoder to output the predicted video segment, thus completing pixel-level reconstruction.

[0018] S4. After the text-based video model is trained, noise is sampled from the Gaussian-distributed noise. The location information of the video to be generated, the text description, and the sampled noise are input into the trained text-based video model, and the output video conforms to the text description and location information.

[0019] Based on the above scheme, each step can be implemented in the following preferred manner.

[0020] Preferably, in each spatiotemporal location perception layer of step S2, the specific processing flow is as follows: Text features are obtained by text encoding the text description; spatial location features are obtained by Fourier mapping the location information; the text features and spatial location features are added together; the added features are input into the first perceptron to obtain the first feature; the first feature is passed through the first temporal attention layer to obtain the first temporal feature; the first temporal feature is input into the dynamic gating network to output the gating result; simultaneously, the feature output from the previous spatiotemporal location perception layer is used as the input to the self-attention layer, which outputs the second feature; if the gating result is a gate... If the gate is open, the first temporal feature is concatenated with the second feature, and the concatenated feature is used as the input to the localization attention layer. After obtaining the third feature output by the localization attention layer, the third feature is added to the second feature to obtain the fourth feature. The fourth feature and the text feature are used as the input to the cross-attention layer. If the gate is closed, a skip-layer mechanism is implemented for the localization attention layer, and the second feature and the text feature are used as the input to the cross-attention layer. After the cross-attention layer outputs the fifth feature, the fifth feature is input to the second temporal attention layer to obtain the second temporal feature, which is then used as the input to the next spatiotemporal location perception layer.

[0021] Preferably, the specific processing flow of the localization attention layer is as follows: after the concatenated features pass through the self-attention layer, a sixth feature is obtained. Token selection is performed on the sixth feature to output the visual token at the corresponding position. The visual token is passed through an activation function to obtain a seventh feature. The seventh feature is multiplied by the learnable parameters to obtain the output of the localization attention layer.

[0022] Preferably, the specific processing flow of the dynamic gating network is as follows: The first temporal feature is first subjected to global average pooling to obtain the pooled feature. The pooled feature is multiplied point by point with the learnable layer embedding to obtain the weight. The weight is then summed with the pooled feature to obtain the weighted feature. The weighted feature is then input into the second perceptron, and the probability of the gate being opened is calculated using the Sigmoid function. It is then determined whether the probability of the gate being opened is greater than or equal to a preset probability threshold. If it is, the gating result indicating that the gate is open is output; otherwise, the gating result indicating that the gate is closed is output.

[0023] Preferably, the probability threshold is set to 0.5.

[0024] Preferably, the second sensor employs a 2-layer MLP.

[0025] Preferably, the text description is encoded using a pre-trained CLIP model to obtain text features.

[0026] Preferably, the first sensor uses a 2-layer MLP.

[0027] Preferably, the text-based video model uses mean squared error loss during training.

[0028] Preferably, in step S2, the feature encoder and the feature decoder are respectively a variational autoencoder encoder and encoder.

[0029] The method of the present invention has the following advantages compared with the current text-based video methods:

[0030] First, the method of this invention unifies textual video frames with position controllability for discrete (e.g., layout) and continuous (e.g., depth map).

[0031] Secondly, this invention uses a dynamic gating network to adaptively selectively skip layers in the localization attention layer. The motivation for this network design is that neural networks have different preferences for high-order abstract semantics and low-order location information. A learnable layer embedding is constructed, and a similarity-based probability value is calculated using the localization information and the layer embedding of each layer to determine whether the localization attention layer is skipped.

[0032] Finally, the present invention shows considerable improvement in inter-frame continuity and video text consistency on mainstream datasets, and the visualization results also show that the present invention can generate smooth, position-controllable videos.

[0033] In summary, the method of this invention can generate continuous videos that correspond to and conform to location information and text under the control of location information and text, and unifies the control of discrete (e.g., layout) and continuous (e.g., depth map) location information. This invention effectively improves the video continuity and text consistency of text-based video models, and has considerable practical application value. In the future, in the field of text-based video, the method of this invention can perform better under diverse location conditions, providing strong support for applications such as e-commerce, advertising recommendation, and intelligent content creation. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the process of the present invention;

[0035] Figure 2 This is a framework diagram of the text-based video model of the present invention;

[0036] Figure 3 This is a schematic diagram of the dynamic gating network of the present invention;

[0037] Figure 4 This is a schematic diagram of the text-based video results of the position control according to the present invention;

[0038] Figure 5 This is a qualitative comparison diagram of the present invention with other methods. Detailed Implementation

[0039] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0040] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0041] Position-controllable text-based video is defined as follows: given a text description and its corresponding position sequence (such as layout, depth map, skeletal point map, etc.), the model processes the data and outputs a video. Text-based video has wide applications in many fields, such as short drama creation, advertising design, e-commerce, and game design. In recent years, text-based video has received increasing attention; however, position-controllable text-based video remains an unsolved problem and is one of the challenges in this field.

[0042] To address the aforementioned problems, the present invention aims to realize a position-controllable text-based video model based on a diffusion model. Firstly, the present invention proposes a spatiotemporal position-aware layer, comprising a self-attention layer, a localization attention layer, a cross-attention layer, and a temporal attention layer. Specifically, the self-attention layer integrates visual tokens, the localization attention layer interacts with positional information and visual tokens, the cross-attention layer interacts with textual information and visual tokens, and the temporal attention layer ensures the inter-frame continuity of the video. Furthermore, considering the preference of deep and shallow layers of neural networks for abstract semantics and low-order localization information, the present invention also proposes a dynamic gating network for an adaptive layer-skipping mechanism for the localization attention layer.

[0043] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned position-controllable text-generated video method based on a diffusion model includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0044] S1. Obtain the training dataset for training the text-based video model. The training dataset consists of target video segments with the same resolution, and each target video segment corresponds to a text description and a location information.

[0045] It should be noted that the original video segments in the training dataset can be any video, and their corresponding text descriptions can be either provided by the dataset itself or pseudo-labels generated by a pre-trained multimodal model.

[0046] Additionally, it should be noted that the aforementioned training dataset I trainThis can be achieved directly using existing datasets such as WebVid. The implementation of WebVid's dataset is existing technology; for details, please refer to the following literature: Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021). Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 1728-1738). Further details will not be elaborated upon here.

[0047] In step S1 of this embodiment, the training dataset I is obtained. train Then, each original video segment is cropped according to the center, and the cropped original video segment is used as the target video segment. The length and width of each target video segment are 512 resolution.

[0048] S2. Construct a text-based video model. The text-based video model consists of a feature encoder, a position-aware UNet network, and a feature decoder for image recovery. The encoder and decoder of the UNet network have a symmetrical structure. The encoder of the UNet network includes several stages of downsampling, and the decoder of the UNet network includes several stages of upsampling. The downsampled feature maps at each level are fused with the upsampled feature maps at each level through residual connections.

[0049] The downsampling process has the same number of stages as the upsampling process. Each stage of the upsampling or downsampling process is implemented by several spatiotemporal position awareness layers that are cascaded in sequence. These spatiotemporal position awareness layers include a self-attention layer, a localization attention layer with token selection, a cross attention layer, a temporal attention layer, and a dynamic gating network with learnable layer embedding.

[0050] It should be noted that in this invention, the feature encoder and the feature decoder are respectively a variational autoencoder (VAE) encoder and encoder, and their implementation is existing technology, which will not be described in detail here.

[0051] It should be noted that in each spatiotemporal location perception layer of step S2 of this invention, the specific processing flow is as follows: Text features are obtained by text encoding the text description; spatial location features are obtained by Fourier mapping the location information; the text features and spatial location features are added together; the added features are input into the first perceptron to obtain the first feature; the first feature is passed through the first temporal attention layer to obtain the first temporal feature; the first temporal feature is input into the dynamic gating network to output the gating result; simultaneously, the feature output by the previous spatiotemporal location perception layer is used as the input of the self-attention layer, which outputs the second feature; if the gating result is... If the gate is open, the first temporal feature and the second feature are concatenated. The concatenated feature is used as the input to the localization attention layer. After obtaining the third feature output by the localization attention layer, the third feature and the second feature are added together to obtain the fourth feature. The fourth feature and the text feature are used as the input to the cross-attention layer. If the gate is closed, a skip-layer mechanism is implemented for the localization attention layer. The second feature and the text feature are used as the input to the cross-attention layer. After the cross-attention layer outputs the fifth feature, the fifth feature is input to the second temporal attention layer to obtain the second temporal feature, which is then used as the input to the next spatiotemporal location perception layer.

[0052] It should be noted that, in this invention, the specific processing flow of the above-mentioned localization attention layer is as follows: after the concatenated features pass through the self-attention layer, the sixth feature is obtained; token selection is performed on the sixth feature to output the visual token at the corresponding position; the visual token is passed through the activation function to obtain the seventh feature; the seventh feature is multiplied by the learnable parameters to obtain the output of the localization attention layer.

[0053] It should be noted that the specific processing flow of the above-mentioned dynamic gating network in this invention is as follows: The first temporal feature is first subjected to global average pooling to obtain the pooled feature. The pooled feature is multiplied point by point with the learnable layer embedding to obtain the weight. The weight is then summed with the pooled feature to obtain the weighted feature. The weighted feature is then input into the second perceptron, and the probability of the gate opening is calculated by Sigmoid. It is then determined whether the probability of the gate opening is greater than or equal to a preset probability threshold. If it is, the gating result indicating that the gate is open is output; if it is not, the gating result indicating that the gate is closed is output.

[0054] The following describes the specific implementation process of each part of the above-mentioned text-to-video model. The text-to-video model of the present invention adopts a basic text-to-image model, including a feature encoder, a UNet network and a feature decoder. In the present invention, the transformer layer in UNet is replaced with a spatial-temporal position-aware layer, each spatial-temporal position-aware layer comprises a self-attention layer, a positioning attention layer with token selection, a cross-attention layer, a temporal attention layer and a dynamic gating network. The position-controllable capability is realized by relying on the positioning attention layer, and the video generation capability is realized by relying on the temporal attention layer. Wherein, the positioning attention layer enables position tokens and visual tokens to interact through self-attention, then selects visual tokens to be transmitted to the next layer, thereby realizing the fusion of position information and visual information; the temporal attention layer improves the continuity between video frames by performing self-attention on the temporal dimension. A dynamic gating network is used to implement a skip-layer mechanism for redundant positioning attention layers, because deep layers and shallow layers of neural networks have different sensitivities to position information and semantic information. A learnable layer embedding is constructed in the dynamic gating network, and the probability of skipping the layer is calculated through the similarity between the positioning information and the layer embedding.

[0055] Specifically, the encoder of the UNet network comprises a 4-stage upsampling process, and correspondingly, the decoder of the UNet network also comprises a 4-stage downsampling process, as Figure 2 (b) shows. Each upsampling process or downsampling process is implemented through a spatial-temporal position-aware layer. As Figure 2 (a) shows, after performing text encoding on a text description through a pre-trained CLIP model, the spatial-temporal position-aware layer obtains text features, obtains spatial position features after performing Fourier mapping on position information, adds the text features and the spatial position features ( Figure 2 in a ), inputs the added features into a first perceptron with a 2-layer MLP structure to obtain a first feature z 1 , passes the first feature through the first temporal attention layer to obtain a first temporal feature g = TempAttn(z 1 ), wherein TempAttn(·) represents the first temporal attention layer.

[0056] The first temporal feature is input into the dynamic gating network ( Figure 3 dynamic gate network therein) as shown in Figure 3 , a global average pooling operation is first performed on the first temporal feature g to obtain a pooled feature, the pooled feature and the learnable layer embedding v are multiplied point-wise ( Figure 3 ⊙ therein) to obtain a weight α, and the weight and the pooled feature are subjected to weighted summation ( Figure 3The weighted features are obtained by ∑), and then input into the second perceptron of a 2-layer MLP structure. The probability r of gating opening is calculated using the Sigmoid function, and then binarized using 0.5 as the probability threshold. Figure 2 As shown in (c), the feature z output by the previous spatiotemporal location sensing layer is... 1 As input to the self-attention layer, the second feature z is output by the self-attention layer. 2 =SelfAttn(z 1 ), SelfAttn(·) represents the self-attention layer.

[0057] When the probability of the gate opening is greater than or equal to 0.5, the dynamic gating network outputs 1, indicating that the gate is open (d). At this time, no skipping mechanism is implemented in the localization attention layer, and the first temporal feature g is combined with the second feature z. 2 Perform splicing, and combine the spliced ​​features z p =[z 2 The concatenated features, g], are used as input to the localization attention layer. After passing through the self-attention layer, the sixth feature z is obtained. 6 =SelfAttn([z 2 [,g]), where [,] represents the concatenation operation. Token selection is performed on the sixth feature, outputting the visual token at the corresponding position: Token = TS(SelfAttn([z]) 2 In this context, TS represents token selection; g is the first temporal feature, representing the token for location information. The visual token is processed through the activation function tanh(γ) to obtain the seventh feature z. 7 =tanh(γ)·TS(SelfAttn([z 2 Multiplying the seventh feature by the learnable parameter β yields the third feature z of the localization attention layer output. 3 =β·tanh(γ)·TS(SelfAttn([z 2 After , g])), add the third feature and the second feature together (( Figure 2 c )), thus obtaining the fourth feature z 4 The fourth feature and the text feature d are used together as input to the cross-attention layer. Here, γ represents a learnable parameter that controls the generation process.

[0058] When the probability of the gate opening is less than 0.5, the dynamic gating network outputs 0, indicating that the gate is closed. In this case, a skip-layer mechanism is implemented on the localization attention layer, connecting it to the next layer via residuals. Specifically, the second feature and the text feature d are used together as input to the cross-attention layer. The fifth feature z is output by the cross-attention layer CrossAttn(·). 5 Then, the fifth feature is input into the second temporal attention layer to obtain the second temporal feature, which is then used as the input to the next spatiotemporal location perception layer.

[0059] S3. Train the above textual video model on the above training dataset. When training the textual video model, only update the parameters of the localization attention layer, temporal attention layer and dynamic gating network in the above spatiotemporal location awareness layer.

[0060] In each iteration of the Wensheng video model training process, the target video segment, along with text description and location information, is input into the Wensheng video model. The noisy target video segment is mapped into a latent representation by a pre-trained feature encoder. The latent representation is then input into the UNet network for downsampling and upsampling to obtain the sampled features, which are then input into a pre-trained feature decoder to output the predicted video segment, thus completing pixel-level reconstruction.

[0061] It should be noted that during training, the overall input to the text-based video model is the target video segment along with its corresponding text description and location information. The target video segment is first mapped into a latent representation by the feature encoder, then undergoes downsampling and upsampling via a location-aware UNet network, and finally, pixel-level reconstruction is completed by the feature decoder. The training objective is to reconstruct the target video and supervise the process using the mean squared error loss (L2 loss) function. The text-based video model is optimized by minimizing the mean squared error loss until it converges. During this training process, the feature encoder and feature decoder are pre-trained, and their parameters are fixed and not updated. The parameters of the self-attention layer and cross-attention layer are also fixed and not updated.

[0062] The mean squared error loss L mentioned above is the L2 loss between the noise sampled from the current step and the noise predicted by the Wensheng video model, and its functional form is:

[0063]

[0064] Where ∈ represents the noise sampled in the current step; f(z) t (t,c) represents the noise predicted by the text-based video model; c is the conditional control including location information and text description; f represents the above text-based video model, t is the temporal embedding of the current step, and z is the text embedding of the current step.t This is the input for the current step.

[0065] S4. After the text-based video model is trained, noise is sampled from the Gaussian-distributed noise. The location information of the video to be generated, the text description, and the sampled noise are input into the trained text-based video model, and the output video conforms to the text description and location information.

[0066] It should be noted that in step S4 of this invention, during the actual text-generated video generation, the trained text-generated video model outputs a video that conforms to the text description and location information. The input consists of the text description and location information of the video to be generated. The text description and location information are input into the trained text-generated video model, which samples noise from Gaussian distributed noise. The trained text-generated video model iteratively denoises the sampled noise and outputs a video that conforms to the text and location information description. In addition, the location information can be layout, depth map, skeleton points, etc.

[0067] The method provided by this invention can effectively improve temporal continuity and text consistency. In order to better demonstrate the specific implementation and technical effects of this invention, the location-controllable text-generated video method based on the diffusion model shown in steps S1 to S4 of the above preferred implementation is applied to a specific example.

[0068] Example

[0069] The implementation method of this embodiment is as described above (i.e., S1 to S4), and the specific steps will not be described in detail again. The following only shows the effect based on the case data.

[0070] This invention was validated on the mainstream laboratory dataset DAVIS under specific settings, and the experimental results fully demonstrate the effectiveness of the method. The DAVIS dataset specifically contains high-resolution video sequences ranging from 1 to 4 seconds, and is a common dataset in video generation and segmentation tasks. The implementation of the DAVIS dataset is existing technology; see: Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., & Van Gool, L. (2017). The 2017 davis challenge on video objectsegmentation. arXiv preprint arXiv:1704.00675.

[0071] This embodiment compares the method trained using this approach with other mainstream methods. Training was performed on the WebVid dataset, and testing was conducted on DAVIS to verify the effectiveness of the proposed method. The final recognition accuracy results are shown in Table 1. These mainstream methods are Control-A-Video (CAV). [1] Control Video (CV) [2] and Gen-1 [3] These are all cutting-edge gait recognition models in the current technology. The implementation methods of these methods are all existing technologies and can be used as a reference.

[0072] [1]: Chen, W., Wu, J., Xie, P., Wu, H., Li, J., Xia,

[0073] [2]: Zhang, Y., Wei, Y., Jiang, D., Zhang, X., Zuo, W., & Tian, ​​Q. (2023). ControlVideo: Training-free Controllable Text-to-Video Generation. arXivpreprint arXiv:2305.13077.

[0074] [3]: Esser, P., Chiu, J., Atighehchian, P., Granskog, J., & Germanidis, A. (2023). Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp.7346-7356).

[0075] Table 1 shows the test results of this example on the DAVIS dataset.

[0076] CAV 94.67 23.08 CV 96.91 25.28 Gen-1 94.57 25.95 This embodiment 97.83 27.07

[0077] Table 1 shows that the proposed method significantly improves the performance of temporal consistency and prompt consistency on the DAVIS dataset. Temporal consistency is calculated by averaging the CLIP values ​​of adjacent frames, while prompt consistency is calculated by averaging the CLIP values ​​of the text and each frame of the generated video. This approach is more consistent with real-world application scenarios, laying a solid foundation for real-world applications and model deployment, and can be well generalized to more complex location scenarios. The CLIP value calculation method is an existing technique; for details, please refer to: Radford, A., Kim, JW, Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

[0078] Figure 4 This is a schematic diagram of the text-based video results for position control according to the present invention. Figure 4 In this context, "Grounding Condition" indicates the positioning condition, and "Prompt" indicates a prompt. Since the text encoder only supports English, therefore... Figure 4 The prompts use plain English input. Figure 5 This is a qualitative comparison diagram of the present invention and other methods, showing the text-generated video results output by the method of the present invention and other methods under the same prompt. Figure 5 There are three sets of prompts, and each method generates three frames from the video segment corresponding to each set of prompts. Figure 5 The first row contains images from the actual video segment, the second row contains images from the video segment generated by the CAV method, the third row contains images from the video segment generated by the CV method, the fourth row contains images from the video segment generated by the Gen-1 method, and the fifth row contains images from the video segment generated by the method of this invention.

[0079] As can be seen from the above technical solutions, the text-based video model of the present invention is easy to implement, provides an application direction for deployment in real-world scenarios, and can realize location-controllable text-based video functionality.

[0080] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A location-controllable text-based video method based on a diffusion model, characterized in that, Includes the following steps: S1. Obtain a training dataset for training the text-based video model. The training dataset consists of target video segments with the same resolution, and each target video segment corresponds to a text description and a location information. S2. Construct a text-based video model, which consists of a feature encoder, a position-aware UNet network, and a feature decoder. The encoder and decoder of the UNet network have a symmetrical structure. The encoder of the UNet network includes several stages of downsampling, and the decoder of the UNet network includes several stages of upsampling. The downsampled feature maps at each level are fused with the upsampled feature maps at each level through residual connections. The downsampling process has the same number of stages as the upsampling process. Each stage of the upsampling or downsampling process is implemented by several spatiotemporal position awareness layers that are cascaded in sequence. The spatiotemporal position awareness layer includes a self-attention layer, a localization attention layer with token selection, a cross attention layer, a temporal attention layer, and a dynamic gating network with learnable layer embedding. S3. Train the text-based video model on the training dataset. When training the text-based video model, only update the parameters of the localization attention layer, the temporal attention layer, and the dynamic gating network in the spatiotemporal location awareness layer. In each iteration of the Wensheng video model training process, the target video segment, text description, and location information are input into the Wensheng video model. The noisy target video segment is mapped into a latent representation by a pre-trained feature encoder. The latent representation is input into the UNet network for downsampling and upsampling to obtain the sampled features, which are then input into a pre-trained feature decoder to output the predicted video segment, thus completing pixel-level reconstruction. S4. After the text-based video model is trained, noise is sampled from the Gaussian-distributed noise. The location information of the video to be generated, the text description, and the sampled noise are input into the trained text-based video model, and the output video conforms to the text description and location information. In each spatiotemporal location perception layer of step S2, the specific processing flow is as follows: Text features are obtained by text encoding the text description; spatial location features are obtained by Fourier mapping the location information; the text features and spatial location features are added together; the added features are input into the first perceptron to obtain the first feature; the first feature is passed through the first temporal attention layer to obtain the first temporal feature; the first temporal feature is input into the dynamic gating network to output the gating result; simultaneously, the feature output from the previous spatiotemporal location perception layer is used as the input to the self-attention layer, which outputs the second feature; if the gating result is "gate open"... The first temporal feature is concatenated with the second feature, and the concatenated feature is used as the input to the localization attention layer. After obtaining the third feature output by the localization attention layer, the third feature is added to the second feature to obtain the fourth feature. The fourth feature and the text feature are used as the input to the cross-attention layer. If the gating result is that the gate is closed, a skipping mechanism is implemented for the localization attention layer, and the second feature and the text feature are used as the input to the cross-attention layer. After the cross-attention layer outputs the fifth feature, the fifth feature is input to the second temporal attention layer to obtain the second temporal feature, which is then used as the input to the next spatiotemporal location perception layer.

2. The location-controllable text-based video method based on a diffusion model as described in claim 1, characterized in that, The specific processing flow of the localization attention layer is as follows: After the concatenated features pass through the self-attention layer, the sixth feature is obtained. Token selection is performed on the sixth feature to output the visual token at the corresponding position. The visual token is passed through the activation function to obtain the seventh feature. The seventh feature is multiplied by the learnable parameters to obtain the output of the localization attention layer.

3. The location-controllable text-based video method based on a diffusion model as described in claim 1, characterized in that, The specific processing flow of the dynamic gating network is as follows: First, the first temporal feature is subjected to global average pooling to obtain the pooled feature. The pooled feature is multiplied point by point with the learnable layer embedding to obtain the weight. The weight is then summed with the pooled feature to obtain the weighted feature. The weighted feature is then input into the second perceptron, and the probability of the gate being opened is calculated using the Sigmoid function. It is then determined whether the probability of the gate being opened is greater than or equal to a preset probability threshold. If it is, the gating result indicating that the gate is open is output; otherwise, the gating result indicating that the gate is closed is output.

4. The location-controllable text-based video method based on a diffusion model as described in claim 3, characterized in that, The probability threshold is set to 0.

5.

5. The location-controllable text-based video method based on a diffusion model as described in claim 3, characterized in that, The second perceptron uses a 2-layer MLP.

6. The location-controllable text-based video method based on a diffusion model as described in claim 1, characterized in that, The text description is encoded using a pre-trained CLIP model to obtain text features.

7. The location-controllable text-based video method based on a diffusion model as described in claim 1, characterized in that, The first sensor uses a 2-layer MLP.

8. The location-controllable text-based video method based on a diffusion model as described in claim 1, characterized in that, The text-based video model uses mean squared error loss during training.

9. The location-controllable text-based video method based on a diffusion model as described in claim 1, characterized in that, In step S2, the feature encoder and the feature decoder are respectively a variational autoencoder encoder and decoder.

Images