Method and device for generating video from text and storage medium

Abstract

The invention discloses a method and a device for generating a video from a text, and a storage medium. The method comprises the following steps: S11, generating a first noise latent code of a target video frame based on an input text prompt; s12, generating a cross attention map of each subject mark in the input text prompt based on the input text prompt and the first noise latent code; s13, Gaussian filtering is carried out on the cross attention map, so that the cross attention map is smooth; s14, updating the first noise latent code based on the cross attention map, generating a second noise latent code, and replacing the first noise latent code with the second noise latent code; s15, de-noising the second noise latent code by using a diffusion model with inter-frame attention control; s16, judging whether the total sampling step number reaches a preset step number or not and whether all the main body marks reach a set minimum attention threshold value or not, and if not, executing the step S12; and S17, outputting the target video frame based on the second noise latent code. The text video method provided by the invention ensures semantic accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a method, device and storage medium for generating video from text. Background Art

[0002] Diffusion model: A generative model that gradually adds noise to data samples and then reverses the inference to generate new samples. This model performs well in areas such as image generation and video generation and can capture complex distribution characteristics.

[0003] Semantic guidance: The semantic information and attention mechanism of the input data (such as text and images) are used in the generation process to ensure that the generated results are consistent with the original data at the semantic level.

[0004] “Text-to-video” refers to the process of generating video content from text. This process involves converting text information into visual dynamic images, which usually requires the combination of natural language processing and computer vision technology to ensure that the video content accurately expresses the semantics of the text.

[0005] DDIM: In Chinese, it means denoising diffusion implicit model.

[0006] DDPM: The Chinese meaning is denoising diffusion probability model.

[0007] In the existing "text-to-video" technology, there are problems with semantic inaccuracy in text-to-video generation, such as ignoring certain subjects in the input text prompt, or incorrectly binding attribute words. There are two main types of existing diffusion model-based text-to-video generation methods. The first is a generation method based on the latent space diffusion model, which uses the latent space diffusion model to pre-train on large-scale text-video pairs, so that the model can generate diverse high-quality videos. The second is to achieve high-quality video generation without any training or optimization by leveraging existing text-to-image synthesis methods (such as Stable Diffusion).

[0008] However, large-scale training solutions consume a lot of computing resources and are costly, and generally have problems with missing subjects and attribute binding in multi-subject generation scenarios. Although the zero-shot video generation method without additional training saves computing costs, the above-mentioned semantic inaccuracy problem still exists and is more serious. Summary of the invention

[0009] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one object of the present invention is to provide a method, device and storage medium for generating video from text, which can solve the problem of semantic inaccuracy in generating video from text.

[0010] The technical solution adopted by the present invention is: In a first aspect, the present invention provides a method for generating a video from text, the method comprising the steps of: S11: generating a first noise latent code of a target video frame based on an input text prompt; S12: generating a cross-attention map of each subject marker in the input text prompt based on the input text prompt and the first noise latent code; S13: performing Gaussian filtering on the cross-attention map to smooth the cross-attention map; S14: updating the first noise latent code based on the cross-attention map to generate a second noise latent code, and replacing the first noise latent code with the second noise latent code; S15: denoising the second noise latent code using a diffusion model with inter-frame attention control; S16: determining whether the total number of sampling steps reaches a predetermined number of steps, and whether all subject markers reach a set minimum attention threshold, if so, executing step S17; otherwise, if not, executing step S12; S17: outputting the target video frame based on the second noise latent code.

[0011] The step S11 includes: using an existing pre-trained model to generate the first noise latent code, specifically including: randomly sampling the first latent code Z from a standard Gaussian distribution t 1; for this Z t 1 Perform the DDIM reverse step of Δt steps to obtain Z t '1; define the global motion direction, calculate and apply the global translation vector k for each frame, and apply the constructed motion flow 1:m to the Z t '1, get the corresponding sequence Z t '1:m, that is, for the first frame Z t '1 Perform m steps of gradually strengthening the translation deformation operation to obtain the Z of m frames t '1:m; Perform the DDPM forward step on the deformed latent code to obtain the initial noise latent code sequence Z of the final m frames t 1:m, based on the Z t 1:m obtain the first noise latent code Z t .

[0012] Among them, step S12 includes: the cross-attention map is averaged by multiple attention maps with a resolution of 16*16, the cross-attention map contains multiple attention maps, each attention map corresponds to each mark, and the value of an element of the matrix of the cross-attention map represents the probability of a subject mark appearing in an image block.

[0013] The step S14 includes: focusing on the most neglected topic tag at the current time step, calculating the loss function L; based on the loss function L, converting the current first noise latent code Z t Perform gradient update to obtain the second noise latent code Z t ′.

[0014] Wherein, step S15 includes: performing denoising by using a cross-frame attention mechanism.

[0015] In a second aspect, the present invention provides a text-generated video device, which includes: a first noise latent code generation module, which is used to generate a first noise latent code of a target video frame based on an input text prompt; a cross-attention map generation module, which is used to generate a cross-attention map of each subject marker in the input text prompt based on the input text prompt and the first noise latent code; a cross-attention map smoothing module, which is used to perform Gaussian filtering on the cross-attention map to smooth the cross-attention map; a second noise latent code generation module, which is used to update the first noise latent code based on the cross-attention map, generate a second noise latent code, and replace the first noise latent code with the second noise latent code; a denoising module, which is used to denoise the second noise latent code using a diffusion model with inter-frame attention control; an iterative judgment module, which is used to judge whether the total number of sampling steps reaches a predetermined number of steps, and if not, returns to the cross-attention map generation module; a target video frame generation module, which is used to output the target video frame based on the second noise latent code.

[0016] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method as described above.

[0017] The beneficial effects of the present invention are: Compared with the existing video generation scheme, the present invention solves the problem of subject missing and incorrect attribute binding in the video generation process by using semantic guidance of cross-attention control, generates the initial latent code of all video frames through the existing pre-trained model, and performs semantic guidance in each subsequent denoising process. The intensity of semantic guidance increases gradually during the whole denoising process, and inter-frame attention control is added to improve the inter-frame consistency problem. The present invention realizes video generation with semantic consistency through a simple method that only requires text input.

[0018] Furthermore, this method can be directly generated on the basis of an existing training model without redundant training, and can be applied to a pre-trained model with better results. On this basis, semantic guidance is performed to generate a video with accurate semantics and high overall video frame consistency. Therefore, this method is a relatively simple, convenient and effective method for generating video, and has low requirements for equipment and users. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of an embodiment of the method for generating video from text of the present invention; Figure 2 yes Figure 1A structural schematic diagram of an application embodiment; Figure 3 yes Figure 1 A flowchart of an embodiment of step S11; Figure 4 yes Figure 1 A schematic structural diagram of an embodiment of step S12; Figure 5 yes Figure 1 A schematic structural diagram of an embodiment of step S13; Figure 6 yes Figure 1 A schematic structural diagram of an embodiment of step S15; Figure 7 It is a structural schematic diagram of an embodiment of a device for generating video from text of the present invention. DETAILED DESCRIPTION

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0021] Please also read Figure 1 and Figure 2 , Figure 1 FIG. 1 is a flow chart of an embodiment of the method for generating video from text of the present invention. Figure 1 As shown, the method includes: S11: Generate a first noise latent code of a target video frame based on an input text prompt; In step S11, the input text prompt may be a sentence or a phrase (eg, "a lion with a crown"). The first noise latent code is generated using an existing pre-trained model, for example, using an open source Text2Video-Zero model.

[0022] Latent code is a low-dimensional representation of data (such as images or text), which is obtained by compressing the high-dimensional information of the original data.

[0023] In the past, the huge 3D denoising network brought expensive computational overhead in video generation. One solution is to transfer the denoising process of the diffusion model from the pixel space to the low-dimensional latent space, thereby reducing the network computation. The latent code contains the core features that describe the data, and the generative model generates new images through the latent code. This low-dimensional representation can retain the key information of the data while removing redundant and unnecessary details, making the calculation and generation process more efficient.

[0024] See also Figure 3 , Figure 3 yes Figure 1 A flow chart of an embodiment of step S11 is shown in FIG. Figure 2As shown, step S11 includes the following sub-steps: S111: Randomly sample the first latent code Z from a standard Gaussian distribution t 1; S112: This Z t 1 Perform the DDIM reverse step of Δt steps to obtain Z t '1; S113: Define the global motion direction, calculate and apply the global translation vector k for each frame, and apply the constructed motion flow 1:m to the Z t '1, get the corresponding sequence Z t '1:m, that is, for the first frame Z t '1 Perform m steps of gradually strengthening the translation deformation operation to obtain the Z of m frames t '1:m; S114: Perform DDPM forward step on the deformed latent code to obtain the final m-frame initial noise latent code sequence Z t 1:m, based on the Z t 1:m obtain the first noise latent code Z t .

[0025] S12: generating a cross-attention map of all subject tags in the input text prompt based on the input text prompt and the first noise latent code; Let the input text prompt be denoted as P, and all subject tokens (e.g., nouns) in P be denoted as S, where S = {S1, S1, ... S k}, for example: P = "a lion with a crown", then S = {"lion", "crown"}. The first noise latent code is denoted by Z t , let the cross attention map be A t .

[0026] See also Figure 4 , at time step t, given the first noise latent code Z t And the input text prompt P, generate a cross attention map denoted as A t In step S12, the cross-attention map At is obtained by averaging the attention maps with a resolution of 16*16 from multiple heads. The cross-attention map contains multiple attention maps, each of which corresponds to each subject tag. The value of any element of the matrix of the cross-attention map represents the probability of a subject tag appearing in an image block. i Extract 16 × 16 normalized spatial attention map , which represents the subject marker S i The impact on each image patch.

[0027] S13: Perform Gaussian filtering on the cross-attention map to smooth the cross-attention map; Through experiments, it is found that when the model does not generate a complete subject but generates some parts of the subject, A single image patch in may also have a high attention value, so the attention map calculated above It may not fully reflect whether the generated image contains the relevant subject, and thus cannot guide subsequent generation tasks. For example, a high attention value may only generate the outline of some body parts of the subject, forming some seemingly reasonable but actually inaccurate appearance features, thus affecting the quality and authenticity of the video.

[0028] To avoid this, Figure 5 As shown, this scheme Gaussian filtering is performed to ensure that the attention value of the maximum activated image block depends on the linear combination of the attention values ​​of its neighboring image blocks. Reducing the discontinuity of local noise makes the attention map smoother and more consistent, which helps the model to allocate attention more accurately.

[0029] S14: updating the first noise latent code based on the cross attention map, generating a second noise latent code, and replacing the first noise latent code with the second noise latent code; From common sense, a successfully generated subject should correspond to an image block that will significantly focus on the subject tag, that is, for each subject tag s in S, There should be at least one image patch with a high attention value in . This embodiment focuses on the most neglected topic tag at the current time step, so the optimized loss function can be defined as:

[0030] in,

[0031] Because different subject markers may be activated at different time steps, all neglected subject markers may be activated at a certain time step.

[0032] After calculating the loss function L, the current first noise latent code Z t Perform gradient update to obtain the second noise latent code Z t ′: where a t is the gradient update step size, the gradient sign Represents the loss function L in the current first noise latent code Z t The direction and rate of change of the

[0033] S15: denoising the second noise latent code using a diffusion model with inter-frame attention control; In order to solve the problem of flickering and inconsistency between overall video frames caused by applying the above semantic guidance and optimization to the video frame by frame, this paper adopts inter-frame attention control to improve the consistency of appearance and movement between frames in the entire generated video.

[0034] There are some alternative options for inter-frame attention control, including full-frame attention mechanism and causal attention mechanism, which can also improve the consistency between video frames. However, the full-frame attention mechanism is calculated with each video frame, and the causal attention mechanism is calculated with all frames before the current frame, so their computational complexity increases sharply with the increase of the number of frames. In addition to the above two O(n 2 ), there are also O(n) methods, such as using only the key and value of the first frame to generate the image of the entire video sequence, but the overall video motion continuity is poor.

[0035] Here, if Figure 6 As shown, this scheme uses a simplified causal attention mechanism. We start from the current frame z k Extract query features from the first frame z1 and the previous frame z k-1 Extract key and value features and implement Q, K, and V.

[0036]

[0037] Where Q, K, V are z1 and z k-1 The result of the concatenation and then projection transformation. By using the cross-frame attention mechanism, the flickering and inconsistency problems between the overall video frames caused by applying the above semantic guidance and optimization to the video frame by frame are alleviated, and the consistency of appearance and motion between the generated frames is significantly improved.

[0038] S16: Determine whether the total number of sampled steps reaches a predetermined number of steps, and whether all subject marks reach a set minimum attention threshold. If so, execute step S17; otherwise, execute step S12; Through experiments, it is found that if the attention value corresponding to a subject tag does not reach a certain threshold in the early stage of denoising, the subsequent denoising process will not generate the corresponding subject. Therefore, a new gradient update mechanism needs to be set here to iteratively update Z t Until all subject markers reach a set minimum attention threshold.

[0039] However, Z tMultiple updates of may lead to deviations in the latent code distribution of video frames, resulting in incoherent images of single video frames. Therefore, this paper sets this iterative update to be performed progressively in a small number of time steps instead of optimizing the entire denoising process.

[0040] Specifically, if each subject tag is ultimately required to reach a minimum attention value of at least 0.8, and the total number of sampling steps is 50, the iteration strategy can be set to: the sampling time step is (t1=0, t2=10, t3=20), and the corresponding minimum attention threshold is (T1=0.05, T2=0.5, T3=0.8).

[0041] This gradual refinement and gradual update strategy can prevent the model from letting Z t deviates from the dataset distribution while promoting more accurate and faithful video generation to the input text prompts S17: Output the target video frame based on the second noise latent code.

[0042] This embodiment proposes a new Vincent video method, and the improvements mainly include two aspects: (1) Generate semantic guidance: Fine-tune the latent code of each video frame during the denoising process so that the latent code can better focus on the semantic information conveyed by the input text prompt.

[0043] (2) Optimization of latent code iteration strategy: Experiments have found that the latent code determines the existence of each subject and the spatial position of each subject in the early denoising step. Multiple fine-tuning of the latent code may lead to deviations in the latent code distribution of a single frame, resulting in poor generation quality. Therefore, the iterative update in this paper is only performed progressively in a small number of time steps, and a minimum attention threshold is set for each subject at each progressive stage.

[0044] Embodiment 2 See also Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the structure of an embodiment of the text-generated video device of the present invention. Figure 7 As shown, the device includes a first noise latent code generation module 11, a cross-attention map generation module 12, a cross-attention map smoothing module 13, a second noise latent code generation module 14, a denoising module 15, an iterative judgment module 16 and a target video frame generation module 17.

[0045] A first noise latent code generating module 11, configured to generate a first noise latent code of a target video frame based on an input text prompt; A cross-attention map generating module 12, for generating a cross-attention map of each subject marker in the input text prompt based on the input text prompt and the first noise latent code; A cross-attention map smoothing module 13 is used to perform Gaussian filtering on the cross-attention map to make the cross-attention map smooth; A second noise latent code generating module 14 is used to update the first noise latent code based on the cross attention map, generate a second noise latent code, and replace the first noise latent code with the second noise latent code; A denoising module 15, configured to denoise the second noise latent code using a diffusion model with inter-frame attention control; The iterative judgment module 16 is used to judge whether the total number of sampling steps reaches the predetermined number of steps. If so, the target video frame generation module 17 is entered; otherwise, if not, the cross-attention map generation module 12 is returned. The target video frame generating module 17 is configured to output the target video frame based on the second noise latent code.

[0046] Specifically, the working method of each module has been described in detail in the first embodiment and will not be repeated here.

[0047] Embodiment 3 The present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in the first embodiment.

[0048] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for generating video from text, characterized in that: Includes steps: S11: Generate a first noise latent code of a target video frame based on an input text prompt; S12: generating a cross-attention map of each subject tag in the input text prompt based on the input text prompt and the first noise latent code; S13: Performing Gaussian filtering on the cross-attention map to smooth the cross-attention map; S14: updating the first noise latent code based on the cross attention map, generating a second noise latent code, and replacing the first noise latent code with the second noise latent code; S15: denoising the second noise latent code using a diffusion model with inter-frame attention control; S16: Determine whether the total number of sampled steps reaches a predetermined number of steps, and whether all subject marks reach a set minimum attention threshold. If so, execute step S17; Otherwise, if it is not reached, execute step S12; S17: Output the target video frame based on the second noise latent code.

2. The method according to claim 1, characterized in that The step S11 includes: using an existing pre-trained model to generate the first noise latent code, specifically including: Randomly sample the first latent code Z from a standard Gaussian distribution t 1; For the Z t 1 Perform the DDIM reverse step of Δt steps to obtain Z t '1; Define the global motion direction, calculate and apply the global translation vector k for each frame, and apply the constructed motion flow 1:m to the Z t '1, get the corresponding sequence Z t '1:m, that is, for the first frame Z t '1 Perform m steps of gradually strengthening the translation deformation operation to obtain the Z of m frames t '1:m; Perform the DDPM forward step on the deformed latent code to obtain the initial noise latent code sequence Z of the final m frames t 1:m, based on the Z t 1:m obtains the first noise latent code Z t .

3. The method according to claim 1, characterized in that The step S12 comprises: The cross-attention map is obtained by averaging multiple attention maps with a resolution of 16*16. The cross-attention map includes multiple attention maps, each of which corresponds to each mark. The value of an element of the matrix of the cross-attention map represents the probability of a subject mark appearing in an image block.

4. The method according to claim 1, characterized in that: The step S14 comprises: Focus on the most neglected topic tag at the current time step and calculate the loss function L; Based on the loss function L, the current first noise latent code Z t Perform gradient update to obtain the second noise latent code Z t ′.

5. The method according to claim 1, characterized in that The step S15 includes: performing denoising by using a cross-frame attention mechanism.

6. A text-generated video device, characterized in that: include: A first noise latent code generating module, configured to generate a first noise latent code of a target video frame based on an input text prompt; a cross-attention map generation module, configured to generate a cross-attention map of each subject marker in the input text prompt based on the input text prompt and the first noise latent code; A cross-attention map smoothing module, used for performing Gaussian filtering on the cross-attention map to smooth the cross-attention map; A second noise latent code generating module, configured to update the first noise latent code based on the cross attention map, generate a second noise latent code, and replace the first noise latent code with the second noise latent code; A denoising module, configured to denoise the second noise latent code using a diffusion model with inter-frame attention control; An iterative judgment module, used to judge whether the total number of sampling steps reaches a predetermined number of steps, and if not, returns to the cross-attention map generation module; A target video frame generating module is used to output the target video frame based on the second noise latent code.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images