Text-to-video method and apparatus, and storage medium

Abstract

The application discloses a text-to-video method and device and a storage medium, and the method comprises the following steps: S11, generating a first noise latent code of a target video frame based on an input text prompt; S12, generating a cross-attention graph of each subject label in the input text prompt based on the input text prompt and the first noise latent code; S13, performing Gaussian filtering on the cross-attention graph to smooth the cross-attention graph; S14, updating the first noise latent code based on the cross-attention graph, generating a second noise latent code, and replacing the first noise latent code with the second noise latent code; S15, denoising the second noise latent code by using a diffusion model with inter-frame attention control; S16, judging whether the total number of sampling steps reaches a predetermined step number and whether all subject labels reach a set minimum attention threshold value, and if not, performing step S12; and S17, outputting the target video frame based on the second noise latent code. The text-to-video method guarantees semantic accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video generation, in particular to a text-to-video generation method and device and storage medium. BACKGROUND

[0002] Diffusion model: a generative model that generates new samples by progressively adding noise to data samples and then reversing the process. This model performs well in image generation, video generation, and other fields, and can capture complex distribution characteristics.

[0003] Semantic guidance: use the semantic information and attention mechanism of input data (such as text, images) during the generation process to ensure that the generated results are consistent with the original data at the semantic level.

[0004] Text-to-video: refers to the process of generating video content from text. This process involves converting text information into visual dynamic images, usually requiring the combination of natural language processing and computer vision technology to ensure that the video content accurately represents the semantics of the text.

[0005] DDIM: Chinese for Denoising Diffusion Implicit Model.

[0006] DDPM: Chinese for Denoising Diffusion Probabilistic Model.

[0007] In existing text-to-video generation technology, there are problems of semantic inaccuracy in text-to-video generation, such as ignoring certain subjects in the input text prompt or incorrectly binding attribute words. There are two main types of existing text-to-video generation methods based on diffusion models. One is the generation method based on latent space diffusion model, which uses latent space diffusion model to pre-train on large-scale text-video pairs, enabling the model to generate diverse high-quality videos. The second is to use existing text-to-image synthesis methods (such as Stable Diffusion) to achieve high-quality video generation without any training or optimization.

[0008] However, large-scale training schemes consume a large amount of computing resources and are costly, and generally have the problem of missing subjects and attribute binding in multi-subject generation scenarios. Zero-shot video generation methods that do not perform additional training, while saving computing costs, still have the problem of semantic inaccuracy and are more severe. SUMMARY

[0009] The present application aims to at least partially solve one of the technical problems in the related art. To this end, one object of the present application is to provide a text-to-video generation method, device and storage medium that can solve the problem of semantic inaccuracy in text-to-video generation.

[0010] The technical scheme adopted by the present application is:

[0011] In a first aspect, the present application provides a text-to-video method, comprising the steps of: S11: generating a first noise latent code of a target video frame based on an input text prompt; S12: generating a cross-attention map for each subject label in the input text prompt based on the input text prompt and the first noise latent code; S13: performing Gaussian filtering on the cross-attention map to smooth the cross-attention map; S14: updating the first noise latent code based on the cross-attention map to generate a second noise latent code, and replacing the first noise latent code with the second noise latent code; S15: denoising the second noise latent code using a diffusion model with inter-frame attention control; S16: determining whether the total number of sampling steps reaches a predetermined step number and whether all subject labels reach a set minimum attention threshold, and if so, performing step S17; otherwise, performing step S12; S17: outputting the target video frame based on the second noise latent code.

[0012] In the step S11, an existing pre-trained model is used to generate the first noise latent code, specifically including: randomly sampling a first latent code Z t 1 from a standard Gaussian distribution; t performing a DDIM reverse step on the Z t 1 for Δt steps to obtain Z t '1:m is applied to the Z t '1 to obtain a corresponding sequence Z t 1:m, a m-step step-by-step strengthening translation deformation operation is performed on the first frame Z t 1 to obtain m frames of Z t 1:m, a DDPM forward step is performed on the deformed latent code to obtain an initial noise latent code sequence Z t 1:m based on the Z t .

[0013] In the step S12, the cross-attention map is obtained by averaging multiple attention maps with a resolution of 16*16, the cross-attention map includes multiple attention maps, each attention map corresponds to each label, and the value of an element of the matrix of the cross-attention map represents the probability of a subject label appearing in an image block.

[0014] In the step S14, the most neglected subject label at the current time step is focused on, a loss function L is calculated, and the current first noise latent code Z t is updated based on the loss function L to obtain a second noise latent code Z t′.

[0015] Herein, step S15 includes: performing denoising by using a cross-frame attention mechanism.

[0016] In a second aspect, the present invention provides a text-generated video device, which includes: a first noise latent code generation module, which is used to generate a first noise latent code of a target video frame based on an input text prompt; a cross-attention map generation module, which is used to generate a cross-attention map of each subject marker in the input text prompt based on the input text prompt and the first noise latent code; a cross-attention map smoothing module, which is used to perform Gaussian filtering on the cross-attention map to smooth the cross-attention map; a second noise latent code generation module, which is used to update the first noise latent code based on the cross-attention map, generate a second noise latent code, and replace the first noise latent code with the second noise latent code; a denoising module, which is used to denoise the second noise latent code using a diffusion model with inter-frame attention control; an iterative judgment module, which is used to judge whether the total number of sampling steps reaches a predetermined number of steps, and if not, returns to the cross-attention map generation module; and a target video frame generation module, which is used to output the target video frame based on the second noise latent code.

[0017] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method as described above.

[0018] The beneficial effects of the present invention are:

[0019] Compared to existing video generation solutions, this method addresses issues such as missing subjects and incorrect attribute binding during the generation process by leveraging semantic guidance with cross-attention control. An existing pre-trained model is used to generate the initial latent codes for all video frames. Semantic guidance is then applied to each subsequent denoising step, with the intensity of semantic guidance increasing progressively throughout the denoising process. Inter-frame attention control is also incorporated to improve inter-frame consistency. This method achieves semantically consistent video generation through a concise method requiring only text input.

[0020] Furthermore, this method can be generated directly based on existing training models, eliminating the need for redundant training and can be applied to pre-trained models with better performance. Furthermore, semantic guidance is applied to generate semantically accurate videos with high overall inter-frame consistency. Therefore, this method is a relatively simple, convenient, and effective method for generating video, and it has low requirements for both equipment and users. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of an embodiment of the method for generating video from text of the present invention;

[0022] Figure 2 yes Figure 1 A schematic structural diagram of an application embodiment;

[0023] Figure 3 yes Figure 1 A flowchart of an embodiment of step S11;

[0024] Figure 4 yes Figure 1 A schematic structural diagram of an embodiment of step S12;

[0025] Figure 5 yes Figure 1 A schematic structural diagram of an embodiment of step S13;

[0026] Figure 6 yes Figure 1 A schematic structural diagram of an embodiment of step S15;

[0027] Figure 7 It is a structural diagram of an embodiment of a device for generating video from text according to the present invention. DETAILED DESCRIPTION

[0028] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0029] Please also refer to Figure 1 and Figure 2 , Figure 1 1 is a flow chart of an embodiment of the method for generating video from text of the present invention. Figure 1 As shown, the method includes:

[0030] S11: Generate a first noise latent code of the target video frame based on the input text prompt;

[0031] In step S11, the input text prompt can be a sentence or a phrase (e.g., "a lion with a crown"). An existing pre-trained model is used to generate the first noise latent code, for example, the open source Text2Video-Zero model is used to generate the first noise latent code.

[0032] Latent code is a low-dimensional representation of data (such as images or text), which is obtained by compressing the high-dimensional information of the original data.

[0033] In previous video generation, large 3D denoising networks incurred high computational overhead. One solution is to shift the denoising process of the diffusion model from the pixel space to a low-dimensional latent space, thereby reducing the network's computational overhead. The latent code contains the core features describing the data, and the generative model uses the latent code to generate new images. This low-dimensional representation preserves the key information of the data while removing redundant and unnecessary details, making the computation and generation process more efficient.

[0034] See also Figure 3 , Figure 3 yes Figure 1 A flow chart of an embodiment of step S11 is shown in FIG. Figure 2 As shown, step S11 includes the following sub-steps:

[0035] S111: Randomly sample the first latent code Z from a standard Gaussian distribution t 1;

[0036] S112: This Z t 1 Perform the DDIM reverse step of Δt step to obtain Z t '1;

[0037] S113: Define the global motion direction, calculate and apply the global translation vector k for each frame, and apply the constructed motion flow 1:m to the Z t '1, get the corresponding sequence Z t '1:m, that is, for the first frame Z t '1 Perform m steps of gradually strengthening the translation deformation operation to obtain the Z of m frames t '1:m;

[0038] S114: Perform DDPM forward step on the deformed latent code to obtain the final m-frame initial noise latent code sequence Z t 1:m, based on the Z t 1:m obtains the first noise latent code Z t .

[0039] S12: generating a cross-attention map of all subject tags in the input text prompt based on the input text prompt and the first noise latent code;

[0040] Let the input text prompt be denoted as P, and all subject tokens (e.g., nouns) in P be denoted as S, where S = {S1, S1, ... S k}, for example: P = "a lion with a crown", then S = {"lion", "crown"}. Let the first noise latent code be Z t , let the cross attention map be recorded as A t .

[0041] See also Figure 4 , at time step t, given the first noise latent code Z t And the input text prompt P, generate a cross attention map denoted as A t In step S12, the cross attention map At is averaged from multiple attention maps with a resolution of 16*16. The cross attention map contains multiple attention maps, each of which corresponds to each subject marker. The value of any element of the matrix of the cross attention map represents the probability of a subject marker appearing in an image block. For each subject marker S i Extract 16 × 16 normalized spatial attention map , which represents the subject marker S i The impact on each image patch.

[0042] S13: Perform Gaussian filtering on the cross-attention map to smooth the cross-attention map;

[0043] Through experiments, we found that when the model does not generate a complete subject, but generates some parts of the subject, A single image block in may also have a high attention value, so the attention map calculated above This may not fully reflect whether the generated image contains the relevant subject, thus failing to guide subsequent generation tasks. For example, a high attention value may only generate the outline of some body parts of the subject, resulting in some seemingly reasonable but actually inaccurate appearance features, thus affecting the quality and authenticity of the video.

[0044] To avoid this, Figure 5 As shown, this scheme Gaussian filtering is performed to ensure that the attention value of the maximum activated image block depends on the linear combination of the attention values ​​of its neighboring image blocks. This reduces the discontinuity of local noise and makes the attention map smoother and more consistent, which helps the model allocate attention more accurately.

[0045] S14: updating the first noise latent code based on the cross attention map, generating a second noise latent code, and replacing the first noise latent code with the second noise latent code;

[0046] From common sense, a successfully generated subject should correspond to an image block that will significantly focus on the subject tag, that is, for each subject tag s in S, There should be at least one image patch with a high attention value in . This embodiment focuses on the most neglected topic tag at the current time step, so the optimized loss function can be defined as:

[0047]

[0048] in,

[0049] Because different subject markers may be activated at different time steps, all neglected subject markers may be activated at a certain time step.

[0050] After calculating the loss function L, the current first noise latent code Z t Perform gradient update to obtain the second noise latent code Z t ′:

[0051]

[0052] where a t is the gradient update step size, the gradient sign Represents the loss function L at the current first noise latent code Z t The direction and rate of change of the.

[0053] S15: Denoising the second noise latent code using a diffusion model with inter-frame attention control;

[0054] To solve the problem of flickering and inconsistency between overall video frames caused by applying the above semantic guidance and optimization to the video frame by frame, this paper adopts inter-frame attention control to improve the consistency of appearance and movement between frames in the entire generated video.

[0055] There are some alternative options for inter-frame attention control, including full-frame attention mechanism and causal attention mechanism, which can also improve the consistency between video frames. However, the full-frame attention mechanism is calculated with each video frame, and the causal attention mechanism is calculated with all frames before the current frame, so their computational complexity increases sharply with the increase of the number of frames. In addition to the above two O(n 2 ) There are also O(n) methods, such as using only the key and value of the first frame to generate images for the entire video sequence, but the overall video motion continuity is poor.

[0056] Here, as Figure 6 As shown, this scheme uses a simplified causal attention mechanism, where we start from the current frame z k Extract query features from the first frame z1 and the previous frame z k-1 Extract key and value features and implement Q, K, and V.

[0057]

[0058] Where Q, K, and V are z1 and z k-1Results of the connection and the post-projection conversion. By using the cross-frame attention mechanism, the overall video inter-frame flicker and inconsistency problems caused by applying the above semantic guidance and optimization frame by frame to the video are alleviated, and the consistency of the appearance and motion between the generated frames is significantly improved.

[0059] S16: Determine whether the total number of sampling steps reaches a predetermined number of steps, and whether all subject labels reach a set minimum attention threshold. If so, perform step S17; otherwise, perform step S12.

[0060] Through experiments, it is found that if the attention value corresponding to a subject label does not reach a certain threshold in the early stage of denoising, the subsequent denoising process will not generate the corresponding subject. Therefore, a new gradient update mechanism is needed here to iteratively update Z t until all subject labels reach a set minimum attention threshold.

[0061] However, multiple updates of Z t may cause the latent code distribution of the video frame to deviate, resulting in discontinuity of the image of the single frame of the video. Therefore, this iterative update is set to be gradually performed in a small part of the time steps, rather than optimized for the entire denoising process.

[0062] Specifically, if the final requirement is that each subject label reaches a minimum attention value of at least 0.8, and the total sampling step is 50, the iterative strategy can be set as: the sampling time steps are (t1=0, t2=10, t3=20), and the minimum attention thresholds are (T1=0.05, T2=0.5, T3=0.8).

[0063] This step-by-step refinement and gradual update strategy can prevent the model from making Z t deviate from the data set distribution, while promoting more accurate and faithful video generation to the input text prompt

[0064] S17: Output the target video frame based on the second noise latent code.

[0065] The embodiment proposes a new text-to-video method, and the improvement mainly includes two aspects:

[0066] (1) Generating semantic guidance: fine-tuning the latent code of each video frame in the denoising process, so that the latent code pays more attention to the semantic information conveyed by the input text prompt.

[0067] (2) Latent code iteration strategy optimization: Experiments have found that the latent code determines the presence and spatial location of each subject in the early denoising step. Repeated fine-tuning of the latent code may lead to deviations in the latent code distribution of a single frame, resulting in poor generation quality. Therefore, the iterative update in this paper is only performed incrementally in a small number of time steps, and a minimum attention threshold is set for each subject at each incremental stage.

[0068] Example 2

[0069] See also Figure 7 , Figure 7 FIG. 1 is a structural diagram of an embodiment of a device for generating video from text according to the present invention. Figure 7 As shown, the device includes a first noise latent code generation module 11, a cross-attention map generation module 12, a cross-attention map smoothing module 13, a second noise latent code generation module 14, a denoising module 15, an iterative judgment module 16 and a target video frame generation module 17.

[0070] A first noise latent code generating module 11 is configured to generate a first noise latent code of a target video frame based on an input text prompt;

[0071] a cross-attention map generating module 12 for generating a cross-attention map of each subject marker in the input text prompt based on the input text prompt and the first noise latent code;

[0072] A cross-attention map smoothing module 13 is used to perform Gaussian filtering on the cross-attention map to smooth the cross-attention map;

[0073] A second noise latent code generating module 14 is configured to update the first noise latent code based on the cross attention map, generate a second noise latent code, and replace the first noise latent code with the second noise latent code;

[0074] A denoising module 15 is configured to denoise the second noise latent code using a diffusion model with inter-frame attention control;

[0075] The iterative judgment module 16 is used to judge whether the total number of sampling steps reaches the predetermined number of steps. If so, the target video frame generation module 17 is entered; otherwise, if not, the cross attention map generation module 12 is returned.

[0076] The target video frame generating module 17 is configured to output the target video frame based on the second noise latent code.

[0077] Specifically, the working method of each module has been described in detail in the first embodiment and will not be repeated here.

[0078] Example 3

[0079] The application further provides a computer readable storage medium, which stores computer executable instructions for causing a computer to execute the method in the embodiment one.

[0080] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and the equivalent modifications or replacements are all included in the scope defined by the claims of the application.

Claims

1. A method for generating video from text, characterized in that: Including steps: S11: Generate the first noise latent code of the target video frame based on the input text prompt, specifically including: randomly sampling the first latent code Z from the standard Gaussian distribution t 1; for the Z t 1 Perform the DDIM reverse step of Δt step to obtain Z t '1; define the global motion direction, calculate and apply the global translation vector k for each frame, and apply the constructed motion flow 1:m to the Z t '1, get the corresponding sequence Z t '1:m, that is, for the first frame Z t '1 Perform m steps of gradually strengthening the translation deformation operation to obtain the Z of m frames t '1:m; perform the DDPM forward step on the deformed latent code to obtain the initial noise latent code sequence Z of the final m frames t 1:m, based on the Z t 1:m obtains the first noise latent code Z t ; S12: Based on the input text prompt and the first noise latent code, generating a cross-attention map for each subject marker in the input text prompt, wherein the cross-attention map is averaged by multiple attention maps with a resolution of 16*16, the cross-attention map includes multiple attention maps, each attention map corresponds to each subject marker, and the value of an element of the matrix of the cross-attention map represents the probability of a subject marker appearing in an image block; S13: Performing Gaussian filtering on the cross-attention map to smooth the cross-attention map; S14: updating the first noise latent code based on the cross attention map, generating a second noise latent code, and replacing the first noise latent code with the second noise latent code; S15: denoising the second noise latent code using a diffusion model with inter-frame attention control; S16: Determine whether the total number of sampled steps reaches a predetermined number of steps, and whether all subject marks reach a set minimum attention threshold. If so, execute step S17; otherwise, execute step S12; S17: Output the target video frame based on the second noise latent code.

2. The method according to claim 1, characterized in that The step S14 includes: Focus on the most neglected subject marker at the current time step and calculate the loss function L; Based on the loss function L, the current first noise latent code Z t Perform gradient update to obtain the second noise latent code Z t ′.

3. The method according to claim 1, characterized in that The step S15 includes: performing denoising by using a cross-frame attention mechanism.

4. A text-generated video device, characterized in that: include: The first noise latent code generation module is used to generate the first noise latent code of the target video frame based on the input text prompt, specifically comprising: randomly sampling the first latent code Z from the standard Gaussian distribution t 1; for the Z t 1 Perform the DDIM reverse step of Δt step to obtain Z t '1; define the global motion direction, calculate and apply the global translation vector k for each frame, and apply the constructed motion flow 1:m to the Z t '1, get the corresponding sequence Z t '1:m, that is, for the first frame Z t '1 Perform m steps of gradually strengthening the translation deformation operation to obtain the Z of m frames t '1:m; perform the DDPM forward step on the deformed latent code to obtain the initial noise latent code sequence Z of the final m frames t 1:m, based on the Z t 1:m obtains the first noise latent code Z t ; a cross-attention map generation module, configured to generate a cross-attention map for each subject marker in the input text prompt based on the input text prompt and the first noise latent code, wherein the cross-attention map is averaged by multiple attention maps with a resolution of 16*16, the cross-attention map comprises multiple attention maps, each attention map corresponds to each subject marker, and the value of an element of the matrix of the cross-attention map represents the probability of a subject marker appearing in an image block; A cross-attention map smoothing module is used to perform Gaussian filtering on the cross-attention map to smooth the cross-attention map; A second noise latent code generation module is configured to update the first noise latent code based on the cross attention map, generate a second noise latent code, and replace the first noise latent code with the second noise latent code; a denoising module, configured to denoise the second noise latent code using a diffusion model with inter-frame attention control; An iterative judgment module is used to judge whether the total number of sampling steps reaches a predetermined number of steps. If not, the module returns to the cross-attention map generation module. A target video frame generating module is configured to output the target video frame based on the second noise latent code.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 3.

Images