Video generation model training method, video generation model reasoning method and video generation model training device

Abstract

The invention provides a video generation model training method and device and a video generation model reasoning method and device. The video generation model training method comprises the steps that label noise is injected into a first hidden space variable converted from any video sample data to obtain a second hidden space variable; performing time causal cross attention calculation and visual cross attention calculation on the second hidden space variable to determine predicted noise according to a calculated third hidden space variable; and optimizing model parameters of the video generation model according to loss values calculated by the predicted noise and the label noise. In the training reasoning process of the video generation model, time causal cross attention calculation and visual cross attention calculation are adopted in an attention mechanism to process data information and space information in hidden space variables, the data calculation amount and the carrying amount are greatly reduced on the premise that the video generation quality is hardly influenced, and the video generation efficiency is improved. And the information processing efficiency of the video generation model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a video generation model training method, reasoning method and device. Background Art

[0002] The video generation model is a large AI model used to generate videos. In related technologies, the video generation model usually uses a global attention mechanism to process data information and spatial information in latent space variables during operation.

[0003] Due to the quadratic growth relationship of the attention mechanism's computational complexity, the computational cost of using a global attention mechanism in the model operation process is extremely high, resulting in long processing time and low processing efficiency.

[0004] In view of this, how to improve the processing efficiency of video generation models has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a video generation model training method, an inference method and a device, which are used to solve the problems of long time consumption and low processing efficiency in the prior art.

[0006] The present invention provides a video generation model training method, which injects label noise into a first latent space variable converted from any video sample data to obtain a second latent space variable; Performing temporal causal cross-attention calculation and visual cross-attention calculation on the second latent space variable to obtain a third latent space variable, so as to determine prediction noise according to the third latent space variable; The model parameters of the video generation model are optimized according to the loss value calculated by the predicted noise and the label noise.

[0007] According to a video generation model training method provided by the present invention, the step of performing temporal causal cross attention calculation and visual cross attention calculation on the second latent space variable to obtain a third latent space variable includes: Using the current network layer of the video generation model, a temporal causal cross attention calculation and a visual cross attention calculation are performed on the second latent space variable to obtain a first calculation result and a second calculation result respectively; Integrate the first calculation result and the second calculation result to obtain a new latent space variable, and input the new latent space variable into a target network layer, where the target network layer is the next network layer of the current network layer; Traverse all network layers of the video generation model and output the third latent space variables.

[0008] According to a video generation model training method provided by the present invention, the current network layer is used to perform temporal causal cross attention calculation on the second latent space variable, including: Initializing the second latent space variable to a full space point set according to the latent space frame number of the second latent space variable; Perform cross-attention calculation on each frame of full-space point in the full-space point set and some historical full-space points, and output the first calculation result.

[0009] According to a video generation model training method provided by the present invention, the cross-attention calculation is performed on each frame of the full-space point set and part of the historical full-space points, respectively, including: For the i-th frame full-space point, select a first number of frames of historical full-space points that are closest to the i-th frame full-space point in time sequence to perform the cross attention calculation, and obtain the i-th attention calculation result; Traverse the full-space points of each frame in the full-space point set to determine the first calculation result according to the cross-attention calculation results corresponding to all the full-space points.

[0010] According to a video generation model training method provided by the present invention, after selecting a first number of frame historical full-space points that are closest to the i-th frame full-space point in time sequence for cross-attention calculation for the i-th frame full-space point, the method further includes: Randomly select at least one historical full-space point that is before the i-n1th frame in time sequence; Perform cross attention calculation on all the historical full-space points, and include the calculation results into the i-th attention calculation result; Among them, n1 is the first number.

[0011] According to a video generation model training method provided by the present invention, after selecting a first number of frame historical full-space points that are closest to the i-th frame full-space point in time sequence for cross-attention calculation for the i-th frame full-space point, the method further includes: The i-th frame full-space points are cross-attended together with the first number of frames of historical full-space points, and the calculation result is included in the i-th attention calculation result.

[0012] According to a video generation model training method provided by the present invention, when selecting the first number of historical full-space points of the frames closest to the i-th frame full-space point in time sequence for the temporal causal cross-attention calculation for the i-th frame full-space point, if the i-n2-th frame does not exist, skip and continue to select the next frame full-space point in time sequence until the full-space point set is traversed or the first number of historical full-space points of the frames are selected; Among them, n2 is smaller than the first number.

[0013] According to a video generation model training method provided by the present invention, the current network layer is used to perform visual cross-attention calculation on the second latent space variable, including: Initializing the second latent space variable to a full frame set at a time point according to the total number of pixels of the second latent space variable; Each group of full frames at each time point in the set of full frames at each time point is subjected to cross-attention calculation with the full frames at other surrounding time points, and a second calculation result is output.

[0014] According to a video generation model training method provided by the present invention, each group of full frames at a time point in the full frame set at a time point is subjected to cross attention calculation with other full frames at other time points around each group, including: For the jth group of full frames at time points, select a second number of groups of adjacent full frames at time points that are closest in position to the jth group of full frames at time points to perform cross attention calculations, and obtain a jth attention calculation result; Each group of the full frames at the time points in the full frame set at the time points is traversed to determine the second calculation result according to the cross-attention calculation results corresponding to all the full frames at the time points.

[0015] According to a video generation model training method provided by the present invention, after selecting a second number of adjacent time point full frames that are closest in position to the jth group of time point full frames for cross attention calculation, the method further includes: Randomly selecting at least one group of full frames at other time points except the full frames at the adjacent time points that have been selected; Cross-attention calculations are performed on all the full frames of the adjacent time points and all the full frames of the other time points, and the calculation results are included in the j-th attention calculation results.

[0016] According to a video generation model training method provided by the present invention, after selecting a second number of adjacent time point full frames that are closest in position to the jth group of time point full frames for cross attention calculation, the method further includes: A cross-attention calculation is performed on the full frames of the j-th group of time points and all the full frames of the adjacent time points, and the calculation results are included in the j-th attention calculation results.

[0017] The present invention also provides a video generation model reasoning method, comprising: The video generation model generates a fourth latent space variable based on user input information and random Gaussian distribution latent space noise, and determines the prediction noise according to the latent space variable obtained after performing temporal causal cross-attention calculation and visual cross-attention calculation on the fourth latent space variable; Denoising the fourth latent space variable according to the predicted noise, and re-inputting the denoised latent space variable as a new fourth latent space variable into the video generation model until a preset stop condition is met, and outputting the iteratively denoised latent space variable; The decoder in the video generation model generates video data corresponding to the user input information according to the latent space variables after the iterative denoising.

[0018] The present invention also provides a video generation model training device, comprising: A noise injection unit, used for injecting label noise into a first latent space variable converted from any video sample data to obtain a second latent space variable; a noise prediction unit, configured to perform a temporal causal cross-attention calculation and a visual cross-attention calculation on the second latent space variable to obtain a third latent space variable, so as to determine a predicted noise according to the third latent space variable; A model optimization unit is used to optimize the model parameters of the video generation model according to the loss value calculated by the predicted noise and the label noise.

[0019] The present invention also provides a video generation model reasoning device, comprising: a noise identification unit, configured to generate a fourth latent space variable based on user input information and random Gaussian distribution latent space noise, and determine prediction noise according to the latent space variable obtained after performing temporal causal cross attention calculation and visual cross attention calculation on the fourth latent space variable; a denoising processing unit, configured to denoise the fourth latent space variable according to the predicted noise, and re-input the denoised latent space variable as a new fourth latent space variable into the video generation model until a preset stop condition is met, and output the iteratively denoised latent space variable; The video inference unit is used to generate video data corresponding to the user input information according to the latent space variables after the iterative denoising using a decoder in the video generation model.

[0020] The video generation model training method, reasoning method and device provided by the present invention process the data information and spatial information in the latent space variables by adopting temporal causal cross-attention calculation and visual cross-attention calculation in the attention mechanism during the training and reasoning process of the video generation model. This greatly reduces the amount of data calculation and transportation without affecting the quality of video generation, and effectively improves the information processing efficiency of the video generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a schematic diagram of the 3D global attention mechanism in the prior art.

[0023] Figure 2 It is a schematic diagram of the 2D+1D local attention mechanism in the prior art.

[0024] Figure 3 It is a flow chart of the video generation model training method provided by the present invention.

[0025] Figure 4 It is a schematic diagram of the 2D+1D global attention mechanism provided by the present invention.

[0026] Figure 5 It is a flow chart of the video generation model reasoning method provided by the present invention.

[0027] Figure 6 It is a structural schematic diagram of the video generation model training device provided by the present invention.

[0028] Figure 7 It is a structural schematic diagram of the video generation model inference device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0032] The main mode of the video generation model is through the diffusion model. Based on a latent noise, it gradually reduces noise and generates multiple frames of pictures, that is, videos. This process can be understood as starting from completely random (or highly noisy) data and gradually removing noise to the final video frame sequence.

[0033] At present, the model structure of the mainstream video generation model is a multi-module splicing implementation, mainly including the backbone model, the text control model and the variational autoencoder (VAE). Among them, the backbone model generally adopts the diffusion transformer (DiT) structure, including multiple series of network layers, each of which includes a spatial attention module, a visual cross attention module and an MLP module. By combining the generation process of the diffusion model with the architecture of the Transformer, it aims to improve the expressiveness and efficiency of the generation model. The powerful representation ability of the Transformer is used to better model the complex dependencies in the data, and the generation mechanism of the diffusion model is used to gradually generate high-quality data samples. In summary, the backbone model, as the computational core of the video generation model, removes the latent space noise by correlating the latent space variables (video data in the latent space) at different spatial points and time points through the network layer. The text control model is mainly used to inject text control information, which can be implemented using a bidirectional attention language understanding model or a multimodal understanding model. VAE is used to compress data and greatly reduce the computational complexity of the backbone model. The whole is divided into two parts: the encoder VAE encoder and the decoder VAE decoder. The VAE encoder compresses the original data (such as video) to the latent space size for training the backbone model to learn denoising; the VAE decoder converts the latent space variables back to the original data space for converting the latent space variables after denoising of the backbone model into video data during inference.

[0034] In the prior art, the backbone model mainly uses 3D global attention to realize the association between different spatial points and time points of latent space variables.

[0035] Figure 1 is a schematic diagram of the 3D global attention mechanism recorded in the prior art, such as Figure 1 As shown in the figure, the 3D global attention mechanism ensures the connection between all points in the latent space variables by mixing global data and spatial information. The self-attention calculation between points in the three dimensions of the latent space (including the two-dimensional spatial dimension and the one-dimensional temporal dimension) is performed together.

[0036] Because of the quadratic growth of the 3D attention mechanism, the computational cost of the 3D global attention method is extremely high in the field of high-resolution and long videos. Assume that the size of the latent space is , the computational amount and data handling amount of the bottleneck operator (i.e., the Q*K matrix multiplication calculation in the attention module) are: and Among them, F is the number of latent space frames, H is the latent space pixel height, W is the latent space pixel width, and D is the latent space dimension size.

[0037] Assuming that the resolution of all three dimensions is increased to twice the previous value, the amount of attention calculation and data handling will become the previous value. times. For example, the original The amount of computation corresponding to the hidden space size is , then when the resolution of all three dimensions is increased to twice the previous value, it means The amount of computation corresponding to the hidden space size is , so the 2x resolution corresponds to The amount of computing and data handling has increased by times.

[0038] To solve this problem, existing technologies also use a 2D+1D local attention method to achieve the association between different spatial points and time points in the latent space variables.

[0039] Figure 2 is a schematic diagram of the 2D+1D local attention mechanism recorded in the prior art, such as Figure 2 As shown in the figure, the 2D+1D local attention method is to independently mix partial and local data and spatial information in the attention module. Specifically, this method divides a three-dimensional latent space variable into a two-dimensional spatial dimension and a one-dimensional time dimension, and the attention module is divided into two parts, the spatial attention module and the visual cross attention module. The two independently perform self-attention calculations between points in the space of the same frame, and the self-attention calculations of the same spatial point between different frames. Generally speaking, this method can greatly reduce the amount of calculations, especially in long video tasks.

[0040] However, the 2D+1D local attention method can reduce the bottleneck operator calculation and data handling in the attention to and However, this attention method loses a lot of important data information and spatial information in the latent space variables, which may eventually lead to a significant decrease in the quality of the generated video, especially in long video tasks, where there will be obvious defects in the incoherence of the video content.

[0041] Considering that the existing video generation model has drawbacks in the process of associating different spatial points and time points of latent space variables (i.e., video data in latent space) in the attention module as the core calculation. The present invention provides a new video generation model, including in the process of model training and actual model reasoning, on the basis of the existing 2D+1D local attention method, by introducing a small amount of current global time information and spatial information that is most relevant to the current point, it can greatly improve the global association between latent space points in the 2D+1D attention mechanism.

[0042] Figure 3 It is a flow chart of the video generation model training method provided by the present invention, such as Figure 3 As shown, it mainly includes but is not limited to the following steps: Step 301 : inject label noise into a first latent space variable converted from any video sample data to obtain a second latent space variable.

[0043] Step 302: Perform temporal causal cross-attention calculation and visual cross-attention calculation on the second latent space variable to obtain a third latent space variable, so as to determine the prediction noise according to the third latent space variable.

[0044] Step 303: Optimize the model parameters of the video generation model according to the loss value calculated by the predicted noise and the label noise.

[0045] It should be noted that the model architecture of the video generation model adopted in the present invention is basically the same as the video generation model adopted in the prior art. The main difference is reflected in the specific training process of the model, especially in the generation of prediction errors using the backbone network. There are essential differences in the attention mechanisms performed by the spatial attention module and visual cross-attention in each network layer.

[0046] The following is a detailed description of the above method for training the video generation model.

[0047] In step 301, B video sample data of size [f,h,w,3] (where 3 is the number of color channels) are input to the video generation model to be trained. The VAE encoder in the video generation model receives the input video sample data and scales it into a low-dimensional first latent space variable Y of size [B,F,H,W,C]. Where f, h and w are the frame size, pixel height and pixel width of the video respectively; B is the batch size; F is the latent space frame number; H is the latent space pixel height; W is the latent space pixel width; C is the latent space channel size.

[0048] Prepare the label noise at time t for noise addition in advance. As a latent space variable, label noise can be implemented using the Denoising Diffusion Probabilistic Models (DDPM).

[0049] Inject the label noise at time t into the first latent space variable Y to obtain the second latent space variable : ; in, , is a random value; is a Gaussian noise with a mean of 0 and a variance of 1, and The size of is consistent with the first latent space variable Y, which is [B, F, H, W, C]; , , ,For example .

[0050] The label noise at time t obtained at this time will be used as the target of model training to guide model learning.

[0051] In step 302, the second latent space variable is obtained Perform attention learning calculations, including temporal causal cross-attention calculations and visual cross-attention calculations, to obtain the third latent space variables.

[0052] Figure 4 is a schematic diagram of the 2D+1D global attention mechanism provided by the present invention, such as Figure 4 As shown, in the spatial attention module of the present invention, a learning calculation of temporal causal cross attention is adopted, that is, the full space point of the current frame is associated with the historical full space point of the previous frame. At the same time, the learning calculation of visual cross attention is adopted in space, so that all frames corresponding to each group of time point full frames corresponding to the current point are only associated with the time point full frames corresponding to a part of the points around the point.

[0053] Through the above two cross-attention mechanisms, an approximate global correlation between any point and its most relevant point can be improved, while reducing the overall computational workload and data handling volume.

[0054] As an optional embodiment, before executing step 302, the second latent space variable Perform scale transformations, including: The second latent space variable After the latent space projection layer, the channel dimension C is converted into the latent space dimension D to obtain the latent space variable Y , whose size is [B, FHW, D], where B is the batch dimension; F*H*W is the token dimension; and D is the size of the latent space dimension. Then, the scale-transformed latent space variable Y Perform attention learning calculations to obtain the third latent space variables.

[0055] As an optional embodiment, the way of integrating the first calculation result and the second calculation result using the MLP module can be achieved through linear projection and activation function, for example, by integrating the information in the spatial dimension and the time dimension to generate a new latent space variable and re-input it into the next network layer. The new latent space variable contains a richer feature representation, which can better describe the video data and improve the generation quality of the model.

[0056] After the MLP module enlarges the latent space dimension D (for example, 4*D), it passes through a nonlinear activation function (such as GELU), and then the enlarged latent space dimension is projected back to the original size D.

[0057] After obtaining the final latent space variable output by the last network layer of the backbone model, it is input into the linear projection module to obtain the predicted noise output by the video generation model in this round of training. , whose size is [B, F*H*W, C], can be converted to [B, F, H, W, C], where Represents all parameters of the model.

[0058] In step 303, the noise is predicted by calculating The label noise at the previous input time t The error between them is taken as the optimization goal, and the model parameters of the video generation model are optimized with minimizing this error.

[0059] Among them, the prediction noise The label noise at the previous input time t The error between them can be calculated by the mean square error: .

[0060] During the training and inference process of the video generation model, the present invention processes the data information and spatial information in the latent space variables by adopting temporal causal cross-attention calculation and visual cross-attention calculation in the attention mechanism. This greatly reduces the amount of data calculation and transportation without almost affecting the quality of video generation, and effectively improves the information processing efficiency of the video generation model.

[0061] Based on the content of the above embodiment, as an optional embodiment, performing temporal causal cross attention calculation and visual cross attention calculation on the second latent space variable to obtain the third latent space variable specifically includes: Using the current network layer of the video generation model, a temporal causal cross attention calculation and a visual cross attention calculation are performed on the second latent space variable to obtain a first calculation result and a second calculation result respectively; Integrate the first calculation result and the second calculation result to obtain a new latent space variable, and input the new latent space variable into a target network layer, where the target network layer is the next network layer of the current network layer; Traverse all network layers of the video generation model and output the third latent space variables.

[0062] refer to Figure 4 As shown in Figure 1, the spatial attention module in each network layer of the video generation model during training mainly performs the following operations: Initialize the input latent space variable Y , according to the latent space variable Y The number of latent space frames F is initialized as multiple cross-attention latent space variables , and call it a full-space point, and its size becomes , and then perform a loop operation with i=1,2,…,F, including: For each full-space point of each frame, a temporal causal cross-attention calculation is performed on it and some historical full-space points of the previous frame, and the first calculation result is output. Temporal causal cross-attention ensures that there is a temporal causal relationship between the full-space point of the current frame and the historical full-space point of the previous frame. Therefore, it is different from the existing 2D+1D attention mechanism. In each round of training, for each full-space point of each frame, a cross-attention calculation is performed on it and some historical full-space points, which can effectively avoid information omission.

[0063] The size of the full-space points in each frame is converted from [B, F*H*W, D] to [B*F, H*W, D], that is, B*F is the batch dimension, H*W is the token dimension, and D is the latent space dimension.

[0064] Then, the cross-attention latent space variables are updated so that the first calculation result contains the cross-attention information of all spatial points of all frames.

[0065] Furthermore, in each network layer of the backbone network of the video generation model, for each group of full frames at each time point, visual cross-attention calculation is performed on it and the surrounding full frames at other time points, and a second calculation result is output.

[0066] Visual criss-cross attention reduces the amount of computation and preserves information in the temporal dimension. Then, the criss-cross attention latent space variables are iteratively updated to contain the visual criss-cross attention information of the full frame at all time points.

[0067] Finally, the first calculation result and the second calculation result are integrated using the MLP module, and the new latent space variables obtained after the integration can be input into the next network layer of the backbone network again.

[0068] In this way, by traversing all the network layers of the backbone network, the final third latent space variables can be obtained.

[0069] The video generation model training method provided by the present invention adopts a cross-attention calculation method in the backbone model. By introducing only a small amount of time and space information most relevant to the current point, the information in the spatial and temporal dimensions is integrated, the feature representation ability of the model is improved, and the global correlation between latent space points in the attention mechanism is greatly improved. Without affecting the quality of video generation, the repeated reading and writing of data is reduced and the amount of data calculation and transportation is greatly reduced.

[0070] As an optional embodiment, the using the current network layer to perform temporal causal cross attention calculation on the second latent space variable includes: Initializing the second latent space variable to a full space point set according to the latent space frame number of the second latent space variable; Perform cross-attention calculation on each frame of full-space point in the full-space point set and some historical full-space points, and output the first calculation result.

[0071] Among them, initializing the second latent space variable to a full space point set according to the latent space frame number of the second latent space variable has been described in detail in the previous embodiment, which will not be repeated here.

[0072] Optionally, performing cross attention calculation on the full space point of each frame and some historical full space points, and outputting the first calculation result mainly includes: ; Perform cross-attention calculation .in, query matrix; , , is the model weight matrix; Initialized criss-cross attention latent space variables (i.e., all spatial points of the frame); Key Matrix; Value matrix.

[0073] Based on the content of the above embodiment, as an optional embodiment, each frame of the full space point in the full space point set is cross-attention calculated with some historical full space points, specifically including: For the i-th frame full-space point, select a first number of frames of historical full-space points that are closest to the i-th frame full-space point in time sequence to perform the cross attention calculation, and obtain the i-th attention calculation result; Traverse the full-space points of each frame in the full-space point set to determine the first calculation result according to the cross-attention calculation results corresponding to all the full-space points.

[0074] In the present invention, latent space variables will pass through multiple network layers, and the spatial attention module in each network layer will perform temporal causal spatial cross-attention calculations.

[0075] Specifically, initialize the latent space variable Y , according to the number of latent space frames F, generate multiple full space points with a size of , and then perform a cyclic calculation of i=1,2,…,F, including: For the full space point of the i-th frame, take A cross-attention calculation is performed on the historical full-space point closest to the full-space point of the i-th frame to obtain a corresponding attention calculation result, which is recorded as the i-th attention calculation result.

[0076] As an optional embodiment, after the spatial attention module selects a first number of historical full-space points that are closest in time sequence to the full-space point of the i-th frame for cross-attention calculation, it further includes: Randomly select at least one historical full-space point that is before the i-n1th frame in time sequence; Perform cross attention calculation on all the historical full-space points, and include the calculation results into the i-th attention calculation result; Among them, n1 is the first number.

[0077] At least one historical full-space point is randomly selected from the frame before the i-n1th frame in time sequence, and the full-space point of the i-th frame is cross-attended with all randomly selected historical full-space points, and all historical full-space points are cross-attended, and the calculation results are included in the i-th attention calculation result. The i-th attention calculation result proposed at this time refers to the result of cross-attention calculation between the i-th frame full-space point and the first number of historical full-space points closest to the i-th frame full-space point. The i-th attention calculation result is spliced ​​into the latent space variable of the full-space point of the i-th frame to obtain a new latent space variable.

[0078] By traversing all the spatial points of all frames and repeating the above steps, the final i-th attention calculation result can be obtained.

[0079] The present invention increases the diversity of spatial associations in model learning by randomly selecting historical frames, generates higher quality video data, and makes the model more robust and adaptable to different video data. At the same time, randomly selecting historical frames effectively reduces the amount of calculation compared to the existing 3D global attention mechanism, and improves the efficiency of model training and reasoning.

[0080] As an optional embodiment, when performing temporal causal cross attention calculation, for the i-th frame full space point, selecting a first number of historical full space points that are closest to the i-th frame full space point in time sequence to perform cross attention calculation, it also includes: The i-th frame full-space point and the first number of historical full-space points are cross-attendant calculated together, and the calculation result is included in the i-th attention calculation result.

[0081] In this embodiment, for the full space point of the i-th frame, not only the points close to it are At the same time, other historical full-space points can be randomly selected, such as random The length is The frame group is composed of the historical full-space points. At the same time, in this embodiment, the temporal causal cross attention calculation can also include the self-attention calculation of the full-space point of the current i-th frame and itself.

[0082] First, select the n1 historical full-space points that are closest to the i-th frame in time sequence, perform cross-attention calculation on the full-space points of the i-th frame together with all the historical full-space points selected in the manner described in the above embodiment, obtain a new i-th attention calculation result, and splice it into the full-space point latent space variable of the i-th frame.

[0083] Finally, the full-space points of each frame in the full-space point set are traversed, and the final first calculation result is output.

[0084] This embodiment trains the video generation model by combining the current frame with the historical frame, so that the trained video generation model can generate clearer and more coherent video data. This is mainly reflected in the combination of the current frame and the historical frame, making the model more robust and able to adapt to different video data. In addition, because only the most recent historical frame is associated, future information leakage is avoided and the amount of calculation is reduced.

[0085] As another optional embodiment, when performing cross-attention calculation, not only the n1 frames of historical full-space points that are closest to the i-th frame full-space point in time sequence are considered, but also at least one randomly selected historical full-space point and the cross-attention calculation of the current frame full-space point are combined.

[0086] This embodiment adopts temporal causal cross attention calculation during training, and by combining random historical frames and current frames, the model is made more robust and can adapt to different video data generation requirements.

[0087] Based on the content of the above embodiment, as an optional embodiment, when selecting a first number of historical full-space points closest to the full-space point of the i-th frame in time sequence for cross-attention calculation, if there is no i-n2-th frame, skip selecting the next frame in time sequence until the full-space point set is traversed or the historical full-space points of the first number of frames are selected; Among them, n2 is smaller than the first number.

[0088] Specifically, for the full space point of the i-th frame, take the Frames, and Random before frame The length is If the frame does not exist, it is skipped. Figure 4 As an example, For 2, and Schematic diagram of the temporal causal spatial attention mechanism when both are equal to 1.

[0089] When i is 1, since the i-th frame full space point has no historical full space point, the number of historical full space points it corresponds to is 0. When i is 2, since the i-th frame full space point has only one historical full space point, the number of historical full space points it corresponds to is 1 (the value of n2 is 1 at this time). However, after i is greater than or equal to 3, When the value is 2, each full-space point in the i-th frame has two historical full-space points.

[0090] is the actual number of frames taken, for example ,until The maximum value of .

[0091] In this embodiment, the size of the i-th attention calculation result obtained after the temporal causal cross attention calculation can be converted from [B, F*H*W, D] to [B*F, H*W, D], where B*F is the batch dimension, H*W is the token dimension, and D is the latent space dimension.

[0092] Finally, calculate , perform cross attention calculation .

[0093] in, Query Matrix ， 、 、 is the model weight matrix, The full space points obtained after initialization (which can be called cross-attention latent space variables), Key Matrix, Value matrix, Refers to the first calculation result output.

[0094] It should be noted that, in this embodiment, the temporal causal cross-attention calculation also includes the results of the self-attention calculation of the full space point of the current frame and itself.

[0095] The present invention provides three ways to implement cross-attention calculation, which respectively realize the effectiveness of the spatial attention module by randomly selecting historical frames, combining the current frame with historical frames, and combining random historical frames with the current frame. These methods reduce the amount of calculation while improving the generation quality and are interpretable, providing a variety of new solutions for video generation tasks.

[0096] Based on the content of the above embodiment, as an optional embodiment, the current network layer is used to perform visual cross attention calculation on the second latent space variable, including: Initializing the second latent space variable to a full frame set at a time point according to the total number of pixels of the second latent space variable; Each group of full frames at each time point in the set of full frames at each time point is subjected to cross-attention calculation with other full frames at other time points around each group, and a second calculation result is output.

[0097] refer to Figure 4 As shown in the figure, during the training of the video generation model, the visual cross-attention calculation performed by each network layer of the video generation model mainly includes the following steps: First, according to the latent space variable Y The total number of pixels is further initialized into a time point full frame set consisting of a time point full frame, whose size is , and then perform loop operations of j=1,2,…,H*W, including: For each group of full frames at each time point, perform visual cross-attention calculation on it and the surrounding full frames at other time points, and output the second calculation result.

[0098] Among them, visual cross attention reduces the amount of calculation and retains information in the time dimension. Finally, the cross attention latent space variable (i.e., full frame at all time points) is updated to contain the cross attention information of all full frames at all time points.

[0099] Among them, for each group of full frames at the time point, cross attention calculation is performed on it and the surrounding full frames at other time points, and a second calculation result is output, which mainly includes: ; Among them, cross attention calculation , here is the model weight matrix.

[0100] Based on the content of the above embodiment, as an optional embodiment, each group of full frames at a time point in the full frame set at the time point is subjected to cross attention calculation with other full frames at other time points around each group, and a second calculation result is output, specifically including: For the jth group of full frames at time points, select a second number of groups of adjacent full frames at time points that are closest in position to the jth group of full frames at time points to perform cross attention calculations, and obtain a jth attention calculation result; Each group of the full frames at the time points in the full frame set at the time points is traversed to output the second calculation result according to the cross-attention calculation results corresponding to all the full frames at the time points.

[0101] This embodiment provides a visual cross-attention calculation mechanism based on full frames of adjacent time points, which adopts a block spatial information interaction method, so it can also be called a block cross-attention calculation mechanism.

[0102] First, for the jth group of full frames at time points, select the second number of groups of adjacent full frames at time points that are closest to the jth group of full frames at time points, such as Figure 4 As shown, the second number is 5*5-1=24.

[0103] It should be noted that if there are less than 24 adjacent full frames at time points within a 5*5 range with the jth group of full frames at time points as the center, then the existing ones can be selected. Figure 4 Taking the top group of the block time attention part shown in as an example, for the group of time point full frames in the upper left corner, there are only 8 adjacent time point full frames within its 5*5 range (plus itself, 9 time point full frames).

[0104] The j-th group of full frames at time points and the selected adjacent full frames at time points are cross-attended to splice the j-th attention calculation result into the j-th group of full frames at time points in latent space to obtain new latent space variables. Finally, each group of full frames at time points in the set of full frames at time points is traversed to repeat the cross-attention calculation, and the second calculation result can be output.

[0105] As another optional embodiment, during the training process, when performing visual cross attention calculation for the jth group of time point full frames, selecting a second number of adjacent time point full frames that are closest in position to the jth group of time point full frames for cross attention calculation, it also includes: Randomly selecting at least one group of full frames at other time points except the full frames at the adjacent time points that have been selected; Cross-attention calculations are performed on all the full frames of the adjacent time points and all the full frames of the other time points, and the calculation results are included in the j-th attention calculation results.

[0106] In this embodiment, a visual cross-attention calculation is provided by combining a random time point and a second number of groups of adjacent time point full frames selected in the previous embodiment that are closest to the jth group of time point full frames, and the specific implementation steps include but are not limited to: Initialize the input latent space variable Y , obtain multiple full-space point sets consisting of full-space points, whose size is , and then repeat the cycle of j=1,2,…, H*W.

[0107] First, for the jth group of full frames at time point, take the surrounding points (e.g. ), as well as the full frame at each time point and random outside these points The size is of points (e.g. ), if the point does not exist, it is skipped.

[0108] is the actual number of frames taken, here The maximum value of ,in is the number of randomly selected full frames at all other time points except the full frames at adjacent time points.

[0109] The size of the input latent space variable can be converted from [B, F*H*W, D] to [B*F, H*W, D] to meet the computational requirements of the cross-attention mechanism, where B*F is the batch dimension, H*W is the token dimension, and D is the latent space dimension.

[0110] Finally, calculate , perform visual cross-attention calculation .

[0111] in, Query Matrix ， 、 、 is the model weight matrix, The full frame of the time point obtained after initialization (which can be called the cross-attention latent space variable), Key Matrix; Value matrix, Refers to the second calculation result output.

[0112] As an optional embodiment, regarding the visual cross-attention calculation, when for the jth group of time point full frames, a second number of groups of adjacent time point full frames that are closest in position to the jth group of time point full frames are selected for cross-attention calculation, it also includes: performing cross-attention calculation on the jth group of time point full frames and all the adjacent time point full frames, and incorporating the calculation results into the jth attention calculation result.

[0113] In short, the video generation model training method provided by the present invention also integrates the full frame at the current time point and its own self-attention calculation when performing visual cross-attention calculation.

[0114] The video generation model training method provided by the present invention, the above three implementation schemes respectively realize the effectiveness calculation of visual cross attention by randomly selecting full frames at adjacent time points, combining full frames at adjacent time points and full frames at random time points, and combining full frames at adjacent time points, full frames at random time points and self-attention calculation. These methods improve the generation quality while reducing the amount of calculation during model training, and are interpretable, providing a variety of new solutions for model training tasks.

[0115] In the actual training process, each network layer of the backbone network will perform cross-attention processing on the input latent space features, and use the MLP layer to enlarge the latent space dimension D of the obtained latent space vector, for example, 4*D, and then pass it through a nonlinear activation function (such as GELU), and then project the enlarged latent space dimension back to the original dimension D.

[0116] Through this nonlinear transformation of the MLP layer, the expressiveness of the video generation model is further enhanced, and richer feature information of the latent space vector is captured, so that the trained video generation model can learn more complex feature representations and adapt to different video data and scenes. At the same time, by processing in each network layer, the amount of calculation of the subsequent network layer can be reduced, and the efficiency of model training and reasoning can be improved.

[0117] The latent space variable output by the last network layer of the backbone model is input into the linear projection module to obtain the predicted noise. , with dimensions of [B,F*H*W,C], can be converted to [B,F,H,W,C], where Represents all model parameters of the video generation model.

[0118] Finally, we can predict the noise With label noise The prediction loss between them is used to optimize the model parameters of the video generation model, which will not be elaborated here.

[0119] Figure 5 It is a flow chart of the video generation model reasoning method provided by the present invention, such as Figure 5 As shown, the video generation model reasoning method provided by the present invention mainly includes but is not limited to the following steps: Step 501, the video generation model generates a fourth latent space variable based on user input information and random Gaussian distribution latent space noise, and determines the prediction noise according to the latent space variable obtained after performing temporal causal cross attention calculation and visual cross attention calculation on the fourth latent space variable; Step 502, denoising the fourth latent space variable according to the predicted noise, and re-inputting the denoised latent space variable as a new fourth latent space variable into the video generation model until a preset stop condition is met, and outputting the iteratively denoised latent space variable; Step 503: The decoder in the video generation model generates video data corresponding to the user input information according to the latent space variables after the iterative denoising.

[0120] In step 501, the user input text prompt or reference image is converted into user input information to guide the video generation content in the model reasoning process. A random Gaussian distributed latent space noise is generated and the user input information is integrated into the latent space noise to generate a fourth latent space variable. The dimension of the fourth latent space variable is consistent with the representation dimension of the video data in the latent space.

[0121] Initialize the fourth latent space variable to a set of full-space points, each full-space point corresponds to a frame in the video, and perform temporal causal cross-attention calculations on each full-space point with a certain number of frames before it (e.g., n1 frames), that is, the current full-space point is associated with at least one full-space point before it, but not with future full-space points. Of course, earlier full-space points can also be randomly selected for cross-attention calculations to increase the diversity of model learning.

[0122] Furthermore, the fourth latent space variable needs to be initialized to a set of full frames at each time point, where each full frame at each time point corresponds to a set of spatial points in the video. For each full frame at each time point, a visual cross-attention calculation is performed with a certain number of full frames at the surrounding time points, that is, the current spatial point is associated with the surrounding spatial points. Similarly, further spatial points can be randomly selected for cross-attention calculation to increase the diversity of model learning.

[0123] Finally, the results of temporal causal cross attention calculation and visual cross attention calculation are integrated and nonlinearly transformed through the MLP module to enhance the feature representation capability. The integrated latent space variables are then input into the linear projection module to obtain the prediction noise.

[0124] In step 502, the Gaussian distribution mean of the latent space variable is calculated according to the current iteration step number t and the model parameter θ. The latent space noise is denoised according to the calculated Gaussian distribution mean. The denoised latent space variable is used as the new latent space noise and re-input into the video generation model.

[0125] Repeat the above steps until the preset stopping condition is met (for example, the preset number of iterations is reached or the difference between the predicted noise and the actual noise is less than a threshold).

[0126] In step 503, the latent space variables after iterative denoising are input into the decoder of the VAE, decoded into video data, and the generated video data is output, corresponding to the user input information.

[0127] The video generation model inference method provided by the present invention processes data information and spatial information in latent space variables by adopting temporal causal cross-attention calculation and visual cross-attention calculation in the attention mechanism. Under the premise of almost no impact on the video generation quality, it greatly reduces the amount of data calculation and transportation, and effectively improves the information processing efficiency of the video generation model.

[0128] As an optional embodiment, when the present invention uses the trained video generation model to perform actual reasoning, it mainly executes the following steps.

[0129] First, conduct Diffusion model denoising inference, where It refers to the maximum number of iterations of the diffusion model in the diffusion process, that is, the number of iterations required to transform the data into Gaussian distribution noise. The denoising inference process mainly includes: exist When , the Gaussian distribution latent space noise with a mean of 0 and a variance of 1 is initialized and used as the input to the video generation model, denoted as the first latent space noise, whose size is [B, F, H, W, C]; When , the input is the latent space noise after iterative denoising output by the video generation model in the previous iteration. Among them, B is the batch size, F is the number of latent space frames, H and W are the latent space pixel height and width respectively, and C is the number of latent space channels.

[0130] The first latent space noise is input into the trained video generation model to obtain the predicted noise output by the video generation model, whose size is [B, F*H*W, C]. Since this process has been described in detail in the video generation model training method provided in the above embodiment, it will not be repeated here.

[0131] In each iteration, the Gaussian distribution mean of the latent space video is calculated according to the current iteration step t and model parameter θ. : ; in, and represent the noise proportion and cumulative noise proportion at the tth step in the diffusion model, respectively.

[0132] Then, according to the current iteration step t and model parameter θ, the latent space variable Y of the iterative denoising is: ; in, ~N(0,1) represents Gaussian noise with mean 0 and variance 1.

[0133] The denoised latent space noise is then used as the new first latent space noise and re-input into the video recognition model for further denoising.

[0134] The above steps are iterated until the preset termination condition is met. In each iteration, the latent space variable Y is updated according to the current iteration step and model parameters to gradually approach the real video data.

[0135] The preset termination condition may be t=1.

[0136] Finally, the decoder of the VAE model enlarges the latent space variable Y of size [B, F, H, W, C] and converts it into B video data of size [f, h, w, 3]. Among them, f, h, w are the number of frames, height and width of the video respectively, and 3 refers to the number of color channels.

[0137] The video generation method provided by the present invention adopts a cross-attention calculation method in the backbone model, and greatly improves the global correlation between latent space points in the attention mechanism by introducing only a small amount of time and space information that is most relevant to the current point. Without affecting the quality of video generation, it reduces repeated reading and writing of data and greatly reduces the amount of data calculation and transportation.

[0138] Figure 6 is a structural diagram of the video generation model training device provided by the present invention, such as Figure 6 As shown, mainly including but not limited to: The noise injection unit 61 is used to inject label noise into the first latent space variable converted from any video sample data to obtain a second latent space variable.

[0139] The noise prediction unit 62 is used to perform temporal causal cross-attention calculation and visual cross-attention calculation on the second latent space variable to obtain a third latent space variable, so as to determine the predicted noise according to the third latent space variable.

[0140] The model optimization unit 63 is used to optimize the model parameters of the video generation model according to the loss value calculated by the predicted noise and the label noise.

[0141] It should be noted that the video generation model training device provided by the present invention can execute the video generation model training method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0142] The video generation model training device provided by the present invention processes data information and spatial information in latent space variables by adopting temporal causal cross-attention calculation and visual cross-attention calculation in the attention mechanism during the model training process. Under the premise of almost no impact on the quality of video generation, the amount of data calculation and transportation is greatly reduced, and the information processing efficiency of the video generation model is effectively improved.

[0143] Figure 7 is a schematic diagram of the structure of the video generation model inference device provided by the present invention, such as Figure 7 As shown, mainly including but not limited to: A noise identification unit 71 is used to generate a fourth latent space variable based on user input information and random Gaussian distribution latent space noise, and determine the prediction noise according to the latent space variable obtained after performing temporal causal cross attention calculation and visual cross attention calculation on the fourth latent space variable; A noise reduction processing unit 72 is used to reduce the noise of the fourth latent space variable according to the predicted noise, and re-input the denoised latent space variable as a new fourth latent space variable into the video generation model until a preset stop condition is met, and output the iteratively denoised latent space variable; The video inference unit 73 is used to generate video data corresponding to the user input information according to the latent space variables after the iterative denoising using the decoder in the video generation model.

[0144] The video generation model inference device provided by the present invention processes data information and spatial information in latent space variables by adopting temporal causal cross-attention calculation and visual cross-attention calculation in the attention mechanism. Under the premise of almost no impact on the quality of video generation, it greatly reduces the amount of data calculation and transportation, and effectively improves the information processing efficiency of the video generation model.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video generation model training method, characterized in that: include: Inject label noise into the first latent space variable converted from any video sample data to obtain a second latent space variable; Performing temporal causal cross-attention calculation and visual cross-attention calculation on the second latent space variable to obtain a third latent space variable, so as to determine prediction noise according to the third latent space variable; The model parameters of the video generation model are optimized according to the loss value calculated by the predicted noise and the label noise.

2. The video generation model training method according to claim 1, characterized in that: The step of performing temporal causal cross attention calculation and visual cross attention calculation on the second latent space variable to obtain a third latent space variable includes: Using the current network layer of the video generation model, a temporal causal cross attention calculation and a visual cross attention calculation are performed on the second latent space variable to obtain a first calculation result and a second calculation result respectively; Integrate the first calculation result and the second calculation result to obtain a new latent space variable, and input the new latent space variable into a target network layer, where the target network layer is the next network layer of the current network layer; Traverse all network layers of the video generation model and output the third latent space variables.

3. The video generation model training method according to claim 2, characterized in that: The current network layer is used to perform temporal causal cross attention calculation on the second latent space variable, including: Initializing the second latent space variable to a full space point set according to the latent space frame number of the second latent space variable; Perform cross-attention calculation on each frame of full-space point in the full-space point set and some historical full-space points, and output the first calculation result.

4. The video generation model training method according to claim 3, characterized in that: The cross-attention calculation is performed on each frame of the full-space point in the full-space point set and on some historical full-space points, respectively, including: For the i-th frame full-space point, select a first number of frames of historical full-space points that are closest to the i-th frame full-space point in time sequence to perform the cross attention calculation, and obtain the i-th attention calculation result; Traverse the full-space points of each frame in the full-space point set to determine the first calculation result according to the cross-attention calculation results corresponding to all the full-space points.

5. The video generation model training method according to claim 4, characterized in that: After selecting a first number of frame historical full-space points that are closest to the i-th frame full-space point in time sequence for cross-attention calculation for the i-th frame full-space point, the method further includes: Randomly select at least one historical full-space point that is before the i-n1th frame in time sequence; Perform cross attention calculation on all the historical full-space points, and include the calculation results into the i-th attention calculation result; Among them, n1 is the first number.

6. The video generation model training method according to any one of claims 4-5, characterized in that: After selecting a first number of frame historical full-space points that are closest to the i-th frame full-space point in time sequence for cross-attention calculation for the i-th frame full-space point, the method further includes: The i-th frame full-space points are cross-attended together with the first number of frames of historical full-space points, and the calculation result is included in the i-th attention calculation result.

7. The video generation model training method according to claim 4, characterized in that: When selecting the first number of frames of historical full-space points closest to the i-th frame full-space point in time sequence for the temporal causal cross-attention calculation for the i-th frame full-space point, if the i-n2-th frame does not exist, skip selecting the next frame of full-space points in time sequence until the full-space point set is traversed or the first number of frames of historical full-space points are selected; Among them, n2 is smaller than the first number.

8. The video generation model training method according to claim 2, characterized in that: The visual cross attention calculation is performed on the second latent space variable using the current network layer, including: Initializing the second latent space variable to a full frame set at a time point according to the total number of pixels of the second latent space variable; Each group of full frames at each time point in the set of full frames at each time point is subjected to cross-attention calculation with other full frames at other time points around each group, and a second calculation result is output.

9. The video generation model training method according to claim 8, characterized in that: Each group of full frames at a time point in the full frame set at the time point is subjected to cross attention calculation with other full frames at other time points around the full frames, including: For the jth group of full frames at time points, select a second number of groups of adjacent full frames at time points that are closest in position to the jth group of full frames at time points to perform cross attention calculations, and obtain a jth attention calculation result; Each group of the full frames at the time points in the full frame set at the time points is traversed to determine the second calculation result according to the cross-attention calculation results corresponding to all the full frames at the time points.

10. The video generation model training method according to claim 9, characterized in that: After selecting a second number of groups of adjacent time point full frames that are closest in position to the jth group of time point full frames for cross attention calculation, the method further includes: Randomly selecting at least one group of full frames at other time points except the full frames at the adjacent time points that have been selected; Cross-attention calculations are performed on all the full frames of the adjacent time points and all the full frames of the other time points, and the calculation results are included in the j-th attention calculation results.

11. The video generation model training method according to any one of claims 9 to 10, characterized in that: After selecting a second number of groups of adjacent time point full frames that are closest in position to the jth group of time point full frames for cross attention calculation, the method further includes: A cross-attention calculation is performed on the full frames of the j-th group of time points and all the full frames of the adjacent time points, and the calculation results are included in the j-th attention calculation results.

12. A video generation model reasoning method, characterized in that: include: The video generation model generates a fourth latent space variable based on user input information and random Gaussian distribution latent space noise, and determines the prediction noise according to the latent space variable obtained after performing temporal causal cross-attention calculation and visual cross-attention calculation on the fourth latent space variable; Denoising the fourth latent space variable according to the predicted noise, and re-inputting the denoised latent space variable as a new fourth latent space variable into the video generation model until a preset stop condition is met, and outputting the iteratively denoised latent space variable; The decoder in the video generation model generates video data corresponding to the user input information according to the latent space variables after the iterative denoising.

13. A video generation model training device, characterized in that: include: A noise injection unit, used for injecting label noise into a first latent space variable converted from any video sample data to obtain a second latent space variable; a noise prediction unit, configured to perform a temporal causal cross-attention calculation and a visual cross-attention calculation on the second latent space variable to obtain a third latent space variable, so as to determine a predicted noise according to the third latent space variable; A model optimization unit is used to optimize the model parameters of the video generation model according to the loss value calculated by the predicted noise and the label noise.

14. A video generation model inference device, characterized in that: include: a noise identification unit, configured to generate a fourth latent space variable based on user input information and random Gaussian distribution latent space noise, and determine prediction noise according to the latent space variable obtained after performing temporal causal cross attention calculation and visual cross attention calculation on the fourth latent space variable; a denoising processing unit, configured to denoise the fourth latent space variable according to the predicted noise, and re-input the denoised latent space variable as a new fourth latent space variable into the video generation model until a preset stop condition is met, and output the iteratively denoised latent space variable; The video inference unit is used to generate video data corresponding to the user input information according to the latent space variables after the iterative denoising using a decoder in the video generation model.

Images