Video generation method and apparatus, device, and medium

Abstract

Embodiments of the present application provide a video generation method and apparatus, a device, and a medium. The method can be applied to the technical field of video content generation, and is used for improving the video generation quality. The method comprises: acquiring a sample video frame sequence from a sample video, determining a first step count and sample original noise, and performing data noise addition processing on the sample video frame sequence to obtain video input data; inputting the video input data, a sample text encoded feature corresponding to sample description text, and first embedding information corresponding to the first step count into an initial generation model; and performing noise prediction on the sample video frame sequence by means of M spatiotemporal residual components and M spatiotemporal attention components in the initial generation model to obtain sample predicted noise, correcting a network parameter in the initial generation model on the basis of the sample original noise and the sample predicted noise, and determining the initial generation model comprising the corrected network parameter as a video generation model.

Description

Video generation method, device, equipment and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 9, 2024, with application number 202410179048.7 and application name “Video Generation Method, Device, Equipment and Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, device, and medium. Background Art

[0003] With the rapid development of AI-generated content technology and diffusion model research, multimedia content generation research has received widespread attention. For example, using AI-generated content technology and diffusion models, it is possible to combine text with video data that matches the text content.

[0004] In the current field of video generation, a Markov chain with a diffusion step can be defined. By continuously adding random noise to the data until pure Gaussian noise data is obtained, the inverse diffusion process is learned, and the video is generated through inverse noise reduction inference. However, the learning capabilities of current video modeling are limited by the modeling scale. Video is a continuous time series with long-term dependencies between video frames. If the number of video frames to be generated changes, the generated video quality will decrease. Summary of the Invention

[0005] The embodiments of the present application provide a video generation method, apparatus, device, and medium, which can improve the quality of video generation.

[0006] An embodiment of the present application provides a video generation method, including:

[0007] Obtain a sample video and a sample description text corresponding to the sample video, obtain a sample video frame sequence in the sample video, and determine the first step number and sample original noise for the sample video frame sequence;

[0008] According to the first step number and the original noise of the sample, the sample video frame sequence is subjected to data noise processing to obtain video input data, and the sample description text is subjected to text encoding processing to obtain the sample text encoding features;

[0009] Obtain an initial generative model; the initial generative model includes M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model are alternately connected, where M is an integer greater than 1;

[0010] Input the video input data into the first of the M spatiotemporal residual components, input the first embedding information corresponding to the first step into the M spatiotemporal residual components, and input the sample text encoding features into the M spatiotemporal attention components;

[0011] In M spatiotemporal residual components and M spatiotemporal attention components, noise prediction is performed on the video input data according to the sample text encoding features and the first embedding information to obtain sample prediction noise;

[0012] According to the original noise of the sample and the predicted noise of the sample, the network parameters of the initial generation model are corrected, and the initial generation model including the corrected network parameters is determined as the video generation model; the video generation model is used to generate video data that matches the video description text.

[0013] In one aspect, an embodiment of the present application provides a video generation device, including:

[0014] A sample acquisition module is used to obtain a sample video and a sample description text corresponding to the sample video, obtain a sample video frame sequence in the sample video, and determine the first step number and sample original noise for the sample video frame sequence;

[0015] The sample noise adding module is used to perform data noise adding processing on the sample video frame sequence according to the first step number and the original noise of the sample to obtain the video input data, and perform text encoding processing on the sample description text to obtain the sample text encoding feature;

[0016] A model acquisition module is used to obtain an initial generative model; the initial generative model includes M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model are alternately connected, where M is an integer greater than 1;

[0017] A sample input module is used to input the video input data into the first spatiotemporal residual component of the M spatiotemporal residual components, input the first embedding information corresponding to the first step into the M spatiotemporal residual components, and input the sample text encoding features into the M spatiotemporal attention components;

[0018] A noise prediction module is configured to perform noise prediction on the video input data in the M spatiotemporal residual components and the M spatiotemporal attention components according to the sample text encoding features and the first embedding information to obtain sample prediction noise;

[0019] The parameter correction module is used to correct the network parameters of the initial generation model based on the original sample noise and the sample predicted noise, and determine the initial generation model containing the corrected network parameters as the video generation model; the video generation model is used to generate video data that matches the video description text.

[0020] In one aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in the above aspect of the embodiment of the present application.

[0021] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor, so that a computer device with a processor executes the method provided in the above aspect of the embodiment of the present application.

[0022] According to one aspect of the present application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in the above aspect.

[0023] In an embodiment of the present application, during the training phase of the initial generative model, a sample video and a sample description text corresponding to the sample video can be obtained, a sample video frame sequence for training the initial generative model is obtained from the sample video, the first step number is randomly sampled, and sample original noise is generated; based on the first step number and the sample original noise, the sample video frame sequence is subjected to noise processing to obtain video input data. The initial generative model may include M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components are alternately connected, where M is an integer greater than 1. The video input data, the sample text encoding features corresponding to the sample description text, and the first embedding information corresponding to the first step number can all be input into the initial generative model, and the sample video frame sequence is subjected to noise prediction through the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model to obtain sample predicted noise. The entire initial generative model can be trained by the mean square error loss between the sample predicted noise and the sample original noise. Through the spatiotemporal residual component and spatiotemporal attention component in the initial generation model, the initial generation model can learn the temporal action information in videos of different time scales, improve the comprehensiveness of the temporal action information, and thus improve the generation quality of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a schematic diagram of a network architecture provided in an embodiment of the present application;

[0025] FIG2 is a schematic diagram of a video generation scenario provided by an embodiment of the present application;

[0026] FIG3 is a flow chart of a video generation method according to an embodiment of the present application;

[0027] FIG4 is a schematic diagram of a structure of multi-time scale input data provided by an embodiment of the present application;

[0028] FIG5 is a schematic diagram of the structure of an initial generation model provided in an embodiment of the present application;

[0029] FIG6 is a schematic diagram of the structure of a spatiotemporal attention component in an initial generation model provided by an embodiment of the present application;

[0030] FIG7 is a flow chart of the training phase of an initial generation model provided in an embodiment of the present application;

[0031] FIG8 is a schematic diagram of training an initial generation model provided in an embodiment of the present application;

[0032] FIG9 is a second flow chart of a video generation method provided in an embodiment of the present application;

[0033] FIG10 is a sampling diagram of a video generation model provided in an embodiment of the present application;

[0034] FIG11 is a flow chart of a sampling phase of a video generation model provided in an embodiment of the present application;

[0035] FIG12 is a schematic structural diagram of a video generating device provided in an embodiment of the present application;

[0036] FIG13 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0038] The embodiments of the present application specifically relate to video content generation technology in artificial intelligence content generation technology, and specifically to a video generation diffusion model based on multi-time scale modeling (hereinafter referred to as a video generation model for ease of understanding). In the training phase, videos with different frame numbers are used for model training, so that the video generation model can capture long-term and short-term temporal action information during the training process; the learning modeling of temporal action information is more comprehensive, and the video generation model obtained by training can generate videos of different lengths, and can ensure that the video images at any length have consistency and action coherence, thereby improving the generation quality of the video. Among them, the diffusion model can include a diffusion process and an inverse diffusion process. The diffusion process represents the process of gradually adding noise to an image (or video), and the inverse diffusion process represents the process of gradually denoising the noise and restoring the original image (or video) from the noise.

[0039] Please refer to Figure 1, which is a schematic diagram of the structure of a network architecture provided in an embodiment of the present application. The network architecture may include a server 10d and a terminal cluster. The terminal cluster may include one or more terminal devices. There is no limit on the number of terminal devices included in the terminal cluster. As shown in Figure 1, the terminal cluster may specifically include terminal device 10a, terminal device 10b, and terminal device 10c, etc.; all terminal devices in the terminal cluster (for example, terminal device 10a, terminal device 10b, and terminal device 10c, etc.) can be connected to the server 10d through a network connection, so that each terminal device can exchange data with the server 10d through the network connection.

[0040] The terminal devices of the terminal cluster may include smart phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), wearable devices (such as smart watches, smart bracelets, etc.), smart voice interaction devices, smart home appliances (such as smart TVs, etc.), vehicle-mounted devices, aircraft and other electronic devices. This application does not limit the type of terminal devices. It is understandable that each terminal device in the terminal cluster shown in Figure 1 can be installed with a business application. When the business application runs in each terminal device, it can interact with the server 10d shown in Figure 1 above. Among them, the business application running in each terminal device can correspond to an independent client or an embedded sub-client integrated in a client. This application does not limit this.

[0041] Business applications may include, but are not limited to, browsers, in-vehicle applications, smart home applications, entertainment applications (e.g., gaming applications), multimedia applications (e.g., video applications, short video applications), conferencing applications, and other applications with video content generation capabilities. If the terminal device included in the terminal cluster is an in-vehicle device, then the in-vehicle device may be an intelligent terminal in a smart transportation scenario, and the business applications running on the in-vehicle device may be referred to as in-vehicle applications.

[0042] Among them, server 10d can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not limit the type of server.

[0043] It should be understood that the business applications installed in the various terminal devices shown in Figure 1 can call the pre-trained video generation model, such as the trained video generation model can be put online in the business application. If object A wants to produce video data for publication in the content interaction platform, then the object A can upload the video description text it wants to express in the business application installed in the terminal device (for example, terminal device 10a); optionally, object A can also set the length of the video it wants to generate in the business application. Among them, the video description text involved in the embodiment of the present application may refer to a natural sentence used to describe the video content that the object wants to produce, and the video description text may include but is not limited to: text description of the product, story or emotional expression text, film and television scripts, press releases, teaching content, medical information, etc.

[0044] The terminal device 10a can obtain the video description text uploaded by object A and call the video generation model in the business application; it can generate random Gaussian noise according to the video length set by object A, and use the Gaussian noise as a starting point, combined with the video text encoding features corresponding to the video description text given by object A, and continuously use the above-mentioned video generation model to predict noise; the Gaussian noise input into the video generation model is denoised by the predicted video prediction noise to obtain a video with less noise, until the noise is completely removed, and the target video data that conforms to the video description text is obtained; object A can publish the generated target video data to the content interaction platform.

[0045] For ease of understanding, the aforementioned video generation model may refer to a video generation diffusion model for generating a video that conforms to a given video description text content. The video generation diffusion model in the training phase (i.e., the video generation diffusion model that has not completed training) may be referred to as the initial generation model, and the video generation diffusion model in the sampling phase (the video generation diffusion model that has completed training) may be referred to as the video generation model. Among them, the initial generation model and the video generation model are video generation diffusion models at different stages, that is, the initial generation model and the video generation model have the same network structure. It should be understood that the video generation process involved in the embodiments of the present application may include the training phase and the sampling phase of the model.

[0046] During the training phase, different degrees of Gaussian noise can be gradually added to the input sample video frame sequence. The process of adding Gaussian noise to the sample video frame sequence can be called the forward process of the sample video frame (also called the diffusion process). Then, the noisy data, the number of steps t (here, the number of steps t can refer to the number of times the noise of the noisy data is diffused based on the sample video frame sequence), and the sample text encoding features encoded by the text encoder (also called text conditions) can be input into the initial generation model. The noise added by the number of steps t (i.e., the sample prediction noise) is predicted by the initial generation model. The loss of the sample prediction noise and the noise actually added in the forward process is calculated to optimize the initial generation model.

[0047] In the sampling stage, the trained video generation model is used for iterative denoising. Starting from random Gaussian noise, combined with the video text encoding features corresponding to the given video description text, the trained video generation model is continuously used to predict the noise. Through gradual denoising, Gaussian noise can be converted into video data with less noise until the noise is completely removed, and the target video data that conforms to the video description text is obtained, thus achieving the task of generating video.

[0048] It should be noted that the process of training the initial generation model using video data and its corresponding video description text, and the process of automatically generating target video data using the video generation model can be performed by a computer device, which can be the server 10d in the network architecture shown in Figure 1 (the server 10d can be the background server corresponding to the business application), or it can be any terminal device in the terminal cluster, or it can be a computer program (including program code, for example, an application client of a business application installed in the terminal device), etc. This application does not limit this.

[0049] Among them, the trained video generation model, as a video content automatic generation technology, can be applied in application scenarios such as advertising creative production, interactive media content generation, film and television production, news reporting, online education, health care, games, etc., which can improve video generation efficiency; or it can also be applied to computer vision scenarios such as text generation video, picture generation video, video editing, text generation picture, text generation, video content understanding, etc. This application does not limit the application scenarios of the video generation model.

[0050] For example, in the context of creative advertising, video generation models can help advertisers (e.g., advertisers) quickly create product introduction videos. They only need to provide a text description of the product, and the video generation model automatically generates video data containing product images, usage scenarios, and user reviews. This allows consumers to more intuitively understand the product, enhancing its image and appeal.

[0051] In interactive media content generation scenarios, video generation models can be used to create interesting and engaging video content. For example, any subject can input textual content expressing a story or emotion. The video generation model can then convert this textual content into target video data containing relevant images, music, and animations, making the interactive content more vivid and interesting, and improving the interactivity of the subject within the content interaction platform.

[0052] In film and television production scenarios, given the content of a script, a video generation model can be used to convert the script into high-quality video footage, thereby improving the production efficiency of movies or animations.

[0053] In news reporting scenarios, once a news editor completes a press release, the video generation model can automatically convert the text content of the press release into a video news item containing relevant images, video clips, background music, and other elements, improving news distribution efficiency. Furthermore, users can choose to read the text news item or watch the video news item generated from the press release, providing a richer way to obtain information.

[0054] In online education scenarios, teachers only need to provide the text of the teaching content. The video generation model can automatically convert the text into a teaching video containing relevant images, animations, explanations, and other elements. This not only saves teachers time, but also makes the teaching content more engaging and helps students better understand and remember the teaching content.

[0055] In healthcare scenarios, doctors can use video generation models to explain complex medical information such as diseases and treatment plans. For example, a doctor can input a text description of a disease, and the video generation model can generate a video containing relevant images, animations, and explanations to help patients and their families better understand the disease and treatment plans.

[0056] In a game scenario, a game developer can input a textual requirement, and the video generation model can generate a video containing virtual game characters, animations, and game scenes, thereby improving the developer's game development efficiency. Alternatively, a game enthusiast can input a textual description of a game level, and the video generation model can generate a video containing the game scene animation, level-by-level steps, and narration, thereby helping game enthusiasts improve the efficiency of producing game commentary videos. Optionally, this video generation model can also be applied to other game scenarios, which is not limited in this application.

[0057] The following describes the application process of the trained video generation model using an interactive media content generation scenario as an example. Referring to Figure 2, which is a schematic diagram of a video generation scenario provided by an embodiment of the present application, when subject A wants to create video data on a content interaction platform, it can activate the content interaction platform on terminal device 20a and trigger the "Video Production" control on the content interaction platform. At this point, terminal device 20a can respond to the triggering operation on the "Video Production" control and display a video production page 20b on the content interaction platform. The video production page 20b may include a text input area 20c and a "Generate Video" control 20d.

[0058] Object A can edit the text content in the text input area 20c. After editing is completed, a trigger operation can be performed on the "Generate Video" control 20d in the video production page 20b. At this time, the terminal device 20a can respond to the trigger operation on the "Generate Video" control 20d, and determine the text content edited by object A in the text input area 20c as the video description text 20e for producing video data, and call the video generation model 20g in the content interaction platform. The network structure of the video generation model 20g will be described in subsequent steps.

[0059] The terminal device 20a can randomly initialize a Gaussian noise 20f (also called initialization noise). The dimension of the Gaussian noise 20f can be expressed as B'×S"×3×H×W, where B' can be expressed as the number of videos to be generated, S" can be expressed as the number of frames of the duration of the video to be generated (also called the number of video frames), H can be expressed as the height of the video to be generated, and W can be expressed as the width of the video to be generated. The number of steps t corresponding to the Gaussian noise 20f can be sampled, and the step encoding information corresponding to the step t can be obtained. The step encoding information corresponding to the step t has the same dimension as the above-mentioned Gaussian noise 20f. Among them, the number of steps t here can be expressed as the degree of noise addition, and the total number of diffusion steps T can be pre-set for the video generation model. S , the number of steps t can be from [1, T S ] is any integer randomly sampled from ; for example, assuming the total number of diffusion steps T S If it is set to 1000, the number of steps t can be any value between 1 and 1000. The terminal device 20a can use a text encoder to perform text encoding processing on the video description text 20e to obtain the video text encoding feature corresponding to the video description text 20e.

[0060] Furthermore, the Gaussian noise 20f, the video text encoding features corresponding to the video description text 20e, and the step encoding information corresponding to the step number t can be input into the video generation model 20g; through the forward process of the video generation model 20g, the video prediction noise is obtained; based on the video prediction noise, the Gaussian noise 20f can be denoised to obtain noisy video data corresponding to the step number t-1, and the noisy video data corresponding to the step number t-1 can be input into the video generation model 20g, and the above denoising process can be repeated. Optionally, or based on the video prediction noise, the noisy video data corresponding to the step number t-τ can be sampled, where τ can be any integer greater than 1, which can improve the sampling efficiency of the video. By gradually denoising the Gaussian noise 20f, the target video data 20h that is noise-free and consistent with the video description text 20e is generated, which can improve the video generation efficiency. The target video data 20h finally generated can be displayed in the video production page 20b. Object A can trigger the "Publish" control 20i in the video production page 20b to publish the target video data 20h in the content interaction platform so that users in the content interaction platform can participate in the interaction of the target video data 20h and improve the interactivity of the published content in the content interaction platform.

[0061] It should be understood that the number of steps t in the training phase can represent the degree of noise addition; the number of steps t in the sampling phase can represent the degree of denoising required. Regardless of whether it is in the training phase or the sampling phase, the same number of steps t has the same meaning. Optionally, for ease of understanding, the number of steps t in the training phase can be called the diffusion step number, and the number of steps t in the sampling phase can be called the denoising step number. This application does not limit this.

[0062] Please refer to Figure 3, which is a flowchart of a video generation method provided in an embodiment of the present application. It is understandable that the video generation method can be executed by a computer device, which can be a server (server 10d as shown in Figure 1) or a terminal device (any terminal device in the terminal cluster as shown in Figure 1), and this application does not limit this. As shown in Figure 3, the video generation method can include the following steps S101 to S106:

[0063] Step S101 , obtaining a sample video and a sample description text corresponding to the sample video, obtaining a sample video frame sequence in the sample video, and determining the first step number and sample original noise for the sample video frame sequence.

[0064] In the embodiment of the present application, during the training phase, a batch of videos and corresponding video text descriptions can be collected to form (video, text) pairs for training the initial generative model. For ease of understanding, in the embodiment of the present application, the videos used to train the initial generative model can be referred to as sample videos, and the video description text corresponding to the sample videos can be referred to as sample description text.

[0065] The sample video can be subjected to frame extraction processing, and an initial video frame sequence with a fixed number of frames can be extracted from the sample video. The initial video frame sequences corresponding to one or more sample videos can form a sample video frame sequence tensor. For example, the frame extraction interval duration corresponding to the sample video can be determined based on the video frame number threshold and the video duration corresponding to the sample video; the frame extraction interval duration can represent the time interval between two consecutive video frames extracted from the sample video. According to the frame extraction interval duration, multiple video frames are obtained from the sample video (such as extracting a fixed number of video frames from the sample video), and the multiple video frames are combined into an initial video frame sequence; the number of video frames contained in the initial video frame sequence is equal to the video frame number threshold, and the video frame number threshold can represent the maximum number of video frames that the initial generation model can support, or be understood as the maximum length of the supported video frames. The video frame number threshold can be customized according to the actual scene requirements, and this application does not limit this.

[0066] A sample video frame sequence for training the initial generative model can be obtained from the initial video frame sequence; wherein the sample video frame sequence can be a subsequence of any initial video frame sequence whose number of video frames is less than or equal to a video frame number threshold, and the subsequence here can refer to a sequence that includes all or part of the video frames in the initial video frame sequence. In other words, the embodiment of the present application does not limit the number of video frames included in the sample video frame sequence for training the initial generative model. For example, sample video frame sequences at multiple time scales can be obtained from a single initial video frame sequence.

[0067] Optionally, the initial video frame sequence corresponding to one or more sample videos can form a video frame sequence tensor of dimension B×S0×3×H×W, where B, S0, H, and W represent the batch size, the threshold number of video frames supported by the initial generative model, the video frame height, and the video frame width, respectively. During the training phase of the initial generative model, the initial generative model can be trained in a distributed manner. For each training step, the initial input (sample video frame sequence) of the initial generative model on each GPU (Graphics Processing Unit) (rank, which can be represented as a group of memory chips) can be an initial video frame sequence of dimension B×S0×3×H×W. To achieve multi-timescale modeling, each rank randomly extracts sample video frame sequences with varying numbers of video frames from an initial video frame sequence of dimensions B×S0×3×H×W. The dimensions of this sample video frame sequence can be recorded as B×S×3×H×W, where S∈[1, S0] represents any value less than or equal to the video frame number threshold S0, representing the number of video frames selected from the initial video frame sequence. In other words, the initial generative model can be trained using sample video frame sequences with varying numbers of video frames. This allows the initial generative model to efficiently capture both long-term and short-term temporal action information during training, providing a more comprehensive learning model for temporal action information.

[0068] Please refer to Figure 4, which is a schematic diagram of the construction of a multi-time-scale input data provided by an embodiment of the present application. After extracting an initial video sequence with a fixed S0 frame from a sample video, sample video frame sequences of different lengths can be randomly extracted from the initial video frame sequence, that is, sample video frame sequences with different numbers of video frames can be randomly extracted to obtain multi-time-scale input data for training the initial generation model. As shown in Figure 4, the multi-time-scale input data may include a sample video frame sequence 30b, a sample video frame sequence 30c, a sample video frame sequence 30d, a sample video frame sequence 30e, etc. with different numbers of video frames. These multi-time-scale input data can be used as the initial input of the initial generation model 30a, and the initial generation model 30a can be used to perform noise prediction on each sample video frame sequence to obtain the sample prediction noise corresponding to each sample video frame sequence.

[0069] For example, after noise prediction is performed on sample video frame sequence 30b, sample predicted noise 30i can be obtained, and the sample predicted noise 30i has the same dimension as the sample video frame sequence 30b; after noise prediction is performed on sample video frame sequence 30c, sample predicted noise 30h can be obtained, and the sample predicted noise 30h has the same dimension as the sample video frame sequence 30c; after noise prediction is performed on sample video frame sequence 30d, sample predicted noise 30g can be obtained, and the sample predicted noise 30g has the same dimension as the sample video frame sequence 30d; after noise prediction is performed on sample video frame sequence 30e, sample predicted noise 30f can be obtained, and the sample predicted noise 30f has the same dimension as the sample video frame sequence 30e. In other words, the dimensions of the input and output of the initial generation model 30a are the same. The network structure of the initial generation model 30a and the noise prediction process of the initial generation model 30a will be described in subsequent steps.

[0070] Furthermore, data noise processing can be performed on each of the above sample video frame sequences to obtain noisy data; that is, during the training phase, noise can be added to the sample video frame sequence used to train the initial generative model, and the noisy data can be input into the initial generative model. In this embodiment of the application, the noisy data can be referred to as video input data. Specifically, the diffusion parameter corresponding to the initial generative model can be obtained (which can be denoted as T S ), according to the diffusion parameter T S , the first step number corresponding to the sample video frame sequence can be sampled; the diffusion parameter is used to characterize the total number of diffusion steps of the initial generation model, and the first step number is less than or equal to the diffusion parameter. Gaussian noise with the same dimension as the sample video frame sequence can be generated and determined as the sample original noise corresponding to the sample video frame sequence.

[0071] In a feasible implementation, a step number t1 (for ease of understanding, the step number t1 here can be referred to as the first step) can be randomly sampled according to the denoising diffusion probabilistic model (DDPM) to randomly generate a Gaussian noise with the same dimension as the sample video frame sequence. The Gaussian noise here can be used as the sample original noise added to the sample video frame sequence. The first step number t1 can be used to represent the degree of noise added to the sample video frame sequence. The first step number t1 can be [1, T S ] any integer value in the range, where T S It can be a pre-set diffusion parameter, the diffusion parameter T S It can be used to characterize the total number of diffusion steps of the initial generation model; for example, the total number of diffusion steps (diffusion parameter T S ) is set to 1000, the first step number t1 can be any value in [1, 1000]. For example, t1 can be any value such as 100, 500, 900, 1000, etc. The sample original noise is randomly sampled from the standard normal distribution N(0, 1) to generate a Gaussian noise with the same dimension as the sample video frame sequence.

[0072] DDPM can include two processes: forward process and reverse process. The forward process can also be called diffusion process, which can be understood as the process of S times (T here S It can refer to the total number of diffusion times set in advance, for example, T S The value can be any value such as 1000, 500, etc., and this application does not limit this.) Noise addition, each diffusion adds a small random noise, gradually adding noise to the input original image or video into a normal distribution, and turning the original image or video into an isotropic noise. The forward process can be understood as making the data noisy, and the reverse process can be understood as a denoising process. Therefore, the reverse process can be called a denoising process. S The noise is removed through a backward prediction, and multiple iterations gradually restore the noise to the original image or video without noise. In other words, the reverse process can be understood as the process of generating data. DDPM can estimate the noise through the forward and backward processes, thereby restoring the noise to the original image or video.

[0073] Among them, the key to DDPM is to predict noise. The embodiment of the present application can use a conditional model (for example, a 3DU-Net obtained by multi-time scale modeling) as a model for predicting noise. By training the 3D U-Net model, the noise predicted by it (for example, sample prediction noise) is close to the actual added noise (for example, sample original noise). The 3D U-Net model adopts a network structure that includes downsampling and upsampling; for example, the network structure of the 3D U-Net model can include but is not limited to: Convolutional Neural Networks (CNN), Feedforward Neural Network, Long Short-Term Memory (LSTM), Recurrent Convolutional Neural Network (RCNN), Attention Mechanism, Variational Autoencoder (VAE), a deformation of any of the above network structures, or a combination of any two or more of the above network structures, etc. The embodiment of the present application does not limit the network structure of the 3D U-Net model.

[0074] It can be understood that the initial generation model in the embodiment of the present application can be considered as a diffusion model, and its training phase is essentially the training process of the 3DU-Net model.

[0075] Step S102: Based on the first step and the original noise of the sample, the sample video frame sequence is subjected to data noise addition processing to obtain video input data, and the sample description text is subjected to text encoding processing to obtain sample text encoding features.

[0076] Specifically, the sequence weight corresponding to the sample video frame sequence can be determined based on the first step, and the original noise weight corresponding to the original noise of the sample can be determined based on the sequence weight, where the sum of the squares of the sequence weight and the original noise weight is 1. The product of the sequence weight and the sample video sequence, and the product of the original noise weight and the original noise of the sample are added together to obtain video input data, which can be input into the initial generation model.

[0077] Among them, when setting the diffusion parameter T S (ie, the total number of diffusion steps), the diffusion parameter T S Pre-set hyperparameters This hyperparameter It can be called noise schedule; where β tIndicates the diffusion rate corresponding to the step number t. The noise schedule can include the diffusion rate corresponding to each step number. t can take values ​​of 1, 2, ..., T S In the forward process, the diffusion rate β t It will gradually increase with the number of steps, and with the diffusion rate β t As increases, the proportion of noise will become larger and larger, which can be roughly regarded as a gradual increase from 0 to 1.

[0078] The above sequence weight can be determined by the first step number and the diffusion rate corresponding to all steps less than the first step number; when the first step number is recorded as t1, the sequence weight can be expressed as The original noise weight can be expressed as For example, when the first step number is 100, the sequence weight can refer to the result obtained by multiplying the diffusion rates corresponding to the first 100 steps and taking the square root of the multiplication result. For the sample video frame sequence x0 obtained above, the sample video frame sequence x0 can be subjected to data noise processing based on the first step number t1 of random sampling and the randomly generated sample original noise (which can be recorded as ε) to obtain the video input data x0 with noise. t1 The video input data x t1 It can be shown as formula (1): t1 =α t1 x0+σ t1 ∈,∈~N(0,1) (1)

[0079] Among them, in obtaining the sequence weight α t1 and the original noise weight σ t1 Afterwards, the sample video frame sequence and the sample original noise can be weighted and summed according to formula (1) to obtain the video input data x for input into the initial generation model. t1 .

[0080] For the sample description text corresponding to the sample video, the sample description text can be subjected to text feature extraction by a text encoder to obtain the sample text encoding features corresponding to the sample description text. In one or more embodiments, in the process of using a text encoder to extract text features from the sample description text, the sample description text can be subjected to word segmentation processing to obtain multiple unit characters (the unit characters here can be single characters or single words in the sample description text), and then each unit character can be vectorized to obtain a vector representation corresponding to each unit character. For example, one-hot (unique hot code) can be used to vectorize each unit character, or word2vec (a word vector model) can be used to vectorize each unit character, which is not limited in this application. The vector representations corresponding to each unit character contained in the sample description text can be combined to obtain the initial text features corresponding to the sample description text; the initial text features are input into the text encoder, and the initial text features are subjected to feature extraction (such as encoding processing) by the text encoder, and the sample text encoding features corresponding to the sample description text can be output.

[0081] Among them, the network structure of the text encoder may include but is not limited to the current text processing model, such as CLIP (Contrastive Language-Image Pre-training, a multimodal model of image + language), T5 (Transfer Text-to-Text Transformer), BERT (Bidirectional Encoder Representation from Transformers, a natural language processing model) structure, BART structure (a natural language processing model improved on the basis of BERT), RNN (Recurrent Neural Network, recurrent neural network), LSTM (Long Short-Term Memory, long short-term memory network, etc., or it can be a deformation of any of the above natural language processing networks, or a combination of any two or more of the above networks, etc. The network structure of the text encoder of this application is not limited. The text encoder involved in the embodiment of the present application can be a pre-trained text encoder, such as a text encoder that has been trained in advance; or it can be a text encoder that needs to be trained in the training phase of the initial generation model, and this application does not limit this.

[0082] The processing of the sample description text by the text encoder may include but is not limited to: the sample description text may be divided into D unit characters, and the unit word vectors corresponding to the D unit characters are obtained; D is a positive integer, such as D can be 1, 2, .... According to the semantic information of the D unit characters in the sample description text, the text vectors corresponding to the D unit characters are obtained; according to the text positions of the D unit characters in the sample description text, the position vectors corresponding to the D unit characters are obtained; the unit word vectors, the text vectors and the position vectors are combined (the combination here can be understood as vector addition) to obtain the initial text features corresponding to the sample description text; the initial text features are input into the text encoder, and the initial text features are encoded by the text encoder to obtain the sample text encoding features corresponding to the sample description text.

[0083] Among them, the computer device can convert each unit character in the sample description text into a word vector (i.e., unit word vector) by querying the word vector table, wherein the word vector table can contain word vectors corresponding to all commonly used characters. The word vector table can be understood as a "dictionary" containing all commonly used character vectors, and the unit character can refer to a single word or a single word in the sample description text. The text vector can be used to characterize the global semantic information of the sample description text and can be fused with the unit word vector; since the semantic information carried by the unit characters appearing in different positions of the sample description text is different (for example, the semantic information carried by "I wait for you" and "You wait for me" is different), a different position vector can be attached to each of the D unit characters in the sample description text for distinction; then, the unit word vector, text vector, and position vector can be added, and the result of the addition is used as the initial text feature corresponding to the sample description text.

[0084] Step S103, obtaining an initial generation model; the initial generation model includes M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generation model are alternately connected.

[0085] In step S104, the video input data is input into the first spatiotemporal residual component of the M spatiotemporal residual components, the first embedding information corresponding to the first step is input into the M spatiotemporal residual components, and the sample text encoding features are input into the M spatiotemporal attention components.

[0086] In the embodiment of the present application, the training phase of the initial generation model can be understood as the training of the 3D U-Net model. The 3D U-Net in the initial generation model may include M spatiotemporal residual components and M spatiotemporal attention components. These M spatiotemporal residual components and M spatiotemporal attention components can be alternately connected in the 3D U-Net of the initial generation model. M is an integer greater than 1. The embodiment of the present application does not limit the value of M, that is, the depth of the 3D U-Net. The 3D U-Net in the initial generation model can be an encoder (downsampling)-decoder (upsampling) structure. Among them, the alternating connection between the M spatiotemporal residual components and the M spatiotemporal attention components can be understood as every two spatiotemporal residual components being connected through a spatiotemporal attention component; similarly, except for the last spatiotemporal attention component in the initial generation model, every two spatiotemporal attention components in the remaining M-1 spatiotemporal attention components can be connected through a spatiotemporal residual component. For example, assuming that the initial generation model includes 3 spatiotemporal residual components and 3 spatiotemporal attention components (M is 3 in this case), then the connection relationship between these 3 spatiotemporal residual components and 3 spatiotemporal attention components can be expressed as: the 1st spatiotemporal residual component → the 1st spatiotemporal attention component → the 2nd spatiotemporal residual component → the 2nd spatiotemporal attention component → the 3rd spatiotemporal residual component → the 3rd spatiotemporal attention component.

[0087] The above-mentioned video input data can be used as the input data of the 3D U-Net in the initial generation model. Of course, in addition to the video input data, the input data of the 3D U-Net can also include sample text encoding features and the first embedding information (time embedding) corresponding to the first step. Among them, the video input data can be directly input into the first spatiotemporal residual component in the initial generation model; the first embedding information can be input into each spatiotemporal residual component in the initial generation model to guide the noise prediction degree of the 3D U-Net in the initial generation model; the sample text encoding features can be input into each spatiotemporal attention component in the initial generation model, which can be used to guide the prediction of the 3D U-Net noise in the initial generation model. If the sample text encoding features corresponding to the sample description text are introduced in the training stage of the initial generation model, the noise prediction of the initial generation model can be called conditional generation under text.

[0088] Optionally, during the training phase, the sample description text can be randomly dropped with a certain probability p. The drop operation here can be understood as changing the sample description text to an empty string, or it can be understood as setting the sample text encoding feature to zero and changing it to a zero vector. At this time, the noise prediction of the initial generation model can be called unconditional generation without text; the above probability p can be any value set according to actual needs, and this application does not limit this.

[0089] Among them, the sample text encoding features corresponding to the sample description text can be used to guide the noise prediction of 3D U-Net, and the training of the initial generation model supports noise prediction under text conditions. 3D U-Net shares parameters, so time embedding is needed to remind the model of the current number of steps, which is used to guide the degree of sample prediction noise output by 3D U-Net. The first step can be converted into an initial diffusion vector, such as using one-hot (unique hot code) to represent the first step as a vector, or using other encoding methods to represent the first step as a vector, which is not limited in this application. Furthermore, the initial diffusion vector corresponding to the first step can be encoded through one or more fully connected layers to obtain the first embedding information corresponding to the first step, and the first embedding information here can refer to the step encoding information corresponding to the first step.

[0090] It can be understood that the above-mentioned video input data, first embedding information and sample text encoding features can be input into the initial generation model separately; or the video input data and the first embedding information can be spliced, and the sample text encoding features and the spliced ​​features can be input into the initial generation model separately. This application does not limit this.

[0091] Please refer to Figure 5, which is a structural diagram of an initial generation model provided by an embodiment of the present application. The embodiment of the present application takes the initial generation model as an example, in which 7 spatiotemporal residual components and 7 spatiotemporal attention components (the value of M in this case is 7) are included to describe the network structure of the initial generation model. Among them, the above-mentioned 7 spatiotemporal residual components and 7 spatiotemporal attention components can constitute a 3D U-Net model in the initial generation model. As shown in Figure 5, the 7 spatiotemporal residual components in the initial generation model 40a are respectively spatiotemporal residual component 40b, spatiotemporal residual component 40d, spatiotemporal residual component 40f, spatiotemporal residual component 40h, spatiotemporal residual component 40j, spatiotemporal residual component 40m and spatiotemporal residual component 40p. The 7 spatiotemporal attention components in the initial generation model 40a are respectively spatiotemporal attention component 40c, spatiotemporal attention component 40e, spatiotemporal attention component 40g, spatiotemporal attention component 40i, spatiotemporal attention component 40k, spatiotemporal attention component 40n and spatiotemporal attention component 40q. Among them, these 7 spatiotemporal residual components and 7 spatiotemporal attention components are alternately connected. It is understandable that each spatiotemporal residual component in the initial generation model 40a can have the same network structure, and each attention component in the initial generation model 40a can have the same network structure.

[0092] For ease of understanding, the structures of the spatiotemporal residual component 40d and the spatiotemporal attention component 40e in the initial generative model 40a are described below using the spatiotemporal residual component 40d and the spatiotemporal attention component 40e as examples. As shown in FIG5 , the spatiotemporal residual component 40d may include a spatial 2D convolution layer (which may be referred to as a spatial convolution layer) and a temporal 1D convolution layer (which may be referred to as a temporal convolution layer). The temporal convolution layer supports input of arbitrary length and is therefore compatible with video input data of different frame numbers, allowing the initial generative model to capture both long-term and short-term temporal action information during training. Assuming that the input data of the spatiotemporal residual component 40d has a dimension of B×S×C×H×W, where B represents the batch size, S represents the number of video frames in the sample video frame sequence, C represents the number of channels, H represents the video height, and W represents the video width; then the input data of dimension B×S×C×H×W can be transformed (reshaped) into data of dimension (B×S)×C×H×W, and the data of dimension (B×S)×C×H×W is input to the spatial convolution layer of the spatiotemporal residual component 40d. The output of the spatial convolution layer in the spatiotemporal residual component 40d can be transformed into data of dimension (B×H×W)×C×S, and the data of dimension (B×H×W)×C×S is input to the temporal convolution layer of the spatiotemporal residual component 40d. Finally, the output of the spatiotemporal residual component 40d has a dimension converted to B×S×C×H×W.

[0093] It should be noted that the number of spatial convolution layers and temporal convolution layers in each spatiotemporal residual component contained in the initial generation model 40a can be one or more, and the present application does not limit the specific structure of the spatial convolution layers and temporal convolution layers in each spatiotemporal residual component. Residual connections can also be performed between the various spatiotemporal residual components in the initial generation model 40a. Specifically, residual connections can be performed between two spatiotemporal residual components with symmetrical positions in the initial generation model 40a. For example, the spatiotemporal residual component 40b and the spatiotemporal residual component 40p have symmetrical positions in the initial generation model 40a. The spatiotemporal residual component 40b and the spatiotemporal residual component 40p can be residually connected, that is, the output of the spatiotemporal residual component 40b can be used as the input of the spatiotemporal residual component 40p. Spatiotemporal residual component 40d and spatiotemporal residual component 40m have symmetrical positions in the initial generative model 40a. A residual connection can be performed between spatiotemporal residual component 40d and spatiotemporal residual component 40m, that is, the output of spatiotemporal residual component 40d can serve as the input of spatiotemporal residual component 40m. Spatiotemporal residual component 40f and spatiotemporal residual component 40j have symmetrical positions in the initial generative model 40a. A residual connection can be performed between spatiotemporal residual component 40f and spatiotemporal residual component 40j, that is, the output of spatiotemporal residual component 40f can serve as the input of spatiotemporal residual component 40j.

[0094] As shown in Figure 5, the spatiotemporal attention component 40e can include a spatial attention subcomponent and a temporal attention subcomponent. The temporal attention subcomponent supports input of arbitrary length and is therefore compatible with video input data of different frame numbers, allowing the initial generative model to capture both long-term and short-term temporal action information during training. Assuming that the input data of the spatiotemporal attention component 40e has a dimension of B×S×C×H×W, the input data of dimension B×S×C×H×W (the input data in this case can be the output of the previous spatiotemporal residual component) can be transformed (reshaped) into data of dimension (B×S)×C×(H×W), and the data of dimension (B×S)×C×(H×W) is input into the spatial attention subcomponent of the spatiotemporal attention component 40e. The output of the spatial attention subcomponent in the spatiotemporal attention component 40e can be transformed into data with a dimension of (B×H×W)×C×S, and the data with a dimension of (B×H×W)×C×S can be input into the temporal attention subcomponent of the spatiotemporal attention component 40e. Finally, the dimension of the output of the spatiotemporal attention component 40e is converted to B×S×C×H×W.

[0095] Among them, each spatiotemporal attention component in the initial generation model 40a can be a Transformer Block (Transformer module, Transformer is a model that uses the attention mechanism to improve the model training speed), the spatial attention subcomponent in each spatiotemporal attention component can be a spatial Transformer Block, and the temporal attention component can be a temporal Transformer Block. Optionally, the network structure of the spatiotemporal attention component in the initial generation model 40a is not limited to the above-mentioned Transformer Block, or can be a Conformer module (Transformer module based on convolution enhancement), etc., which is not limited in this application.

[0096] Step S105: In the M spatiotemporal residual components and the M spatiotemporal attention components, noise prediction is performed on the video input data according to the sample text encoding features and the first embedding information to obtain sample prediction noise.

[0097] Among them, after the video input data, the sample text encoding features and the first embedding information are input into the initial generation model, the data processing process of the video input data, the sample text encoding features and the first embedding information in the initial generation model can be called a noise prediction process. Specifically, each spatiotemporal residual component in the initial generation model can have the same structure, and each spatiotemporal attention component can also have the same structure. That is to say, the processing operations of the M spatiotemporal residual components in the initial generation model on their respective input data can be considered similar, and the processing operations of the M spatiotemporal attention components on their respective input data can also be considered similar. For ease of understanding, the following description takes the i-th spatiotemporal residual component and the i-th attention component in the initial generation model as an example.

[0098] In one or more embodiments, the first embedding information and the input data of the i-th spatiotemporal residual component can be feature extracted by the i-th spatiotemporal residual component among the M spatiotemporal residual components to obtain the sample residual features of the i-th spatiotemporal residual component; i is a positive integer less than or equal to M. The sample text encoding features and the sample residual features of the i-th spatiotemporal residual component can be subjected to attention processing by the i-th spatiotemporal attention component among the M spatiotemporal attention components to obtain the sample attention features of the i-th spatiotemporal attention component, and the sample attention features of the last spatiotemporal attention component among the M spatiotemporal attention components (which can be referred to as the M-th spatiotemporal attention component) are determined as the sample prediction noise. Wherein, when i is 1, the input data of the i-th spatiotemporal residual component includes the video input data and the first embedding information; when i is not 1, the input data of the i-th spatiotemporal residual component includes the first embedding information and the sample attention features of the i-1-th spatiotemporal attention component.

[0099] Among them, the i-th spatiotemporal residual component in the initial generation model includes a spatial convolution layer and a temporal convolution layer. When i is 1, the input data of the i-th spatiotemporal residual component includes video input data and first embedding information, such as the spatiotemporal residual component 40b shown in Figure 5. When i is greater than 1 and less than or equal to M / 2+1, the input data of the i-th spatiotemporal residual component includes the output features of the i-1-th spatiotemporal attention component (which can be called sample attention features) and the first embedding information; as shown in Figure 5, the spatiotemporal residual component 40d, the spatiotemporal residual component 40f and the spatiotemporal residual component 40h. When i is greater than M / 2+1, the input data of the i-th spatiotemporal residual component includes the sample attention features and the first embedding information of the i-1-th spatiotemporal attention component, and the output features (which can be called sample residual features) of the spatiotemporal residual component that has a symmetrical position with the i-th spatiotemporal residual component in the initial generation model; for example, as shown in Figure 5, the spatiotemporal residual component 40j, the spatiotemporal residual component 40m and the spatiotemporal residual component 40p, wherein the input data of the spatiotemporal residual component 40j includes the sample attention features and the first embedding information of the spatiotemporal attention component 40i, and the sample residual features of the spatiotemporal residual component 40f.

[0100] Among them, for the i-th spatiotemporal residual component in the initial generation model (the i-th spatiotemporal residual component here can be any spatiotemporal residual component in the initial generation model), the first embedding information and the input data of the i-th spatiotemporal residual component can be added to obtain a combined input feature, and the dimension of the combined input feature can be transformed from B×S×C×H×W to (B×S)×C×H×W; through the spatial convolution layer in the i-th spatiotemporal residual component, a spatial convolution operation is performed on the combined input feature with a dimension of (B×S)×C×H×W to obtain a spatial convolution feature, and the spatial convolution feature with a dimension of (B×S)×C×H×W is converted to Perform dimensionality transformation to obtain convolution transformation features of dimension (B×H×W)×C×S; perform temporal convolution operation on the convolution transformation features of dimension (B×H×W)×C×S through the temporal convolution layer in the i-th spatiotemporal residual component to obtain temporal convolution features, the dimension of which can be converted from (B×H×W)×C×S to B×S×C×H×W; obtain the sample residual features of the i-th spatiotemporal residual component based on the temporal convolution features of dimension B×S×C×H×W and the input data of the i-th spatiotemporal residual component; the sample residual features of the i-th spatiotemporal residual component have a dimension of B×S×C×H×W. Optionally, in addition to the spatial convolution layer and the temporal convolution layer, the i-th spatiotemporal residual component may also include other network layers, such as activation network layers, fully connected layers, pooling layers, etc. The embodiment of the present application does not limit the network structure of the spatiotemporal residual component.

[0101] The i-th spatiotemporal attention component in the initial generation model can include a spatial attention sub-component and a temporal attention sub-component. The sample text encoding features corresponding to the above sample description text can be input into the spatial attention sub-component in each spatiotemporal attention component, and the sample residual features of the i-th spatiotemporal residual component can be input into the spatial attention component in the i-th spatiotemporal attention. It can be understood that the dimension of the data input into the spatial attention subcomponent needs to be converted to (B×S)×C×(H×W); through the spatial attention subcomponent in the i-th spatiotemporal attention component, the sample text encoding features with dimensions of (B×S)×C×(H×W) and the sample residual features output by the i-th spatiotemporal residual component with dimensions of (B×S)×C×(H×W) are spatially attended to obtain spatial attention features; the spatial attention features can then be dimensionally transformed to obtain attention transformation features, the dimensions of which can be recorded as (B×H×W)×C×S, and the attention transformation features are temporally attended to by the temporal attention subcomponent in the i-th spatiotemporal attention component to obtain the sample attention features of the i-th spatiotemporal attention component; the dimensions of the sample attention features need to be converted to B×S×C×H×W.

[0102] It is understandable that the attention processing involved in the embodiment of the present application may refer to the forward calculation process of its respective input data using the various spatiotemporal attention components (attention mechanism introduced in the initial generation model) in the initial generation model (or the trained video generation model), and the attention processing may include spatial attention processing and temporal attention processing. Among them, spatial attention processing may refer to the forward calculation process of its respective input data using the spatial attention subcomponents (which may include spatial self-attention layers, spatial mutual attention layers, feedforward network layers, etc.) in each spatiotemporal attention component. Temporal attention processing may refer to the forward calculation of its respective input data using the temporal attention subcomponents (which may include one or more temporal self-attention layers, feedforward network layers, etc.) in each spatiotemporal attention component. The input data of spatial attention processing and temporal attention processing are different, the dimensions of the input data supported are also different, and there are differences in the processing of the input data; spatial attention processing can be used to capture spatial information in sample videos under text conditions, and temporal attention processing can be used to capture temporal information in sample videos; the specific implementation process of spatial attention processing and temporal attention processing will be described in the subsequent content.

[0103] Among them, the structure of the spatial attention subcomponent and the temporal attention subcomponent in the i-th spatiotemporal attention component can be seen in Figure 6, which is a structural diagram of the spatiotemporal attention component in an initial generation model provided by an embodiment of the present application. The spatial attention subcomponent in the i-th spatiotemporal attention component can be shown as the spatial attention subcomponent 50a in Figure 6. The spatial attention subcomponent 50a can include but is not limited to a spatial self-attention layer, a spatial cross-attention layer, a feedforward network layer 1, etc.; the dimensions of the input data of the spatial self-attention layer, the spatial cross-attention layer, and the feedforward network layer are all (B×S)×C×(H×W). The temporal attention subcomponent in the i-th spatiotemporal attention component can be shown as the temporal attention subcomponent 50b in Figure 6. The temporal attention subcomponent 50b can include but is not limited to a temporal self-attention layer 1, a temporal self-attention layer 2, a feedforward network layer 2, etc.; the dimensions of the input data of the temporal self-attention layer 1, the temporal self-attention layer 2, and the feedforward network layer 2 are all (B×H×W)×C×S.

[0104] In one or more embodiments, the spatial attention subcomponent (for example, the spatial attention subcomponent 50a) in the i-th spatiotemporal attention component performs spatial attention processing on the sample text encoding features and the sample residual features of the i-th spatiotemporal residual component, and the implementation process of obtaining the spatial attention features may include: after the sample description text is encoded by the text encoder 50c to obtain the sample text encoding features 50d, the sample text encoding features 50d and the sample residual features of the i-th spatiotemporal residual component may be input into the spatial attention subcomponent 50a in the i-th spatiotemporal attention component; according to the transformation weight matrix corresponding to the spatial self-attention layer in the spatial attention subcomponent 50a, the sample residual features of the i-th spatiotemporal residual component are transformed into a spatial query matrix (which can be denoted as Q1), a spatial key matrix (which can be denoted as K1) and a spatial value matrix (which can be denoted as V1). Among them, the transformation matrix corresponding to the spatial self-attention layer may include three parameter matrices, such as the parameter matrix W q , parameter matrix W k And the parameter matrix W v , the transformation weights corresponding to the spatial self-attention layer can be learned during the training process. The sample residual features of the i-th spatiotemporal residual component are combined with the parameter matrix W in the transformation weight matrix q The spatial query matrix Q1 can be obtained by performing point multiplication; the sample residual features of the i-th spatiotemporal residual component are combined with the parameter matrix W in the transformation weight matrix k The spatial key matrix K1 can be obtained by performing point multiplication; the sample residual features of the i-th spatiotemporal residual component are combined with the parameter matrix W in the transformation weight matrix vPerforming dot multiplication can obtain the spatial value matrix V1. Each query vector in the above spatial query matrix can be used to encode the similarity relationship between each feature and other features, and the similarity relationship can determine the dependency information between the feature and the previous feature.

[0105] The spatial query matrix Q1 and the transposed matrix of the spatial key matrix K1 can be multiplied to obtain the first candidate weight matrix (which can be recorded as ), the first candidate weight matrix can be the inner product (also called dot product, dot product) of each row vector in the spatial query matrix Q1 and the spatial key matrix K1; in order to prevent the inner product from being too large, the number of the first column corresponding to the spatial query matrix Q1 can be obtained (the spatial query matrix Q1 and the spatial key matrix K1 have the same number of columns, which can also be called vector dimension). The square root of the first candidate weight matrix and the first column number (which can be recorded as ) is normalized to obtain the first attention weight matrix, and the spatial self-attention feature is obtained by the dot product between the first attention weight matrix and the spatial value matrix V1. Among them, the first attention weight matrix can be expressed as The softmax function is a function used for normalization processing. The softmax function can be used to calculate the self-attention coefficient of a single feature for other features. Each row in is softmaxed. The dot product between the first attention weight matrix and the spatial value matrix V1 is determined as the output feature of the spatial self-attention layer (which can be expressed as ), at this time the output features of the spatial self-attention layer can be used as spatial self-attention features.

[0106] Furthermore, the sample text encoding feature 50d corresponding to the above-mentioned sample description text can be specifically input into the spatial mutual attention layer in the spatial attention subcomponent 50a. Through the spatial mutual attention layer in the spatial attention subcomponent 50a, the above-mentioned spatial self-attention feature (the output feature of the spatial self-attention layer) and the sample text encoding feature 50d are subjected to attention operation to guide the noise prediction of the 3D U-net to obtain the spatial mutual attention feature; the attention operation here can be to perform the data processing process of the aforementioned spatial self-attention layer on the feature composed of the spatial self-attention feature and the sample text encoding feature 50d, which will not be repeated here. Through the feedforward network layer in the spatial attention subcomponent (such as the feedforward network layer 1 in the spatial attention subcomponent 50a in Figure 6), the spatial mutual attention feature is subjected to feature transformation to obtain the spatial attention feature; for example, the implicit weight matrix and bias vector corresponding to the feedforward network layer 1 in the spatial attention subcomponent 50a can be obtained, and the spatial attention feature is determined based on the bias vector and the dot product between the implicit weight matrix and the spatial attention feature.

[0107] In one or more embodiments, the attention transformation feature is subjected to temporal attention processing by the temporal attention subcomponent in the i-th spatiotemporal attention component (such as the temporal attention subcomponent 50b in Figure 6), and the process of obtaining the sample attention feature of the i-th spatiotemporal attention component may include: the attention transformation matrix can be transformed into a temporal query matrix (which can be denoted as Q2), a temporal key matrix (which can be denoted as K2), and a temporal value matrix (which can be denoted as V2) according to the transformation weight matrix corresponding to the temporal self-attention layer in the temporal attention subcomponent 50b (for example, the temporal self-attention layer 1 shown in Figure 6). The temporal query matrix Q2 is point-multiplied with the transposed matrix of the temporal key matrix K2 to obtain a second candidate weight matrix (which can be denoted as ), obtain the second column number corresponding to the time series query matrix Q2; add the second candidate weight matrix and the square root of the second column number (which can be recorded as ) is normalized to obtain the second attention weight matrix (which can be recorded as ), the temporal self-attention features are obtained by the dot product between the second attention weight matrix and the temporal value matrix V2; the temporal self-attention features are transformed through the feedforward network layer in the temporal attention subcomponent to obtain the sample attention features of the i-th spatiotemporal attention component. As shown in Figure 6, the temporal attention subcomponent can include two temporal self-attention layers (temporal self-attention layer 1 and temporal self-attention layer 2). The two temporal self-attention layers process the input data in the same way. The processing process can be referred to the data processing process of the spatial self-attention layer above, and will not be repeated here.

[0108] It is understandable that, similar to the data processing process in the aforementioned i-th spatiotemporal residual component and the i-th spatiotemporal attention component, the input data (which may include video input data, first embedding information, and sample text encoding features) can be processed by the M spatiotemporal residual components and M spatiotemporal attention components of the initial generation model to obtain the sample prediction noise corresponding to the video input data. For example, the sample attention feature of the last spatiotemporal attention component in the initial generation model can be used as the sample prediction noise output by the initial generation model for the video input data.

[0109] Step S106, based on the sample original noise and the sample predicted noise, the network parameters in the initial generation model are modified, and the initial generation model including the modified network parameters is determined as the video generation model; the video generation model is used to generate video data that matches the video description text.

[0110] Specifically, the mean square error loss between the original noise of the sample and the predicted noise of the sample, i.e., the MSE loss (Mean-Squared Loss), can be obtained; the network parameters of the initial generation model are iteratively trained according to the mean square error loss until the initial generation model meets the training end condition, and the training is stopped. The initial generation model at the end of the training is determined as the video generation model. The mean square error loss can be shown as formula (2):

[0111] Where L(θ) in formula (2) represents the mean square error loss corresponding to the initial generation model, θ represents the trainable parameters (network parameters) in the initial generation model; f θ (x t1 ,t1,c1) represents the sample prediction noise output by the initial generation model, f θ represents the 3D U-Net model in the initial generation model (which can be called a conditional model), x t1 Represents the video input data, t1 represents the first step encoding feature, and c1 represents the sample text encoding feature.

[0112] The above-mentioned training end conditions may include but are not limited to a preset maximum number of iterations, or the value of the total loss of the model is less than or equal to a preset error value, etc., and this application does not limit this. When the initial generation model meets the training end conditions, the training can be stopped, and the current network parameters can be saved, and the initial generation model containing the current network parameters can be determined as the video generation model. Among them, the video generation model is the initial generation model that has been trained, which can be used to generate target video data that conforms to the text content from arbitrary Gaussian noise combined with video description text.

[0113] Please refer to Figure 7, which is a flow chart of the training phase of an initial generation model provided by an embodiment of the present application. As shown in Figure 7, the training phase of the initial generation model may include the following steps S11 to S17:

[0114] S11, data preparation.

[0115] Specifically, a batch of videos and their corresponding video text descriptions are obtained to form (video, text) pairs for training the initial generative model. The (video, text) pairs here can serve as sample pairs for training the initial generative model. A (video, text) pair includes a video and its corresponding video description text. In this embodiment of the application, the video in the (video, text) pair can be referred to as a sample video, and the text in the (video, text) pair (i.e., the video description text) can be referred to as a sample description text.

[0116] S12, video frame extraction.

[0117] Specifically, assuming that the initial generative model supports the input of S (a pre-set threshold number of video frames) frames for training, a fixed number of T video frames can be extracted from each video to form an initial video frame sequence tensor with a dimension of B×T×3×H×W.

[0118] S13, Multi-time scale data production.

[0119] Specifically, when the initial generative model is distributedly trained, the initial input dimensions of the initial generative model on each GPU rank can be B×S0×3×H×W at each training step. To achieve multi-timescale modeling, each rank can randomly extract a sample video frame sequence with a different number of video frames, denoted as x0, with dimensions B×S×3×H×W, where S∈[1, S0]. This allows the initial generative model to perform multi-timescale modeling based on videos of different lengths.

[0120] S14, adds noise to the input data.

[0121] Specifically, according to the above formula (1), according to DDPM and the first step number t1, a Gaussian noise ε with the same latitude as the sample video frame sequence is randomly generated. The Gaussian noise ∈ at this time can be called the sample original noise. According to the sample original noise and the sample original noise ∈, the sample video frame sequence is subjected to data noise processing to obtain the video input data x t1 (Also called noisy input data), the video input data x t1 It can be calculated according to formula (1).

[0122] S15, Multi-timescale modeling.

[0123] Specifically, multi-time-scale temporal modeling can be based on the designed 3D U-Net model (which can be recorded as the conditional model f θ ), specifically, a temporal convolution layer (or temporal convolution module, which can be added after the spatial convolution layer in the 2D U-Net model) and a temporal attention subcomponent (which can be added after the spatial attention subcomponent in the 2D U-Net model) can be added to the current 2D U-Net model. Since both the temporal convolution layer and the temporal attention subcomponent support inputs of arbitrary length, they are compatible with video input data with different numbers of video frames. Among them, the network structure of the initial generation model obtained by multi-time scale modeling can be found in the description of the embodiments corresponding to Figures 5 and 6, and will not be repeated here.

[0124] S16, text conditional encoding and fusion.

[0125] Specifically, the sample description text corresponding to the sample video is input into the text encoder. Assuming that the length of the sample description text is L (L is a positive integer), the sample description text is encoded by the text encoder to obtain a B×L×D dimension. text The sample text encoding features (which can be recorded as c1); where B represents the batch size, L represents the length of the sample description text, and D text Represents the vector dimension of a unit character. The sample text encoding feature c1 can be input into the spatial mutual attention layer contained in each spatial attention subcomponent in the initial generation model, and perform attention operation with the original input of the spatial mutual attention layer (spatial self-attention feature) to guide the noise prediction of the 3D U-Net model. Optionally, during the training process, the sample description text can be randomly Dropped (i.e., changed to an empty string) with a certain probability p, and the initial generation model is trained to support unconditional generation without text. In other words, during the training phase, the initial generation model can be trained to support both conditional generation with text and unconditional generation without text, which can promote the applicable scenarios of the initial generation model and improve the noise prediction accuracy of the initial generation model.

[0126] S17, noise prediction and loss calculation.

[0127] Specifically, the above video input data x t1 , the sample text encoding feature c1 (text condition) and the first embedding information corresponding to the first step t1 are input into the initial generation model (for example, the 3D U-Net model designed above), and the sample prediction noise f is predicted θ (x t1 ,t1,c1). According to DDPM, the sample original noise ∈ can be combined with the sample prediction noise f θ (x t1 ,t1,c1) calculates the MSE loss (which can be calculated according to the above formula (2)), and uses this MSE loss to train the entire initial generation model. The trained video generation model can combine video description text from arbitrary Gaussian noise to generate target video data that matches the text content.

[0128] Please refer to Figure 8, which is a schematic diagram of the training of an initial generation model provided by an embodiment of the present application. As shown in Figure 8, through steps S11 to S13 in the embodiment corresponding to Figure 7, a sample video frame sequence 60a can be obtained from the sample video. At this time, the sample video frame sequence 60a can be used as the initial input for the initial generation 60h. It can be based on the pre-set diffusion parameter T SA first step number t1 is randomly sampled. The first step number t1 and the corresponding diffusion rates for all steps less than the first step number t1 are obtained from the set noise scale 60b, thereby obtaining the sequence weight corresponding to the sample video frame sequence 60a. Simultaneously, a Gaussian noise ∈ is randomly sampled from a standard normal distribution and used as the sample original noise 60c added to the sample video frame sequence 60a. Based on the sample original noise 60c, the sequence weight, and the original noise weight (which can be determined by the sequence weight), the sample video frame sequence 60a is subjected to data noise processing to obtain video input data 60d. The first step number t1 can be encoded as first embedding information 60g, which can be input into each spatiotemporal residual component in the initial generative model 60h.

[0129] The sample description text 60e corresponding to the sample video can be input into a text encoder, which performs text encoding processing on the sample description text 60e to obtain a sample text encoding feature 60f corresponding to the sample description text 60e. The sample text encoding feature 60f can be input into each spatiotemporal attention component in the initial generation model 60h, specifically the spatial mutual attention layer in each spatiotemporal attention component, to guide the noise prediction of the initial generation model 60h. In other words, after the video input data 60d, the first embedding information 60g, and the sample text encoding feature 60f are input into the initial generation model 60h, the sample prediction noise 60i can be output by the initial generation model 60h. The mean square error loss between the sample original noise 60c and the sample prediction noise 60i is calculated, and the initial generation model 60h is trained using the mean square error loss. The trained initial generation model 60h can be called a video generation model.

[0130] In the embodiment of the present application, during the training process of the initial generative model, sample video frame sequences of different frame numbers are used to train the initial generative model. Through the spatiotemporal residual component and spatiotemporal attention component in the initial generative model, the initial generative model can efficiently capture long-term and short-term temporal action information during the training process, that is, learn temporal action information in videos of different time scales, improve the comprehensiveness of temporal action information, and thus improve the quality of video generation.

[0131] Please refer to Figure 9, which is a second flow chart of a video generation method provided in an embodiment of the present application. It is understandable that the video generation method can be executed by a computer device, which can be a server or a terminal device, and this application does not limit this. As shown in Figure 9, the video generation method can include the following steps S201 to S206:

[0132] Step S201: Obtain video description text and initialization noise, and determine the second step number corresponding to the initialization noise.

[0133] Specifically, the trained video generation model can be used to generate target video data with different durations, and the process of generating target video data using the video generation model can be called the sampling stage. The data processing process in the sampling stage may include: first, it is necessary to obtain the video description text used to generate the video data, and the video description text can be input into the video generation model as a text condition to guide the video generation model to generate target video data that conforms to the video description text; then, a Gaussian noise can be randomly initialized for the video generation model, and the Gaussian noise can be called initialization noise; the dimension of the initialization noise can be expressed as B'×S"×3×H×W, S"∈[1, S], where B', S", H, and W represent the number of videos to be generated, the number of frames of duration (for example, the number of video frames to be generated), the height of the generated video, and the width of the generated video, respectively. The number of steps corresponding to the above initialization noise can be called the second step number (for ease of understanding, the second step number can be recorded as t2, and the initialization noise can be recorded as z t2 ), the second step number t2 can be used to represent the initialization noise z t2 The noise level, the second step number t2 can be [1, T S ] any integer value in the range, T S is the pre-set diffusion parameter.

[0134] It should be understood that since a multi-time-scale modeling approach is adopted in the training stage, that is, the video generation model is trained using sample video frame sequences with different numbers of video frames, the video generation process can support the generation of videos of any length in the range of [1, S0]. In other words, the video generation model can generate target video data of different lengths.

[0135] Step S202: perform text encoding processing on the video description text to obtain video text encoding features, and input the initialization noise, video text encoding features, and second embedding information corresponding to the second step number into the video generation model.

[0136] Specifically, the video description text can be input into a trained text encoder, which then processes the video description text to obtain video text encoding features corresponding to the video description text. The text encoding process of the video description text can be found in the description of the text encoding process of the sample description text in step S102 of the embodiment corresponding to FIG. 3 , and will not be further described here.

[0137] The second step number t2 can be represented by a vector, such as a one-hot encoding of the first step number, or other encoding methods can be used to represent the first step number. This application does not limit this. Further, the vector representation of the second step number t2 can be encoded to obtain the second embedded information corresponding to the second step number t2.

[0138] The initialization noise z t2 , the second embedding information and the video text encoding features are input into the trained video generation model, where the initialization noise z t2 The first spatiotemporal residual component of the video generation model can be input, the second embedding information can be input into each spatiotemporal residual component of the video generation model, and the video text encoding features can be input into the spatial mutual attention layer of each spatiotemporal attention component of the video generation model. The network structure of the video generation model is the same as that of the initial generation model mentioned above. For details, please refer to the embodiments corresponding to Figures 5 and 6, which will not be repeated here.

[0139] Step S203, through the j-th spatiotemporal residual component in the video generation model, feature extraction is performed on the second embedding information and the input data of the j-th spatiotemporal residual component to obtain the noise residual feature of the j-th spatiotemporal residual component; when j is 1, the input data of the j-th spatiotemporal residual component includes the second embedding information and the initialization noise; when j is not 1, the input data of the j-th spatiotemporal residual component includes the video text encoding feature and the noise attention feature of the j-1-th spatiotemporal attention component.

[0140] In step S204, the j-th spatiotemporal attention component in the video generation model performs attention processing on the video text encoding features and the noise residual features of the j-th spatiotemporal residual component to obtain the noise attention features of the j-th spatiotemporal attention component.

[0141] Specifically, the video generation model may include M spatiotemporal residual components and M spatiotemporal attention components connected in an alternating manner. The M spatiotemporal residual components in the video generation model can perform the same operation on their respective input data. Similarly, the M spatiotemporal attention components in the video generation model can perform the same operation on their respective input data. For ease of decomposition, the embodiment of the present application is described by taking the j-th spatiotemporal residual component and the j-th spatiotemporal attention component connected in the video generation model as an example, where j is a positive integer less than or equal to M; in other words, the j-th spatiotemporal residual component can be any spatiotemporal residual component in the video generation model, and the j-th spatiotemporal attention component can be any spatiotemporal attention component in the video generation model.

[0142] The jth spatiotemporal residual component in the video generation model can include a spatial convolution layer and a temporal convolution layer. When j is 1, the input data of the jth spatiotemporal residual component includes the initialization noise z t2 and the second embedding information. When j is greater than 1 and less than or equal to M / 2+1, the input data of the j-th spatiotemporal residual component may include the output features of the j-1-th spatiotemporal attention component (which may be called noise attention features) and the second embedding information. When j is greater than M / 2+1, the input data of the j-th spatiotemporal residual component may include the noise attention features of the j-1-th spatiotemporal attention component, the second embedding information, and the output features of the spatiotemporal residual component in the video generation model that has a symmetrical position with the j-th spatiotemporal residual component (which may be called noise residual features). The j-th spatiotemporal attention component in the video generation model may include a spatial attention subcomponent and a temporal attention subcomponent. The spatial attention subcomponent may include a spatial self-attention layer, a spatial mutual attention layer, and a feedforward network layer. The temporal attention subcomponent may include one or more temporal self-attention layers (for example, two temporal self-attention layers) and a feedforward network layer. The input data of the j-th spatiotemporal attention component is the output features (noise residual features) of the j-th spatiotemporal residual component. Among them, the noise z is initialized t2 The processing process of the second embedded information and the video text encoding features in the video generation model can refer to the relevant description in step S105 of the embodiment corresponding to Figure 3, and will not be repeated here.

[0143] In the sampling phase, the noise z is initialized t2 After the second embedding information and the video text encoding features are input into the video generation model, N-step denoising can be performed according to sampling methods such as DDIM (Denoising diffusion implicit models) and DDPM, and finally the target video data that is noise-free and conforms to the semantics of the video description text is obtained. The dimension of the target video data can be expressed as 3×H×W; where N is a positive integer, such as N can be less than or equal to T S (diffusion parameter), N can be set according to actual needs. The embodiment of the present application does not limit the sampling method and the value of N.

[0144] It should be noted that during the training of the initial generative model, the sample text encoding features corresponding to the sample description text are randomly dropped with a certain probability p (changing the sample description text to an empty string, which can be understood as setting the sample text encoding features to zero to form a zero matrix). This allows the trained video generation model to support both conditional generation with text and unconditional generation without text. Therefore, during the sampling phase, a classifier-free guidance technique can be used. Two types of noise can be predicted simultaneously during each denoising step: one is the predicted noise under the condition of text (which can be simply referred to as conditional prediction noise), and the other is the predicted noise under the condition of no text (which can be simply referred to as unconditional prediction noise). The conditional prediction noise can refer to the use of the video text encoding features corresponding to the video description text as text conditions to guide the noise prediction of the 3D U-Net during the noise prediction process using the 3D U-Net in the video generation model. The unconditional prediction noise can refer to the predicted noise obtained after changing the video description text to an empty string (e.g., changing the video text encoding features to a zero matrix) during the noise prediction process using the 3D U-Net in the video generation model.

[0145] Step S205: Determine the video prediction noise based on the noise attention feature of the Mth spatiotemporal attention component in the video generation model, and denoise the initialization noise based on the video prediction noise and the second step number to obtain noisy video data corresponding to the third step number; the third step number is less than the second step number.

[0146] Step S206: De-noising the noisy video data corresponding to the third step number is performed using a video generation model until target video data matching the video description text is generated.

[0147] Specifically, the video prediction noise predicted by the 3D U-Net can be obtained based on the output results of the 3D U-Net in the video generation model. The output results of the 3D U-Net can refer to the output features (noise attention features) of the last spatiotemporal attention component (i.e., the Mth spatiotemporal attention component) in the 3D U-Net. When the video generation model only uses conditional generation with text (which can be simply referred to as a conditional model), the noise attention features of the Mth spatiotemporal attention component in the video generation model can be considered as noise predictions obtained under the guidance of the video text encoding features. In this case, the noise attention features of the Mth spatiotemporal attention component in the video generation model can be determined as the video prediction noise.

[0148] Optionally, if the video generation model uses both conditional generation with text (conditional model) and unconditional generation without text (which can be referred to as unconditional model for short), the Classifier-free Guidance technology can be used in the sampling stage. Then, when using the 3D U-Net in the video generation model for noise prediction, the noise attention features of the Mth spatiotemporal attention component in the video generation model can include the noise attention features under text conditions and the noise attention components under no-text conditions. The noise attention features under text conditions refer to the attention features obtained by using the video text encoding features, or can refer to the attention features obtained when using the video text encoding features as text conditions; the noise attention features under no-text conditions refer to the attention features obtained by using the video text encoding features after setting to zero, or can refer to the attention features obtained by changing the video text encoding features to a zero matrix, that is, when there are no text conditions.

[0149] Conditional prediction noise can be generated based on the noise attention feature under text conditions, and unconditional prediction noise can be generated based on the noise attention feature under no-text conditions. Specifically, the noise attention feature under text conditions can be used as conditional prediction noise, and the noise attention feature under no-text conditions can be used as unconditional prediction noise. Then, the noise difference between the conditional prediction noise and the unconditional prediction noise can be obtained, and the unconditional prediction noise and the product of the noise intensity parameter and the noise difference can be added to obtain the video prediction noise. At this time, the video prediction noise can be the video prediction noise corresponding to the second step number predicted by the video generation model. In other words, the video prediction noise can be obtained by fusion of the conditional prediction noise and the unconditional prediction noise. Denoising by using the video prediction noise can improve the quality of video generation. The video prediction noise can be shown as formula (3):

[0150] Among them, in formula (3) Indicates the video prediction noise corresponding to the second step number when the Classifier-free Guidance technology is used in the sampling stage; represents the unconditional prediction noise, Indicates that the video generation model uses the zeroed video text encoding features when generating unconditional prediction noise; ∈ θ =f θ (z t2 ,t2,c2) represents the conditional prediction noise, z t2represents the initialization noise of the 3D U-Net input to the video generation model, t2 represents the second step number, c2 represents the video text encoding feature corresponding to the video description text; ω represents the noise intensity parameter, which is the guiding scale for controlling the noise intensity and is used to control the direction of the generation effect between the unconditional model and the conditional model.

[0151] After obtaining the video prediction noise corresponding to the second step number, the noisy video data corresponding to the third step number can be calculated based on the video prediction noise, the initialization noise and the second step number; the noisy video data corresponding to the third step number, the third embedding information corresponding to the third step number, and the video text encoding features can be input into the video generation model, and the above denoising process is repeated through the video generation model until the noise-free video data z0 is obtained. At this time, the video data z0 can be used as the target video data that conforms to the video description text content.

[0152] Please refer to Figure 10, which is a sampling diagram of a video generation model provided by an embodiment of the present application. This embodiment of the present application takes the DDPM method as an example to describe the sampling process of the video generation model; as shown in Figure 10, a Gaussian noise can be randomly initialized for the video generation model 70a, and the Gaussian noise can be used as the initialization noise z t2 , then the initialization noise z t2 The corresponding step number t2 can be called the second step number. The initialization noise z t2 , the step number encoding information (second embedded information) corresponding to the step number t2 and the video text encoding features are input into the video generation model 70a. The video prediction noise corresponding to the step number t2 can be obtained through the video generation model 70a. The noise prediction process of the video generation model 70a can refer to the relevant descriptions in the aforementioned steps S203 to S205, which will not be repeated here.

[0153] According to the video prediction noise corresponding to the above step t2, the probability distribution corresponding to the step t2-1 (t2-1 at this time can be used as the third step) can be obtained. The probability distribution corresponding to the step t2-1 can be recorded as p(z t2-1 |z t2 ,z0), in accordance with According to the probability distribution, random sampling can be performed to obtain the noisy video data z corresponding to the step number t2-1 t2-1 , the above process can be called a denoising process performed using the video generation model; where z t2-1 represents the noisy video data corresponding to step t2-1, z t2 represents the initialization noise (which can be understood as the noisy video data corresponding to step number t2), z0 represents the noise-free video data (such as the target video data finally generated); βt2 represents the diffusion rate corresponding to step number t2, That is, the cumulative product of the diffusion rates corresponding to all steps less than or equal to step number t2-1; That is, the cumulative product of the diffusion rates corresponding to all steps less than or equal to the step number t2. Using the reparameterization technique, the random sampling process according to the above probability distribution can be converted to: randomly sampling a Gaussian noise ε0, and then we can calculate in Represents the video prediction noise corresponding to step number t2, In other words, the noise can be predicted by randomly sampling Gaussian noise ∈ 0 and the video corresponding to the step number t2 Initialize the noise z t2 , and the diffusion rate corresponding to all steps less than or equal to step t2, directly calculate the noisy video data z corresponding to step t2-1 t2-1 The noisy video data z corresponding to step t2-1 t2-1 , the step number encoding information corresponding to the step number t2-1 can be input into the video generation model 70a as new input data together with the video text encoding features, and the above denoising process is repeated until the noise-free video data z0 is generated, and the noise-free video data z0 can be used as the final target video data.

[0154] For example, assuming that the second step number t2 = 1000, after performing a denoising process, the noisy video data corresponding to step number 999 (ie, z 999 ), continue to use the video generation model 70a to analyze the noisy video data z 999 Perform denoising to obtain the noisy video data z corresponding to step number 998 998 , and the above operations are continuously repeated until the noise-free video data z0 is obtained. In other words, in the sampling stage, it is necessary to use the video generation model 70a to perform 1000 (N is 1000) steps of denoising processing to obtain the noise-free video data z0. It should be noted that the number of steps in the sampling stage (including the second step and the third step, etc.) can be understood as the number of denoising steps. The number of steps in the training stage is to gradually add noise to the sample video frame sequence. The larger the number of steps, the greater the noise intensity added to the sample video frame sequence; the number of steps in the sampling stage is to gradually denoise the initialization noise. The smaller the number of steps, the more noise is removed for the initialization noise. The number of steps at this time can also be called the denoising step.

[0155] Alternatively, if the DDIM method is used for denoising in the sampling phase, then there is no need to add random noise (such as Gaussian noise ∈ 0 in the DDPM method) in the process of generating the target video data (i.e., the denoising process). This allows for a smaller number of sampling steps to speed up the video data generation process. For example, assuming that the diffusion parameter in the training phase is set to T s =1000, the DDIM sampling method is adopted in the sampling stage, and the sampling step number N is set to 50. Then, in the process of generating the target video data using the trained video generation model, the initialization noise corresponding to the step number 1000, the embedding information corresponding to the step number 1000, and the video text encoding features can be input into the video generation model. The 3D U-Net in the video generation model outputs the video prediction noise corresponding to the step number 1000. According to the denoising formula corresponding to DDIM (the denoising formula corresponding to DDIM is disclosed in the prior art and is not listed here), the noisy video data corresponding to the step number 980 can be calculated. Then, the noisy video data corresponding to the step number 980, the embedding information corresponding to the step number 980, and the video text encoding features can be input into the video generation model. The 3D U-Net in the video generation model outputs the video prediction noise corresponding to the step number 960. According to the denoising formula corresponding to DDIM, the noisy video data corresponding to the step number 960 can be calculated. The above operation is repeated until the noise-free video data z0 (target video data that meets the video description text content) is obtained. Compared with the DDPM sampling method, the DDIM sampling method is used for denoising in the embodiment of the present application, which can increase the generation speed of the target video data.

[0156] Please refer to Figure 11, which is a flow chart of the sampling phase of a video generation model provided by an embodiment of the present application. As shown in Figure 11, the sampling phase of the video generation model may include the following steps S21 to S23:

[0157] S21, noise initialization.

[0158] Specifically, a Gaussian noise can be randomly initialized, which can be used as the initialization noise z input into the video generation model. t2 , the initialization noise z t2 The dimension can be expressed as B×S”×3×H×W, S”∈[1, S], where t2 can be called the second step number.

[0159] S22, text conditional encoding.

[0160] Specifically, a video description text used to create a video can be obtained, and the video description text can be input into a text encoding encoder. The video description text is then encoded by the text encoder to obtain a video text encoding feature corresponding to the video description text. The video text encoding feature can be called text conditional encoding.

[0161] S23, denoising and video generation.

[0162] Specifically, the noise z is initialized t2 The video text encoding features and the embedded information corresponding to step t2 are input into the video generation model. Based on sampling methods such as DDPM and DDIP, N-step denoising is performed to ultimately obtain noise-free target video data that conforms to the video description text content. The denoising and video generation processes can be found in the descriptions of steps S203 to S206 above and will not be repeated here.

[0163] In an embodiment of the present application, during the training process of the initial generation model, a sample video frame sequence of different frame numbers is used to train the initial generation model. Through the spatiotemporal residual component and spatiotemporal attention component in the initial generation model, the initial generation model can efficiently capture long-term and short-term temporal action information during the training process, that is, learn the temporal action information in videos of different time scales, improve the comprehensiveness of temporal action information, and thus improve the generation quality of the video; through the above-mentioned training method, videos of different lengths can be generated in the sampling stage, and the picture has high consistency and action coherence at any length, thereby improving the generation quality of the video.

[0164] It is understandable that in the specific implementation of this application, personal information such as the user's facial image may be involved. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain permission or consent from relevant institutions or departments, or the users themselves, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions.

[0165] Please refer to Figure 12, which is a schematic diagram of the structure of a video generation device provided by an embodiment of the present application. As shown in Figure 12, the video generation device 1 includes: a sample acquisition module 11, a sample noise addition module 12, a model acquisition module 13, a sample input module 14, a noise prediction module 15, and a parameter correction module 16;

[0166] The sample acquisition module 11 is used to obtain a sample video and a sample description text corresponding to the sample video, obtain a sample video frame sequence in the sample video, and determine the first step number and sample original noise for the sample video frame sequence;

[0167] The sample noise adding module 12 is used to perform data noise adding processing on the sample video frame sequence according to the first step number and the original noise of the sample to obtain video input data, and perform text encoding processing on the sample description text to obtain sample text encoding features;

[0168] A model acquisition module 13 is used to acquire an initial generative model; the initial generative model includes M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model are alternately connected, where M is an integer greater than 1;

[0169] A sample input module 14 is configured to input the video input data into the first spatiotemporal residual component of the M spatiotemporal residual components, input the first embedding information corresponding to the first step into the M spatiotemporal residual components, and input the sample text encoding features into the M spatiotemporal attention components;

[0170] A noise prediction module 15 is configured to perform noise prediction on the video input data in the M spatiotemporal residual components and the M spatiotemporal attention components according to the sample text encoding features and the first embedding information to obtain sample predicted noise;

[0171] The parameter correction module 16 is used to correct the network parameters in the initial generation model based on the original sample noise and the predicted sample noise, and determine the initial generation model including the corrected network parameters as the video generation model; the video generation model is used to generate video data that matches the video description text.

[0172] In one or more embodiments, the sample acquisition module 11 acquires a sample video frame sequence in a sample video to perform the following operations:

[0173] Determine the frame extraction interval duration corresponding to the sample video according to the video frame number threshold and the video duration corresponding to the sample video;

[0174] According to the frame extraction interval, multiple video frames are obtained from the sample video, and the multiple video frames are combined into an initial video frame sequence; the number of video frames included in the initial video frame sequence is equal to the video frame number threshold;

[0175] A sample video frame sequence for training an initial generation model is obtained from the initial video frame sequence; the sample video frame sequence is a subsequence of any initial video frame sequence whose number of video frames is less than or equal to a video frame number threshold.

[0176] In one or more embodiments, the sample acquisition module 11 determines the first step number and the sample original noise for the sample video frame sequence, and is used to perform the following operations:

[0177] Obtain the diffusion parameter corresponding to the initial generation model, and sample the first step number corresponding to the sample video frame sequence according to the diffusion parameter; the diffusion parameter is used to characterize the total number of diffusion steps of the initial generation model, and the first step number is less than or equal to the diffusion parameter;

[0178] A Gaussian noise having the same dimension as the sample video frame sequence is generated, and the Gaussian noise is determined as the sample original noise corresponding to the sample video frame sequence.

[0179] In one or more embodiments, the sample noise adding module 12 performs data noise adding processing on the sample video frame sequence according to the first step number and the sample original noise to obtain video input data for performing the following operations:

[0180] Determine the sequence weight corresponding to the sample video sequence according to the first step number, and determine the original noise weight corresponding to the sample original noise according to the sequence weight; the sum of the squares of the sequence weight and the original noise weight is 1;

[0181] The product of the sequence weight and the sample video sequence, and the product of the original noise weight and the sample original noise are added together to obtain the video input data.

[0182] In one or more embodiments, the sample denoising module 12 performs text encoding processing on the sample description text to obtain sample text encoding features for performing the following operations:

[0183] Divide the sample description text into D unit characters and obtain the unit word vectors corresponding to the D unit characters; D is a positive integer;

[0184] According to the semantic information of D unit characters in the sample description text, obtain the text vectors corresponding to the D unit characters respectively;

[0185] According to the text positions of the D unit characters in the sample description text, obtain the position vectors corresponding to the D unit characters respectively;

[0186] Combine the unit word vector, text vector, and position vector to obtain the initial text features corresponding to the sample description text;

[0187] The initial text features are input into the text encoder, and the initial text features are encoded by the text encoder to obtain the sample text encoding features corresponding to the sample description text.

[0188] In one or more embodiments, the noise prediction module 15 performs denoising on the video input data in the M spatiotemporal residual components and the M spatiotemporal attention components based on the sample text encoding features and the first embedding information to obtain sample prediction noise for performing the following steps:

[0189] Through the i-th spatiotemporal residual component among the M spatiotemporal residual components, feature extraction is performed on the first embedded information and the input data of the i-th spatiotemporal residual component to obtain the sample residual feature of the i-th spatiotemporal residual component; i is a positive integer less than or equal to M;

[0190] Through the i-th spatiotemporal attention component among the M spatiotemporal attention components, the sample text encoding features and the sample residual features of the i-th spatiotemporal residual component are processed to obtain the sample attention features of the i-th spatiotemporal attention component;

[0191] The sample attention feature of the last spatiotemporal attention component among the M spatiotemporal attention components is determined as the sample prediction noise;

[0192] Among them, when i is 1, the input data of the i-th spatiotemporal residual component includes the video input data and the first embedding information; when i is not 1, the input data of the i-th spatiotemporal residual component includes the first embedding information and the sample attention features of the i-1-th spatiotemporal attention component.

[0193] In one or more embodiments, the i-th spatiotemporal residual component in the initial generative model includes a spatial convolution layer and a temporal convolution layer; when i is greater than M / 2+1, the input data of the i-th spatiotemporal residual component also includes sample residual features of the spatiotemporal residual component in the initial generative model that has a symmetrical position with the i-th spatiotemporal residual component;

[0194] The noise prediction module 15 performs feature extraction on the first embedded information and the input data of the i-th spatiotemporal residual component through the i-th spatiotemporal residual component among the M spatiotemporal residual components to obtain the sample residual feature of the i-th spatiotemporal residual component, which is used to perform the following operations:

[0195] Add the first embedding information and the input data of the i-th spatiotemporal residual component to obtain the combined input feature;

[0196] Through the spatial convolution layer in the i-th spatiotemporal residual component, the combined input features are spatially convolved to obtain spatial convolution features, and the spatial convolution features are dimensionally transformed to obtain convolution transformation features;

[0197] Through the temporal convolution layer in the i-th spatiotemporal residual component, the convolution transformation features are subjected to temporal convolution operations to obtain temporal convolution features. According to the temporal convolution features and the input data of the i-th spatiotemporal residual component, the sample residual features of the i-th spatiotemporal residual component are obtained.

[0198] In one or more embodiments, the i-th spatiotemporal attention component in the initial generative model includes a spatial attention subcomponent and a temporal attention subcomponent;

[0199] The noise prediction module 15 performs attention processing on the sample text encoding features and the sample residual features of the i-th spatiotemporal residual component through the i-th spatiotemporal attention component among the M spatiotemporal attention components to obtain the sample attention features of the i-th spatiotemporal attention component, which is used to perform the following operations:

[0200] Through the spatial attention subcomponent in the i-th spatiotemporal attention component, the sample text encoding features and the sample residual features of the i-th spatiotemporal residual component are spatially attended to obtain the spatial attention features;

[0201] The spatial attention feature is dimensionally transformed to obtain the attention transformation feature, and the attention transformation feature is processed by the temporal attention subcomponent in the i-th spatiotemporal attention component to obtain the sample attention feature of the i-th spatiotemporal attention component.

[0202] In one or more embodiments, the noise prediction module 15 performs spatial attention processing on the sample text encoding features and the sample residual features of the i-th spatiotemporal residual component through the spatial attention subcomponent in the i-th spatiotemporal attention component to obtain spatial attention features for performing the following operations:

[0203] According to the transformation weight matrix corresponding to the spatial self-attention layer in the spatial attention subcomponent, the sample residual features of the i-th spatiotemporal residual component are transformed into a spatial query matrix, a spatial key matrix, and a spatial value matrix;

[0204] Perform a dot product operation on the spatial query matrix and the transposed matrix of the spatial key matrix to obtain the first candidate weight matrix, and obtain the first column number corresponding to the spatial query matrix;

[0205] Normalize the ratio of the first candidate weight matrix to the square root of the first column number to obtain the first attention weight matrix, and obtain the spatial self-attention feature according to the dot product between the first attention weight matrix and the spatial value matrix;

[0206] Through the spatial mutual attention layer in the spatial attention subcomponent, the spatial self-attention feature and the sample text encoding feature are subjected to attention operation to obtain the spatial mutual attention feature;

[0207] The spatial attention features are obtained by performing feature transformation on the spatial mutual attention features through the feedforward network layer in the spatial attention subcomponent.

[0208] In one or more embodiments, the noise prediction module 15 performs temporal attention processing on the attention transformation feature through the temporal attention subcomponent in the i-th spatiotemporal attention component to obtain a sample attention feature of the i-th spatiotemporal attention component, which is used to perform the following operations:

[0209] According to the transformation weight matrix corresponding to the temporal self-attention layer in the temporal attention subcomponent, the attention transformation matrix is ​​transformed into a temporal query matrix, a temporal key matrix, and a temporal value matrix;

[0210] Perform a dot multiplication operation on the time series query matrix and the transposed matrix of the time series key matrix to obtain the second candidate weight matrix, and obtain the second column number corresponding to the time series query matrix;

[0211] Normalize the ratio of the second candidate weight matrix to the square root of the second column number to obtain the second attention weight matrix, and obtain the temporal self-attention feature according to the dot product between the second attention weight matrix and the temporal value matrix;

[0212] The temporal self-attention features are transformed through the feedforward network layer in the temporal attention subcomponent to obtain the sample attention features of the i-th spatiotemporal attention component.

[0213] In one or more embodiments, the parameter modification module 16 modifies the network parameters in the initial generation model based on the sample original noise and the sample predicted noise, and determines the initial generation model including the modified network parameters as the video generation model, for performing the following operations:

[0214] Get the mean square error loss between the sample original noise and the sample predicted noise;

[0215] The network parameters of the initial generation model are iteratively trained according to the mean square error loss until the initial generation model meets the training end conditions, then the training is stopped, and the initial generation model at the end of the training is determined as the video generation model.

[0216] In one or more embodiments, the video generation device 1 further includes: a noise initialization module 17, a text encoding module 18, and a denoising module 19;

[0217] The noise initialization module 17 is used to obtain the video description text and the initialization noise, and determine the second step number corresponding to the initialization noise;

[0218] A text encoding module 18 is configured to perform text encoding processing on the video description text to obtain a video text encoding feature, and input the initialization noise, the video text encoding feature, and the second embedding information corresponding to the second step number into the video generation model;

[0219] The denoising module 19 is configured to perform feature extraction on the second embedding information and the input data of the j-th spatiotemporal residual component through the j-th spatiotemporal residual component in the video generation model to obtain the noise residual feature of the j-th spatiotemporal residual component, and perform attention processing on the video text encoding feature and the noise residual feature of the j-th spatiotemporal residual component through the j-th spatiotemporal attention component in the video generation model to obtain the noise attention feature of the j-th spatiotemporal attention component; when j is 1, the input data of the j-th spatiotemporal residual component includes the second embedding information and the initialization noise; when j is not 1, the input data of the j-th spatiotemporal residual component includes the video text encoding feature and the noise attention feature of the j-1-th spatiotemporal attention component, where j is a positive integer less than or equal to M;

[0220] The denoising module 19 is further configured to determine the video prediction noise based on the noise attention feature of the M-th spatiotemporal attention component in the video generation model, and denoise the initialization noise based on the video prediction noise to obtain noisy video data corresponding to a third number of steps; the third number of steps is less than the second number of steps;

[0221] The denoising module 19 is further configured to perform denoising processing on the noisy video data corresponding to the third step number through a video generation model until target video data matching the video description text is generated.

[0222] In one or more embodiments, the noise attention feature of the Mth spatiotemporal attention component in the video generation model includes a noise attention feature under text conditions and a noise attention component under no-text conditions; the noise attention feature under text conditions refers to an attention feature obtained by using video text encoding features, and the noise attention feature under no-text conditions refers to an attention feature obtained by using video text encoding features after being set to zero;

[0223] The denoising module 19 determines the video prediction noise based on the noise attention feature of the Mth spatiotemporal attention component in the video generation model, and is used to perform the following operations:

[0224] Generate conditional prediction noise based on the noise attention features under text conditions, and generate unconditional prediction noise based on the noise attention features under no-text conditions;

[0225] The noise difference between the conditional prediction noise and the unconditional prediction noise is obtained, and the unconditional prediction noise and the product of the noise intensity parameter and the noise difference are added to obtain the video prediction noise.

[0226] According to an embodiment of the present application, the steps involved in the video generation method shown in FIG3 above can be performed by various modules in the video generation device 1 shown in FIG12. For example, step S101 shown in FIG3 can be performed by the sample acquisition module 11 shown in FIG12, step S102 shown in FIG3 can be performed by the sample noise addition module 12 shown in FIG12, step S103 shown in FIG3 can be performed by the model acquisition module 13 shown in FIG12, step S104 shown in FIG3 can be performed by the sample input module 14 shown in FIG12, step S105 shown in FIG3 can be performed by the noise prediction module 15 shown in FIG12, step S106 shown in FIG3 can be performed by the parameter correction module 16 shown in FIG12, and so on.

[0227] According to one embodiment of the present application, the various modules in the video generating device 1 shown in Figure 12 can be separately or all merged into one or several modules (or units) to constitute, or some of the modules can be further split into at least two smaller units (or sub-units) in function, and the same operation can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In actual applications, the functions of one module can also be achieved by at least two modules, or the functions of at least two modules can be achieved by one module. In other embodiments of the present application, the video generating device 1 can also include other modules. In actual applications, these functions can also be implemented with the assistance of other modules, and can be implemented by the collaboration of at least two modules.

[0228] In an embodiment of the present application, during the training phase of the initial generative model, a sample video and a sample description text corresponding to the sample video can be obtained, a sample video frame sequence for training the initial generative model is obtained from the sample video, the first step number is randomly sampled, and sample original noise is generated; based on the first step number and the sample original noise, the sample video frame sequence is subjected to noise processing to obtain video input data. The initial generative model may include M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components are alternately connected, where M is an integer greater than 1. The video input data, the sample text encoding features corresponding to the sample description text, and the first embedding information corresponding to the first step number are input into the initial generative model, and the sample video frame sequence is subjected to noise prediction through the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model to obtain sample predicted noise. The entire initial generative model can be trained by the mean square error loss between the sample predicted noise and the sample original noise. Through the spatiotemporal residual component and spatiotemporal attention component in the initial generation model, the initial generation model can learn the temporal action information in videos of different time scales, improve the comprehensiveness of the temporal action information, and thus improve the generation quality of the video.

[0229] Please refer to Figure 13, which is a structural diagram of a computer device provided in an embodiment of the present application. As shown in Figure 13, the computer device 1000 can be a terminal device, for example, the terminal device 10a in the embodiment corresponding to Figure 1 above, or it can be a server, for example, the server 10d in the embodiment corresponding to Figure 1 above, which will not be limited here. For ease of understanding, this application takes the computer device as an example of a terminal device. The computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the processor 1001. As shown in FIG13 , the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0230] The network interface 1004 in the computer device 1000 can also provide network communication functions, and the optional user interface 1003 can also include a display and a keyboard. In the computer device 1000 shown in Figure 13, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0231] Obtain a sample video and a sample description text corresponding to the sample video, obtain a sample video frame sequence in the sample video, and determine the first step number and sample original noise for the sample video frame sequence;

[0232] According to the first step number and the original noise of the sample, the sample video frame sequence is subjected to data noise processing to obtain video input data, and the sample description text is subjected to text encoding processing to obtain the sample text encoding features;

[0233] Obtain an initial generative model; the initial generative model includes M spatiotemporal residual components and M spatiotemporal attention components, and the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model are alternately connected, where M is an integer greater than 1;

[0234] Input the video input data into the first of the M spatiotemporal residual components, input the first embedding information corresponding to the first step into the M spatiotemporal residual components, and input the sample text encoding features into the M spatiotemporal attention components;

[0235] In M spatiotemporal residual components and M spatiotemporal attention components, noise prediction is performed on the video input data according to the sample text encoding features and the first embedding information to obtain sample prediction noise;

[0236] According to the original noise of the sample and the predicted noise of the sample, the network parameters of the initial generation model are corrected, and the initial generation model including the corrected network parameters is determined as the video generation model; the video generation model is used to generate video data that matches the video description text.

[0237] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the video generation method described in any of the embodiments shown in Figures 3, 7, 9, and 11 above, and can also execute the video generation device 1 described in the embodiment shown in Figure 12 above, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated here.

[0238] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the video generation device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the description of the video generation method in any of the embodiments of Figures 3, 7, 9 and 11 above, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device for execution, or executed on multiple computer devices located at one location, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0239] In addition, it should be noted that: the embodiment of the present application also provides a computer program product, which may include a computer program, and the computer program may be stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, so that the computer device performs the description of the video generation method in any of the embodiments in Figures 3, 7, 9 and 11 above, and therefore, will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiments involved in this application, please refer to the description of the method embodiments of this application.

[0240] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different media contents, rather than to describe a specific order. In addition, the terms "including" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0241] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0242] The methods and related devices provided in the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided in the embodiments of the present application, and can be specifically implemented by computer programs for each process and / or box in the method flow charts and / or structural diagrams, as well as the combination of the processes and / or boxes in the flow charts and / or block diagrams. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the computer program executed by the processor of the computer or other programmable data processing device produces a device for implementing the function specified in one process or multiple processes in the flow chart and / or one box or multiple boxes in the structural diagram. These computer programs can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the computer program stored in the computer-readable memory produces a product including an instruction device, which implements the function specified in one process or multiple processes in the flow chart and / or one box or multiple boxes in the structural diagram. These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the computer program executed on the computer or other programmable device provides steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the structural diagram.

[0243] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0244] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video generation method, characterized in that: The method is executed by a computer device, and includes: Obtaining a sample video and a sample description text corresponding to the sample video, obtaining a sample video frame sequence in the sample video, and determining a first step number and a sample original noise for the sample video frame sequence; According to the first step number and the original noise of the sample, the sample video frame sequence is subjected to data noise processing to obtain video input data, and the sample description text is subjected to text encoding processing to obtain sample text encoding features; Obtain an initial generative model; the initial generative model includes M spatiotemporal residual components and M spatiotemporal attention components, the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model are alternately connected, and M is an integer greater than 1; Inputting the video input data into the first spatiotemporal residual component of the M spatiotemporal residual components, inputting the first embedding information corresponding to the first step into the M spatiotemporal residual components, and inputting the sample text encoding features into the M spatiotemporal attention components; In the M spatiotemporal residual components and the M spatiotemporal attention components, noise prediction is performed on the video input data according to the sample text encoding features and the first embedding information to obtain sample prediction noise; According to the sample original noise and the sample predicted noise, the network parameters of the initial generation model are modified, and the initial generation model including the modified network parameters is determined as the video generation model; the video generation model is used to generate video data that matches the video description text.

2. The method according to claim 1, characterized in that The obtaining of a sample video frame sequence from the sample video includes: Determine the frame extraction interval duration corresponding to the sample video according to the video frame number threshold and the video duration corresponding to the sample video; Acquire multiple video frames from the sample video according to the frame extraction interval, and form an initial video frame sequence from the multiple video frames; the number of video frames included in the initial video frame sequence is equal to the video frame number threshold; A sample video frame sequence for training the initial generation model is obtained from the initial video frame sequence; the sample video frame sequence is a subsequence of any one of the initial video frame sequences whose number of video frames is less than or equal to the video frame number threshold.

3. The method according to any one of claims 1 to 2, characterized in that The step of determining the first step number and the sample original noise for the sample video frame sequence comprises: Obtaining a diffusion parameter corresponding to the initial generation model, and sampling the first step number corresponding to the sample video frame sequence according to the diffusion parameter; the diffusion parameter is used to represent the total number of diffusion steps of the initial generation model, and the first step number is less than or equal to the diffusion parameter; Gaussian noise having the same dimension as the sample video frame sequence is generated, and the Gaussian noise is determined as the sample original noise corresponding to the sample video frame sequence.

4. The method according to any one of claims 1 to 3, characterized in that The step of performing data noise addition processing on the sample video frame sequence according to the first step number and the sample original noise to obtain video input data includes: Determining a sequence weight corresponding to the sample video sequence according to the first step number, and determining an original noise weight corresponding to the sample original noise according to the sequence weight; the sum of the squares of the sequence weight and the original noise weight is 1; The product of the sequence weight and the sample video sequence and the product of the original noise weight and the sample original noise are added together to obtain video input data.

5. The method according to any one of claims 1 to 4, characterized in that The performing text encoding processing on the sample description text to obtain the sample text encoding features includes: Divide the sample description text into D unit characters, and obtain the unit word vectors corresponding to the D unit characters respectively; D is a positive integer; Obtaining text vectors corresponding to the D unit characters respectively according to semantic information of the D unit characters in the sample description text; According to the text positions of the D unit characters in the sample description text, obtaining position vectors corresponding to the D unit characters respectively; Combining the unit word vector, the text vector, and the position vector to obtain initial text features corresponding to the sample description text; The initial text features are input into a text encoder, and the initial text features are encoded by the text encoder to obtain sample text encoding features corresponding to the sample description text.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: performing denoising processing on the video input data in the M spatiotemporal residual components and the M spatiotemporal attention components according to the sample text encoding features and the first embedding information to obtain sample prediction noise, including: Performing feature extraction on the first embedding information and the input data of the i-th spatiotemporal residual component through the i-th spatiotemporal residual component among the M spatiotemporal residual components to obtain a sample residual feature of the i-th spatiotemporal residual component; i is a positive integer less than or equal to M; Performing attention processing on the sample text encoding feature and the sample residual feature of the i-th spatiotemporal residual component through the i-th spatiotemporal attention component among the M spatiotemporal attention components to obtain the sample attention feature of the i-th spatiotemporal attention component; Determine the sample attention feature of the last spatiotemporal attention component among the M spatiotemporal attention components as the sample prediction noise; Among them, when i is 1, the input data of the i-th spatiotemporal residual component includes the video input data and the first embedding information; when i is not 1, the input data of the i-th spatiotemporal residual component includes the first embedding information and the sample attention features of the i-1-th spatiotemporal attention component.

7. The method according to claim 6, characterized in that The i-th spatiotemporal residual component includes a spatial convolution layer and a temporal convolution layer; when i is greater than M / 2+1, the input data of the i-th spatiotemporal residual component also includes sample residual features of the spatiotemporal residual component in the initial generative model that has a symmetrical position with the i-th spatiotemporal residual component; The step of performing feature extraction on the first embedding information and the input data of the i-th spatiotemporal residual component through the i-th spatiotemporal residual component among the M spatiotemporal residual components to obtain a sample residual feature of the i-th spatiotemporal residual component includes: Adding the first embedding information and the input data of the i-th spatiotemporal residual component to obtain a combined input feature; Performing a spatial convolution operation on the combined input features through the spatial convolution layer in the i-th spatiotemporal residual component to obtain spatial convolution features, and performing a dimensionality transformation on the spatial convolution features to obtain convolution transformation features; Through the temporal convolution layer in the i-th spatiotemporal residual component, a temporal convolution operation is performed on the convolution transformation feature to obtain a temporal convolution feature, and based on the temporal convolution feature and the input data of the i-th spatiotemporal residual component, the sample residual feature of the i-th spatiotemporal residual component is obtained.

8. The method according to claim 6, characterized in that The i-th spatiotemporal attention component includes a spatial attention subcomponent and a temporal attention subcomponent; The method of performing attention processing on the sample text encoding feature and the sample residual feature of the i-th spatiotemporal residual component by the i-th spatiotemporal attention component among the M spatiotemporal attention components to obtain the sample attention feature of the i-th spatiotemporal attention component includes: Performing spatial attention processing on the sample text encoding feature and the sample residual feature of the i-th spatiotemporal residual component through the spatial attention subcomponent in the i-th spatiotemporal attention component to obtain a spatial attention feature; The spatial attention feature is dimensionally transformed to obtain an attention transformation feature, and the attention transformation feature is subjected to temporal attention processing by the temporal attention subcomponent in the i-th spatiotemporal attention component to obtain a sample attention feature of the i-th spatiotemporal attention component.

9. The method according to claim 8, characterized in that The spatial attention subcomponent in the i-th spatiotemporal attention component performs spatial attention processing on the sample text encoding feature and the sample residual feature of the i-th spatiotemporal residual component to obtain a spatial attention feature, including: According to the transformation weight matrix corresponding to the spatial self-attention layer in the spatial attention subcomponent, the sample residual features of the i-th spatiotemporal residual component are transformed into a spatial query matrix, a spatial key matrix, and a spatial value matrix; Performing a dot product operation on the spatial query matrix and the transposed matrix of the spatial key matrix to obtain a first candidate weight matrix, and obtaining the first column number corresponding to the spatial query matrix; Normalizing the ratio of the first candidate weight matrix to the square root of the first column number to obtain a first attention weight matrix, and obtaining a spatial self-attention feature according to a dot product between the first attention weight matrix and the spatial value matrix; Performing an attention operation on the spatial self-attention feature and the sample text encoding feature through the spatial mutual attention layer in the spatial attention subcomponent to obtain a spatial mutual attention feature; The spatial mutual attention feature is transformed through the feedforward network layer in the spatial attention subcomponent to obtain the spatial attention feature.

10. The method according to claim 8, characterized in that The step of performing temporal attention processing on the attention transformation feature by the temporal attention subcomponent in the i-th spatiotemporal attention component to obtain a sample attention feature of the i-th spatiotemporal attention component includes: According to the transformation weight matrix corresponding to the temporal self-attention layer in the temporal attention subcomponent, the attention transformation matrix is transformed into a temporal query matrix, a temporal key matrix, and a temporal value matrix; Perform a dot multiplication operation on the time series query matrix and the transposed matrix of the time series key matrix to obtain a second candidate weight matrix, and obtain the number of second columns corresponding to the time series query matrix; Normalizing the ratio of the second candidate weight matrix to the square root of the second column number to obtain a second attention weight matrix, and obtaining a temporal self-attention feature according to a dot product between the second attention weight matrix and the temporal value matrix; The temporal self-attention feature is transformed through the feedforward network layer in the temporal attention subcomponent to obtain the sample attention feature of the i-th spatiotemporal attention component.

11. The method according to any one of claims 1 to 10, characterized in that The method of modifying the network parameters of the initial generation model according to the sample original noise and the sample predicted noise, and determining the initial generation model including the modified network parameters as the video generation model, comprises: Obtaining a mean square error loss between the sample original noise and the sample predicted noise; The network parameters of the initial generation model are iteratively trained according to the mean square error loss until the initial generation model meets the training end condition, the training is stopped, and the initial generation model at the end of the training is determined as the video generation model.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Obtaining a video description text and initialization noise, and determining a second step number corresponding to the initialization noise; Performing text encoding processing on the video description text to obtain video text encoding features, and inputting the initialization noise, the video text encoding features, and second embedding information corresponding to the second step number into the video generation model; Through the j-th spatiotemporal residual component in the video generation model, feature extraction is performed on the second embedding information and the input data of the j-th spatiotemporal residual component to obtain the noise residual feature of the j-th spatiotemporal residual component; through the j-th spatiotemporal attention component in the video generation model, attention processing is performed on the video text encoding feature and the noise residual feature of the j-th spatiotemporal residual component to obtain the noise attention feature of the j-th spatiotemporal attention component; when j is 1, the input data of the j-th spatiotemporal residual component includes the second embedding information and the initialization noise; when j is not 1, the input data of the j-th spatiotemporal residual component includes the video text encoding feature and the noise attention feature of the j-1-th spatiotemporal attention component, and j is a positive integer less than or equal to M; determining video prediction noise based on the noise attention feature of the Mth spatiotemporal attention component in the video generation model, and performing denoising on the initialization noise based on the video prediction noise and the second number of steps to obtain noisy video data corresponding to a third number of steps; wherein the third number of steps is less than the second number of steps; The noisy video data corresponding to the third step is denoised using the video generation model until target video data matching the video description text is generated.

13. The method according to any one of claims 1 to 12, characterized in that The noise attention features of the Mth spatiotemporal attention component in the video generation model include the noise attention features under the text condition, and Noise attention component in the no-text condition; The noise attention feature under the text condition refers to the attention feature obtained by using the video text encoding feature, and the noise attention feature under the no-text condition refers to the attention feature obtained by using the video text encoding feature after setting to zero; The determining of the video prediction noise according to the noise attention feature of the Mth spatiotemporal attention component in the video generation model includes: Generating conditional prediction noise based on the noise attention features under the text condition, and generating unconditional prediction noise based on the noise attention features under the no-text condition; A noise difference between the conditional prediction noise and the unconditional prediction noise is obtained, and the unconditional prediction noise and a product of a noise intensity parameter and the noise difference are added to obtain video prediction noise.

14. A video generating device, characterized in that: include: A sample acquisition module is used to obtain a sample video and a sample description text corresponding to the sample video, obtain a sample video frame sequence from the sample video, and determine the first step number and sample original noise for the sample video frame sequence; A sample noise adding module is used to perform data noise adding processing on the sample video frame sequence according to the first step number and the sample original noise to obtain video input data, and perform text encoding processing on the sample description text to obtain sample text encoding features; A model acquisition module, configured to acquire an initial generative model; the initial generative model comprises M spatiotemporal residual components and M spatiotemporal attention components, wherein the M spatiotemporal residual components and the M spatiotemporal attention components in the initial generative model are alternately connected, where M is an integer greater than 1; A sample input module, configured to input the video input data into the first spatiotemporal residual component of the M spatiotemporal residual components, input the first embedding information corresponding to the first step into the M spatiotemporal residual components, and input the sample text encoding features into the M spatiotemporal attention components; a noise prediction module, configured to perform noise prediction on the video input data in the M spatiotemporal residual components and the M spatiotemporal attention components according to the sample text encoding features and the first embedding information to obtain sample predicted noise; A parameter correction module is used to correct the network parameters of the initial generation model based on the sample original noise and the sample predicted noise, and determine the initial generation model containing the corrected network parameters as the video generation model; the video generation model is used to generate video data that matches the video description text.

15. A computer device, characterized in that: including memory and processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 13.

17. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images