Video generation method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

This application provides a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. The method includes: determining text information for describing the video; identifying words describing the foreground in the text information; identifying second noise data belonging to the foreground in first noise data of the video; and generating the video based on the words describing the foreground, the first noise data, the text information, and the second noise data. This application can enrich the details of the foreground and background in the video, improving the quality of the generated video.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, video generation technology is gradually becoming a research hotspot.

[0003] In related technologies, video generation techniques mainly use Generative Adversarial Networks (GANs), autoregressive models, and diffusion models, combined with conditions to guide the generation of videos. For example, a diffusion model can be used to generate a video based on a sentence. Summary of the Invention

[0004] This application provides a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the quality of the generated video.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video generation method, the method comprising: determining text information for describing a video; identifying words describing the foreground in the text information; identifying second noise data belonging to the foreground in first noise data of the video; and generating a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0007] This application provides a video generation apparatus, including:

[0008] The text determination module is used to determine the text information used to describe the video;

[0009] A foreground word recognition module is used to identify words describing the foreground in the text information;

[0010] A noise recognition module is used to identify second noise data belonging to the foreground in the first noise data of the video;

[0011] The video generation module is used to generate a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0012] This application provides an electronic device, the electronic device comprising:

[0013] Memory is used to store executable instructions or computer programs.

[0014] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video generation method provided in the embodiments of this application.

[0015] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the video generation method provided in this application.

[0016] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the video generation method provided in this application.

[0017] The embodiments of this application have the following beneficial effects:

[0018] By generating a video based on words describing the foreground, second noise data belonging to the foreground from the first noise data, text information, and the first noise data, the generation of foreground and background can be more accurately guided during the video generation process, thereby improving the realism and richness of details in the video. Furthermore, compared to manually specifying the foreground in the video, this embodiment of the application improves video generation efficiency and the accuracy of the generated foreground by recognizing words describing the foreground from text information and recognizing second noise data belonging to the foreground from the first noise data. Therefore, this embodiment of the application can enrich the details of the foreground and background in the video and improve the quality of the generated video. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the architecture of the video generation system provided in the embodiments of this application;

[0020] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0021] Figure 3 This is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure 1 ;

[0022] Figure 4 This is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure 2 ;

[0023] Figure 5 This is another flowchart illustrating the video generation method provided in this application embodiment;

[0024] Figure 6 This is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure 3 ;

[0025] Figure 7 This is another optional flowchart illustrating the video generation method provided in the embodiments of this application;

[0026] Figure 8 This is a flowchart of a control method for video generation based on a diffusion model provided in an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of the process of the large language model outputting foreground words provided in the embodiments of this application;

[0028] Figure 10 This is a schematic diagram of the interface for outputting foreground words by the large language model provided in this application embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0032] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0033] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0034] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0035] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0036] 1) Diffusion model: This is a data generation method that transforms noisy data into clean data, commonly used in image and video generation. For example, in video generation, the diffusion model is a deep learning model based on a probabilistic process, used to generate the final video based on given initial conditions (such as noisy data and text information).

[0037] 2) Noise data: This is random data used in the video generation process, typically Gaussian noise or other types of random noise. Noise data is the starting point for video generation and is ultimately transformed into video through a series of processing steps.

[0038] 3) Text information used to describe the video: This refers to a piece of text information used to describe the content of the video to be generated. The text information can include specific descriptions such as the video's theme, scene, characters, and actions, which are used to guide the diffusion model in generating a video that conforms to the described content.

[0039] 4) Adding noise: This refers to the process of adding noise to data such as images or audio. Noise can include Gaussian noise, etc. In the image processing of the diffusion model, adding noise is also called diffusion. In this process, the diffusion model gradually introduces noise, starting from the original image and gradually blurring the image until it becomes a state of random noise.

[0040] 5) Denoising: Denoising refers to the process of removing noise from data such as images or audio. In video generation, denoising is often used in conjunction with diffusion models. Denoising refers to the reverse diffusion process of diffusion models, which restores or generates clear video content by gradually removing noise.

[0041] 6) Foreground: This refers to the object or scene in the foreground of the video, usually the focus of the video. In the video generation process, the foreground typically refers to elements that need to be emphasized or highlighted.

[0042] 7) Background: This refers to the background or environment located at the back of the frame in a video, usually serving to highlight the foreground. The background provides richer environmental information and enhances the realism of the video.

[0043] 8) Self-attention processing: This is a deep learning technique used to calculate the degree of influence of each element in the input data on other elements. It can be achieved by introducing a self-attention mechanism into the neural network, thereby improving the model's ability to understand and generate input data.

[0044] 9) Cross-attention processing: This is an application of an attention mechanism primarily used to establish connections between different modalities (such as text and images). In video generation, it helps the model understand the relationship between textual information and video content, thereby generating videos that better match the description. Through the cross-attention mechanism, the diffusion model can dynamically adjust the degree of attention given to textual information during the generation process to ensure that the generated video content matches the description.

[0045] 10) Vision Transformer (VIT): A deep learning model architecture for images that can divide an image into a series of fixed-size image patches (called Patches) and encode these image patches into a series of basic units (Tokens). These Tokens are processed through a self-attention mechanism to extract image features.

[0046] 11) Basic Unit (Token): This is the smallest unit of processing in a vision model (Vision Transformer). In deep learning and natural language processing, a token refers to the basic unit into which text or images are segmented. For example, for text information, a token is usually a word or sub-word; for image data, it is an image patch.

[0047] 12) Mask: A mask is a representation of a basic unit (token), used to indicate which parts the model should focus on and which parts it should ignore. In image processing, a mask is typically a binary matrix of the same size as the input image, where 1 represents the region to be focused on and 0 represents the region to be ignored. In video generation, a mask can be used to indicate which foreground regions the model should prioritize when generating the video.

[0048] 13) Large Language Model (LLM): This refers to a natural language processing model with a large number of parameters. These models are usually pre-trained on large-scale text datasets and are able to understand and generate high-quality natural language text.

[0049] 14) Foreground bounding box: This refers to a bounding box with a predefined shape, such as a rectangle or circle, used to mark the boundary of the foreground in an image or video. In video generation tasks, foreground bounding boxes help diffusion models more accurately locate and process the foreground in the video, thereby improving the quality of the generated video. Foreground bounding boxes can be manually specified by the user or obtained through automatic detection.

[0050] 15) Latent Space: In machine learning and deep learning models, this refers to a low-dimensional continuous space to which the encoder maps the original input data. In the latent space, data is represented as a set of continuous variables, typically with a compact representation that captures the inherent structure and features of the input data.

[0051] 16) Latent Space Shape: This refers to the dimensions and structure of the data in the latent space. In deep learning models, the latent space shape is usually represented by a set of numbers that correspond to the various dimensions of the data in the latent space. For example, in a diffusion model, if the shape of the latent space is [1, 32, 8, 16, 16], this means that the data in the latent space has the following dimensions: Batch Size [1] indicates that one sample is processed at a time, Channels

[32] indicates the number of channels in the latent space, Length [8] indicates the length of the time series, Height

[16] indicates the height, and Width

[16] indicates the width.

[0052] Most video generation technologies utilize diffusion-based model frameworks, such as Make-a-video, Text2video-zero, and Imagen Video. Within these diffusion frameworks, one main network is based on a U-Net architecture, while another is based on a Transformer architecture. Initially, Transformer models were designed for sequence tasks rather than image tasks. However, with increased research, their scalability has allowed them to emerge in other domains, including image processing. With ample training data, Transformer models can often outperform traditional image model main networks like U-Net and ResNet. In image generation, various conditions and methods guide generation, such as Low-Rank Adapter (LoRA), text prompts, and graphical conditions. Each new condition-guided generation mode requires multiple rounds of training to achieve optimal results. Conditional guidance methods for video generation are still lacking in research, and video generation has an additional dimension compared to image generation, requiring significantly more training resources. Using the Transformer model as the main network to train the video generation model is even more costly. The following section introduces some commonly used model architectures in related technologies.

[0053] GAN-based Image Generators: These extend image generators based on Generative Adversarial Networks (GANs) to video generation models. However, GAN-based video generation methods often suffer from mode collapse during training. Autoregressive Models: These use autoregressive models to learn data distributions to generate videos. While these models typically offer good video quality and more stable convergence, training them requires significant computational resources. Diffusion Models: Diffusion models, particularly those based on Transformers, have not been fully explored. Transformer Backbones: In the image domain, the Transformer architecture has become the mainstream model, but methods for integrating it into diffusion models for video generation remain unclear. Compared to images, video faces significant challenges in generating high-quality videos due to its complexity and high dimensionality, including complex spatiotemporal information in high-resolution frames (Video Generation Challenges). U-Net in Video Generation: U-Net (a convolutional neural network-based architecture) has gained prominence in image and video generation work. However, for potential diffusion models based on Transformers, the inductive bias of U-Net is not critical. Long-range Contextual Relationships: Attention-based architectures offer an intuitive option for capturing long-range contextual relationships in videos. However, effectively integrating these architectures into video generation tasks remains a challenge. Pre-trained Models: While pre-trained models are commonly used for image generation, the effectiveness of pre-trained Transformer models in video generation has not been fully investigated.

[0054] In summary, the relevant technologies face the following technical challenges: Pre-trained models for video generation based on diffusion model frameworks suffer from high training costs when attempting to introduce interactive, graph-based conditional control. Common practices in video generation include unconditional and conditional video generation. Conditional video generation often involves generating a video based on a sentence, several images (video prediction, frame interpolation, etc.), or filling in black borders. The relevant technologies lack training-free conditional control methods based on Transformer-based networks, and the generated videos often exhibit limited foreground and background detail, a noticeable disconnect between foreground and background, and poor video quality.

[0055] In view of the problems existing in related technologies, this application provides a video generation method, apparatus, electronic device, computer-readable storage medium and computer program product, which can enrich the details of the foreground and background in the video and improve the quality of the generated video.

[0056] The following describes exemplary applications of the video generation device provided in this application embodiment. The video generation device provided in this application embodiment is an electronic device used to implement a video generation method. The electronic device provided in this application embodiment can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or it can be implemented as a server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment. The following will describe exemplary applications when the video generation device is implemented as a server or terminal.

[0057] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video generation system provided in this application embodiment. To achieve video generation, a video generation application can be provided. For example, the video generation application can be a dedicated application for video generation, or it can be a functional module in other applications (such as a video generation module in a game application). The video generation system 100 provided in this application embodiment includes at least a terminal 400, a network 300, and a server 200, wherein the server 200 is the server for the video generation application. The server 200 can constitute the video generation device of this application embodiment, that is, the video generation method of this application embodiment is implemented through the server 200. The terminal 400 is connected to the server 200 through the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0058] See Figure 1Users can perform interactive operations on the video generation application client via terminal 400. These operations can include inputting text information, setting first noise data, and clicking "start video generation." After receiving the user's interaction, the client can encapsulate the text information and first noise data into a video generation request and send it to server 200 via network 300. Upon receiving the video generation request, server 200 responds by determining the text information used to describe the video; identifying words describing the foreground in the text information; identifying second noise data belonging to the foreground in the first noise data of the video; and generating the video based on the words describing the foreground, the first noise data, the text information, and the second noise data. After generating the video, server 200 can also send the video to terminal 400 to display it to the user.

[0059] In some embodiments, the video generation method of this application embodiment can also be executed by the terminal 400 itself. That is, after the terminal 400 receives the interactive operation input by the user through the client, the terminal 400 determines the text information used to describe the video; the terminal 400 identifies words describing the foreground in the text information; the terminal 400 identifies second noise data belonging to the foreground in the first noise data of the video; and the terminal 400 generates the video based on the words describing the foreground, the first noise data, the text information, and the second noise data. After the video is generated and determined, the generated video is displayed on the client interface of the terminal 400.

[0060] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] Memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 450 may optionally include one or more storage devices physically located remote from processor 410. Memory 450 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0064] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 452 for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.; presentation module 453 for enabling the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., display screen, speaker, etc.); input processing module 454 for detecting and translating one or more user inputs or interactions from one or more input devices 432.

[0065] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A video generation apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: a text determination module 4551, a foreground word recognition module 4552, a noise recognition module 4553, and a video generation module 4554. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0066] In other embodiments, the video generation apparatus provided in this application can be implemented in hardware. As an example, the video generation apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0067] It should be noted that, based on the understanding of the following text, those skilled in the art can apply the video generation method provided in the embodiments of this application to any scenario where video is generated using artificial intelligence technology, such as film and television animation production scenarios, game development scenarios, live interactive scenarios, education and training scenarios, cultural preservation and reconstruction scenarios, medical and health visualization scenarios, experimental simulations, etc. The following description uses a scenario where video is generated based on pre-input text information using artificial intelligence technology as an example.

[0068] Figure 3 This is a flowchart illustrating the video generation method provided in the embodiments of this application. Figure 1 The following will combine Figure 3 The steps shown are explained as follows: Figure 3 As shown, taking the server as the execution subject of the video generation method as an example, the method includes the following steps 101 to 104:

[0069] Step 101: Determine the text information used to describe the video.

[0070] Here, text information refers to a piece of text used to describe the content of the video to be generated. The text information can include specific descriptions such as the video's theme, scene, characters, and actions. The text information can be entered by the user in the client interface of the video generation application. For example, the user-entered text information could be "A white cheetah with big eyes gallops across the grassland."

[0071] Step 102: Identify words in the text information that describe the foreground.

[0072] Here, the foreground is the object or scene in the foreground of the video. The text information can include multiple words, among which the words describing the foreground are words used to describe the scene, object or action in the foreground of the video. First, the text information can be preprocessed, including word segmentation, removal of stop words, etc., to ensure that the subsequent word recognition process is more accurate. Next, natural language processing technology (such as large language model (LLM)) is used to identify the words describing the foreground in the text information. Taking the text information "a white cheetah with big eyes is galloping on the grassland" as an example, after preprocessing the text information, the multiple words obtained are "white", "big eyes", "cheetah", "grassland", "galloping". Using the large language model (LLM) to identify the preprocessed text information, the words describing the foreground are "white", "big eyes", "cheetah", "galloping", etc. Here, the words describing the foreground obtained by filtering can have foreground markers, such as foreground markers as subscripts [1].

[0073] In related technologies, methods that manually determine words describing the foreground often rely solely on nouns. For example, in the text "A white cheetah with big eyes is galloping across the grassland," the manually determined foreground word is simply "cheetah," ignoring highly relevant terms like "white" and "big eyes," as well as the action word "galloping." This can lead to inaccurate foreground guidance during video playback. In contrast, this application's embodiment uses a large language model to automatically identify words describing the foreground from the text, improving the ability to segment foreground and background in complex scenes. This provides more precise guidance for foreground and background during video generation, enriching the details of the foreground and background in the video.

[0074] Step 103: Identify the second noise data belonging to the foreground in the first noise data of the video.

[0075] Here, in the video generation method provided in this application embodiment, first noise data and text information are used as inputs to a diffusion model. In the diffusion model, the first noise data is a set of random values ​​extracted from a probability distribution (such as a Gaussian distribution). The first noise data is used to initialize the video generation process and is gradually denoised into image frames in the video through the diffusion model. The shape of the first noise data corresponds to the dimension of the image frames in the generated video. For example, in the diffusion model, if the shape of the generated image frame is [Batch Size, Channels, Length, Height, Width], then the first noise data also has the same shape as the image frame. For example, the first noise data can be a truncated Gaussian distribution in the latent space shape. For example, if the shape of the latent space is [1, 32, 8, 16, 16], then a truncated Gaussian distribution noise of that latent space shape is generated. It should be noted that for each image frame in the video, the value of the first noise data extracted from the probability distribution is different, but the shape is the same.

[0076] After randomly obtaining the first noise data, the first noise data can be divided into blocks to obtain multiple third noise data. Based on the position information of the foreground, the second noise data belonging to the foreground is determined from the multiple third noise data. The position information of the foreground is used to indicate the location of the foreground in the first noise data, that is, to specify the area where the foreground is displayed in the video. Here, the obtained second noise data can have a foreground marker, such as the foreground marker being a subscript [1].

[0077] In some embodiments, see Figure 4 , Figure 4 The second noise data belonging to the foreground in the first noise data of the video identified in step 103 can be achieved through the following steps 1031 to 1032:

[0078] Step 1031: Determine the position information of the foreground of the image frame in the video.

[0079] Here, the video may include multiple image frames. Each image frame may include a predefined foreground frame, within which the foreground will be displayed. The foreground frame is a border with a predefined shape, such as a rectangle or circle, used to mark the boundary of the foreground within the image frame. The foreground frame can be obtained by the user manually inputting its position information. This position information includes the coordinates of the center point, width, and height of the foreground frame. For example, the foreground position information can be represented as [X, Y, H, W], where X and Y represent the x and y coordinates of the center point of the foreground frame, and H and W represent the pixel values ​​occupied by the height and width of the foreground frame, respectively. Assuming the image frame size is 100px × 100px, and the center point coordinates are set to (0, 0), if the foreground position information can be represented as [0, 0, 20, 20], then the foreground frame is a 20×20 rectangle with a center point coordinate of (0, 0) within the image frame.

[0080] In some embodiments, determining the foreground position information of an image frame in the video in step 1031 can be achieved as follows: First, determine the first position information of the first foreground of the first image frame and the second position information of the second foreground of the second image frame. Then, determine the third position information of the first image frame in the video and the fourth position information of the second image frame in the video. Finally, based on the first position information, the second position information, the third position information, the fourth position information, and the number of image frames in the video, determine the foreground position information for each image frame.

[0081] Here, the video may include multiple image frames, where the first image frame and the second image frame are any two different image frames from the multiple image frames. For example, the first image frame can be the start frame of the video, and the second image frame can be the end frame of the video. The first position information of the first foreground and the second position information of the second foreground can be the same or different. The user can input the first position information of the first foreground and the second position information of the second foreground. The multiple image frames in the video are arranged in chronological order, with each image frame representing a point in time within the video. The third position information refers to the specific position of the first image frame in the video, and the fourth position information refers to the specific position of the second image frame in the video, such as a timestamp or frame number. Taking the third and fourth position information as timestamps as an example, the timestamp can be in seconds or milliseconds. For example, if the first image frame of the video is the start frame, then the third position information of the first image frame is a timestamp of 0 seconds; if the second image frame is an adjacent image frame to the start frame, then the fourth position information is 0.033 seconds (if the frame rate is 30fps). Taking the third and fourth position information as frame numbers as an example, suppose the user specifies the first image frame as the starting frame of the video, that is, the third position information is frame number 1. At the same time, the user also specifies the second image frame as the 5th frame in the video after the first image frame, that is, the fourth position information is frame number 5.

[0082] The positional change patterns of the first and second foreground elements in the video can be calculated based on the first, second, third, and fourth positional information. For each image frame between the first and second image frames, the positional information of the foreground element in the intermediate frame is calculated based on the first and second positional information, the number of image frames, and the positional change pattern. This application does not limit the method for determining the positional change pattern; for example, a continuous target recognition algorithm can be used for calculation.

[0083] Compared to manually setting the foreground position, which can lead to a noticeable disconnect between the foreground and background in the generated video, the embodiments of this application can automatically determine the foreground position corresponding to each image frame based on two preset foreground positions and the number of image frames in the video. This ensures the temporal and spatial continuity of the foreground in each frame of the generated video, and helps to improve the accuracy and consistency of the foreground during the video generation process.

[0084] Step 1032: Based on the location information of the foreground, mark the second noise data in the first noise data.

[0085] Here, the first noise data can be divided into blocks to obtain multiple third noise data. Second noise data is then determined from these multiple third noise data based on the foreground's location information.

[0086] This application embodiment determines the generation of second noise data with foreground markers based on the foreground location information, i.e., noise data used to generate the foreground. This will provide guidance for the subsequent video generation process, ensuring that the foreground is displayed in the position where the user expects the foreground to be displayed in the generated video, thereby improving video quality.

[0087] In some embodiments, the step 1032 of marking the second noise data in the first noise data based on the position information of the foreground can be implemented in the following way: First, the first noise data is divided into blocks to obtain the third noise data; then, based on the position information of the foreground, the fourth noise data is determined from the third noise data, wherein the fourth noise data is the third noise data that overlaps with the foreground; then, the overlap area between the foreground and the fourth noise data is determined, and the ratio of the overlap area to the area of ​​the fourth noise data is determined as the overlap ratio; finally, based on the overlap ratio and a preset ratio threshold, the second noise data is marked in the fourth noise data.

[0088] Here, a Vision Transformer (VIT) model can be used to segment the first noise data into blocks, resulting in multiple third noise data (Tokens). The foreground position information is scaled based on the shape of the first noise data and the size of the image frame to obtain the target position information. Alternatively, the foreground position information can be scaled based on the ratio of the width of the first noise data to the width of the image frame to obtain the target position information. The target position information of the foreground can be represented by a foreground bounding box of a preset shape. Based on the target position information of the foreground, the foreground bounding box is scaled onto multiple third noise data, resulting in at least one third noise data overlapping with the foreground bounding box. Each overlapping third noise data is designated as a fourth noise data. It should be noted that "overlapping" here means that at least one of the third noise data is located within the foreground bounding box, that is, the area of ​​the third noise data partially or completely overlaps with the area of ​​the foreground. For example, the width and height of the first noise data are both 50, while the size of the image frame is 100×100, and the position information of the foreground is [0, 0, 20, 20]. Then, the ratio of the width of the first noise data to the width of the image frame is 1 / 2. Based on the width ratio, the position information of the foreground is scaled to obtain the target position information [0, 0, 10, 10]. The foreground box is placed on the 50×50 first noise data according to the target position information [0, 0, 10, 10]. If the first noise data is divided into 9 third noise data, and 4 of the third noise data overlap with the area selected by the foreground box, then these 4 third noise data are determined as 4 fourth noise data.

[0089] For each fourth noise data point, the area of ​​the fourth noise data and the scaled area of ​​the foreground can be obtained. Based on the area of ​​the fourth noise data and the scaled area of ​​the foreground, the overlap area between the fourth noise data and the foreground is determined. The calculation method for the overlap area can refer to the calculation method for the area of ​​a rectangle, and is not limited in detail in this embodiment. The ratio of the overlap area to the area of ​​the fourth noise data is determined as the overlap ratio. Assuming the overlap area is A and the area of ​​the fourth noise data is B, the overlap ratio is A / B. The preset ratio threshold is a threshold used to determine whether a certain fourth noise data can be regarded as second noise data. This embodiment does not limit the value of the preset ratio threshold, and it can be set according to actual needs. When the overlap ratio is greater than or equal to the preset ratio threshold, the fourth noise data is determined as second noise data.

[0090] This application embodiment utilizes VIT to segment the first noise data, which improves the processing efficiency of the noise data, and generates second noise data with foreground markers based on the foreground's position information. This will provide guidance for the subsequent video generation process, ensuring that the foreground in the generated video is displayed at the position where the user expects the foreground to be displayed, thereby improving the quality of the generated video.

[0091] In this embodiment of the application, the second noise data is marked in the fourth noise data based on the overlap ratio and a preset ratio threshold. This can be achieved in the following ways: if the overlap ratio is greater than or equal to the preset ratio threshold, the fourth noise data is determined as the second noise data; or, if the overlap ratio is less than the preset ratio threshold, a random number is generated, and if the random number is less than or equal to the overlap ratio, the fourth noise data is determined as the second noise data.

[0092] Here, for the fourth noise data, if the overlap ratio between the fourth noise data and the foreground is greater than or equal to a preset threshold, the fourth noise data is directly identified as the second noise data. If the overlap ratio is less than the preset threshold, the overlap ratio is determined as a sampling probability, and the fourth noise data is randomly identified as belonging to the second noise data based on the sampling probability. For example, 0-1 sampling can be performed based on the sampling probability: a random number is randomly generated within the interval 0-1, and if the random number is less than or equal to the sampling probability, the fourth noise data is identified as the second noise data.

[0093] For example, with a preset ratio threshold of 0.5, when the overlap ratio between the fourth noise data and the foreground is 0.6, the fourth noise data is determined as the second noise data. When the overlap ratio between the fourth noise data and the foreground is 0.4, 0.4 is determined as the sampling probability. A 0-1 sampling is performed, randomly selecting a random number from the interval [0,1]. If the random number is 0.3, which is less than the sampling probability of 0.4, the fourth noise data is determined as the second noise data. If the random number is 0.45, which is greater than the sampling probability of 0.4, the fourth noise data is not the second noise data.

[0094] This application embodiment improves the accuracy of determining the second noise data by setting a preset ratio threshold, ensuring that only fourth noise data with sufficient overlap with the foreground is marked as second noise data for foreground generation. The preset ratio threshold can be flexibly adjusted according to actual needs, thereby controlling which fourth noise data is marked as second noise data and increasing the controllability of the foreground during video generation. Simultaneously, by handling cases with an overlap ratio less than the preset ratio threshold through random sampling, areas with some correlation to the foreground but a low overlap ratio can be appropriately considered while maintaining the prominence of foreground elements, thus better balancing the relationship between the foreground and background and improving the quality of the generated video.

[0095] In some embodiments, Figure 5 This is another flowchart illustrating the video generation method provided in this application embodiment, such as... Figure 5 As shown, after identifying the second noise data belonging to the foreground in the first noise data of the video in step 103, the video generation method provided in this application embodiment further includes the following steps 201 to 202:

[0096] Step 201: Determine the distance between the center point of the second noise data and the center point of the foreground.

[0097] Here, after determining the second noise data from multiple third noise data based on the overlap ratio, for each second noise data, the center point of the second noise data and the center point of the scaled foreground can also be obtained. Both the center point of the second noise data and the center point of the foreground are represented in coordinate form. The distance between the center point of the second noise data and the center point of the foreground can be calculated based on the coordinates of the center point of the second noise data and the center point of the foreground.

[0098] Step 202: If the distance is greater than a preset distance threshold, change the second noise data to the fifth noise data belonging to the background.

[0099] Here, the preset distance threshold refers to a threshold used to determine whether the distance between the center point of the second noise data and the center point of the foreground is sufficiently close. The preset distance threshold is set according to actual needs and is used to determine whether to change the second noise data into fifth noise data belonging to the background. A smaller preset distance threshold means that only areas very close to the center of the foreground will be considered second noise data, while a larger preset distance threshold will cause more areas in the first noise data to be considered second noise data. For each second noise data, when the distance between the center point of the second noise data and the center point of the foreground is less than or equal to the preset distance threshold (e.g., 10 pixels), the second noise data remains unchanged. When the distance between the center point of the second noise data and the center point of the foreground is greater than the preset distance threshold (e.g., 10 pixels), the foreground label of the second noise data is removed, and it is changed to fifth noise data belonging to the background.

[0100] This application embodiment adjusts the second noise data based on the distance between the center point of the second noise data and the center point of the foreground, ensuring that the second noise data more accurately corresponds to the foreground, providing more accurate guidance for the subsequent video generation process, and improving the quality and naturalness of the generated video.

[0101] Step 104: Generate a video based on words describing the foreground, first noise data, text information, and second noise data.

[0102] Here, text information and first noise data can be input into a pre-trained video generation model (e.g., a Transformer-based diffusion model). The pre-trained video generation model processes the text information and first noise data based on words describing the foreground and second noise data to obtain the video.

[0103] This application embodiment generates video by comprehensively considering words describing the foreground, second noise data, text information, and first noise data. This allows for more precise guidance in the generation of foreground and background during video generation, thereby improving the realism and richness of details in the video. Furthermore, compared to manually specifying the foreground in the video, this application embodiment obtains words describing the foreground through word recognition of text information and identifies second noise data belonging to the foreground through identification of the first noise data. This improves video generation efficiency and the accuracy of the generated foreground. Therefore, this application embodiment can enrich the details of the foreground and background in the video, improving the quality of the generated video.

[0104] In some embodiments, see Figure 6 , Figure 6 The step 104, which generates a video based on words describing the foreground, first noise data, text information, and second noise data, can be achieved through steps 1041 to 1043:

[0105] Step 1041: Encode the text information to obtain a text vector.

[0106] In this context, the word vectors corresponding to the words describing the foreground in the text vectors have foreground labels.

[0107] Here, after inputting the text information and the first noise data into the pre-trained video generation model, the encoder in the video generation model can encode each word in the text information separately, obtaining the word vector of each word. The word vectors of multiple words are combined to obtain the text vector. Among them, the word vectors corresponding to the words describing the foreground have foreground labels.

[0108] Step 1042: Map the first noise data to obtain the target noise vector.

[0109] Among them, the noise vector corresponding to the second noise data in the target noise vector has a foreground label.

[0110] Here, since the first noise data is initially a three-dimensional vector representation, the visual information processing model (VIT) can be used to segment the noise data into blocks, resulting in multiple third noise data points. Each third noise data point is then mapped to obtain a one-dimensional noise vector. Position encoding is performed on the noise vector corresponding to each third noise data point to obtain the target noise vector. The noise vector corresponding to the second noise data point has a foreground marker.

[0111] Step 1043: Generate video based on text vector and target noise vector.

[0112] Here, the text vector and the target noise vector can be combined and used as input to the video generation model. The video is then generated using a pre-trained video generation model (such as a Transformer-based diffusion model).

[0113] The embodiments of this application generate videos based on a combination of text vectors and target noise vectors, which can effectively improve the overall quality and visual effects of the videos. Furthermore, both the text vectors and target noise vectors contain foreground markers, which can more accurately guide the generation of the foreground during the video generation process, thereby improving the realism and richness of video details.

[0114] In this embodiment, the video is generated through T iterations based on text vectors and target noise vectors, where T is a positive integer greater than 1. Step 1043, generating the video based on text vectors and target noise vectors, can be achieved as follows: For the t-th iteration, the following processing is performed: If the t-th iteration is a first predetermined iteration, the intermediate noise data of the t-th iteration is denoised based on the foreground markers of the text vectors, the foreground markers of the target noise vectors, and the text vectors to obtain denoised data for the t-th iteration, and this denoised data is determined as the intermediate noise data for the (t+1)-th iteration; If the t-th iteration is a second predetermined iteration, the intermediate noise data of the t-th iteration is denoised based on the text vectors to obtain denoised data for the t-th iteration, and this denoised data is determined as the intermediate noise data for the (t+1)-th iteration, where the intermediate noise data of the first iteration is the target noise vector, 1≤t≤T; the denoised data of the T-th iteration is then determined as the video.

[0115] Here, the first set round refers to the rounds in the iteration process that require denoising based on foreground markers, and the second set round refers to the rounds in the iteration process that do not require denoising based on foreground markers. In other words, the second set round is the round in T iterations excluding the first set round.

[0116] For example, T is 1000, the first set rounds are rounds 1-10, and the second set rounds are rounds 11-1000. During the 1000-round iterative processing, the first 10 iterations denoise the intermediate noise data based on foreground markers and text vectors to obtain denoised data. The last 990 iterations denoise the intermediate noise data based only on text vectors to obtain denoised data. First, the first round of processing is performed. The intermediate noise data in the first round is the target noise vector. Based on the foreground markers of the text vector, the foreground markers of the target noise vector, and the text vector, the target noise vector is denoised to obtain the denoised data for the first round. This denoised data is then used as the intermediate noise data for the second round. In the second round of processing, since the size of the intermediate noise data in the second round may differ from the size of the initial noise data, it is necessary to first determine the second noise data from the intermediate noise data in the second round based on the foreground position information, and then generate the target noise vector corresponding to the intermediate noise data in the second round. Based on the foreground markers of the text vector, the foreground markers of the target noise vector in the second round, and the text vector, denoising is performed on the intermediate noise data of the second round to obtain the denoised data of the second round. This denoised data of the second round is then used as the intermediate noise data of the third round. This process is repeated until the 11th iteration. The intermediate noise data of the 11th round is then denoised based on the text vector to obtain the denoised data of the 11th round, and this denoised data of the 11th round is then used as the intermediate noise data of the 12th round. When the denoised data of the 1000th round is obtained, it is identified as an image frame. Multiple image frames are then sorted and combined to obtain the video.

[0117] In this embodiment, the first 10 iterations ensure that the foreground and background focus more on their own generation during the initial state, ignoring the complex interactions of the fusion process. This results in higher video quality and more realistic details. Furthermore, the subsequent 990 iterations ensure that the intersection of the foreground and background participates in the calculation, merging the foreground and background and reducing the sense of separation between them in the video.

[0118] For example, T is 1000, and T can be divided into a first set round and a second set round based on a preset interval. The first set round and the second set round switch every preset interval. The preset interval can be calculated based on T. For example, the preset interval can be 3, meaning the round type switches every 3 rounds. Starting from round 1, the operation of the first set round is performed every 3 rounds, for example, round 1, round 4, round 7, etc. The operation of the second set round is performed in rounds other than the first set round, for example, round 2, round 3, round 5, round 6, etc.

[0119] This application embodiment, by alternately calculating the foreground and background during the multi-round iterative processing of video generation, makes the foreground and background blending in the generated video more natural and improves the video quality.

[0120] In this embodiment, the intermediate noise data of round t is denoised based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector to obtain the denoised data of round t. This can be achieved in the following way: First, based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector, noise prediction processing is performed on the intermediate noise data of round t to obtain the predicted noise of round t; then, the predicted noise of round t is removed from the intermediate noise data of round t to obtain the denoised data of round t.

[0121] For example, for the first iteration, noise prediction processing can be performed on the intermediate noise data of the first iteration based on the foreground markers of the text vector, the foreground markers of the target noise vector, and the text vector to obtain the predicted noise of the first iteration. Then, the predicted noise of the first iteration is subtracted from the intermediate noise data of the first iteration to obtain the denoised data of the first iteration.

[0122] The embodiments of this application can gradually remove noise through an iterative denoising process, thereby improving the overall quality and visual effect of the video. At the same time, considering the foreground markers of the text vectors and the foreground markers of the target noise vectors during the denoising process can make the foreground and background details in the generated video richer.

[0123] In this embodiment, noise prediction processing is performed on the intermediate noise data of round t based on the foreground markers of the text vector, the foreground markers of the target noise vector, and the text vector to obtain the predicted noise of round t. This can be achieved in the following way: First, a first attention processing is performed on the intermediate noise data of round t and the round variable to obtain a first attention feature; then, based on the foreground markers of the text vector and the foreground markers of the target noise vector, a second attention processing is performed on the text vector and the intermediate noise data of round t to obtain a second attention feature; and finally, based on the foreground markers of the text vector and the foreground markers of the target noise vector, a self-attention processing is performed on the intermediate noise data of round t to obtain a third attention feature; then, the first attention feature, the second attention feature, and the third attention feature are fused to obtain the predicted noise of round t.

[0124] Here, taking t=1 as an example, the intermediate noise data in the first round is the target noise vector. First attention processing is applied to the target noise vector and the round variable to obtain the first attention feature. The first attention processing process involves using the target noise vector as the query matrix and the round variable as the key matrix. The weight parameter matrix is ​​determined based on the product of the query matrix and the key matrix. The round variable is used as the value matrix, and the product of the weight parameter matrix and the value matrix is ​​normalized using the normalized exponential function (softmax function) to obtain the first attention feature. In the second attention processing, the target noise vector is used as the query matrix, and the text vector is used as the key matrix and value matrix, undergoing a similar calculation process to the first attention processing. In the third attention processing, the target noise vector is used as the query matrix, key matrix, and value matrix, undergoing a similar calculation process to the first attention processing. The first, second, and third attention features are then fused and weighted to obtain the predicted noise for the first round.

[0125] This application embodiment applies three different attention processing processes to text vectors, noise data, and round variables, and fuses the results of the three attention processing processes to obtain predicted noise. This can make the generated video more consistent with the description in the text information and more continuous.

[0126] In this embodiment, based on the foreground markers of the text vector and the foreground markers of the target noise vector, a second attention processing is performed on the text vector and the intermediate noise data in the t-th round to obtain a second attention feature. This can be achieved in the following way: First, for the word vectors in the text vector and the noise vectors in the intermediate noise data in the t-th round, if either the word vector or the noise vector has a foreground marker, a set numerical vector is determined as an attention sub-feature; or, if both the word vector and the noise vector have foreground markers, or neither has a foreground marker, cross-attention processing is performed on the word vector and the noise vector to obtain an attention sub-feature; then, the attention sub-features are combined to obtain the second attention feature.

[0127] Here, the numerical vector can be set to zero. Taking t=1 as an example, in the second attention processing, for word vectors in the text vector and noise vectors in the target noise vector, when the word vector has a foreground marker but the noise vector does not, or vice versa, the zero vector is determined as the attention sub-feature, meaning that attention calculation is not performed on this word vector and noise vector. When both word vectors and noise vectors have foreground markers, or neither has a foreground marker, cross-attention processing is performed on the word vectors and noise vectors to obtain attention sub-features. Cross-attention processing refers to using the noise vector as the query matrix and the word vector as the key matrix. The weight parameter matrix is ​​determined based on the product of the query matrix and the key matrix. The word vector is used as the value matrix, and the product of the weight parameter matrix and the value matrix is ​​normalized using the normalized exponential function (softmax function) to obtain the attention sub-features. Combining the corresponding attention sub-features of each word vector and noise vector yields the second attention feature.

[0128] This application embodiment enables the foreground and background to generate images separately by performing cross-attention calculation only on word vectors and noise vectors that have the same foreground label or neither of them has a foreground label, thus enriching the detail of the foreground and background.

[0129] Figure 7 This is another optional flowchart illustrating the video generation method provided in the embodiments of this application, such as... Figure 7 As shown, the method includes the following steps 301 to 309:

[0130] Step 301: The terminal receives the user's interactive operation.

[0131] Here, user interaction operations can include inputting text information, setting the first noise data, clicking to start generating video, etc.

[0132] Step 302: The terminal responds to the interactive operation and generates a video generation request.

[0133] Here, after receiving the user's interactive operation, the terminal can encapsulate the text information and the first noise data into the video generation request.

[0134] Step 303: The terminal sends a video generation request to the server.

[0135] Step 304: In response to the video generation request, the server determines the text information used to describe the video.

[0136] Here, the text information is used to describe the video to be generated. The specific process of determining the text information used to describe the video can be referred to step 101 in the above embodiment, and will not be repeated here.

[0137] Step 305: The server identifies words in the text information that describe the foreground.

[0138] Here, the specific process of identifying words describing the foreground in the text information can refer to step 102 in the above embodiment, and will not be repeated here.

[0139] Step 306: The server identifies the second noise data belonging to the foreground in the first noise data of the video.

[0140] Here, the specific process of identifying the second noise data belonging to the foreground in the first noise data of the video can refer to step 103 in the above embodiment, and will not be repeated here.

[0141] Step 307: The server generates a video based on words describing the foreground, first noise data, text information, and second noise data.

[0142] Here, the specific process of generating a video based on words describing the foreground, first noise data, text information, and second noise data can be referred to step 104 in the above embodiment, and will not be repeated here.

[0143] Step 308: The server sends the video to the terminal.

[0144] Step 309: The terminal displays the video on the current interface.

[0145] This application embodiment generates video by comprehensively considering words describing the foreground, second noise data, text information, and first noise data. This allows for more precise guidance in the generation of foreground and background during video generation, thereby improving the realism and richness of details in the video. Furthermore, compared to manually specifying the foreground in the video, this application embodiment obtains words describing the foreground through word recognition of text information and identifies second noise data belonging to the foreground through identification of the first noise data. This improves video generation efficiency and the accuracy of the generated foreground. Therefore, this application embodiment can enrich the details of the foreground and background in the video, improving the quality of the generated video.

[0146] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0147] It should be noted that the video generation method provided in this application embodiment can be applied to any scenario in which video is generated using artificial intelligence technology, such as film and television animation production scenarios, game development scenarios, live interactive scenarios, education and training scenarios, cultural preservation and reconstruction scenarios, medical and health visualization scenarios, experimental simulations, etc. The following explanation uses a scenario where video is generated based on pre-input text information using artificial intelligence technology as an example.

[0148] The video generation method provided in this application is a control method for video generation based on a diffusion model. Figure 8 This is a flowchart of a control method for video generation based on a diffusion model, provided in an embodiment of this application. See also... Figure 8 First, the input to the diffusion model is noise data 900 (latent space), text data 901 (condition 1) (corresponding to the text information in the above embodiment), and a graphic guide box 902 (condition 2) (corresponding to the foreground box in the above embodiment). Assuming the diffusion process of the diffusion model has 1000 time steps, the input noise data (corresponding to the first noise data in the above embodiment) is the noise data at the 1000th time step of the diffusion process, such as Gaussian noise in latent space distributed as [1, 32, 8, 16, 16]. Assuming the video has 10 image frames, each image frame randomly obtains noise from a Gaussian distribution, meaning the noise data for each image frame is different. The graphic guide box can be a varying window of a given foreground pixel range in the image frame, represented by a center point, and can be [X, Y, H, W], where X and Y represent the coordinates of the center point of the graphic guide box, and H and W represent the pixel values ​​occupied by the height and width of the graphic guide box, respectively. The same graphic guide box is used in a single video generation process. Alternatively, multiple graphic guide boxes can be input to map the graphic guide boxes to image frames: input a graphic guide box for a starting frame and a graphic guide box for an ending frame, and then deduce the graphic guide box in each image frame based on the frame number of the image frame.

[0149] The text data 901 (text-prompt) could be "Pandas are playing hide-and-seek in the bamboo forest". Figure 9 This is a flowchart illustrating the output of foreground words using a large language model according to an embodiment of this application. Step 501: Input text data. Step 502: Separate the text data based on the large language model to obtain foreground words and background words. Step 503: Mask the foreground and background words. That is, mark the foreground words (corresponding to the words describing the foreground in the above embodiment) with a foreground label "1" and the background words with a background label "0". Using a large language model (LLM, such as llama3 8B) to output the foreground words and related words in the text data 901, then the panda is the foreground word, and the bamboo forest is the background word. Figure 10 This is a schematic diagram of the interface for outputting foreground words by the large language model provided in this embodiment. The prompt word strategy in the large language model processing can be a chain-of-thought (CoT), which can preserve richer details of the foreground topic.

[0150] The diffusion model is a pre-trained model comprising a Transformer main network and a diffusion denoising sampling network. The diffusion model includes an encoding module 903 (Embeddings to Tokens), a multi-head spatiotemporal self-attention module 904, a layer normalization module 905, and a linear reshaping module 906. The encoding module 903 converts text data into embedding vectors (corresponding to the text vectors in the above embodiments), where one word in the text data corresponds to one token. Noise data is processed in blocks using the VIT method to obtain token vectors (converting from three-dimensional to one-dimensional vector representation, corresponding to the target noise vector in the above embodiments). In this step, the encoding module 903 also performs positional encoding on the noise data. The encoding module 903 scales the graphical guide box accordingly to the proportional space of the noise data and performs foreground labeling (masking) on ​​the token at each position in the graphical guide box (corresponding to the third noise data in the above embodiments).

[0151] The foreground labeling process is as follows: To better integrate the foreground with the background during the spatial-attention process of the multi-head spatiotemporal self-attention module 904, based on the above mechanism, the foreground labeling corresponding to each patch in the graphical bounding box is reset using the following strategy during the first K time steps of the diffusion model. If the intersection-over-union (IoU) ratio between the graphical bounding box and the patch (corresponding to the fourth noise data in the above embodiment) exceeds 50% (corresponding to the preset ratio threshold in the above embodiment), then that position is considered foreground noise (corresponding to the second noise data in the above embodiment). The IoU ratio (corresponding to the overlap ratio in the above embodiment) is the intersection area (overlap area) of the graphical bounding box and the corresponding position divided by the sum of the areas of the positions (the area of ​​the fourth noise data), and is an indicator for judging the degree of overlap between two positions. If the IoU ratio is less than 50%, then the IoU ratio is used as a probability, and the position is randomly identified as foreground based on this probability. For locations already marked as foreground, there is a certain probability of removing the foreground marker. This probability is determined by the Euclidean space position of the location relative to the foreground location before expansion; that is, by calculating the distance between the center of the location and the center of the graphic guide frame. If the distance is greater than a preset distance threshold, the foreground marker can be removed. This embodiment of the application uses the above method to overcome the problem of foreground and background separation in videos.

[0152] The multi-head spatiotemporal self-attention module 904 includes three attention mechanisms: spatiotemporal cross attention, spatiotemporal text cross attention, and spatial self-attention. In video generation, spatiotemporal cross attention (corresponding to the first attention processing in the above embodiments) allows the diffusion model to focus on spatial relationships between different time steps. The query matrix Q is noise data, and the key matrix K and value matrix V are both time steps (corresponding to the round variables in the above embodiments). Spatiotemporal text cross attention (corresponding to the second attention processing in the above embodiments) focuses on relevant spatial regions in the image or video based on text prompts. The query matrix Q is noise data, and the key matrix K and value matrix V are both text data. Spatial self-attention (corresponding to the self-attention processing in the above embodiments) focuses on spatial relationships within the image or video, such as the interaction between different noise data. The query matrix Q is noise data, and the key matrix K and value matrix V are both noise data. A masking operation is performed on these three attention mechanisms: areas with inconsistent foreground and background are masked, and attention calculation is not performed on areas with consistent foreground and background. In other words, attention is calculated only if two tokens have the same mask attribute (both foreground or both background); otherwise, attention is not calculated. This mechanism can be used directly on pre-trained diffusion models without further training.

[0153] Based on the settings of the Transformer module's main network, the corresponding multi-head spatiotemporal self-attention module 904 can be intermittently replaced in the first K (K=10) time steps of model inference truncation. Intermittent replacement is defined based on the time step T during the diffusion process; for example, replacing every three steps results in T%3. Intermittent replacement helps alleviate the foreground-background fusion problem. It allows the cross-attention modules in the multi-head spatiotemporal self-attention module 904 to alternately calculate the feature maps of the foreground and background in each time step, making the fusion more natural. For example, assuming a total diffusion of 50 time steps, the original mechanism prevents the foreground and background from seeing each other from step 50 to step 40. Intermittent replacement, however, prevents the foreground and background from seeing each other in time steps 50, 48, 46, 44, 42, and 40. In these steps, the foreground and background overlap (i.e., the two tokens have different mask attributes) from participating in attention calculation, while attention calculation is performed in all other time steps, ignoring foreground markers, resulting in a more natural foreground-background fusion in the generated video.

[0154] During the attention calculation process, the attention calculation result satisfies the following formulas (1) and (2).

[0155]

[0156] MaskedAttention(Q,K,V,M) represents the attention calculation result, with the query matrix (Query, Q), key matrix (Key, K), and value matrix (Value, V) being the three components involved in the attention calculation. d represents the dimension of the K matrix, serving to standardize and stabilize the gradient. It is a mask, also known as a mask matrix; M[i,j] = 0 means that the foreground labels of the two tokens i and j are different, and the mask is recorded as 0 in this case. The corresponding position is denoted as negative infinity. The result of the normalization function calculation for negative infinity is 0, meaning no attention calculation is performed. M[i,j] = 0 represents that the foreground labels of tokens i and j are the same; in this case, the mask is denoted as 1, and the mask matrix... The corresponding position is recorded as 0, which does not affect the normal attention calculation.

[0157] After calculation using three attention mechanisms, three attention results are obtained. These three results are then fused into a single attention feature (which can be concatenated, averaged, etc., depending on the model structure). This attention feature is then input into the layer normalization module 905. As a common part of prediction networks, the layer normalization module 905 serves the following purposes: reducing internal covariate bias: By normalizing the output of each layer, it helps reduce internal covariate bias, improving the generalization ability to the input data. This means it can stabilize the input distribution of each layer, making the model training process more stable and reducing the model's dependence on the data distribution. Improving gradient flow: Normalization helps reduce the vanishing or exploding gradient problem, making the training of deep networks easier. The output of the layer normalization module 905 is then input into the linear reshaping module 906. The linear reshaping module 906 is a fully connected network followed by an average pooling layer, primarily used to align the network output shape with the desired shape. The output of the linear reshaping module 906 yields the prediction noise at time step t. The diffusion model subtracts the predicted noise at time step t from the original noisy data based on the sampler's rules until time step = 0, thus obtaining the final generated video.

[0158] It should be noted that, taking 1000 time steps as an example, when the diffusion model is trained, if noise data is input at time step t=1000, the predicted noise at time step t=999 will be output. However, when the actual video is generated, it can skip steps to generate the predicted noise, for example, outputting the predicted noise at time step 100, time step 800, time step 500 all the way down to time step 0.

[0159] This application provides an additional control method to pre-trained video generation models that lack conditional box control. Based on graphical guide boxes and text data, it enhances the video generation model's ability to align with human intent. In pre-trained video generation models, retraining is unnecessary; the method simply inserts itself into all self-attention calculation modules (in actual video generation, token positions without identical mask labels are multiplied by a large negative number; otherwise, they participate in attention calculation). This method can be used in any diffusion model with a Transformer backbone network. Furthermore, this condition does not incur additional computational overhead and has good adaptability. In the multi-step generation of the diffusion model, assuming t=50, this masking mechanism can be used in the first K steps and canceled in the next 50-k steps. The masking mechanism ensures that in the initial generation state, the foreground and background focus more on their own generation, ignoring the complex interactions of the fusion part. This results in higher generation quality and more realistic details. This application's embodiment enriches the details of the foreground and background in the video, improving the quality of the generated video.

[0160] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0161] The following description continues to illustrate the exemplary structure of the video generation apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the video generation device 455 in the memory 450 may include:

[0162] The text determination module 4551 is used to determine the text information used to describe the video; the foreground word recognition module 4552 is used to recognize the words describing the foreground in the text information; the noise recognition module 4553 is used to recognize the second noise data belonging to the foreground in the first noise data of the video; and the video generation module 4554 is used to generate the video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0163] In some embodiments, the noise recognition module 4553 is further configured to determine the position information of the foreground of an image frame in a video; and to mark the second noise data in the first noise data based on the position information of the foreground.

[0164] In some embodiments, the noise recognition module 4553 is further configured to determine first position information of the first foreground of the first image frame and second position information of the second foreground of the second image frame; determine third position information of the first image frame in the video; determine fourth position information of the second image frame in the video; and determine position information of the foreground for each image frame based on the first position information, the second position information, the third position information, the fourth position information and the number of image frames in the video.

[0165] In some embodiments, the noise identification module 4553 is further configured to divide the first noise data into blocks to obtain third noise data; determine fourth noise data from the third noise data based on the position information of the foreground, wherein the fourth noise data is third noise data that overlaps with the foreground; determine the overlapping area between the foreground and the fourth noise data, and determine the ratio of the overlapping area to the area of ​​the fourth noise data as the overlap ratio; and mark the second noise data in the fourth noise data based on the overlap ratio and a preset ratio threshold.

[0166] In some embodiments, the noise identification module 4553 is further configured to determine the fourth noise data as the second noise data when the overlap ratio is greater than or equal to a preset ratio threshold; and to randomly generate a random number when the overlap ratio is less than the preset ratio threshold, and to determine the fourth noise data as the second noise data when the random number is less than or equal to the overlap ratio.

[0167] In some embodiments, the video generation apparatus 455 further includes a marker adjustment module for determining the distance between the center point of the second noise data and the center point of the foreground; if the distance is greater than a preset distance threshold, the second noise data is changed to fifth noise data belonging to the background.

[0168] In some embodiments, the video generation module 4554 is further configured to encode text information to obtain a text vector, wherein the word vectors corresponding to the words describing the foreground in the text vector have foreground labels; map the first noise data to obtain a target noise vector, wherein the noise vectors corresponding to the second noise data in the target noise vector have foreground labels; and generate a video based on the text vector and the target noise vector.

[0169] In some embodiments, the video is generated based on a text vector and a target noise vector through a T-round iterative process, where T is a positive integer greater than 1. The video generation module 4554 is further configured to perform the following processing for the t-th iteration: if the t-th iteration is a first predetermined iteration, denoising the intermediate noise data of the t-th iteration based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector to obtain denoised data for the t-th iteration, and determining the denoised data for the t-th iteration as the intermediate noise data for the (t+1)-th iteration; if the t-th iteration is a second predetermined iteration, denoising the intermediate noise data of the t-th iteration based on the text vector to obtain denoised data for the t-th iteration, and determining the denoised data for the t-th iteration as the intermediate noise data for the (t+1)-th iteration, where the intermediate noise data for the first iteration is the target noise vector, 1 ≤ t ≤ T; and determining the denoised data for the T-th iteration as the video.

[0170] In some embodiments, the video generation module 4554 is further configured to perform noise prediction processing on the intermediate noise data of the t-th round based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector to obtain the predicted noise of the t-th round; and remove the predicted noise of the t-th round from the intermediate noise data of the t-th round to obtain the denoised data of the t-th round.

[0171] In some embodiments, the video generation module 4554 is further configured to perform a first attention processing on the intermediate noise data and round variable of the t-th round to obtain a first attention feature; perform a second attention processing on the text vector and the intermediate noise data of the t-th round based on the foreground marker of the text vector and the foreground marker of the target noise vector to obtain a second attention feature; perform a self-attention processing on the intermediate noise data of the t-th round based on the foreground marker of the text vector and the foreground marker of the target noise vector to obtain a third attention feature; and fuse the first attention feature, the second attention feature and the third attention feature to obtain the predicted noise of the t-th round.

[0172] In some embodiments, the video generation module 4554 is further configured to, for word vectors in text vectors and noise vectors in intermediate noise data of the t-th round, determine a set numerical vector as an attention sub-feature if either the word vector or the noise vector has a foreground label; if both the word vector and the noise vector have foreground labels, or neither has a foreground label, perform cross-attention processing on the word vector and the noise vector to obtain attention sub-features; and combine the attention sub-features to obtain a second attention feature.

[0173] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video generation method described in this application.

[0174] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the video generation method provided in this application. For example, ... Figure 3 The video generation method is shown.

[0175] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0176] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0177] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0178] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0179] In summary, the embodiments of this application can enrich the details of the foreground and background in the video, reduce the sense of separation between the foreground and background, and improve the quality of the generated video.

[0180] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of video generation, the method comprising: The method includes: Determine the text information used to describe the video; Identify words describing the foreground in the text information; Determine the position information of the foreground in the image frame of the video; Based on the location information of the foreground, the second noise data of the foreground is marked in the first noise data of the video; wherein, the first noise data is used as the input of the video generation model to initialize the video generation process; A video is generated based on the words describing the foreground, the first noise data, the text information, and the second noise data.

2. The method of claim 1, wherein, Determining the position information of the foreground in the image frame of the video includes: Determine the first position information of the first foreground in the first image frame and the second position information of the second foreground in the second image frame; Determine the third position information of the first image frame in the video, and determine the fourth position information of the second image frame in the video; Based on the first location information, the second location information, the third location information, the fourth location information, and the number of image frames in the video, the location information of the foreground for each image frame is determined.

3. The method of claim 1, wherein, The step of marking the second noise data in the first noise data based on the location information of the foreground includes: The first noise data is divided into blocks to obtain the third noise data; Based on the location information of the foreground, a fourth noise data is determined from the third noise data, wherein the fourth noise data is the third noise data that overlaps with the foreground; Determine the overlap area between the foreground and the fourth noise data, and determine the ratio of the overlap area to the area of ​​the fourth noise data as the overlap ratio; Based on the overlap ratio and the preset ratio threshold, the second noise data is marked in the fourth noise data.

4. The method of claim 3, wherein, The step of marking the second noise data in the fourth noise data based on the overlap ratio and a preset ratio threshold includes: If the overlap ratio is greater than or equal to the preset ratio threshold, the fourth noise data is determined as the second noise data; If the overlap ratio is less than the preset ratio threshold, a random number is randomly generated. If the random number is less than or equal to the overlap ratio, the fourth noise data is determined as the second noise data.

5. The method according to any one of claims 1 to 4, characterized in that, After marking the second noise data of the foreground in the first noise data of the video based on the foreground's location information, the method further includes: Determine the distance between the center point of the second noise data and the center point of the foreground; If the distance is greater than a preset distance threshold, the second noise data is changed to the fifth noise data belonging to the background.

6. The method of claim 1, wherein, The process of generating a video based on the words describing the foreground, the first noise data, the text information, and the second noise data includes: The text information is encoded to obtain a text vector, wherein the word vectors corresponding to the words describing the foreground in the text vector have foreground tags; The first noise data is mapped to obtain a target noise vector, wherein the noise vector corresponding to the second noise data in the target noise vector has a foreground label; A video is generated based on the text vector and the target noise vector.

7. The method of claim 6, wherein, The video is generated based on the text vector and the target noise vector through T iterations, where T is a positive integer greater than 1; the generation of the video based on the text vector and the target noise vector includes: For the t-th iteration, perform the following processing: When the t-th round is the first set round, based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector, the intermediate noise data of the t-th round is denoised to obtain the denoised data of the t-th round, and the denoised data of the t-th round is determined as the intermediate noise data of the (t+1)-th round. When the t-th round is the second predetermined round, based on the text vector, the intermediate noise data of the t-th round is denoised to obtain the denoised data of the t-th round, and the denoised data of the t-th round is determined as the intermediate noise data of the (t+1)-th round, wherein the intermediate noise data of the 1st round is the target noise vector, and 1≤t≤T; The denoised data from round T is determined as video.

8. The method of claim 7, wherein, The process of denoising the intermediate noise data in round t based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector to obtain the denoised data in round t includes: Based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector, noise prediction processing is performed on the intermediate noise data of the t-th round to obtain the predicted noise of the t-th round. The predicted noise of round t is removed from the intermediate noise data of round t to obtain the denoised data of round t.

9. The method of claim 8, wherein, The step of performing noise prediction processing on the intermediate noise data of the t-th round based on the foreground marker of the text vector, the foreground marker of the target noise vector, and the text vector to obtain the predicted noise of the t-th round includes: The intermediate noise data and round-number variables of the t-th round are subjected to first attention processing to obtain first attention features; Based on the foreground markers of the text vector and the foreground markers of the target noise vector, a second attention process is performed on the text vector and the intermediate noise data of the t-th round to obtain the second attention feature; Based on the foreground markers of the text vector and the foreground markers of the target noise vector, self-attention processing is performed on the intermediate noise data of the t-th round to obtain the third attention feature; By fusing the first attention feature, the second attention feature, and the third attention feature, the prediction noise for round t is obtained.

10. The method according to claim 9, characterized in that, The second attention process, based on the foreground markers of the text vector and the foreground markers of the target noise vector, performs a second attention feature on the text vector and the intermediate noise data of the t-th round, including: For the word vectors in the text vectors and the noise vectors in the intermediate noise data of the t-th round, if the word vectors have the foreground markers or the noise vectors have the foreground markers, the set numerical vectors are determined as attention sub-features; When both the word vector and the noise vector have the foreground marker, or neither has the foreground marker, cross-attention processing is performed on the word vector and the noise vector to obtain attention sub-features; The attention features are combined to obtain the second attention feature.

11. A video generating apparatus, characterized by comprising: The device includes: The text determination module is used to determine the text information used to describe the video; A foreground word recognition module is used to identify words describing the foreground in the text information; A noise recognition module is used to determine the position information of the foreground in the image frame of the video; based on the position information of the foreground, to mark the second noise data of the foreground in the first noise data of the video; wherein, the first noise data is used as the input of the video generation model to initialize the video generation process; The video generation module is used to generate a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

12. An electronic device, comprising: The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the video generation method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the video generation method according to any one of claims 1 to 10.

14. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the video generation method according to any one of claims 1 to 10.

Images