Video generation method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a video generation method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining text information for describing a video; identifying words describing the foreground in the text information; identifying second noise data belonging to the foreground in the first noise data of the video; a video is generated based on the words describing the foreground, the first noise data, the text information, and the second noise data. According to the invention, the details of the foreground and the background in the video can be enriched, and the quality of the generated video is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, video generation technology is gradually becoming a research hotspot.

[0003] In the related art, video generation technology mainly uses Generative Adversarial Networks (GANs), autoregressive models and diffusion models, combined with guided generation conditions to generate videos. For example, a diffusion model is used to generate a video based on a paragraph. Summary of the invention

[0004] Embodiments of the present application provide a video generation method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the quality of generated videos.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a video generation method, the method comprising: determining text information used to describe a video; identifying words describing a foreground in the text information; identifying second noise data belonging to the foreground in first noise data of the video; and generating a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0007] The embodiment of the present application provides a video generation device, including:

[0008] A text determination module, used to determine text information used to describe the video;

[0009] a foreground word recognition module, used to recognize words describing the foreground in the text information;

[0010] A noise identification module, used for identifying second noise data belonging to the foreground in the first noise data of the video;

[0011] A video generation module is used to generate a video based on the words describing the foreground, the first noise data, the text information and the second noise data.

[0012] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0013] A memory for storing computer executable instructions or computer programs;

[0014] The processor is used to implement the video generation method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0015] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the video generation method provided in the embodiment of the present application when executed by a processor.

[0016] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the video generation method provided in the embodiment of the present application is implemented.

[0017] The embodiments of the present application have the following beneficial effects:

[0018] By comprehensively generating a video based on the words describing the foreground, the second noise data belonging to the foreground in the first noise data, the text information and the first noise data, the generation of the foreground and the background can be more accurately guided during the video generation process, thereby improving the authenticity and richness of the details in the video. In addition, compared to manually specifying the foreground in the video, the embodiment of the present application can improve the efficiency of video generation and improve the accuracy of the foreground generated in the video by identifying the text information to obtain the words describing the foreground and identifying the first noise data to obtain the second noise data belonging to the foreground. Therefore, the embodiment of the present application can enrich the details of the foreground and the background in the video and improve the quality of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the architecture of the video generation system provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 1 ;

[0022] Figure 4 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 2 ;

[0023] Figure 5 is another flowchart of the video generation method provided in an embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 3 ;

[0025] Figure 7 is another optional flow chart of the video generation method provided in the embodiment of the present application;

[0026] Figure 8 is a flow chart of a control method for video generation based on a diffusion model provided in an embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of a process of outputting foreground words of a large language model provided in an embodiment of the present application;

[0028] Fig.10 It is a schematic diagram of the interface for outputting foreground words of a large language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0030] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0032] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0033] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0035] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0036] 1) Diffusion model: It is a data generation method that transforms noisy data into clean data, which is commonly used in the fields of image generation and video generation. For example, in the field of video generation, the diffusion model is a deep learning model based on a probabilistic process, which is used to generate the final video based on given initial conditions (such as a noisy data and text information).

[0037] 2) Noise data: It is random data used in the video generation process, usually Gaussian noise or other types of random noise. Noise data is the starting point of video generation and is finally converted into video through a series of processing steps.

[0038] 3) Text information used to describe the video: refers to a piece of text information used to describe the content of the video to be generated. The text information can include specific descriptions of the video's theme, scene, characters, actions, etc., to guide the diffusion model to generate a video that meets the description content.

[0039] 4) Noise addition: refers to the process of adding noise to data such as images or audio. Noise can include Gaussian noise, etc. In the process of image processing by the diffusion model, noise addition is also called diffusion. In this process, the diffusion model gradually introduces noise, starting from the original image, and gradually blurs the image until it becomes a random noise state.

[0040] 5) Denoising: Denoising is the process of removing noise from data such as images or audio. In video generation, denoising is often used in conjunction with a diffusion model. Denoising refers to the inverse diffusion process of the diffusion model, which gradually removes noise to restore or generate clear video content.

[0041] 6) Foreground: refers to the object or scene at the front of the video, which is usually the focus of the video. In the video generation process, the foreground usually refers to the element that needs to be emphasized or displayed.

[0042] 7) Background: refers to the background or environment behind the video, which usually serves to set off the foreground. The background provides richer environmental information for the video and enhances the realism of the video.

[0043] 8) Self-attention processing: It is a deep learning technique used to calculate the degree of influence of each element in the input data on other elements. It can be achieved by introducing a self-attention mechanism in the neural network, thereby improving the model's ability to understand and generate input data.

[0044] 9) Cross-attention processing: It is an application of the attention mechanism, which is mainly used to establish connections between different modalities (such as text and images). In video generation, it can help the model understand the connection between text information and video content, so as to better generate videos that match the description. Through the cross-attention mechanism, the diffusion model can dynamically adjust the degree of attention to text information during the generation process to ensure that the generated video content matches the description.

[0045] 10) Vision Transformer (VIT): A deep learning model architecture for images that divides images into a series of fixed-size image blocks (called Patches), encodes these image blocks into a series of basic units (Tokens), and processes these Tokens through a self-attention mechanism to extract image features.

[0046] 11) Basic unit (Token): It is the smallest processing unit of the vision model (Vision Transformer). In deep learning and natural language processing, Token refers to the basic unit into which text or images are segmented. For example, for text information, Token is usually a word or sub-word unit; for image data, it is an image patch.

[0047] 12) Mask: It is a mask for the basic unit (Token). It is a technique used to instruct the model which parts to focus on and which parts to ignore. In image processing, Mask is usually a binary matrix of the same size as the input image, where 1 represents the area to focus on and 0 represents the area to be ignored. In video generation, Mask can be used to indicate the foreground area that the model focuses on when generating videos.

[0048] 13) Large Language Model (LLM): refers to a natural language processing model with a large number of parameters. Such models are usually pre-trained on large-scale text datasets and can understand and generate high-quality natural language text.

[0049] 14) Foreground box: refers to a frame with a preset shape used to mark the foreground boundary in an image or video, such as a rectangular box, a circular box, etc. In the video generation task, the foreground box can help the diffusion model more accurately locate and process the foreground in the video, thereby improving the effect of video generation. The foreground box can be manually specified by the user or obtained through automatic detection.

[0050] 15) Latent Space: refers to a low-dimensional continuous space to which the original input data is mapped by the encoder in machine learning and deep learning models. In the latent space, the data is represented as a set of continuous variables, usually with a compact representation that can capture the intrinsic structure and characteristics of the input data.

[0051] 16) Latent Space Shape: refers to the dimension and structure of the data in the latent space. In deep learning models, the latent space shape is usually represented by a set of numbers that correspond to the dimensions of the data in the latent space. For example, in the diffusion model, if the shape of the latent space is [1, 32, 8, 16, 16], it means that the data in the latent space has the following dimensions: Batch Size is [1], which means processing one sample at a time, Channels is

[32] , which means the number of channels in the latent space, Length is [8], which means the length of the time series, Height is

[16] , which means the height, and Width is

[16] , which means the width.

[0052] Most of the technologies related to video generation use models based on the diffusion model framework, such as Make-a-video, Text2video-zero, and Imagen Video. In the diffusion model framework, one part of the main network is based on the U-Net architecture, and the other part is based on the Transformer model architecture. In the development of deep learning, the Transformer model was initially oriented to sequence tasks rather than image tasks, but as the academic community deepened its research, the Transformer model has good scalability, allowing it to begin to emerge in other fields (including images). When the amount of training data is relatively abundant, the Transformer model is likely to outperform the traditional image model main network such as U-Net and ResNet. In the field of image generation, there are many conditions and methods to guide generation, such as using low-rank adapters (LoRA), text prompts, and graphic conditions to guide image generation. In each new condition-guided generation mode, the image generation model needs to be trained for multiple rounds to achieve better results. There is still a lack of research on related conditional guidance methods in the field of video generation. Video generation has one more dimension than image generation, and requires more training resources. If the Transformer model is used as the main network, the cost of training the video generation model is higher. The following introduces some commonly used model architectures in related technologies.

[0053] GAN-based Image Generators: A model that extends the image generator based on the generative adversarial network (GAN) to video generation, but GAN-based video generation methods often face the problem of mode collapse during training. Autoregressive Models: A model that uses an autoregressive model to learn the data distribution to generate videos. Although this model generally provides good video quality and more stable convergence, training such models requires significant computational resources. Diffusion Models: Diffusion models, Transformer-based diffusion models have not been fully explored. Transformer Backbones: In the image field, the Transformer architecture has become a mainstream model, but the method of integrating the Transformer architecture into the diffusion model of video generation is unclear. Compared with images, due to the complex and high-dimensional characteristics of videos, including complex spatiotemporal information in high-resolution frames, generating high-quality videos faces significant challenges (Video Generation Challenges). U-Net in Video Generation: U-Net (a convolutional neural network-based structure) occupies a prominent position in image and video generation work. However, for the underlying diffusion model based on the Transformer, the inductive bias of U-Net is not critical. Long-range Contextual Relationships: Attention-based architectures provide an intuitive choice for capturing long-range contextual relationships in videos. However, how to effectively integrate these architectures into video generation tasks remains a problem. Pre-trained Models: Although pre-trained models are commonly used for image generation, the effectiveness of pre-trained Transformer models in video generation has not been fully studied.

[0054] In summary, the related technologies have the following technical problems: Pre-trained models for video generation based on the diffusion model framework have high training costs if they want to introduce interactive graphics-based conditional control. In the field of video generation, common practices include unconditional video generation and conditional video generation. In conditional video generation, a video is often generated based on a paragraph of speech, a video is generated from several pictures (video prediction, video interpolation, etc.), or black edges of the video are filled. The related technologies lack a video generation method with training-free conditional control based on the Transformer network, and the generated video has single foreground and background details, with an obvious sense of separation between the foreground and background, and the video quality is poor.

[0055] Based on the problems existing in the related technologies, the embodiments of the present application provide a video generation method, device, electronic device, computer-readable storage medium and computer program product, which can enrich the details of the foreground and background in the video and improve the quality of the generated video.

[0056] The following describes an exemplary application of the video generation device provided in the embodiment of the present application. The video generation device provided in the embodiment of the present application is an electronic device for implementing a video generation method. The electronic device provided in the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Below, the exemplary application of the video generation device when it is implemented as a server or a terminal will be described.

[0057] See also Figure 1 , Figure 1 It is the framework diagram of the video generation system that the embodiment of the application provides, for realizing video generation, a video generation application can be provided, for example, the video generation application can be an application dedicated to video generation, or it can be a functional module (such as a video generation module in a game application, etc.) in other applications. The video generation system 100 that the embodiment of the application provides at least includes a terminal 400, a network 300 and a server 200, wherein the server 200 is a server of the video generation application. The server 200 can constitute the video generation device of the embodiment of the application, i.e., realize the video generation method of the embodiment of the application by the server 200. The terminal 400 connects the server 200 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0058] See also Figure 1, the user can perform interactive operations on the client of the video generation application through the terminal 400, and the interactive operations can be, for example, inputting text information operations, setting first noise data operations, clicking to start generating video operations, etc. After receiving the interactive operations of the user, the client can encapsulate the text information and the first noise data into a video generation request through the terminal, and send the video generation request to the server 200 through the network 300. After receiving the video generation request, the server 200 determines the text information used to describe the video in response to the video generation request; the server 200 identifies the words describing the foreground in the text information; the server 200 identifies the second noise data belonging to the foreground in the first noise data of the video; the server 200 generates a video based on the words describing the foreground, the first noise data, the text information, and the second noise data. After generating the video, the server 200 can also send the video to the terminal 400 to show the video to the user.

[0059] In some embodiments, the video generation method of the embodiment of the present application can also be performed by the terminal 400 itself, that is, after the terminal 400 receives the interactive operation input by the user through the client, the terminal 400 determines the text information used to describe the video; the terminal 400 identifies the words describing the foreground in the text information; the terminal 400 identifies the second noise data belonging to the foreground in the first noise data of the video; the terminal 400 generates the video based on the words describing the foreground, the first noise data, the text information and the second noise data. After the video is generated and determined, the generated video is displayed on the client interface of the terminal 400.

[0060] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 450 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 450 may optionally include one or more storage devices physically away from the processor 410. The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as exemplarily described below.

[0064] An operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 452, for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; a presentation module 453, for enabling information to be presented (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., display screens, speakers, etc.) associated with a user interface 430; an input processing module 454, for detecting one or more user inputs or interactions from one of the one or more input devices 432 and translating the detected inputs or interactions.

[0065] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2The video generation device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a text determination module 4551, a foreground word recognition module 4552, a noise recognition module 4553 and a video generation module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0066] In other embodiments, the video generating device provided in the embodiments of the present application can be implemented in hardware. As an example, the video generating device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generating method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0067] It should be noted that, based on the understanding of the following, those skilled in the art can apply the video generation method provided in the embodiments of the present application to any scenario where videos are generated using artificial intelligence technology, such as film and television animation production scenarios, game development scenarios, live interactive scenarios, education and training scenarios, cultural protection and reconstruction scenarios, medical health visualization scenarios, experimental simulations, etc. The following is an example of using artificial intelligence technology to generate a video based on pre-input text information.

[0068] Figure 3 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 1 , the following will combine Figure 3 The steps shown are explained as Figure 3 As shown, the video generation method is described by taking the execution subject of the server as an example, and the method includes the following steps 101 to 104:

[0069] Step 101, determining text information for describing a video.

[0070] Here, the text information is a piece of text used to describe the video content in the video to be generated. The text information may include specific descriptions such as the theme, scene, character, action, etc. of the video. The text information may be input by the user in the client interface of the video generation application. For example, the text information input by the user may be "a white cheetah with big eyes galloping on the grassland".

[0071] Step 102, identifying words describing the foreground in the text information.

[0072] Here, the foreground is an object or scene at the front of the screen in the video. The text information may include multiple words, wherein the words describing the foreground are words used to describe the scene, object or action at the front of the screen in the video. First, the text information may be preprocessed in a basic manner, including word segmentation, stop word removal, etc., to ensure that the subsequent word recognition process is more accurate. Next, natural language processing technology (such as a large language model (LLM)) is used to identify the words describing the foreground in the text information. Taking the text information "white big-eyed cheetah galloping on the grassland" as an example, after preprocessing the text information, the multiple words obtained are "white", "big eyes", "cheetah", "on the grassland", and "galloping". The large language model (LLM) is used to identify the preprocessed text information, and the words describing the foreground obtained are "white", "big eyes", "cheetah", "galloping", etc. Here, the words describing the foreground obtained by screening may have a foreground tag, such as the foreground tag being a subscript [1].

[0073] In the related art, the method of manually judging the words describing the foreground is that the words manually judged to describe the foreground are only nouns. For example, for the text information "white big-eyed cheetah galloping on the grassland", the word manually judged to describe the foreground is "cheetah", ignoring the words "white" and "big eyes" that are highly related to cheetahs, as well as the action word "galloping", which will cause the problem of inaccurate guidance of the foreground during the video process. The embodiment of the present application uses a large language model to automatically identify text information and obtain words describing the foreground, which can improve the ability to divide words describing the foreground and background of complex scenes, thereby providing more accurate guidance for the foreground and background in the video generation process, and enriching the details of the foreground and background in the video.

[0074] Step 103: identifying second noise data belonging to the foreground in the first noise data of the video.

[0075] Here, in the video generation method provided in the embodiment of the present application, the first noise data and text information are used as inputs of the diffusion model. In the diffusion model, the first noise data is a set of random values ​​extracted from a probability distribution (such as a Gaussian distribution). The first noise data is used to initialize the video generation process, and is gradually denoised by the diffusion model to become an image frame in the video. The shape of the first noise data corresponds to the dimension of the image frame in the generated video. For example, in the diffusion model, if the shape of the generated image frame is [batch size, channels, length, height, width], then the first noise data also has the same shape as the image frame. Exemplarily, the first noise data can be a truncated Gaussian distribution in the shape of the latent space, for example, if the shape of the latent space is [1, 32, 8, 16, 16], then a truncated Gaussian distribution noise of the shape of the latent space is generated. It should be noted that for each image frame in the video, the value of the first noise data extracted from the probability distribution is different, but the shape is the same.

[0076] After randomly extracting the first noise data, the first noise data can be processed into blocks to obtain a plurality of third noise data. The second noise data belonging to the foreground is determined from the plurality of third noise data based on the position information of the foreground. The position information of the foreground is used to indicate the position of the foreground in the first noise data, that is, to specify the area showing the foreground in the video. Here, the obtained second noise data can have a foreground tag, for example, the foreground tag is a subscript [1], etc.

[0077] In some embodiments, see Figure 4 , Figure 4 It is shown that the second noise data belonging to the foreground in the first noise data of the video identified in step 103 can be implemented by the following steps 1031 to 1032:

[0078] Step 1031, determining the position information of the foreground of the image frame in the video.

[0079] Here, the video may include multiple image frames. The image frame may include a preset foreground frame, and the foreground will be displayed in the area of ​​the foreground frame. The foreground frame refers to a frame with a preset shape used to mark the foreground boundary in the image frame, such as a rectangular frame, a circular frame, etc. The foreground frame can be obtained by the user manually inputting the position information of the foreground. The position information of the foreground includes the center point coordinates, width and height of the foreground frame. Exemplarily, the position information of the foreground can be expressed as [X, Y, H, W], where X and Y represent the horizontal and vertical coordinates of the center point of the foreground frame, and H and W represent the pixel values ​​of the height and width of the foreground frame respectively. Assume that the size of the image frame of the video is 100px (pixel) × 100px, and the center point coordinates of the image frame are set to (0, 0). If the position information of the foreground can be expressed as [0, 0, 20, 20], it means that the foreground frame is a 20×20 rectangular frame with the center point coordinates (0, 0) in the image frame.

[0080] In some embodiments, determining the position information of the foreground of the image frame in the video in step 1031 may be implemented in the following manner: first, determining the first position information of the first foreground of the first image frame and the second position information of the second foreground of the second image frame. Then, determining the third position information of the first image frame in the video, and determining the fourth position information of the second image frame in the video. Finally, based on the first position information, the second position information, the third position information, the fourth position information, and the number of image frames in the video, determining the position information of the foreground of each image frame.

[0081] Here, the video may include multiple image frames, and the first image frame and the second image frame are any two different image frames in the multiple image frames. Exemplarily, the first image frame may be a starting frame in the video, and the second image frame may be an ending frame in the video. The first position information of the first foreground and the second position information of the second foreground may be the same or different. The user may input the first position information of the first foreground and the second position information of the second foreground. The multiple image frames in the video are arranged in chronological order, and each image frame represents a time point in the video. The third position information refers to the specific position of the first image frame in the video, and the fourth position information refers to the specific position of the second image frame in the video, such as a timestamp or a frame number. Taking the third position information and the fourth position information as a timestamp as an example, the timestamp may be in seconds or milliseconds. For example, if the first image frame of the video is the starting frame, the third position information of the first image frame is a timestamp of 0 seconds, and the second image frame is an image frame adjacent to the starting frame, then the fourth position information is 0.033 seconds (if the frame rate is 30fps). Taking the third position information and the fourth position information as frame numbers as an example, assuming that the user specifies the first image frame as the starting frame of the video, that is, the third position information is frame number 1. At the same time, the user also specifies the second image frame as the fifth frame after the first image frame in the video, that is, the fourth position information is frame number 5.

[0082] The position change rule of the first foreground and the second foreground in the video can be calculated based on the first position information, the second position information, the third position information and the fourth position information. For each image frame between the first image frame and the second image frame, the position information of the foreground of the intermediate frame is calculated based on the first position information, the second position information, the number of image frames and the position change rule. The embodiment of the present application does not limit the method for determining the position change rule, for example, the target continuous recognition algorithm can be used for calculation.

[0083] Compared with manually setting the foreground position, which results in an obvious sense of separation between the foreground and the background in the generated video, the embodiment of the present application can automatically determine the foreground position corresponding to each image frame based on two preset foreground positions and the number of image frames of the video, so as to ensure the temporal and spatial coherence of the foreground of each frame in the generated video, which helps to improve the accuracy and consistency of the foreground during the video generation process.

[0084] Step 1032: Mark the second noise data in the first noise data based on the position information of the foreground.

[0085] Here, the first noise data may be processed into blocks to obtain a plurality of third noise data, and the second noise data may be determined from the plurality of third noise data based on the position information of the foreground.

[0086] The embodiment of the present application determines to generate second noise data with a foreground mark based on the position information of the foreground, that is, the noise data for generating the foreground, which will provide guidance for the subsequent video generation process, ensure that the foreground in the generated video is displayed at the position where the user expects the foreground to be displayed, and improve the video quality.

[0087] In some embodiments, in step 1032, marking the second noise data in the first noise data based on the position information of the foreground can be achieved in the following manner: first, the first noise data is divided into blocks to obtain third noise data; then, based on the position information of the foreground, fourth noise data is determined from the third noise data, wherein the fourth noise data is the third noise data that overlaps with the foreground; then, the overlapping area between the foreground and the fourth noise data is determined, and the ratio of the overlapping area to the area of ​​the fourth noise data is determined as the overlapping ratio; finally, based on the overlapping ratio and a preset ratio threshold, the second noise data is marked in the fourth noise data.

[0088] Here, the first noise data can be processed in blocks using a vision model (Vision Transformer, VIT) to obtain multiple third noise data (Tokens). The position information of the foreground is scaled based on the shape of the first noise data and the size of the image frame to obtain the target position information. The position information of the foreground can be scaled based on the ratio of the width of the first noise data to the width of the image frame to obtain the target position information. The target position information of the foreground can represent the foreground with a foreground frame of a preset shape, and the foreground frame is scaled to multiple third noise data based on the target position information of the foreground to obtain at least one third noise data that overlaps with the foreground frame, and each third noise data that overlaps is determined as the fourth noise data. It should be noted that the meaning of overlap here is that at least one of the third noise data is located in the foreground frame, that is, the area of ​​the third noise data overlaps partially or completely with the area of ​​the foreground. For example, the width and height of the shape of the first noise data are both 50, and the size of the image frame is 100×100, and the position information of the foreground is [0, 0, 20, 20]. The ratio of the width of the first noise data to the width of the image frame is 1 / 2. The position information of the foreground is scaled based on the width ratio, and the target position information is [0, 0, 10, 10]. The foreground frame is placed on the 50×50 first noise data according to the target position information [0, 0, 10, 10]. If the first noise data is divided into 9 third noise data, and there are 4 third noise data overlapping with the area selected by the foreground frame, then these 4 third noise data are determined as 4 fourth noise data.

[0089] For each fourth noise data, the area of ​​the fourth noise data and the area of ​​the scaled foreground can be obtained. Based on the area of ​​the fourth noise data and the area of ​​the scaled foreground, the overlapping area of ​​the fourth noise data and the foreground is determined. The calculation method of the overlapping area can refer to the calculation method of the rectangular area, which is not limited in detail in the embodiment of the present application. The ratio of the overlapping area to the area of ​​the fourth noise data is determined as the overlapping ratio. Assuming that the overlapping area is A and the area of ​​the fourth noise data is B, the overlapping ratio is A / B. The preset ratio threshold refers to a threshold used to determine whether a certain fourth noise data can be regarded as the second noise data. The embodiment of the present application does not limit the value of the preset ratio threshold, and it can be set based on actual needs. When the overlapping ratio is greater than or equal to the preset ratio threshold, the fourth noise data is determined as the second noise data.

[0090] The embodiment of the present application utilizes VIT to process the first noise data in blocks, which can improve the processing efficiency of the noise data, and generate second noise data with a foreground mark based on the position information of the foreground. This will provide guidance for the subsequent video generation process, ensure that the foreground in the generated video is displayed at the position where the user expects the foreground to be displayed, and improve the quality of the generated video.

[0091] In an embodiment of the present application, based on the overlap ratio and the preset ratio threshold, marking the second noise data in the fourth noise data can be achieved in the following manner: when the overlap ratio is greater than or equal to the preset ratio threshold, determining the fourth noise data as the second noise data; or, when the overlap ratio is less than the preset ratio threshold, randomly generating a random number, and when the random number is less than or equal to the overlap ratio, determining the fourth noise data as the second noise data.

[0092] Here, for the fourth noise data, when the overlap ratio between the fourth noise data and the foreground is greater than or equal to a preset ratio threshold, the fourth noise data is directly determined as the second noise data. When the overlap ratio is less than the preset ratio threshold, the overlap ratio is determined as the sampling probability, and whether the fourth noise data belongs to the second noise data is randomly determined based on the sampling probability. For example, 0-1 sampling can be performed based on the sampling probability: a random number is randomly generated within the range of 0-1, and when the random number is less than or equal to the sampling probability, the fourth noise data is determined as the second noise data.

[0093] Exemplarily, when the preset ratio threshold is 0.5 and the overlap ratio of the fourth noise data to the foreground is 0.6, the fourth noise data is determined to be the second noise data. When the overlap ratio of the fourth noise data to the foreground is 0.4, 0.4 is determined as the sampling probability. Perform 0-1 sampling and randomly select a random number from the interval [0,1]. If the random number is 0.3, which is less than the sampling probability of 0.4, the fourth noise data is determined to be the second noise data. If the random number is 0.45, which is greater than the sampling probability of 0.4, the fourth noise data is not the second noise data.

[0094] The embodiment of the present application sets a preset ratio threshold to ensure that only the fourth noise data that has sufficient overlap with the foreground will be marked as the second noise data for generating the foreground, thereby improving the accuracy of determining the second noise data. The selection of the preset ratio threshold can be flexibly adjusted according to actual needs, so as to control which fourth noise data is marked as the second noise data, thereby increasing the controllability of the foreground during the video generation process. At the same time, by randomly sampling the situation where the overlap ratio is less than the preset ratio threshold, it is possible to appropriately consider areas that have a certain association with the foreground but a low overlap ratio while keeping the foreground elements prominent, thereby better balancing the relationship between the foreground and the background, and improving the quality of the generated video.

[0095] In some embodiments, Figure 5 is another flow chart of the video generation method provided in the embodiment of the present application. Figure 5 As shown, after the second noise data belonging to the foreground in the first noise data of the video is identified in step 103, the video generation method provided in the embodiment of the present application further includes the following steps 201 to 202:

[0096] Step 201, determining the distance between the center point of the second noise data and the center point of the foreground.

[0097] Here, after the second noise data is determined from the plurality of third noise data based on the overlap ratio, for each second noise data, the center point of the second noise data and the center point of the scaled foreground can also be obtained. The center point of the second noise data and the center point of the foreground are both represented in the form of coordinates. The distance between the center point of the second noise data and the center point of the foreground can be calculated based on the coordinates of the center point of the second noise data and the coordinates of the center point of the foreground.

[0098] Step 202: when the distance is greater than a preset distance threshold, the second noise data is changed to fifth noise data belonging to the background.

[0099] Here, the preset distance threshold refers to a threshold used to determine whether the distance between the center point of the second noise data and the center point of the foreground is close enough. The preset distance threshold is set according to actual needs and is used to determine whether the second noise data is changed to the fifth noise data belonging to the background. A smaller preset distance threshold means that only the area very close to the center of the foreground will be regarded as the second noise data, while a larger preset distance threshold will cause more areas in the first noise data to be regarded as the second noise data. For each second noise data, when the distance between the center point of the second noise data and the center point of the foreground is less than or equal to the preset distance threshold (for example, 10 pixels), the second noise data remains unchanged. When the distance between the center point of the second noise data and the center point of the foreground is greater than the preset distance threshold (for example, 10 pixels), the foreground mark of the second noise data is cancelled and changed to the fifth noise data belonging to the background.

[0100] The embodiment of the present application adjusts the second noise data based on the distance between the center point of the second noise data and the center point of the foreground, ensuring that the second noise data corresponds to the foreground more accurately, providing more accurate guidance for the subsequent video generation process, and improving the quality and naturalness of video generation.

[0101] Step 104 , generating a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0102] Here, the text information and the first noise data may be input into a pre-trained video generation model (eg, a Transformer-based diffusion model). The pre-trained video generation model processes the text information and the first noise data based on words describing the foreground and the second noise data to obtain a video.

[0103] The embodiment of the present application generates a video based on the words describing the foreground, the second noise data, the text information and the first noise data, and can more accurately guide the generation of the foreground and the background during the video generation process, thereby improving the authenticity and richness of the details in the video. In addition, compared to manually specifying the foreground in the video, the embodiment of the present application obtains the words describing the foreground by performing word recognition on the text information, and obtains the second noise data belonging to the foreground by recognizing the first noise data, which can improve the efficiency of video generation and improve the accuracy of the foreground generated in the video. Therefore, the embodiment of the present application can enrich the details of the foreground and the background in the video and improve the quality of the generated video.

[0104] In some embodiments, see Figure 6 , Figure 6 It is shown that in step 104, a video is generated based on the words describing the foreground, the first noise data, the text information, and the second noise data, which can be achieved by the following steps 1041 to 1043:

[0105] Step 1041, encode the text information to obtain a text vector.

[0106] Among them, the word vector corresponding to the word describing the foreground in the text vector has a foreground tag.

[0107] Here, after the text information and the first noise data are input into the pre-trained video generation model, the encoder in the video generation model can encode each word in the text information separately to obtain a word vector for each word. The word vectors of multiple words are combined to obtain a text vector. Among them, the word vector corresponding to the word describing the foreground has a foreground tag.

[0108] Step 1042, mapping the first noise data to obtain a target noise vector.

[0109] The noise vector corresponding to the second noise data in the target noise vector has a foreground label.

[0110] Here, since the first noise data is initially represented by a three-dimensional vector, the visual model VIT can be used to process the noise data in blocks to obtain multiple third noise data, and each third noise data is mapped to obtain a one-dimensional noise vector corresponding to each third noise data. The noise vector corresponding to each third noise data is position-encoded to obtain a target noise vector. Among them, the noise vector corresponding to the second noise data has a foreground mark.

[0111] Step 1043, generating a video based on the text vector and the target noise vector.

[0112] Here, the text vector and the target noise vector can be combined as the input of the video generation model. The video is generated by a pre-trained video generation model (such as the Transformer-based diffusion model).

[0113] The embodiment of the present application generates a video based on a comprehensive text vector and a target noise vector, which can effectively improve the overall quality and visual effect of the video. In addition, both the text vector and the target noise vector have foreground markers, which can more accurately guide the generation of the foreground during the video generation process, thereby improving the authenticity and richness of the video details.

[0114] In the embodiment of the present application, the video is generated based on the text vector and the target noise vector through a T-round iterative process, where T is a positive integer greater than 1. In step 1043, the video is generated based on the text vector and the target noise vector, which can be implemented in the following manner: the following processing is performed for the t-th round iteration: when the t-th round is the first set round, based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, the intermediate noise data of the t-th round is denoised to obtain the denoised data of the t-th round, and the denoised data of the t-th round is determined as the intermediate noise data of the t+1-th round; when the t-th round is the second set round, based on the text vector, the intermediate noise data of the t-th round is denoised to obtain the denoised data of the t-th round, and the denoised data of the t-th round is determined as the intermediate noise data of the t+1-th round, where the intermediate noise data of the 1st round is the target noise vector, 1≤t≤T; the denoised data of the T-th round is determined as the video.

[0115] Here, the first set round refers to the round set in the iteration process that needs to perform denoising based on the foreground mark, and the second set round refers to the round set in the iteration process that does not need to perform denoising based on the foreground mark. In other words, the second set round is the round in the T rounds of iteration excluding the first set round.

[0116] Exemplarily, T is 1000, the first set round is the 1st to 10th round, and the second set round is the 11th to 1000th round. In the 1000-round iterative processing process, the first 10 iterations perform denoising on the intermediate noise data based on the foreground mark and the text vector to obtain denoised data, and the last 990 iterations perform denoising on the intermediate noise data based only on the text vector to obtain denoised data. First, the first round of processing is performed, the intermediate noise data of the first round is the target noise vector, and the target noise vector is denoised based on the foreground mark of the text vector, the foreground mark of the target noise vector, and the text vector to obtain the denoised data of the first round, and the denoised data of the first round is determined as the intermediate noise data of the second round. When performing the second round of processing, since the size of the intermediate noise data of the second round may be different from the size of the initial noise data, it is necessary to first determine the second noise data from the intermediate noise data of the second round based on the position information of the foreground, and then generate the second round target noise vector corresponding to the intermediate noise data of the second round. Based on the foreground mark of the text vector, the foreground mark of the second round target noise vector and the text vector, the intermediate noise data of the second round is denoised to obtain the denoised data of the second round, and the denoised data of the second round is determined as the intermediate noise data of the third round. Repeat the above steps until the 11th round of iteration, the intermediate noise data of the 11th round is denoised based on the text vector to obtain the denoised data of the 11th round, and the denoised data of the 11th round is determined as the intermediate noise data of the 12th round. When the denoised data of the 1000th round is obtained, the denoised data of the 1000th round is determined as an image frame, and multiple image frames are sorted and combined to obtain a video.

[0117] On the one hand, the embodiment of the present application can ensure that when the initial state is generated, the foreground and background can pay more attention to their own generation and ignore the complex interaction of the fused part through the first 10 iterations. This can make the generated video higher in quality and more realistic in details. On the other hand, through the last 990 iterations, it can ensure that the intersection of the foreground and background is involved in the calculation, and the foreground and background are fused, reducing the sense of separation between the foreground and background in the video.

[0118] Exemplarily, T is 1000, and T can be divided into a first set round and a second set round based on a preset interval round. The first set round and the second set round are switched every preset interval round. The preset interval round can be calculated based on T. For example, the preset interval round can be 3, that is, the round type is switched every 3 rounds. Starting from the first round, the operation of the first set round is performed every 3 rounds, for example, the first round, the fourth round, the seventh round, etc. The operation of the second set round is performed in a round other than the first set round, for example, the second round, the third round, the fifth round, the sixth round, etc.

[0119] The embodiment of the present application calculates the foreground and background alternately during multiple rounds of iterative processing of video generation, so that the fusion of the foreground and background in the generated video is more natural, thereby improving the quality of the video.

[0120] In an embodiment of the present application, based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, the intermediate noise data of the tth round is denoised to obtain the denoised data of the tth round. This can be achieved in the following way: first, based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, the intermediate noise data of the tth round is subjected to noise prediction processing to obtain the predicted noise of the tth round; then, the predicted noise of the tth round is removed from the intermediate noise data of the tth round to obtain the denoised data of the tth round.

[0121] For example, for the first round of iteration, the intermediate noise data of the first round can be processed for noise prediction based on the foreground tag of the text vector, the foreground tag of the target noise vector, and the text vector to obtain the predicted noise of the first round. Then, the predicted noise of the first round is subtracted from the intermediate noise data of the first round to obtain the denoised data of the first round.

[0122] The embodiment of the present application can gradually remove noise through an iterative denoising process, thereby improving the overall quality and visual effect of the video. At the same time, by considering the foreground mark of the text vector and the foreground mark of the target noise vector during the denoising process, the foreground and background details in the generated video can be richer.

[0123] In an embodiment of the present application, based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, noise prediction processing is performed on the intermediate noise data of the tth round to obtain the predicted noise of the tth round, which can be achieved in the following way: first, a first attention processing is performed on the intermediate noise data of the tth round and the round variable to obtain a first attention feature; and based on the foreground mark of the text vector and the foreground mark of the target noise vector, a second attention processing is performed on the text vector and the intermediate noise data of the tth round to obtain a second attention feature; and based on the foreground mark of the text vector and the foreground mark of the target noise vector, a self-attention processing is performed on the intermediate noise data of the tth round to obtain a third attention feature; then, the first attention feature, the second attention feature and the third attention feature are fused to obtain the predicted noise of the tth round.

[0124] Here, we take t as 1 as an example, and the intermediate noise data of the first round is the target noise vector. The first attention processing is performed on the target noise vector and the round variable to obtain the first attention feature. The first attention processing process is to take the target noise vector as the query matrix, the round variable as the key matrix, determine the weight parameter matrix based on the product of the query matrix and the key matrix, take the round variable as the value matrix, and use the normalized exponential function (softmax function) to normalize the product of the weight parameter matrix and the value matrix to obtain the first attention feature. In the second attention processing process, the target noise vector is taken as the query matrix, the text vector is taken as the key matrix and the value matrix, and a calculation process similar to the first attention processing is performed. In the self-attention processing process, the target noise vector is taken as the query matrix, the key matrix and the value matrix, and a calculation process similar to the first attention processing is performed. By integrating the first attention feature, the second attention feature and the third attention feature, the first attention feature, the second attention feature and the third attention feature can be weighted and summed to obtain the predicted noise of the first round.

[0125] The embodiment of the present application performs three different attention processing processes on text vectors, noise data, and round variables, and fuses the three attention processing results to obtain predicted noise, so that the generated video can be more consistent with the description content in the text information and more continuous.

[0126] In an embodiment of the present application, based on the foreground mark of the text vector and the foreground mark of the target noise vector, a second attention processing is performed on the text vector and the intermediate noise data of the tth round to obtain a second attention feature, which can be achieved in the following way: first, for the word vector in the text vector and the noise vector in the intermediate noise data of the tth round, when the word vector has a foreground mark or the noise vector has a foreground mark, a set numerical vector is determined as an attention sub-feature; or, when both the word vector and the noise vector have a foreground mark, or neither has a foreground mark, cross-attention processing is performed on the word vector and the noise vector to obtain an attention sub-feature; then, the attention sub-features are combined to obtain a second attention feature.

[0127] Here, the numerical vector is set to be a zero vector. Taking t as 1 as an example, in the process of the second attention processing, for the word vector in the text vector and the noise vector in the target noise vector, when the word vector has a foreground tag but the noise vector does not have a foreground tag, or the word vector does not have a foreground tag but the noise vector has a foreground tag, the zero vector is determined as the attention sub-feature, that is, the attention calculation is not performed on the word vector and the noise vector. In the case where both the word vector and the noise vector have a foreground tag, or both do not have a foreground tag, the word vector and the noise vector are cross-attention processed to obtain the attention sub-feature. Cross-attention processing refers to taking the noise vector as the query matrix, the word vector as the key matrix, determining the weight parameter matrix based on the product of the query matrix and the key matrix, taking the word vector as the value matrix, and using the normalized exponential function (softmax function) to normalize the product of the weight parameter matrix and the value matrix to obtain the attention sub-feature. The attention sub-features corresponding to each word vector and noise vector combination are combined to obtain the second attention feature.

[0128] The embodiment of the present application performs cross-attention calculation only on word vectors and noise vectors that have the same foreground tag or do not have a foreground tag, so that images of the foreground and background can be generated separately, thereby enriching the details of the foreground and background.

[0129] Figure 7 is another optional flow chart of the video generation method provided in the embodiment of the present application, such as Figure 7 As shown, the method includes the following steps 301 to 309:

[0130] Step 301: The terminal receives an interactive operation from a user.

[0131] Here, the user's interactive operation may be an operation of inputting text information, an operation of setting first noise data, an operation of clicking to start generating a video, and the like.

[0132] Step 302: The terminal generates a video generation request in response to the interactive operation.

[0133] Here, after receiving the user's interactive operation, the terminal may encapsulate the text information and the first noise data into a video generation request.

[0134] Step 303: The terminal sends a video generation request to the server.

[0135] Step 304: The server determines text information for describing the video in response to the video generation request.

[0136] Here, the text information is used to describe the video to be generated. The specific process of determining the text information used to describe the video can refer to step 101 in the above embodiment, and will not be repeated here.

[0137] Step 305 , the server identifies words describing the foreground in the text information.

[0138] Here, the specific process of identifying the words describing the foreground in the text information can refer to step 102 in the above embodiment, and will not be repeated here.

[0139] Step 306: The server identifies second noise data belonging to the foreground in the first noise data of the video.

[0140] Here, the specific process of identifying the second noise data belonging to the foreground in the first noise data of the video can refer to step 103 in the above embodiment, and will not be repeated here.

[0141] In step 307 , the server generates a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0142] Here, the specific process of generating a video based on the words describing the foreground, the first noise data, the text information and the second noise data can refer to step 104 in the above embodiment, which will not be repeated here.

[0143] Step 308: The server sends the video to the terminal.

[0144] Step 309: The terminal displays the video on the current interface.

[0145] The embodiment of the present application generates a video based on the words describing the foreground, the second noise data, the text information and the first noise data, and can more accurately guide the generation of the foreground and the background during the video generation process, thereby improving the authenticity and richness of the details in the video. In addition, compared to manually specifying the foreground in the video, the embodiment of the present application obtains the words describing the foreground by performing word recognition on the text information, and obtains the second noise data belonging to the foreground by recognizing the first noise data, which can improve the efficiency of video generation and improve the accuracy of the foreground generated in the video. Therefore, the embodiment of the present application can enrich the details of the foreground and the background in the video and improve the quality of the generated video.

[0146] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0147] It should be noted that the video generation method provided in the embodiments of the present application is applied to any scenario where videos are generated using artificial intelligence technology, such as film and television animation production scenarios, game development scenarios, live interactive scenarios, education and training scenarios, cultural protection and reconstruction scenarios, medical health visualization scenarios, experimental simulations, etc. The following is an example of using artificial intelligence technology to generate a video based on pre-input text information.

[0148] The video generation method provided in the embodiment of the present application is a control method for video generation based on a diffusion model. Figure 8 is a flow chart of a control method for video generation based on a diffusion model provided in an embodiment of the present application. Figure 8 First, the input of the diffusion model is noise data 900 (latent space), text data 901 (condition 1) (corresponding to the text information in the above embodiment) and a graphic guide frame 902 (condition 2) (corresponding to the foreground frame in the above embodiment). Assuming that the diffusion process of the diffusion model has 1000 time steps, the input noise data (corresponding to the first noise data in the above embodiment) is the noise data of the 1000th time step of the diffusion process, such as the noise of [1, 32, 8, 16, 16] of Gaussian distribution (Gaussian Noise in latent space). Assuming that the video has 10 frames of image frames, each frame of the image frame randomly obtains noise from the Gaussian distribution, that is, the noise data of each frame of the image frame is different. The graphic guide frame can be a change window of a given foreground pixel range in the image frame, for example, represented by a center point, which can be [X, Y, H, W], where X and Y represent the coordinates of the center point of the graphic guide frame, and H and W represent the pixel values ​​occupied by the height and width of the graphic guide frame respectively. In a video generation process, the graphic guide frame used is the same. Alternatively, multiple graphic guide frames may be input to map the graphic guide frames to image frames: a graphic guide frame of a start frame and a graphic guide frame of an end frame are input, and the graphic guide frame in each image frame is inferred according to the number of image frames.

[0149] The text data 901 (text-prompt) may be "The panda is playing hide-and-seek in the bamboo forest." Fig. 9 It is a flow chart of the output of foreground words by the large language model provided in an embodiment of the present application. Step 501, input text data. Step 502, separate and process the text data based on the large language model to obtain foreground words and background words. Step 503, mask mark the foreground words and background words. That is, the foreground words (corresponding to the words describing the foreground in the above embodiment) are marked as foreground "1", and the background words are marked as background "0". Use a large language model (LLM, such as llama3 8B) to output the foreground words and related words in the text data 901, then panda is the foreground word, and bamboo forest is the background word. Fig.10 This is a schematic diagram of the interface of the large language model outputting foreground words provided in the embodiment of the present application. The prompt word strategy in the large language model processing process can be a chain of thought (CoT), and using this method can make the foreground topic details more abundant.

[0150] The diffusion model is a pre-trained model including a main network of a Transformer module and a diffusion denoising sampling network. The diffusion model includes an encoding module 903 (Embeddings to Tokens), a multi-head spatiotemporal self-attention module 904, a layer normalization module (LayerNorm) 905, and a linear reshape module (Linear and Reshape) 906. The encoding module 903 converts the text data into an Embedding vector (corresponding to the text vector in the above embodiment), one word in the text data corresponds to one Token, and the noise data is processed in blocks using the VIT method to obtain a Tokens vector (from three-dimensional to one-dimensional vector representation, corresponding to the target noise vector in the above embodiment). In this step, the encoding module 903 also performs position encoding on the noise data. The encoding module 903 scales the graphic guide frame to the proportional space of the noise data accordingly, and performs a foreground marking (mask marking) operation on the token at each position in the graphic guide frame (corresponding to the third noise data in the above embodiment).

[0151] The process of foreground marking operation is as follows: In order to better integrate the foreground with the background during the attention processing (Spatial-Attention) of the multi-headed spatiotemporal self-attention module 904, based on the above mechanism, in the first K time steps of the diffusion model, the following strategy is adopted to reset the foreground mark corresponding to each patch in the graphic guide frame. If the intersection over union (IoU) of the graphic guide box (bounding box) and the patch (corresponding to the fourth noise data in the above embodiment) exceeds 50% (corresponding to the preset ratio threshold in the above embodiment), the position is considered to be foreground noise (corresponding to the second noise data in the above embodiment). The intersection over union (corresponding to the overlap ratio in the above embodiment) is the intersection area (overlapping area) of the graphic guide box and the corresponding position divided by the sum of the areas of the positions (the area of ​​the fourth noise data), which is an indicator for judging the overlap of two positions. If the intersection over union is less than 50%, the intersection over union is used as a probability, and the probability is used to randomly determine whether the position belongs to the foreground. For a position that has been marked as a foreground, there is a certain probability of removing the foreground mark, and the probability is determined by the Euclidean space position of the position before the foreground position is expanded, that is, the distance between the center of the position and the center of the graphic guide frame is calculated. If the distance is greater than a preset distance threshold, the foreground mark can be removed. The embodiment of the present application adopts the above method to overcome the problem of separation between the foreground and the background in the video.

[0152] The multi-headed spatiotemporal self-attention module 904 includes three attention mechanisms: spatial-temporal cross attention, spatial-text cross attention, and spatial self-attention. In video generation, spatial-temporal cross attention (corresponding to the first attention processing in the above embodiment) allows the diffusion model to focus on the spatial relationship between different time steps, the query matrix Q is noise data, and the key matrix K and the value matrix V are both time steps (corresponding to the round variables in the above embodiment). The spatiotemporal text cross attention (corresponding to the second attention processing in the above embodiment) focuses on the relevant spatial areas in the image or video according to the text prompt (Text Prompt), the query matrix Q is noise data, and the key matrix K and the value matrix V are both text data. Spatial self-attention (corresponding to the self-attention processing in the above embodiment) focuses on the spatial relationship within the image or video, such as the interaction between different noise data, the query matrix Q is noise data, and the key matrix K and the value matrix V are both noise data. Masking operations are performed on these three attention mechanisms: that is, the foreground and background that are inconsistent are covered, and attention calculation is not performed, and the foreground and background that are consistent can be calculated. That is to say, if the masks of two tokens are marked with the same attribute (both foreground or background), attention calculation is performed, and if they are different attributes, attention calculation is not performed. This mechanism can be used directly on the pre-trained diffusion model without training.

[0153] According to the settings of the main network of the Transformer module, the corresponding multi-head spatiotemporal self-attention module 904 can be replaced intermittently in the first K (K=10) time steps of the model reasoning truncation. Intermittent replacement is defined based on the time step T in the diffusion process. For example, if a replacement is made every three steps, it is T%3. Intermittent replacement will help alleviate the problem of foreground and background fusion. Intermittent replacement allows the cross-attention module in the multi-head spatiotemporal self-attention module 904 to alternately calculate the feature maps of the foreground and background in the time step, making the fusion of this part more natural. For example, assuming a total of 50 time steps of diffusion, the original mechanism is that from the 50th step to the 40th step, the cross-section of the foreground and background cannot see each other. Intermittent replacement is that in the 50th, 48th, 46th, 44th, 42nd, and 40th time steps, the cross-section of the foreground and background (that is, the mask marks of the two Tokens are different attributes) do not participate in the attention calculation, and the foreground mark is not considered in the remaining time steps, and attention calculation is performed, so that the foreground and background fusion in the generated video is more natural.

[0154] During the attention calculation process, the attention calculation results satisfy the following formulas (1) and (2).

[0155]

[0156] Among them, MaskedAttntion(Q,K,V,M) is the result of attention calculation, query matrix (Query, Q), key matrix (Key, K) and value matrix (Value, V) are the three components involved in attention calculation. d is the dimension of K matrix, which plays the role of standardization and stabilization of gradient. is a mask, that is, a mask matrix; M[i,j] = 0 means that the foreground marks of the two tokens i and j are different, and the mask is recorded as 0. The mask matrix The corresponding position in is recorded as negative infinity, and the result of normalizing negative infinity is 0, that is, no attention calculation is performed. M[i,j] = 0 means that the foreground marks of the two tokens i and j are the same, and the mask is recorded as 1. The mask matrix The corresponding position in is recorded as 0, which does not affect the normal attention calculation.

[0157] After the three attention mechanisms are calculated, three attention results are obtained. The three attention results are fused into one (can be concatenated, averaged, etc., depending on the model structure setting) attention feature, and the attention feature is input into the layer normalization module 905. As part of the common prediction network, the layer normalization module 905 has the following functions: reducing internal covariate shift, by normalizing the output of each layer, it helps to reduce the problem of internal covariate shift and improves the generalization ability of input data. This means that it can stabilize the input distribution of each layer, making the model training process more stable and reducing the model's dependence on data distribution. Improve gradient flow: Normalization operation helps reduce the problem of gradient disappearance or explosion, making it easier to train deep networks. Then the output result of the layer normalization module 905 is input into the linear reshaping module (Linear and Reshape) 906. The linear reshaping module 906 is a fully connected network followed by an average pooling layer, mainly to align the network output shape with the required shape. The output of the linear reshaping module 906 obtains the prediction noise of the tth time step. The diffusion model subtracts the predicted noise at the tth time step from the original noise data based on the sampler's rules until time step = 0, and the final generated video is obtained.

[0158] It should be noted that, taking 1000 time steps as an example, during the training process of the diffusion model, the noise data of time step t=1000 is input, and the predicted noise of time step t=999 will be output. However, when the actual video is generated, the generation can be skipped, for example, the predicted noise of the 100th time step, 800th time step, 500th time step and so on to 0th time step is output.

[0159] The embodiment of the present application can add a control method to the pre-trained video generation model that does not contain conditional frame control. Based on the graphic guide frame and text data, the ability of the video generation model to align with human intentions is improved. In the pre-trained video generation model, there is no need to retrain, and it only needs to be inserted into all the self-attention calculation modules of the model (in the actual video generation process, the token position that does not have the same mask mark is multiplied by a large negative number, then this part of the token position does not participate in the attention calculation, otherwise it participates in the attention calculation), which can be used in any diffusion model with Transformer as the backbone network. In addition, this condition does not bring additional computational overhead and has good adaptability. In the multi-step generation of the diffusion model, assuming that t=50 is set at this time, it can be adjusted to use this mask mechanism in the first K steps, and cancel this mask mechanism in the last 50-k steps. The mask mechanism can ensure that when the initial state is generated, the foreground and background can pay more attention to their own generation and ignore the complex interaction of the fusion part. This can make the generation quality higher and the details more realistic. The embodiment of the present application can enrich the details of the foreground and background in the video and improve the quality of the generated video.

[0160] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0161] The following is a description of an exemplary structure of the video generation device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the video generation device 455 of the memory 450 may include:

[0162] A text determination module 4551 is used to determine text information used to describe a video; a foreground word recognition module 4552 is used to recognize words describing the foreground in the text information; a noise recognition module 4553 is used to recognize second noise data belonging to the foreground in the first noise data of the video; and a video generation module 4554 is used to generate a video based on the words describing the foreground, the first noise data, the text information, and the second noise data.

[0163] In some embodiments, the noise identification module 4553 is further used to determine the position information of the foreground of the image frame in the video; based on the position information of the foreground, the second noise data is marked in the first noise data.

[0164] In some embodiments, the noise identification module 4553 is also used to determine the first position information of the first foreground of the first image frame and the second position information of the second foreground of the second image frame; determine the third position information of the first image frame in the video, and determine the fourth position information of the second image frame in the video; based on the first position information, the second position information, the third position information, the fourth position information and the number of image frames in the video, determine the position information of the foreground for each image frame.

[0165] In some embodiments, the noise identification module 4553 is also used to divide the first noise data into blocks to obtain third noise data; based on the position information of the foreground, determine the fourth noise data from the third noise data, wherein the fourth noise data is the third noise data that overlaps with the foreground; determine the overlapping area between the foreground and the fourth noise data, and determine the ratio of the overlapping area to the area of ​​the fourth noise data as the overlapping ratio; based on the overlapping ratio and a preset ratio threshold, mark the second noise data in the fourth noise data.

[0166] In some embodiments, the noise identification module 4553 is also used to determine the fourth noise data as the second noise data when the overlap ratio is greater than or equal to a preset ratio threshold; when the overlap ratio is less than the preset ratio threshold, randomly generate a random number, and when the random number is less than or equal to the overlap ratio, determine the fourth noise data as the second noise data.

[0167] In some embodiments, the video generating device 455 further includes a marking adjustment module for determining the distance between the center point of the second noise data and the center point of the foreground; when the distance is greater than a preset distance threshold, the second noise data is changed to fifth noise data belonging to the background.

[0168] In some embodiments, the video generation module 4554 is also used to encode text information to obtain a text vector, wherein the word vector corresponding to the word describing the foreground in the text vector has a foreground tag; map the first noise data to obtain a target noise vector, wherein the noise vector corresponding to the second noise data in the target noise vector has a foreground tag; and generate a video based on the text vector and the target noise vector.

[0169] In some embodiments, the video is generated based on the text vector and the target noise vector through a T-round iteration process, where T is a positive integer greater than 1. The video generation module 4554 is also used to perform the following processing for the t-th round iteration: when the t-th round is the first set round, based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, the intermediate noise data of the t-th round is denoised to obtain the denoised data of the t-th round, and the denoised data of the t-th round is determined as the intermediate noise data of the t+1-th round; when the t-th round is the second set round, based on the text vector, the intermediate noise data of the t-th round is denoised to obtain the denoised data of the t-th round, and the denoised data of the t-th round is determined as the intermediate noise data of the t+1-th round, where the intermediate noise data of the 1st round is the target noise vector, 1≤t≤T; the denoised data of the T-th round is determined as the video.

[0170] In some embodiments, the video generation module 4554 is also used to perform noise prediction processing on the intermediate noise data of the tth round based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector to obtain the predicted noise of the tth round; remove the predicted noise of the tth round from the intermediate noise data of the tth round to obtain the denoised data of the tth round.

[0171] In some embodiments, the video generation module 4554 is also used to perform a first attention processing on the intermediate noise data and round variables of the tth round to obtain a first attention feature; based on the foreground mark of the text vector and the foreground mark of the target noise vector, perform a second attention processing on the text vector and the intermediate noise data of the tth round to obtain a second attention feature; based on the foreground mark of the text vector and the foreground mark of the target noise vector, perform self-attention processing on the intermediate noise data of the tth round to obtain a third attention feature; and fuse the first attention feature, the second attention feature and the third attention feature to obtain the predicted noise of the tth round.

[0172] In some embodiments, the video generation module 4554 is also used to determine the set numerical vector as the attention sub-feature for the word vector in the text vector and the noise vector in the intermediate noise data of the tth round, when the word vector has a foreground mark or the noise vector has a foreground mark; when both the word vector and the noise vector have a foreground mark, or neither has a foreground mark, perform cross-attention processing on the word vector and the noise vector to obtain an attention sub-feature; and combine the attention sub-features to obtain a second attention feature.

[0173] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the video generation method described in the embodiment of the present application.

[0174] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the video generation method provided by the present application embodiment, for example, Figure 3 The video generation method is shown.

[0175] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0176] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0177] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0178] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0179] In summary, the embodiments of the present application can enrich the details of the foreground and background in the video, reduce the sense of separation between the foreground and background, and improve the quality of the generated video.

[0180] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Determining text information used to describe the video; identifying words in the text information that describe a prospect; identifying second noise data belonging to the foreground in the first noise data of the video; A video is generated based on the words describing the foreground, the first noise data, the text information, and the second noise data.

2. The method according to claim 1, characterized in that The identifying second noise data belonging to the foreground in the first noise data of the video includes: Determining position information of the foreground of the image frame in the video; Based on the position information of the foreground, the second noise data is marked in the first noise data.

3. The method according to claim 2, characterized in that The determining the position information of the foreground of the image frame in the video includes: Determine first position information of a first foreground of a first image frame and second position information of a second foreground of a second image frame; Determine third position information of the first image frame in the video, and determine fourth position information of the second image frame in the video; Based on the first position information, the second position information, the third position information, the fourth position information, and the number of image frames in the video, position information of a foreground for each image frame is determined.

4. The method according to claim 2, characterized in that: The step of marking the second noise data in the first noise data based on the position information of the foreground includes: Dividing the first noise data into blocks to obtain third noise data; Based on the position information of the foreground, determining fourth noise data from the third noise data, wherein the fourth noise data is the third noise data that overlaps with the foreground; determining an overlapping area between the foreground and the fourth noise data, and determining a ratio of the overlapping area to an area of ​​the fourth noise data as an overlapping ratio; Based on the overlap ratio and a preset ratio threshold, the second noise data is marked in the fourth noise data.

5. The method according to claim 4, characterized in that The marking the second noise data in the fourth noise data based on the overlap ratio and a preset ratio threshold comprises: When the overlap ratio is greater than or equal to the preset ratio threshold, determining the fourth noise data as the second noise data; When the overlap ratio is less than the preset ratio threshold, a random number is randomly generated, and when the random number is less than or equal to the overlap ratio, the fourth noise data is determined as the second noise data.

6. The method according to any one of claims 1 to 5, characterized in that: After identifying second noise data belonging to the foreground in the first noise data of the video, the method further includes: Determine the distance between the center point of the second noise data and the center point of the foreground; When the distance is greater than a preset distance threshold, the second noise data is changed to fifth noise data belonging to the background.

7. The method according to claim 1, characterized in that The step of generating a video based on the words describing the foreground, the first noise data, the text information, and the second noise data comprises: Encoding the text information to obtain a text vector, wherein the word vector corresponding to the word describing the foreground in the text vector has a foreground tag; Mapping the first noise data to obtain a target noise vector, wherein a noise vector corresponding to the second noise data in the target noise vector has a foreground mark; A video is generated based on the text vector and the target noise vector.

8. The method according to claim 7, characterized in that The video is generated based on the text vector and the target noise vector through T rounds of iterative processes, where T is a positive integer greater than 1; the video is generated based on the text vector and the target noise vector, including: The following processing is performed for the tth iteration: In the case where the tth round is the first set round, based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, denoising is performed on the intermediate noise data of the tth round to obtain denoised data of the tth round, and the denoised data of the tth round is determined as the intermediate noise data of the t+1th round; In the case where the tth round is the second set round, based on the text vector, the intermediate noise data of the tth round is denoised to obtain the denoised data of the tth round, and the denoised data of the tth round is determined as the intermediate noise data of the t+1th round, wherein the intermediate noise data of the 1st round is the target noise vector, 1≤t≤T; The denoised data of the Tth round is determined as a video.

9. The method according to claim 8, characterized in that The step of performing denoising on the intermediate noise data of the tth round based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector to obtain denoised data of the tth round includes: Based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector, performing noise prediction processing on the intermediate noise data of the tth round to obtain the predicted noise of the tth round; The prediction noise of the tth round is removed from the intermediate noise data of the tth round to obtain denoised data of the tth round.

10. The method according to claim 9, characterized in that The step of performing noise prediction processing on the intermediate noise data of the tth round based on the foreground mark of the text vector, the foreground mark of the target noise vector and the text vector to obtain the predicted noise of the tth round includes: Performing first attention processing on the intermediate noise data and round variables of the tth round to obtain first attention features; Based on the foreground mark of the text vector and the foreground mark of the target noise vector, performing second attention processing on the text vector and the intermediate noise data of the tth round to obtain a second attention feature; Based on the foreground mark of the text vector and the foreground mark of the target noise vector, performing self-attention processing on the intermediate noise data of the tth round to obtain a third attention feature; The first attention feature, the second attention feature and the third attention feature are fused to obtain the prediction noise of the tth round.

11. The method according to claim 10, characterized in that The step of performing second attention processing on the text vector and the intermediate noise data of the tth round based on the foreground mark of the text vector and the foreground mark of the target noise vector to obtain a second attention feature includes: For the word vector in the text vector and the noise vector in the intermediate noise data of the tth round, when the word vector has the foreground tag or the noise vector has the foreground tag, determining a set numerical vector as an attention sub-feature; In a case where both the word vector and the noise vector have the foreground tag, or both do not have the foreground tag, cross-attention processing is performed on the word vector and the noise vector to obtain an attention sub-feature; The attention sub-features are combined to obtain a second attention feature.

12. A video generating device, characterized in that: The device comprises: A text determination module, used to determine text information used to describe the video; a foreground word recognition module, used to recognize words describing the foreground in the text information; A noise identification module, used for identifying second noise data belonging to the foreground in the first noise data of the video; A video generation module is used to generate a video based on the words describing the foreground, the first noise data, the text information and the second noise data.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the video generation method described in any one of claims 1 to 11 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the video generating method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the video generation method according to any one of claims 1 to 11 is implemented.

Images