Video generation method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a video generation method and device, electronic equipment, a computer readable storage medium and a computer program product. The embodiment of the invention can be applied to video generation scenes of games, virtual reality, video processing and the like; the method comprises the following steps of: determining a first semantic feature from a first description text for describing a global video, and determining a second semantic feature from a second description text for describing a video clip; carrying out the following processing through iteration i: generating an ith intermediate variable block sequence by utilizing the ith input variable block sequence, the first semantic feature and the position sequence; determining an ith output variable block sequence based on the ith intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature; and determining a target video based on the Nth output variable block sequence and the position sequence. According to the invention, the quality of the generated video can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a video generation method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Video generation is an important application direction of artificial intelligence. This technology can automatically generate a matching video for the description text entered by the user. When generating a video, in order to meet the requirements indicated by the user through the description text as much as possible, it is also necessary to be able to finely control the clips in the video during the video generation process. However, in the related art, there is a lack of correlation between the latent variable subsequence of the video clip and the description text of the complete video, resulting in a low correlation between the screen of the generated video clip and the screen of the original video, which ultimately makes the quality of the generated video low. Summary of the invention

[0003] Embodiments of the present application provide a video generation method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the quality of generated videos.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a video generation method, the method comprising:

[0006] Determine a first semantic feature from a first description text that describes the entire video, and determine a second semantic feature from a second description text that describes the video segment;

[0007] The following processing is performed by iteration i, 1≤i≤N, N is a positive integer:

[0008] Generate an i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence; when i=1, the i-th input variable block sequence includes latent variable blocks obtained by dividing the latent variables of the initial latent variable sequence into blocks, and when i>1, the i-th input variable block sequence includes the i-1-th output variable block sequence, and the position sequence includes the position information of each of the latent variable blocks;

[0009] Determine an i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature;

[0010] Based on the Nth output variable block sequence and the position sequence, a target video indicated by the first description text and the second description text is determined.

[0011] The embodiment of the present application provides a video generation device, including:

[0012] A feature determination module, used to determine a first semantic feature from a first description text describing the entire video, and to determine a second semantic feature from a second description text describing the video segment;

[0013] A sequence determination module, for generating an i-th intermediate variable block sequence using an i-th input variable block sequence, the first semantic feature and a position sequence; when i=1, the i-th input variable block sequence includes a latent variable block obtained by dividing the latent variables of the initial latent variable sequence into blocks; when i>1, the i-th input variable block sequence includes an i-1-th output variable block sequence, and the position sequence includes position information of each of the latent variable blocks; based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature, determining the i-th output variable block sequence;

[0014] A video determination module is used to determine a target video indicated by the first description text and the second description text based on the Nth output variable block sequence and the position sequence.

[0015] In the above scheme, the sequence determination module is also used to determine a first subsequence from the i-th intermediate variable block sequence based on the time interval and the position sequence, and determine a position subsequence corresponding to the first subsequence from the position sequence; generate a second subsequence based on the second semantic feature, the first subsequence and the position subsequence; and determine the i-th output variable block sequence using the second subsequence and the i-th intermediate variable block sequence.

[0016] In the above scheme, the second semantic features include: multi-level text features of the second description text, and single-level text features of the second description text; the sequence determination module is also used to fuse the multi-level semantic features of the second description text into each latent variable block of the first subsequence, and generate a third subsequence using the fused latent variable blocks; fuse the multi-level text features of the second description text and the single-level text features of the second description text to obtain fused features; and generate the second subsequence based on the third subsequence and the fused features.

[0017] In the above scheme, the sequence determination module is also used to determine a control identifier for the latent variable block in the i-th intermediate variable block sequence based on the time interval and the time position in the position sequence, wherein the control identifier indicates whether the latent variable block needs to be regenerated according to the second description text; from the i-th intermediate variable block sequence, the latent variable block that needs to be regenerated according to the second description text represented by the control identifier is extracted, and the sequence generated by using the extracted latent variable block is determined as the first subsequence.

[0018] In the above scheme, the sequence determination module is also used to use the second subsequence to replace the first subsequence in the i-th intermediate variable block sequence to obtain the i-th variable block sequence to be optimized; and smooth the i-th variable block to be optimized to obtain the i-th output variable block sequence.

[0019] In the above scheme, the feature determination module is also used to extract K second text features from the second description text; wherein K is a positive integer; and determine the second semantic feature based on the K second text features.

[0020] In the above scheme, the K second text features include: multi-level text features of the second description text; the feature determination module is also used to encode the second description text to obtain second encoded features; from the second encoded features, multiple levels of second text sub-features are extracted, and the multi-level text features of the second description text are determined using the multiple levels of the second text sub-features.

[0021] In the above scheme, the video generation module also includes: a variable block module, which is used to divide each latent variable in the initial latent variable sequence into multiple latent variable blocks; and generate the first input variable block sequence using the multiple latent variable blocks of each latent variable.

[0022] In the above scheme, the video determination module is also used to restore the Nth output variable block sequence to an output latent variable sequence based on the position sequence; decode the output latent variable sequence to obtain a video frame sequence, and based on the video frame sequence, determine the target video jointly indicated by the first description text and the second description text.

[0023] In the above scheme, i=M, and the sequence determination module is further used to determine the Mth intermediate variable block sequence as the Mth output variable block sequence, wherein M can take any one or more values ​​between 1 and N.

[0024] An embodiment of the present application provides an electronic device, wherein the electronic device comprises:

[0025] A memory for storing computer executable instructions or computer programs;

[0026] The processor is used to implement the video generation method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0027] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the video generation method provided in the embodiment of the present application when executed by a processor.

[0028] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the video generation method provided in the embodiment of the present application is implemented.

[0029] The embodiments of the present application have the following beneficial effects: in each iteration, the first semantic feature of the first description text that describes the video globally, the input variable block sequence and the position sequence of each iteration are first used to generate an intermediate variable block sequence to realize the injection of the first semantic feature, and then, based on the intermediate variable block sequence, combined with the time interval of the video clip and the second semantic feature of the second description text that describes the video clip, an output variable block sequence of each iteration is generated to realize the injection of the second semantic feature, so that the latent variable block corresponding to the video clip can be associated with the first description text, and the process is continuously repeated through iteration, so that the semantic features of the first description text and the second description text can be fully injected into the latent variable block, so that the transition between the picture of the video clip and the picture of the complete video can be natural, thereby improving the quality of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the architecture of the video generation system provided in an embodiment of the present application;

[0031] Figure 2 The embodiment of this application provides Figure 1 A schematic diagram of the structure of the server in FIG.

[0032] Figure 3 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 1 ;

[0033] Figure 4 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 2 ;

[0034] Figure 5 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 3;

[0035] Fig. 6A This is a short video generation diagram provided in the embodiment of the present application. Figure 1 ;

[0036] Figure 6B This is a short video generation diagram provided in the embodiment of the present application. Figure 2 ;

[0037] Figure 7 It is a video generation system architecture diagram provided in an embodiment of the present application;

[0038] Figure 8 It is a schematic diagram of the overall prompt word encoding and the segment prompt word encoding provided in the embodiment of the present application;

[0039] Fig. 9 It is a model architecture diagram of the video generation part provided in the embodiment of the present application;

[0040] Fig.10 This is a schematic diagram of the structure of the video generation model provided in the embodiment of the present application. Figure 1 ;

[0041] Fig.11 It is a schematic diagram of generating marking information provided by an embodiment of the present application;

[0042] Fig.12 This is a schematic diagram of the structure of the video generation model provided in the embodiment of the present application. Figure 2 . DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0044] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0047] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0048] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0049] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0050] 1) Video generation refers to outputting a video that matches the input text describing the video (hereinafter referred to as the description text) through artificial intelligence processing. For example, if the text input by the user is "a white little rabbit suddenly appeared in the depths of the forest", the generated video may include a green forest scene and a white rabbit jumping out of a certain part of the forest.

[0051] 2) Description text refers to text used to describe the content of the video. In an embodiment of the present application, there may be multiple different description texts for the same video, such as a first description text and a second description text. Among them, the first description text is a text that describes the entire video, that is, the video as a whole, which provides information about the theme and overall style of the video to set a unified background or framework for the video; the second description text is a text that describes a specific part of the video, that is, a text that describes the video clip, which can describe a specific scene, action or detail in the video in detail, so as to finely control the local content in the video generation. For example, "generate a romantic sunset beach walk video" can be used as the first description text, and "in the video, when the protagonist walks to the beach, show the flying seagulls and floating waves" can be used as the second description text.

[0052] 3) Latent sequence is a sequence generated by latent variables. Each latent variable in the latent variable sequence corresponds to a video frame in the video frame sequence. Here, the latent variable can be regarded as a variable that is not directly observed in the video generation process, that is, the most basic feature in the video generation process, which can be obtained by adding or removing noise to obtain a new video frame. In other words, the latent variable is an abstract representation of the video frame, which contains key features such as the motion pattern and texture information of the video frame, which can be used as the basis for generating new video frames (for example, adding descriptive text to the existing latent variable to generate a new latent variable, and decoding the latent variable to obtain a new video frame).

[0053] 4) Position sequence is a sequence generated by the position information of the latent variable block. The latent variable block is obtained by dividing the latent variable into blocks. The position information of the latent variable block includes the spatial position of the latent variable block, that is, the position of the latent variable block in the latent variable, and also includes the time position of the latent variable block, that is, the appearance time of the latent variable block.

[0054] 5) A control flag is a flag used to distinguish whether the latent variable block needs to be regenerated based on the second description text described in the video clip. In other words, each latent variable block has its corresponding control flag, which indicates whether the latent variable block is a latent variable block that needs fine control.

[0055] 6) Diffusion Transformer (DIT) model is a deep learning model that combines the diffusion model and the transformer model. It uses the powerful sequence modeling ability of the transformer and the advantages of the diffusion model in data generation to generate high-quality videos. In DIT, the diffusion process starts with a simple distribution (such as Gaussian noise) and transforms this simple distribution into a target distribution (such as the distribution of training data) through a series of steps. Each step can be regarded as adding some noise; then the transformer is used to predict the distribution of the next step. This process can be regarded as a reverse Markov chain, which starts from noise and gradually changes the distribution to eventually generate complex samples.

[0056] 7) Contrastive Language Image Pre-training (CLIP) is a multimodal deep learning model that uses contrastive learning to learn the association between images and text by pre-training on large-scale text-image datasets. During the pre-training process, the model is able to learn the ability to encode images and text into a unified vector space, which enables the model to understand the content of images and match images with the text that describes them. After pre-training, CLIP can identify elements such as objects, scenes, actions, etc. in images, and can also understand text related to the image, such as labels, descriptions, titles, etc.

[0057] 8) Text-to-Text Transfer Transformer (T5) is a pre-trained natural language processing (NLP) model based on the Transformer model. The T5 model regards all NLP tasks as text-to-text conversion problems, so the T5 model can accept any form of text input and generate corresponding text output.

[0058] 9) Cross Attention, which is usually used in the model to process the association between multiple input sequences, so as to capture the important information interaction between different sequences in the model. The cross attention mechanism can help the diffusion model better understand the association between different parts, so as to more accurately predict the changes and impacts in the diffusion process.

[0059] 10) Self-attention mechanism, which can assign different weights to each element in the sequence (the weight indicates the relevance of the element to other elements), so that the model can dynamically pay attention to other elements in the sequence when processing each element in the sequence. For example, in the diffusion model, the input of the self-attention mechanism is a latent variable or a latent variable that has been fused with semantic features. At this time, the role of self-attention is to make the semantic features in the latent variable more obvious, thereby improving the effect of video generation.

[0060] Video generation is an important application direction of artificial intelligence. This technology can automatically generate a matching video for the description text entered by the user. When generating a video, in order to meet the requirements indicated by the user through the description text as much as possible, it is also necessary to be able to finely control the clips in the video during the video generation process. In related technologies, video generation can be achieved through the following processing: feature extraction of the global description text of the video, and injecting the obtained semantic features together with the initialized latent variable sequence into a model capable of video generation, so that the model can generate a new latent variable sequence that incorporates the semantic features of the description text, and then the model denoises the new latent variable sequence to obtain a video that meets the requirements of the description text.

[0061] For example, a temporal self-attention mechanism is added to the U-shaped network (UNet) of the stable diffusion model to improve the video generation capability of the stable diffusion model. In this scheme, the input is the description text of the entire video and the latent variable sequence initialized by the stable diffusion model. Each latent variable in the latent variable sequence is the basis for generating a video frame. After the description text is encoded by the encoder in the CLIP model, relevant features are generated, and then injected into the UNet combined with the video generation technology together with the latent variable sequence. The UNet generates a new latent variable sequence, and finally the new latent variable sequence is subjected to denoising to obtain a video that meets the user's description text requirements.

[0062] Among them, for video clips that need fine control, we can first query the corresponding latent variable subsequence in the latent variable sequence according to the time interval of the video clip, and then inject the features of the description text of the video clip and the latent variable subsequence into a model dedicated to sub-content generation (also based on the stable diffusion model structure). The sub-content generation model generates a new latent variable subsequence for the video clip, and uses the latent variable subsequence to replace the original latent variable subsequence, thereby completing the fine control of the video clip.

[0063] Alternatively, it is also possible to perform secondary generation for video clips that require fine control. For example, using the video generation method described above, all video frame sequences are first generated, and then the video frames in the video clips that require fine control are extracted and encoded into a new latent variable sequence using the variational autoencoder (VAE) in the stable diffusion model, which is then input into the stable diffusion model again. The description text of the video clip is also input into the stable diffusion model at the same time. In this way, the stable diffusion model can regenerate the video frames of the video clip, and finally insert the generated video frames into the original time position, thereby completing the generation of the entire video.

[0064] However, although the above scheme can control the video content within a specified time segment, the generation of the latent variable sequence of the complete video and the generation of the latent variable subsequence of the video segment can be regarded as two independent latent variable sequence generation. This will cause the latent variable subsequence of the video segment to lack association with the description text of the complete video, resulting in the weak association between the image of the generated video segment and the image of the original video, resulting in the low quality of the generated video. In addition, in the above scheme, directly inserting the generated latent variable subsequence into the original latent variable sequence, or directly replacing the original video segment with the generated video segment, can easily lead to serious sudden picture changes, which ultimately results in low quality of the generated video.

[0065] The embodiments of the present application provide a video generation method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the quality of the generated video. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the electronic device is implemented as a server.

[0066] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video generation system provided in the embodiment of the present application. To support a video generation application, in the video generation system 100, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. It should be noted that in the video generation system 100, a database 500 is also provided to provide data support to the server 200. The database 500 can be configured in the server 200, or it can be independent of the server 200. Figure 1 The illustrated case is that the database 500 is independent of the server 200 .

[0067] Terminal 400 is used to respond to the user's input operation in the text input interface in the graphical interface (graphic interface 400-11 and graphical interface 400-21 are shown as examples), obtain the first description text and the second description text, and send the first description text and the second description text to the server 200 through the network 300.

[0068] The server 200 is used to determine a first semantic feature from a first description text that describes the video globally, and to determine a second semantic feature from a second description text that describes the video clip; perform the following processing by iteration i, 1≤i≤N, N is a positive integer: generate an i-th intermediate variable block sequence using the i-th input variable block sequence, the second semantic feature and the position sequence, wherein when i=1, the i-th input variable block sequence includes a latent variable block obtained by dividing the latent variables of the initial latent variable sequence into blocks, and when i>1, the i-th input variable block sequence includes an i-1-th output variable block sequence, and the position sequence includes the position information of each latent variable block; determine the i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature; determine the target video indicated by the first description text and the second description text based on the N-th output latent variable block sequence; and send the target video to the terminal 400.

[0069] The terminal 400 is further configured to display the target video in the graphic interface 400 - 11 and the graphic interface 400 - 21 .

[0070] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0071] See also Figure 2 , Figure 2 The embodiment of this application provides Figure 1 A schematic diagram of the structure of a server (an implementation of an electronic device) in FIG. Figure 2 The server 200 shown includes: at least one processor 210, a memory 250, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0072] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0074] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0076] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0077] A network communication module 252, used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi) and Universal Serial Bus (USB), etc.;

[0078] In some embodiments, the video generation device provided in the embodiments of the present application can be implemented in software. Figure 2 The video generation device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a feature determination module 2551, a sequence determination module 2552, a video determination module 2553 and a variable block module 2554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0079] In other embodiments, the video generating device provided in the embodiments of the present application can be implemented in hardware. As an example, the video generating device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generating method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0080] In some embodiments, a terminal or a server (both are possible implementations of electronic devices) can implement the video generation method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Nativ e) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a video production APP, a short video APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0081] Below, the video generation method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. As mentioned above, the electronic device implementing the video generation method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated hereinafter.

[0082] See also Figure 3 , Figure 3 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 1 , will combine Figure 3 The steps shown are explained, Figure 3 The main body of the step is the electronic device.

[0083] Step 101: determine a first semantic feature from a first description text that describes the entire video, and determine a second semantic feature from a second description text that describes a video segment.

[0084] The embodiment of the present application is implemented in a scenario where a matching video is generated based on a user's input text. The user's input text includes a first description text for describing the video globally and a second description text for describing a video segment in the video. The electronic device extracts semantic features from the first description text and the second description text respectively to obtain a first semantic feature that can reflect the text meaning and semantic information of the first description text, and obtains a second semantic feature that can reflect the text meaning and semantic information of the second description text.

[0085] It should be noted that the video global here refers to the entire video, so the first description text summarizes the information related to the overall content of the video to be generated, such as the main theme, emotion, style and other information. It can be seen that the purpose of the first description text is to provide a global semantic framework for video generation. Video clip refers to a clip in the video to be generated, such as the 2nd to 4th seconds of the video, so the second description text describes in detail the content, actions or changes in this time period, such as specific events in the scene, the actions and expressions of the characters in the video, etc. It can be seen that the purpose of the second description text is to finely control the video of a specific time segment to obtain more personalized video content.

[0086] For example, the first description text input by the user may be "A group of people are walking on the beach at sunset in summer", which specifies the main scene and participants of the video and provides direction for video generation; the second description text input by the user may be "Seagulls are flying in the distance", which describes a specific segment in the video in more detail and is used to guide the content of the specific segment.

[0087] It should be noted that the electronic device can extract semantic features from the first description text and the second description text in a variety of different ways. The following first describes the extraction process of the second semantic feature.

[0088] In some embodiments of the present application, Figure 3 Determining the second semantic feature from the second description text describing the video clip in step 101 can be achieved by the following process: extracting K second text features from the second description text; and determining the second semantic feature based on the K second text features, where K is a positive integer.

[0089] The electronic device first performs K different feature extractions on the second description text to obtain K different second text features for the same second description text. Afterwards, the K second text features can be directly used to generate second semantic features to complete the semantic feature extraction of the second description text.

[0090] Here, the electronic device can implement N feature extractions on the second description text through K different text encoders, for example, respectively using the text encoder in the contrastive language image pre-training model (Contrastive Language Image Pre-training, CLIP) and the text encoder in the text-to-text transfer transformer (Text-to-Text Transfer Transformer, T5) model to perform feature extraction on the second description text respectively to obtain two different second text features.

[0091] The electronic device may also use K different content feature analysis methods for the second description text to achieve K feature extractions for the second description text, such as using entity (such as place, object name) recognition to achieve feature extraction for the second description text, and using keyword extraction to achieve feature extraction for the second description text, thereby being able to obtain two different second text features, namely, entities and keywords, for the second description text.

[0092] In some embodiments of the present application, the K second text features may include: multi-level text features of the second description text. The multi-level text features refer to text features that integrate text meanings and semantic information at different levels, so that the second semantic features obtained based on the multi-level text features of the second description text can simultaneously retain text features at different levels, for example, it can simultaneously retain high-level features for the overall picture description in the second description text, and can also simultaneously retain low-level features for the picture details description in the second description text, thereby increasing the amount of information in the second semantic features.

[0093] In the case where the K second text features include multi-level text features of the second description text, K second text features are extracted from the second description text, including: encoding the second description text to obtain second encoded features; extracting multiple levels of second text sub-features from the second encoded features, and determining the multi-level text features of the second description text using the multiple levels of second text sub-features.

[0094] The electronic device can encode the second description text using a bag-of-words model or a word embedding model to map the second description text to a feature space (e.g., a vector space, a matrix space, etc.) to obtain a corresponding second encoded feature. Afterwards, the electronic device can extract features from the second encoded feature through a text encoder, such as a text encoder of CLIP, and use the outputs of the last several layers of the text encoder as second text sub-features, such as the outputs of the last 6 layers of the text encoder of CLIP as second text sub-features. Afterwards, the electronic device can directly determine the second text sub-features of multiple levels as multi-level text features of the second description text, or can fuse the obtained second text sub-features of multiple levels (e.g., concatenate, overlay, etc.), and determine the fusion result as the multi-level text feature of the second description text.

[0095] Of course, the K second text features may include: single-level text features of the second description text, where the single-level text features refer to text features with only a single level of text meaning and semantic information, which may be the output of the last layer of the text encoder.

[0096] After obtaining the K second text features, the electronic device can use the obtained K second text features to determine the second semantic feature corresponding to the second description text. Here, the electronic device can fuse the K second text features to obtain the second semantic feature, or perform dimensional conversion on the K second text features respectively (that is, so that the K second text features can be used as a follow-up), and jointly determine the K second text features after the dimensional conversion as the second semantic feature.

[0097] At this point, the generation of the second semantic feature is completed.

[0098] In some embodiments of the present application, Figure 3 Determining the first semantic feature from the first description text that globally describes the video in step 101 can be achieved by the following process: extracting K first text features from the first description text; and determining the first semantic feature based on the K first text features. Similarly, K is a positive integer.

[0099] Among them, the N first text features include: multi-level text features of the first description text. In this case, K first text features are extracted from the first description text, including: encoding the first description text to obtain a first encoding feature; extracting multiple levels of first text sub-features from the first encoding feature, and using the multiple levels of first text sub-features to determine the multi-level text features of the first description text.

[0100] It should be noted that the method of determining the first semantic feature from the first description text is basically similar to the method of determining the second semantic feature from the second description text mentioned above, and will not be repeated here.

[0101] After completing step 101, the electronic device may execute steps 102 to 103 by iterating i, wherein 1≤i≤N, N is a positive integer, and the specific value of N may be set according to actual needs.

[0102] Step 102: Generate an i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence.

[0103] After the electronic device extracts the first semantic feature and the second semantic feature from the user's input, it will start injecting the first semantic feature and the second semantic feature to obtain the latest latent variable sequence. It should be noted that in order to make the details of all areas in the video frame richer, in the embodiment of the present application, the semantic feature is injected in units of the latent variable block of the latent variable, and multiple semantic feature injections are performed by iterating i so that the latent variable block, the first semantic feature and the second semantic feature can be fully integrated. Here, when i=1, the i-th input variable block sequence includes the latent variable block obtained by dividing the latent variables of the initial latent variable sequence into blocks, and when i>1, the i-th input variable block sequence includes the i-1-th output variable block sequence; the position sequence record includes the position information of each latent variable block. Among them, the initial latent variable sequence can be obtained by initializing with random noise, for example, obtaining a latent variable sequence (which can be a historical video or a latent variable sequence corresponding to a blank video), and adding random noise to the latent variable sequence to obtain the initial latent variable sequence.

[0104] In each iteration, the electronic device will combine the first semantic feature and the position sequence on the basis of the input variable block sequence of each iteration to generate a new variable block sequence. The newly generated variable block sequence is recorded as the intermediate variable block sequence of each iteration, and the intermediate variable block sequence is used to inject the second semantic feature later. Among them, when i=1, that is, in the first iteration, the electronic device will use the sequence composed of the latent variable blocks of each latent variable in the initial latent variable sequence, that is, the variable block sequence corresponding to the initial latent variable sequence, as the input variable block sequence of the first iteration. When i>1, that is, in the second and subsequent iterations, the electronic device can use the output variable block sequence of the i-1th iteration, that is, the previous iteration, as the input variable block sequence of this iteration. Of course, the electronic device can also select any one of the output variable block sequences generated in the previous i-1 iterations, that is, all iterations before this iteration, as the input variable block sequence of the i-th iteration.

[0105] It should be noted that the position information of the latent variable block may include the spatial position of the latent variable block, that is, the position of the latent variable block in the latent variable, and may also include the time position of the latent variable block, that is, the appearance time of the latent variable block. Through the position information of the latent variable block, the electronic device can clearly know the time position and spatial position of the latent variable block, so as to clearly know what kind of processing should be performed on the latent variable block to generate a new latent variable block that meets the requirements. Here, the spatial position of the latent variable block can be given by the center coordinates or the coordinates of the upper left corner of the latent variable block, and the time position of the latent variable block can be given according to the video frame identifier of the latent variable to which the latent variable block belongs (here each latent variable will correspond to a video frame, so each latent variable will have a frame identifier) ​​or the appearance time of the video frame corresponding to the latent variable to which it belongs in the video.

[0106] In some embodiments of the present application, the first semantic feature includes: a multi-level text feature of the first description text, and a single-level text feature of the first description text. In this case, the electronic device can first fuse the multi-level text feature of the first description text with the latent variable block in each iteration of the input variable block sequence, and fuse the multi-level text feature of the first description text with the single-level text feature of the first description text, and operate the two fusion results and the position sequence through the attention mechanism to obtain a new variable block sequence. After that, the obtained new variable block sequence can be subjected to regularization, nonlinear transformation, dimensional transformation and other processing, and the processing result is used as an intermediate variable block sequence; the obtained new variable block sequence can also be subjected to regularization, nonlinear transformation and dimensional transformation and other processing, and the fusion result of the new variable block sequence and the single-level semantic feature of the first description text is subjected to regularization, nonlinear transformation and dimensional transformation and other processing, and finally the two processing results are merged to obtain an intermediate variable block sequence.

[0107] It should be noted that the fusion, attention mechanism, regularization, nonlinear transformation and dimension transformation in the above processing can be implemented by a network layer with corresponding functions. Among them, the electronic device can use the zero initial normalization (LayerNormZero) layer to realize the fusion of the multi-level text features of the first description text and the variable block in the input variable block sequence, and the fusion of the multi-level text features of the first description text and the single-level text features of the first description text. The electronic device can use the attention layer (Attention Layer) to implement the attention mechanism, and the input of the attention layer is the fusion result of the multi-level text features of the first description text and the variable block, the fusion result of the multi-level text features of the first description text and the single-level text features of the first description text, and the position sequence, wherein the value vector (Value) in the attention mechanism can be calculated by the fusion result of the multi-level text features of the first description text and the variable block, and the query vector (Query) and the key vector (Key) can be selected from the fusion result of the multi-level text features of the first description text and the single-level text features of the first description text, and the position information of the latent variable block in the position sequence for calculation (the calculation parameters of the query vector (Query) and the key vector (Key) cannot be repeated). Electronic devices can use the normalization layer (LayerNorm) to implement regularization processing and the feedforward layer (Feedforward) to implement nonlinear transformation and dimensional transformation processing.

[0108] In some other embodiments of the present application, the first semantic feature includes: a multi-level text feature of the first descriptive text. In this case, the electronic device may first fuse the multi-level text features of the first descriptive text with the variable blocks in the input variable block sequence of each iteration, and may combine the position subsequence to apply an attention mechanism to the new variable block sequence obtained by the fusion, and then may perform regularization, nonlinear transformation, dimensional transformation and other processing on the obtained new variable block sequence, and use the processing result as an intermediate variable block sequence; or may perform regularization, nonlinear transformation, dimensional transformation and other processing on the obtained new variable block sequence, and at the same time perform regularization, nonlinear transformation, dimensional transformation and other processing on the fusion result of the new variable block sequence and the multi-level semantic features of the first descriptive text, and finally merge the two processing results to obtain an intermediate variable block sequence.

[0109] Step 103: Determine the i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature.

[0110] After obtaining the intermediate variable block sequence of each iteration, the electronic device will inject the second semantic feature into the intermediate variable block sequence in combination with the time interval and position sequence of the video clip to obtain a new variable block sequence, and record the obtained variable block sequence as the output variable block sequence of each iteration. The obtained output variable block sequence can be used as the input variable block sequence of subsequent iterations to generate a variable block sequence with a more detailed and rich representation through iterative processing.

[0111] In the embodiment of the present application, a video segment refers to a segment that needs to be finely controlled when the video is generated, so that the time interval of the video segment refers to the interval formed from the start time of the segment that needs to be finely controlled to the end time of the segment. The time interval can be specified by the user, or it can be determined by the electronic device based on the logical relationship between the events described by the first description text and the second description text (for example, when the first description text is "dance performance, the audience takes their seats when the performance starts, and the audience applauds after the performance ends", and the second description text is "showing a close-up of a dancer's expression", at this time, the electronic device can set the time interval of the required video segment to 1 / 2 to 2 / 3 of the video). Of course, the time interval can also be set randomly by the electronic device.

[0112] See also Figure 4 , Figure 4 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, Figure 3 Step 103 in the above, i.e., determining the i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence, and the second semantic feature, can be implemented by the following processing:

[0113] Step 1031: Determine a first subsequence from the ith intermediate variable block sequence according to the time interval and the position sequence, and determine a position subsequence corresponding to the first subsequence from the position sequence.

[0114] The position sequence records the position information of the latent variable block, which includes the time position and spatial position of the latent variable block. The position sequence does not change in each iteration, and is a sequence generated by the position information of the variable block of the latent variable in the initial latent variable sequence. In this way, in each iteration, the electronic device can read the time position of each latent variable block in the intermediate variable block sequence from the position sequence, and extract the first subsequence from the intermediate variable block sequence based on the read time position and time interval. After determining the first subsequence, the electronic device will extract the position information corresponding to each latent variable block in the first subsequence from the position sequence, and the sequence formed by these position information is the position subsequence.

[0115] In some embodiments of the present application, Figure 4 Step 1031 in , i.e., determining the first subsequence from the ith intermediate variable block sequence according to the time interval and position sequence, can be achieved by the following processing: determining a control flag for the latent variable block in the ith intermediate variable block sequence according to the time position in the time interval and position sequence, wherein the control flag indicates whether the latent variable block needs to be regenerated according to the second description text; extracting from the ith intermediate variable block sequence the latent variable block that needs to be regenerated according to the second description text, which is indicated by the control flag, and determining a sequence generated by using the extracted latent variable block as the first subsequence.

[0116] That is, the electronic device will compare the time position read from the position sequence with the time interval. When the time position of the latent variable block hits the time interval, it means that the latent variable to which the latent variable block belongs corresponds to the video frame that needs to be finely controlled, that is, the latent variable block needs to be regenerated according to the second description text. At this time, the electronic device will set the control identifier of the latent variable block to indicate that it needs to be regenerated according to the second description text (for example, the control identifier of the latent variable block is set to 1); when the time position of the latent variable block does not hit the time block, it means that the latent variable to which the latent variable block belongs does not correspond to the video frame that needs to be finely controlled, so that the latent variable block does not need to be regenerated according to the second description text. At this time, the electronic device will set the control identifier of the latent variable block to indicate that it does not need to be regenerated according to the second description text (for example, the control identifier of the latent variable block is set to 0). Then, the electronic device extracts the latent variable block whose control identifier indicates that it needs to be regenerated according to the second description text in each iteration of the intermediate variable block sequence, and generates a new sequence of the extracted latent variable blocks according to the original order in the intermediate variable block sequence, which is the first subsequence.

[0117] Of course, after the electronic device extracts the latent variable blocks that need to be regenerated from the intermediate variable block sequence, it can also generate a new sequence with these extracted latent variable blocks in a random order, or in an order opposite to the original order in the intermediate variable block sequence, and use the generated sequence as the first subsequence, which is not limited in the embodiment of the present application.

[0118] Step 1032: Generate a second subsequence based on the second semantic feature, the first subsequence, and the position subsequence.

[0119] After obtaining the first subsequence that needs to be finely controlled and the position subsequence corresponding to the first subsequence, the electronic device will combine the second semantic feature and the position subsequence at the same time, inject the second semantic feature into the latent variable blocks in the first subsequence to regenerate these latent variable blocks. The sequence formed by the regenerated latent variable blocks is the second subsequence.

[0120] In some embodiments of the present application, the second semantic feature includes: a multi-level text feature of the second description text, and a single-level text feature of the second description text. In this case, Figure 4 Step 1032 in , i.e., generating the second subsequence based on the second semantic feature, the first subsequence and the position subsequence, can be achieved by the following processing: fusing the multi-level semantic features of the second description text into each latent variable block of the first subsequence, and generating the third subsequence using the fused latent variable blocks; fusing the multi-level text features of the second description text and the single-level text features of the second description text to obtain fused features; and generating the second subsequence based on the third subsequence and the fused features.

[0121] That is to say, the electronic device can first fuse the multi-level text features of the second description text with the latent variable blocks in each iteration of the intermediate variable block sequence, and generate a new variable block sequence by fusing the latent variable blocks with the multi-level text features of the second description text in a random order, or in the order of the latent variable blocks in the intermediate variable block sequence, and the variable block sequence is the third subsequence. Afterwards, the electronic device fuses the multi-level text features of the second description text with the single-level text features of the second description text to obtain the fusion features of the text, and through the attention mechanism, operates the third subsequence, the fusion features and the position subsequence to obtain a new variable block sequence. Afterwards, the obtained new variable block sequence can be subjected to regularization, nonlinear transformation and dimensional transformation, and the processing result can be used as the second subsequence; the obtained new variable block sequence can also be subjected to regularization, nonlinear transformation and dimensional transformation, while the fusion result of the new variable block sequence and the single-level semantic features of the second description text is subjected to regularization, nonlinear transformation and dimensional transformation, and finally the two processing results are merged to obtain the second variable block sequence.

[0122] Similar to the generation process of the intermediate variable block sequence of each iteration, the fusion, attention mechanism, regularization, nonlinear transformation and dimensionality transformation in the above processing can also be implemented through network layers with corresponding functions, such as fusion through the zero initial normalization (LayerNormZero) layer, attention mechanism through the attention layer, regularization processing through the normalization layer (LayerNorm), and nonlinear transformation and dimensionality transformation processing through the feedforward layer (Feedforward).

[0123] In some embodiments of the present application, the second semantic feature includes: a multi-level text feature of the second description text. In this case, the electronic device may first fuse the multi-level text features of the second description text with the latent variable blocks in the first subsequence of each iteration, and then combine the position subsequence to apply an attention mechanism to the new variable block sequence obtained by the fusion, and subsequently perform regularization, nonlinear transformation, dimensionality transformation and other processing, and use the processing result as the second subsequence; the obtained new variable block sequence may also be subjected to regularization, nonlinear transformation, dimensionality transformation and other processing, and the fusion result of the new variable block sequence and the multi-level semantic features of the second description text may be subjected to regularization, nonlinear transformation, dimensionality transformation and other processing, and finally the two processing results may be merged to obtain the second subsequence.

[0124] Step 1033: Determine the i-th output variable block sequence using the second subsequence and the i-th intermediate variable block sequence.

[0125] After obtaining the second subsequence, the electronic device can replace the first subsequence in the intermediate variable block sequence of each iteration with the second subsequence, and directly use the intermediate variable block sequence after the replacement as the output variable block sequence of each iteration. Of course, the electronic device can also take some optimization measures for the intermediate variable block sequence after the replacement to make the final output variable block sequence smoother, so that the picture of the video obtained based on the output variable block sequence is more harmonious and avoids sudden changes in the picture.

[0126] In some embodiments of the present application, Figure 4 Step 1033 in , i.e., determining the i-th output variable block sequence by using the second subsequence and the i-th intermediate variable block sequence, can be achieved by the following processing: using the second subsequence to replace the first subsequence in the i-th intermediate variable block sequence to obtain the i-th variable block sequence to be optimized; smoothing the i-th variable block to be optimized to obtain the i-th output variable block sequence.

[0127] That is to say, after replacing the first subsequence in the intermediate variable block sequence with the second subsequence, the electronic device will smooth the obtained new variable block sequence, that is, the variable block sequence to be optimized, so as to eliminate the inharmoniousness between the second subsequence and other parts in the intermediate variable block sequence through smoothing, and use the variable block sequence after smoothing as the output variable block sequence of each iteration. In this way, it can be ensured that the latent variable blocks of fine control and other content in the obtained output variable block sequence can be smoothly transitioned to avoid sudden picture changes between the finely controlled video segments and other segments in the entire video, further improving the quality of the generated video.

[0128] It should be noted that the electronic device can apply a self-attention mechanism to the variable block sequence to be optimized, that is, read the variable block sequence to be optimized through the self-attention layer to achieve smoothing through the self-attention mechanism. The electronic device can also achieve smoothing by applying linear interpolation to the variable block sequence to be optimized, which is not limited in the embodiments of the present application.

[0129] Step 104: Determine the target video indicated by the first description text and the second description text based on the Nth output variable block sequence and position sequence.

[0130] After the last iteration is completed, the electronic device will perform video decoding based on the output variable block sequence and position sequence of the last iteration. The resulting video is a target video that matches the first description text and the second description text, that is, a target video that meets the user's requirements.

[0131] See also Figure 5 , Figure 5 This is a schematic diagram of the process of the video generation method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, Figure 3 Step 104 in the above, i.e., determining the target video indicated by the first description text and the second description text based on the Nth output variable block sequence and the position sequence, can be implemented by the following processing:

[0132] Step 1041: restore the Nth output variable block sequence to an output latent variable sequence according to the position sequence.

[0133] The electronic device will read the time position and spatial position of each latent variable block from the position sequence, restore the latent variable blocks with the same time position to latent variables according to the spatial position, and sort all the restored latent variables in the order of time position, so that a latent variable sequence can be obtained. In order to distinguish it from the initial latent variable sequence, the obtained latent variable sequence is recorded as the output latent variable sequence.

[0134] Step 1042: decode the output latent variable sequence to obtain a video frame sequence, and determine the target video indicated by the first description text and the second description text based on the video frame sequence.

[0135] The electronic device decodes each latent variable in the output latent variable sequence, that is, removes the noise in the latent variable, so as to obtain the video frame corresponding to each latent variable. After completing the decoding of all latent variables in the output latent variable sequence, the electronic device can obtain a video frame sequence. Afterwards, the electronic device can determine video parameters for the video frame sequence, such as frame rate, resolution, video height and width, etc., and then convert the video frame sequence into a video in combination with the video parameters, and directly determine the obtained video as the target video; or perform processing such as adding audio tracks, clipping, and speed adjustment on the converted video, and use the processed video as the target video.

[0136] It should be noted that the electronic device can complete the above decoding process through a decoder specifically used to decode the latent variables. The decoder can be decoded by a VAE decoder in the DIT, or by a Generative Adversarial Network (GANs), which is not specifically limited in the embodiments of the present application.

[0137] In other embodiments of the present application, Figure 3 Step 104 in the above, i.e., determining the target video indicated by the first description text and the second description text based on the N-th output variable block sequence and the position sequence, can also be achieved by the following processing: decoding each latent variable block in the N-th output variable block sequence to obtain a picture block sequence; converting the picture block sequence into a video frame sequence in combination with the position sequence, and determining the target video indicated by the first description text and the second description text based on the video frame sequence.

[0138] That is to say, the electronic device can also directly decode the latent variable blocks in the output variable block sequence of the last iteration, and then read the time position and spatial position of each latent variable block from the position sequence, and restore the latent variable blocks with the same time position to a video frame according to the spatial position, and sort the obtained video frames according to the time position, so that a video frame sequence can be obtained, and then the target video can be obtained.

[0139] It can be understood that, compared with the related art, there is a lack of association between the latent variable subsequence of the video clip and the description text of the complete video, which ultimately leads to the problem of low quality of the generated video. In the embodiment of the present application, in each iteration, the first semantic feature of the first description text that describes the video globally, the input variable block sequence and the position sequence of each iteration are first used to generate an intermediate variable block sequence to achieve the injection of the first semantic feature. Then, on the basis of the intermediate variable block sequence, combined with the time interval of the video clip and the second semantic feature of the second description text that describes the video clip, the output variable block sequence of each iteration is generated to achieve the injection of the second semantic feature, so that the latent variable block corresponding to the video clip can be associated with the first description text, and the process is continuously repeated through iteration, so that the semantic features of the first description text and the second description text can be fully injected into the latent variable block, so that the transition between the picture of the video clip and the picture of the complete video can be natural, thereby improving the quality of the generated video.

[0140] In some embodiments of the present application, before starting iteration i, the method may further include the following processing: dividing each latent variable in the initial latent variable sequence into multiple latent variable blocks; and generating the first input variable block sequence using the multiple latent variable blocks of each latent variable.

[0141] The electronic device may evenly divide each latent variable in the initial latent variable sequence into blocks to obtain multiple latent variable blocks, or may unevenly divide each latent variable into blocks to obtain multiple latent variable blocks. Afterwards, the electronic device may use the block results of each latent variable, i.e., multiple latent variable blocks, to generate a sequence in a certain order, and the sequence is the input variable block sequence of the first iteration.

[0142] It should be noted that the electronic device can form a subsequence from left to right and from top to bottom for the latent variable blocks in the same latent variable, and then form an input variable block sequence by arranging the subsequences of different latent variables in the time sequence of the video frames corresponding to the latent variables. The electronic device can also arrange all the latent variable blocks in a random order to obtain an input variable block sequence, which is not limited in the embodiments of the present application.

[0143] It should also be noted that, since it takes a certain amount of time and computing resources to complete the processing when determining the output variable block sequence based on the intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence, and the second semantic feature, this step is performed in each iteration, which may affect the efficiency of video generation. In this regard, in an embodiment of the present application, the electronic device can perform the above processing only for some rounds of iterations, but not for other rounds of iterations, that is, directly use the intermediate variable block sequence as the output variable block sequence.

[0144] That is, in some embodiments of the present application, when i=M, after generating the i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence, the method may further include the following processing: determining the M-th intermediate variable block sequence as the M-th output variable block sequence, where M may take any one or more values ​​between 1 and N.

[0145] Thus, when the electronic device is performing the Mth iteration, it can directly skip the processing of the Mth output variable block sequence based on the intermediate variable block sequence of the Mth iteration, the time interval corresponding to the video clip, the position sequence and the second semantic feature, so as to achieve the processing only in part of the N iterations. In this way, the time required for video generation and the required computing resources can be reduced.

[0146] The following describes a specific application scenario of the video generation method according to an embodiment of the present application.

[0147] The video generation method of the embodiment of the present application can be applied to the generation process of a game promotion video. In this case, the first description text can be used to describe the overall style of the game promotion video, and the second description text can be used to describe the display details of the game character in the game promotion video. Thus, the electronic device can determine the first semantic feature from the first description text that describes the overall style of the game promotion video, and determine the second semantic feature from the second description text that describes the display details of the game character; perform the following processing by iteration i, 1≤i≤N, N is a positive integer: generate the i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence, wherein the latent variable block of the first input variable block sequence includes the latent variable block obtained by dividing the latent variables of the initial latent variable sequence, and the position sequence includes the position information of each latent variable block; determine the i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature; when i is iterated to N, determine the target video indicated by the first description text and the second description text based on the N-th output variable block sequence and the position sequence.

[0148] The video generation method of the embodiment of the present application can also be applied to the production of animation videos. In this case, the first description text describes the plot and characters of the entire animation video, and the second description text describes the plot and characters in the storyboard. Therefore, the electronic device can determine the first semantic feature from the first description text that describes the plot and characters of the entire animation video, and determine the second semantic feature from the second description text that describes the plot and characters in the storyboard; perform the following processing by iteration i, 1≤i≤N, N is a positive integer: generate the i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence, wherein the latent variable block of the first input variable block sequence includes the latent variable block obtained by dividing the latent variables of the initial latent variable sequence, and the position sequence includes the position information of each latent variable block; determine the i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature; when i is iterated to N, determine the target video indicated by the first description text and the second description text based on the N-th output variable block sequence and the position sequence.

[0149] Of course, the video generation method of the embodiment of the present application may not be limited to the application scenarios described above.

[0150] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0151] The embodiment of the present application is implemented in a scenario where a short video (a video whose duration is less than a duration threshold) that meets the user's requirements is generated based on the user's overall prompt words and segment prompt words.

[0152] See Figure 6. Fig. 6A This is a short video generation diagram provided in the embodiment of the present application. Figure 1 . When the overall prompt word (first description text) input by the user is "an anime-style video depicting a girl sitting by the window, the wall is covered with leaves, and the leaves are slowly falling", and the fragment prompt word (second description text) is "the girl's hair is fluttering in the wind", a matching short video 6-1 can be generated based on this. Figure 6B This is a short video generation diagram provided in the embodiment of the present application. Figure 2 When the overall prompt word input by the user is "This cartoon-style video shows a scene: four children sitting on the ground, warming themselves around a campfire at night, looking up at the stars in the sky", and the segment prompt word is "the stars gradually fall", a matching short video 6-2 can be generated accordingly.

[0153] Next, the generation of the above-mentioned short video is explained.

[0154] First, see Figure 7 , Figure 7It is a video generation system architecture diagram provided by an embodiment of the present application, and the video generation system architecture 7-1 includes a prompt word encoding part 7-11 and a video generation part 7-12. The prompt word encoding part 7-11 includes an overall prompt word encoding 7-111 and a fragment prompt word encoding 7-112. In the overall prompt word encoding 7-111 and the fragment prompt word encoding 7-112, two encoders are respectively set, such as the text encoder 7-2 of CLIP and the encoder 7-3 of the T5 model, and a text embedding network layer 7-113 is connected behind each encoder to perform dimension conversion on the output of the encoder. The encoded output after dimension conversion will be input into the video generation part 7-12 for processing. The input of the video generation part 7-12 includes a latent variable block sequence (the first input variable block sequence) 7-121, a variable block position vector 7-122 corresponding to the latent variable block sequence, and the 4 encoded outputs above. The latent variable block sequence 7-121 is a sequence formed by a plurality of latent variable blocks 7-14, and the plurality of latent variable blocks 7-14 are obtained by splitting each frame in the latent variable sequence (initial latent variable sequence) 7-13 obtained by initializing the random noise. During the splitting process, the position of each latent variable block in the original latent variable and the time of the latent variable are recorded, and these records are used to generate a vector, which is the variable block position vector 7-122 (position sequence). The video generation part 7-12 can use a transformer 7-4 as the core network, wherein the number of transformers is N. After a series of transformer calculations, the above input can fully integrate the video content to be generated into each latent variable block in the latent variable block sequence to obtain a new latent variable block sequence 7-15 (the Nth output variable block sequence), and then restore the new latent variable block sequence 7-15 to a new latent variable sequence 7-16 (output latent variable sequence), and then decode the obtained new latent variable sequence 7-18 to obtain the desired video (target video). Here, the input of the transformer 7-4 also includes marking information 7-17 for whether each latent variable block in the latent variable block sequence 7-121 needs to be finely controlled.

[0155] As can be seen from the above, in the embodiment of the present application, the overall prompt word encoding and the segment prompt word encoding are used to perform phased feature extraction on the overall prompt word and the segment prompt word input by the user, which not only enables the semantics of the overall prompt word to be better integrated into the entire video, but also allows the segment description words of the video segment that is to be focused on to be injected separately, thereby improving the control ability of video generation. In addition, in order to better extract semantic features, in the embodiment of the present application, two different encoders can be used to perform complementary semantic feature extraction, thereby improving the accuracy of the video to be generated.

[0156] Figure 8 Schematic diagram of the overall prompt word encoding and the segment prompt word encoding provided in the embodiment of the present application. Figure 8 In the overall prompt word encoding 8-1 and the segment prompt word encoding 8-2, the CLIP text encoder 8-3 and the T5 encoder 8-4 can be used simultaneously to encode the overall prompt word, and the CLIP text encoder 8-3 and the T5 encoder 8-4 can be used simultaneously to encode the segment prompt word, thereby obtaining four encoding results 8-5 (K second text features). Next, the electronic device will use the text embedding layer 8-6 (which can be implemented as a multi-layer perceptron) to align the dimensions of the four obtained encoding results (the alignment results are the first semantic features and the second semantic features), so that the dimensions of the four encoding results can meet the dimensional requirements of the input of the subsequent video generation part.

[0157] It should be noted that the text encoder of CLIP is composed of 12 layers of Transformers. In the general encoding process, only the output of the last layer of Transformer is used as the encoding result. In the embodiment of the present application, the output of the last 6 layers of Transformer (multiple levels of first text sub-features and multiple levels of second text sub-features) are used to form the encoding result (multi-level text features). This operation is to allow the encoding result to have different levels of semantics, so that the encoding result can simultaneously retain the description of the overall picture at the high level and the description of the details at the bottom level.

[0158] The T5 encoder is a multi-layer encoder formed by self-attention and feed-forward neural networks. In the embodiment of the present application, the T5 encoder is added to increase the understanding of the overall prompt words and the segment prompt words, so that the obtained semantics can be supplemented to the semantics encoded by the text encoder of CLIP to improve the accuracy of the final generated video.

[0159] Fig. 9 This is a model architecture diagram of the video generation part provided in the embodiment of the present application. Fig. 9 The video generation part of the embodiment of the present application can be divided into three parts, namely, latent variable block sequence generation 9-11, video generation model 9-12 and decoding calculation 9-13.

[0160] Among them, in the latent variable block sequence generation 9-11, the latent variable sequence initialized by random noise (initial latent variable sequence) 9-111 will be disassembled into a latent variable block sequence (the first input variable block sequence) 9-112, and the position of the latent variable block will be used to generate a variable block position vector (position sequence) 9-113, and the latent variable block sequence 9-112 and the variable block position vector 9-113 will be used as the input of the video generation model 9-12 for subsequent calculations. Fig. 9 It can be seen that each frame in the latent variable sequence 9-111 will be split into a series of latent variable blocks, and these latent variable blocks are stretched to form a latent variable block sequence 9-112. The variable block position vector 9-113 is provided to the video generation model 9-12 so that the video generation model 9-12 can perceive the time position and spatial position of each latent variable block in the latent variable sequence. At the same time, after the video generation model 9-12 completes the calculation, it is also necessary to combine the variable block position vector 9-113 to reassemble the latent variable block sequence (the Nth output variable block sequence) 9-121 output by the video generation model 9-12 into a new latent variable sequence 9-131, so as to facilitate the decoding of the video frame 9-132 to obtain the video. It should be noted that the input of the video generation model 9-12 also includes the extracted semantic features 9-2, and the marking information 9-3 of whether fine control is required for each latent variable block in the latent variable block sequence 9-112.

[0161] The video generation model 9-12 is composed of multiple Transformers. The number of Transformers is N, and the value of N can be set freely. Fig.10 This is a schematic diagram of the structure of the video generation model provided in the embodiment of the present application. Figure 1 See also Fig.10 The input of the video generation model has three parts, namely, a latent variable block sequence 10-1, a variable block position vector 10-2, and semantic features extracted from the overall prompt word and the segment prompt word. The semantic features here include the encoding result 10-3 of the overall prompt word by the text encoder of CLIP, the encoding result 10-4 of the overall prompt word by the encoder of T5, the encoding result 10-5 of the segment prompt word by the text encoder of CLIP, and the encoding result 10-6 of the segment prompt word by T5. Among them, the video generation model includes a converter 10-7 for semantic injection into the overall video, and a converter 10-8 for semantic injection into the video segment. Among them, the structures of converter 10-7 and converter 10-8 are similar, and the structure is described below only by taking converter 10-7 as an example.

[0162] In converter 10-7, the encoding result 10-3 and the latent variable block sequence 10-1 are first fused through a zero-initial normalization layer 10-71 (the purpose is to fuse the semantics extracted by the latent variable block and the text encoder of CLIP), and the encoding result 10-3 and the encoding result 10-4 are fused through a zero-initial normalization layer 10-72 (the purpose is to fuse the semantics of the two encoding results). After the fusion is completed, the fusion result is input into the attention layer 10-73 for cross-attention calculation. It should be noted that the input of the attention layer 10-73 also includes the variable block position vector 10-2, so that the converter 10-7 can know the time position and spatial position of the latent variable block. After the attention calculation is completed, the obtained calculation result 10-74 can be divided into two paths for calculation, one of which is to directly input the calculation result 10-74 to the normalization layer 10-75 and the feedforward layer 10-76 for calculation (a short circuit connection will be performed during this period), and the other path first fuses the calculation result 10-74 with the encoding result 10-4, and then inputs the fused result to the normalization layer 10-75 and the feedforward layer 10-76 for calculation (a short circuit connection will also be performed during this period). Finally, the calculation results of these two paths are merged 10-77 to complete the calculation of the converter 10-7 and obtain a new latent variable block sequence 10-9.

[0163] Afterwards, the electronic device will determine the subsequence 10-11 (first subsequence) from the latent variable block sequence 10-9 (intermediate variable block sequence) according to the time range (time interval) 10-10 that needs to be finely controlled and the variable block position vector 10-2. Afterwards, the subvector 10-12 corresponding to the subsequence 10-11 is determined from the variable block position vector 10-2, and the subsequence 10-11, the subvector 10-12, the encoding result 10-5 and the encoding result 10-6 are used as the input of the converter 10-8 for calculation. After the converter 10-8 completes the calculation, a new subsequence 10-13 (second subsequence) is obtained, and the subsequence 10-11 in the latent variable block sequence 10-9 is replaced with the subsequence 10-13, and the replaced latent variable block sequence 10-14 (variable block sequence to be optimized) is input into the self-attention layer 10-15, so that the subsequence 10-13 in the latent variable block sequence 10-14 can be harmonized with other sequences. Finally, the output of the self-attention layer 10-15 is input into the next converter for new fusion. In this way, after step-by-step calculation, the latent variable sequence finally generated is completely harmonized, so that there will be no sudden picture changes in the decoded video frame. Among them, similar to the converter 10-7, the converter 10-8 includes a zero-initial normalization layer 10-81, a zero-initial normalization layer 10-82, an attention layer 10-83, two processing paths of the calculation results 10-84 composed of a normalization layer 10-85 and a feedforward layer 10-86, and merging the calculation results 10-87.

[0164] When determining a subsequence from a latent variable block sequence within a time range and a variable block position vector, the electronic device generates corresponding marking information (control identifier) ​​for each latent variable block in the latent variable block sequence based on the time range and the variable block position vector, where 1 indicates that the latent variable block needs fine control, i.e., regeneration, and 0 indicates that the latent variable block does not need fine control.

[0165] Fig.11 It is a schematic diagram of generating the marking information provided by the embodiment of the present application. If each latent variable in the latent variable sequence 11-1 can be divided into 16 latent variable blocks, if the second frame to the fourth frame of the video, that is, the second latent variable to the fourth latent variable, need to be finely controlled, then these three frames should be extracted as the subsequence 11-2. At this time, the electronic device will set the marking information corresponding to the latent variable blocks of these three frames in the latent variable block sequence to 1, that is, the 17th point position to the 64th point position is set to 1, and the marking information corresponding to the other latent variable blocks is set to 0.

[0166] After multiple Transformer calculations, a semantic latent variable block sequence that incorporates the overall prompt word and the segment prompt word is obtained. The electronic device merges it in the form of latent variables to obtain a latent variable sequence. Finally, the electronic device inputs each latent variable in the latent variable sequence into the VAE decoder for decoding to obtain a video frame sequence, and then obtains the corresponding video (target video).

[0167] It should be noted that although the above solution can improve the quality of the video, it also increases the amount of computation and memory required for video generation. For example, when the length of the latent variable block sequence is 128 and the length of the subsequence that needs to be finely controlled is 64, then Fig.10 In order to reduce memory consumption and computational complexity, in some embodiments, some layers of the N-layer Transformer can be selected to process the segment prompt words, that is, Fig.10 Processing in converter 10-8.

[0168] That is, in some embodiments, the processing in the converter 10 - 7 may be performed only on the latent variable block sequence. Fig.12 This is a schematic diagram of the structure of the video generation model provided in the embodiment of the present application. Figure 2 For the Mth converter among N converters (M can be one or more values ​​from 1 to N), Fig.10 The output of the converter 10-7 in the self-attention layer 12-1 is directly processed, and the processing result is input into the next converter for calculation. In this way, the amount of calculation required for memory consumption can be effectively reduced.

[0169] It can be understood that in the embodiments of the present application, user information, such as the first description text, the second description text and other related data, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of the relevant data need to comply with relevant laws, regulations and standards.

[0170] The following is a description of an exemplary structure of the video generation device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the video generation device 255 of the memory 250 may include:

[0171] A feature determination module 2551 is used to determine a first semantic feature from a first description text describing the entire video, and to determine a second semantic feature from a second description text describing the video segment;

[0172] A sequence determination module 2552 is used to generate an i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence; when i=1, the i-th input variable block sequence includes the latent variable blocks obtained by dividing the latent variables of the initial latent variable sequence into blocks; when i>1, the i-th input variable block sequence includes the i-1-th output variable block sequence, and the position sequence includes the position information of each of the latent variable blocks; based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature, determine the i-th output variable block sequence;

[0173] The video determination module 2553 is used to determine the target video indicated by the first description text and the second description text based on the Nth output variable block sequence and the position sequence.

[0174] In the above scheme, the sequence determination module 2552 is also used to determine a first subsequence from the i-th intermediate variable block sequence based on the time interval and the position sequence, and determine a position subsequence corresponding to the first subsequence from the position sequence; generate a second subsequence based on the second semantic feature, the first subsequence and the position subsequence; and determine the i-th output variable block sequence using the second subsequence and the i-th intermediate variable block sequence.

[0175] In the above scheme, the second semantic features include: multi-level text features of the second description text, and single-level text features of the second description text; the sequence determination module 2552 is also used to fuse the multi-level semantic features of the second description text into each latent variable block of the first subsequence, and generate a third subsequence using the fused latent variable blocks; fuse the multi-level text features of the second description text and the single-level text features of the second description text to obtain fused features; and generate the second subsequence based on the third subsequence and the fused features.

[0176] In the above scheme, the sequence determination module 2552 is also used to determine a control identifier for the latent variable block in the i-th intermediate variable block sequence based on the time interval and the time position in the position sequence, wherein the control identifier indicates whether the latent variable block needs to be regenerated according to the second description text; from the i-th intermediate variable block sequence, the latent variable block that needs to be regenerated according to the second description text represented by the control identifier is extracted, and the sequence generated by using the extracted latent variable block is determined as the first subsequence.

[0177] In the above scheme, the sequence determination module 2552 is also used to use the second subsequence to replace the first subsequence in the i-th intermediate variable block sequence to obtain the i-th variable block sequence to be optimized; and smooth the i-th variable block to be optimized to obtain the i-th output variable block sequence.

[0178] In the above scheme, the feature determination module 2551 is also used to extract K second text features from the second description text; wherein K is a positive integer; and determine the second semantic feature based on the K second text features.

[0179] In the above scheme, the K second text features include: multi-level text features of the second description text; the feature determination module 2551 is also used to encode the second description text to obtain second encoded features; from the second encoded features, multiple levels of second text sub-features are extracted, and the multi-level text features of the second description text are determined using the multiple levels of the second text sub-features.

[0180] In the above scheme, the video generation module 255 also includes: a variable block module 2554, which is used to divide each latent variable in the initial latent variable sequence into multiple latent variable blocks; and generate the first input variable block sequence using the multiple latent variable blocks of each latent variable.

[0181] In the above scheme, the video determination module 2553 is also used to restore the Nth output variable block sequence to an output latent variable sequence based on the position sequence; decode the output latent variable sequence to obtain a video frame sequence, and based on the video frame sequence, determine the target video jointly indicated by the first description text and the second description text.

[0182] In the above solution, i=M, and the sequence determination module 2552 is further used to determine the Mth intermediate variable block sequence as the Mth output variable block sequence, wherein M can take any one or more values ​​between 1 and N.

[0183] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the video generation method described in the embodiment of the present application.

[0184] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the video generation method provided by the present application embodiment, for example, Figure 3 The video generation method is shown.

[0185] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0186] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0187] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0188] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0189] In summary, through the embodiments of the present application, in each iteration, the first semantic feature of the first description text that describes the video globally, the input variable block sequence and the position sequence of each iteration are first used to generate an intermediate variable block sequence to realize the injection of the first semantic feature, and then, on the basis of the intermediate variable block sequence, combined with the time interval of the video clip and the second semantic feature of the second description text that describes the video clip, the output variable block sequence of each iteration is produced to realize the injection of the second semantic feature, so that the latent variable block corresponding to the video clip can be associated with the first description text, and the process is continuously repeated through iteration, so that the semantic features of the first description text and the second description text can be fully injected into the latent variable block, so that the transition between the picture of the video clip and the picture of the complete video can be natural, thereby improving the quality of the generated video; the second semantic feature can also be injected only in part of the iterations in the N iterations, so that the time required for video generation and the required computing resources can be reduced.

[0190] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Determine a first semantic feature from a first description text that describes the entire video, and determine a second semantic feature from a second description text that describes the video segment; The following processing is performed by iteration i, 1≤i≤N, N is a positive integer: Generate an i-th intermediate variable block sequence using the i-th input variable block sequence, the first semantic feature and the position sequence; when i=1, the i-th input variable block sequence includes latent variable blocks obtained by dividing the latent variables of the initial latent variable sequence into blocks; when i>1, the i-th input variable block sequence includes the i-1-th output variable block sequence, and the position sequence includes the position information of each of the latent variable blocks; Determine an i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature; Based on the Nth output variable block sequence and the position sequence, a target video indicated by the first description text and the second description text is determined.

2. The method according to claim 1, characterized in that The step of determining the i-th output variable block sequence based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature comprises: Determine a first subsequence from the i-th intermediate variable block sequence according to the time interval and the position sequence, and determine a position subsequence corresponding to the first subsequence from the position sequence; Generate a second subsequence based on the second semantic feature, the first subsequence and the position subsequence; The i-th output variable block sequence is determined by using the second subsequence and the i-th intermediate variable block sequence.

3. The method according to claim 2, characterized in that The second semantic features include: multi-level text features of the second description text, and single-level text features of the second description text; The generating a second subsequence based on the second semantic feature, the first subsequence and the position subsequence comprises: fusing the multi-level semantic features of the second description text into each latent variable block of the first subsequence, and generating a third subsequence using the fused latent variable blocks; fusing the multi-level text features of the second description text and the single-level text features of the second description text to obtain a fused feature; The second subsequence is generated based on the third subsequence and the fused feature.

4. The method according to claim 2, characterized in that: The determining of the first subsequence from the i-th intermediate variable block sequence according to the time interval and the position sequence comprises: Determining a control flag for the latent variable block in the ith intermediate variable block sequence according to the time interval and the time position in the position sequence, wherein the control flag indicates whether the latent variable block needs to be regenerated according to the second description text; From the i-th intermediate variable block sequence, the latent variable block representing the control mark that needs to be regenerated according to the second description text is extracted, and a sequence generated by using the extracted latent variable block is determined as the first subsequence.

5. The method according to claim 2, characterized in that: The step of using the second subsequence and the i-th intermediate variable block sequence to determine the i-th output variable block sequence comprises: Using the second subsequence, replacing the first subsequence in the i-th intermediate variable block sequence, to obtain the i-th variable block sequence to be optimized; The i-th variable block to be optimized is smoothed to obtain the i-th output variable block sequence.

6. The method according to any one of claims 1 to 5, characterized in that: The determining of the second semantic feature from the second description text describing the video clip includes: Extracting K second text features from the second description text, wherein K is a positive integer; The second semantic feature is determined based on the K second text features.

7. The method according to claim 6, characterized in that The K second text features include: multi-level text features of the second description text; the K second text features extracted from the second description text include: Encoding the second description text to obtain a second encoding feature; A plurality of levels of second text sub-features are extracted from the second coding features, and the multi-level text features of the second description text are determined by using the plurality of levels of the second text sub-features.

8. The method according to any one of claims 1 to 5, characterized in that: Before starting iteration of i, the method further includes: Dividing each latent variable in the initial latent variable sequence into a plurality of latent variable blocks; The first input variable block sequence is generated by using the multiple latent variable blocks of each latent variable.

9. The method according to any one of claims 1 to 5, characterized in that: The determining, based on the Nth output variable block sequence and the position sequence, a target video indicated by the first description text and the second description text, comprises: According to the position sequence, the Nth output variable block sequence is restored to an output latent variable sequence; The output latent variable sequence is decoded to obtain a video frame sequence, and based on the video frame sequence, the target video indicated by the first description text and the second description text is determined.

10. The method according to any one of claims 1 to 5, characterized in that: i=M, after generating the i-th intermediate variable block sequence by using the i-th input variable block sequence, the first semantic feature and the position sequence, the method further includes: The Mth intermediate variable block sequence is determined as the Mth output variable block sequence, wherein M can take any one or more values ​​between 1 and N.

11. A video generating device, characterized in that: The device comprises: A feature determination module, used to determine a first semantic feature from a first description text describing the entire video, and to determine a second semantic feature from a second description text describing the video segment; A sequence determination module, for generating an i-th intermediate variable block sequence using an i-th input variable block sequence, the first semantic feature and a position sequence; when i=1, the i-th input variable block sequence includes a latent variable block obtained by dividing the latent variables of the initial latent variable sequence into blocks; when i>1, the i-th input variable block sequence includes an i-1-th output variable block sequence, and the position sequence includes position information of each of the latent variable blocks; based on the i-th intermediate variable block sequence, the time interval corresponding to the video clip, the position sequence and the second semantic feature, determining the i-th output variable block sequence; A video determination module is used to determine a target video indicated by the first description text and the second description text based on the Nth output variable block sequence and the position sequence.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 10 when executing computer executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images