Video generation method, device, electronic device, storage medium and program product

Abstract

The present application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating a video, including: determining an acquired initial image as the first frame image in a frame image sequence to be generated, and obtaining description text of each frame image in the frame image sequence; generating the i-th frame image based on the description text of the i-th frame image in the frame image sequence, each frame image in the frame image sequence that precedes the i-th frame image, and each frame image that precedes the i-th frame image; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence; traversing i to obtain each frame image in the frame image sequence; performing video synthesis on each frame image in the frame image sequence to obtain a target video. Through the present application, the accuracy of frame images in a video and the consistency between adjacent frame images can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for generating a video. Background Art

[0002] In film and television drama scenarios, when users upload an image, they often hope to use video technology to animate the image into a film and television style. That is, to make the image move and the movement performance similar to the film and television style, such as a short video showing the changes in character expressions, landscape switching, etc.

[0003] Related text-to-image technology is effective in generating cartoons, artistic landscapes, etc., but it is not effective in generating images related to film and television scenes and characters. Or, due to the influence of random noise, the output image and the input image have poor consistency in the scene, and the accuracy of the image content is poor. Summary of the Invention

[0004] Embodiments of the present application provide a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of frame images in a video and the consistency between adjacent frame images.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a method for generating a video, including:

[0007] Determining the obtained initial image as the first frame image in the frame image sequence to be generated, and obtaining description text of each frame image in the frame image sequence;

[0008] generating the i-th frame image based on the description text of the i-th frame image in the frame image sequence, the description texts of the frame images preceding the i-th frame image in the frame image sequence, and the description texts of the frame images preceding the i-th frame image;

[0009] Wherein, i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence;

[0010] Traversing the i to obtain each frame image in the frame image sequence;

[0011] Perform video synthesis on each frame image in the frame image sequence to obtain a target video.

[0012] The present invention provides a video generation device, including:

[0013] an acquisition module, configured to determine the acquired initial image as the first frame image in a frame image sequence to be generated, and to acquire description text of each frame image in the frame image sequence;

[0014] a generating module, configured to generate the i-th frame image based on a description text of the i-th frame image in the frame image sequence, each frame image in the frame image sequence that is located before the i-th frame image, and each frame image that is located before the i-th frame image; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence;

[0015] A traversal module, configured to traverse the i to obtain each frame image in the frame image sequence;

[0016] The synthesis module is used to perform video synthesis on each frame image in the frame image sequence to obtain a target video.

[0017] An embodiment of the present application provides an electronic device, including:

[0018] a memory for storing executable instructions;

[0019] The processor is used to implement the video generation method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium having computer-executable instructions stored therein. When the computer-executable instructions are executed by a processor, the processor will execute the video generation method provided in the embodiment of the present application.

[0021] The present invention provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video generation method provided in the present invention.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] In the embodiment of the present application, the acquired initial image is used as the first frame image of the frame image sequence to be generated. When generating each frame image in the frame image sequence to be generated, the frame images before the frame image, the descriptive text associated with each frame image before the frame image, and the descriptive text of the frame image itself are fully referenced. Then, based on each frame image in the frame image sequence, the target video is generated. This not only improves the accuracy of each frame image in the video, but also ensures the consistency between adjacent frame images. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a schematic diagram of the architecture of a video generation system 100 provided in an embodiment of the present application;

[0025] Figure 2 1 is a schematic structural diagram of an electronic device 500 for implementing a method for generating a video according to an embodiment of the present application;

[0026] Figure 3 Schematic diagram of the process of generating a video provided by an embodiment of the present application;

[0027] Figure 4 Schematic diagram of the process of generating the i-th frame image provided in an embodiment of the present application;

[0028] Figure 5 Schematic diagram of a method for determining text constraint information provided in an embodiment of the present application;

[0029] Figure 6 1 is a schematic diagram of a text feature extraction process based on the CLIP model provided in an embodiment of the present application;

[0030] Figure 7 This is a schematic diagram of a method for determining graphic and text constraint information provided in an embodiment of the present application;

[0031] Figure 8 This is a schematic diagram of the image and text feature extraction process provided by an embodiment of the present application;

[0032] Figure 9 This is a flow chart of the frame image generation process provided by an embodiment of the present application;

[0033] Figure 10 is a diagram of the i-th frame image generation process provided by an embodiment of the present application;

[0034] Figure 11 This is a frame image generation method based on a machine learning model provided in an embodiment of the present application;

[0035] Figure 12 This is a flow chart of a serial frame image generation model provided in an embodiment of the present application;

[0036] Figure 13 Schematic diagram of a training method for a frame image generation model provided in an embodiment of the present application;

[0037] Figure 14 This is a schematic diagram of the video sample processing flow provided by an embodiment of the present application;

[0038] Figure 15 This is a flow chart of a clustering process for video samples provided in an embodiment of the present application;

[0039] Figure 16Schematic diagram of the process for determining the distance threshold provided in an embodiment of the present application;

[0040] Figure 17 This is another flowchart of the method for generating a video provided in an embodiment of the present application;

[0041] Figure 18A This is a schematic diagram summarizing the flow of the method for generating a video provided in an embodiment of the present application;

[0042] Figure 18B This is a visualization diagram of multi-frame image generation provided by an embodiment of the present application;

[0043] Figure 19 Schematic diagram of the structure of the continuous frame generation model provided in the embodiment of the present application;

[0044] Figure 20 This is a schematic diagram of a standard diffusion model provided in an embodiment of the present application;

[0045] Figure 21A This is a schematic diagram of the U-Net process without adding constraint information provided in an embodiment of the present application;

[0046] Figure 21B This is a schematic diagram of a U-Net with added constraint information provided in an embodiment of the present application;

[0047] Figure 22 This is a structural diagram of a BLIP text extraction model provided in an embodiment of the present application;

[0048] Figure 23 This is a diagram of the training data collection process provided by an embodiment of the present application;

[0049] Figure 24 This is a schematic diagram of the storyboard operation for a video sample provided in an embodiment of the present application;

[0050] Figure 25 This is a schematic diagram of the reasoning process of continuous frame images provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0055] It should be pointed out that in the embodiments of the present application, data related to the attributes of user images is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Image recognition: This involves category-level recognition, regardless of the specific instance of the object. This task considers only the object's category (e.g., person, dog, cat, bird, etc.) and identifies the object's category. A typical example is the recognition task in ImageNet, a large open-source dataset for general object recognition, which involves identifying which of 1,000 categories an object belongs to.

[0058] 2) Imagenet: A large-scale open-source dataset for general object recognition.

[0059] 3) Imagenet pre-trained model: A deep learning network model is trained based on Imagenet, and the parameter weights of the model are obtained as the Imagenet pre-trained model.

[0060] 4) Text-based images: The model takes text as input and outputs related images.

[0061] 5) Image-to-image: Taking an image as input, the model outputs images related to it.

[0062] 6) Movie and TV Series Stock Images: This collects all movie and TV series videos. Since the amount of data generated after extracting frames from movie and TV series videos is enormous, data compression is necessary to avoid lengthy retrieval times. Therefore, the videos are storyboarded, and one frame of image is extracted from each storyboard and stored in the stock.

[0063] 7) Constrastive Language-Image Pretraining (CLIP) Model: The input of the CLIP model is a sample pair consisting of images and text. The architecture is divided into two parts: an image encoder and a text encoder. The text branch of CLIP: for each novel sentence, it is encoded by a text encoder (specifically, transform) to obtain N text feature vectors (T1, T2, T3, ..., TN); the image branch of CLIP: for each movie and TV series stock image, it is encoded by an image encoder (specifically, resnet) to obtain N image feature vectors (I1, I2, I3, ..., IN);

[0064] When it is necessary to compare the similarity between images and texts, a feature vector matrix is ​​constructed based on the text feature vector and the image feature vector. Since the diagonal part of the feature vector matrix is ​​the image and text matching, the similarity is the largest, so the inner product of the two eigenvectors on the diagonal of the matrix can be calculated. The larger the inner product, the more similar they are.

[0065] 8) Diffusion Model: A generative image model that learns the information attenuation caused by noise and then uses the learned pattern to generate images. This concept also applies to latent variables because it attempts to learn the noise distribution rather than the data distribution. The noise distribution is modeled using the concept of Markov chains, making it a probabilistic model. The diffusion model consists of two stages: the forward process from the original image to the noise, the diffusion process, and the reverse process from the noise to the original image. Diffusion Generative Model: A generative model that converts a complex true distribution into a simple prior distribution by gradually adding noise, and generates the true distribution from the simple prior distribution by gradually removing noise.

[0066] 9) The Vision Transformer (ViT) model is a model for image classification that uses a Transformer-like structure on image patches. The input image is segmented into fixed-size patches, each of which is then linearly embedded. Position embeddings (vector representations) are added, and the resulting vector sequence is fed into a standard Transformer encoder for learning. For classification, an additional learnable "classification token" is added to the sequence.

[0067] Based on the above explanations of the nouns and terms involved in the embodiments of the present application, the video generation system provided by the embodiments of the present application is described below. Figure 1 , Figure 1 This is an architectural diagram of a video generation system 100 provided in an embodiment of the present application. To support an exemplary application, terminals (terminal 400-1 and terminal 400-2 are shown as examples) are connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using wireless or wired links.

[0068] In some embodiments, a terminal (such as terminal 400-1 and terminal 400-2) is deployed with a target application that can realize the video generation function. The target application can generate a target video with a target style (such as film and television style, animation style, etc.) based on the input initial image and the descriptive text of each frame image in the video to be generated. It is used to send a video generation request for the target video to the server based on the target application, wherein the video generation request carries the initial image as the first frame image in the frame image sequence to be generated, and the descriptive text for the second frame image in the frame image sequence to be generated. The terminal is also used to receive the target video returned by the server and perform a target operation on the target video, and the target operation includes at least one of the following: play, download, share, etc.

[0069] In some embodiments, the server 200 is used to receive a video generation request for a target video sent by a terminal, parse the video generation request, and use the obtained initial image as the first frame image in the frame image sequence to be generated. At the same time, the server 200 parses the request to obtain a description text for the second frame image in the frame image sequence to be generated. Then, based on the description text of the i-th frame image in the frame image sequence, the frame images before the i-th frame image in the frame image sequence, and the description text of the frame images before the i-th frame image, the i-th frame image is generated; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence; i is traversed to obtain each frame image in the frame image sequence; and video synthesis is performed on each frame image in the frame image sequence to obtain the target video.

[0070] In actual applications, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal (such as terminal 400-1 and terminal 400-2) can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited to this. The terminal (such as terminal 400-1 and terminal 400-2) and the server 200 can be directly or indirectly connected via wired or wireless communication, and this application is not limited thereto.

[0071] The embodiments of the present application can also be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to realize data calculation, storage, processing, and sharing.

[0072] Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems require a large amount of computing and storage resources.

[0073] Next, the electronic device implementing the video generation method provided by the embodiment of the present application is described. Figure 2 , Figure 2 Schematic diagram of the structure of an electronic device 500 for implementing a method for generating a video according to an embodiment of the present application. Figure 1 The server 200 or terminal shown is an electronic device 500. Figure 1 Taking the server shown as an example, an electronic device for implementing the method for generating a video according to an embodiment of the present application is described. The electronic device 500 provided by the embodiment of the present application includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0074] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0075] In some embodiments, the user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0076] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 510.

[0077] The memory 550 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0078] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0079] The operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and handle hardware-based tasks; a network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB); a presentation module 553 is used to enable information to be presented (for example, a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 associated with the user interface 530 (for example, a display screen, a speaker, etc.); an input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0080] In some embodiments, the video generation device provided in the embodiments of the present application can be implemented in software. Figure 2 A video generation device 555 stored in a memory 550 is shown, which can be software in the form of a program and plug-in, etc., including the following software modules: an acquisition module 5551, a generation module 5552, a traversal module 5553 and a synthesis module 5554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0081] In other embodiments, the video generation device provided in the embodiments of the present application can be implemented by a combination of software and hardware. As an example, the video generation device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video generation method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0082] In some embodiments, the terminal or server can implement the video generation method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP, a web browser APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. For example, Figure 1 The target application deployed in the terminals shown in the figure (such as terminal 400-1 and terminal 400-2) can be any form of application, module or plug-in mentioned above. The target application can realize the video generation function. Specifically, the terminal can receive the initial image and description text uploaded based on the human-computer interaction interface provided by the target application, and output the generated target video.

[0083] Based on the above description of the video generation system and electronic device provided by the embodiment of the present application, the video generation method provided by the embodiment of the present application is described below. In actual implementation, the video generation method provided by the embodiment of the present application can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration, so that Figure 1 The server 200 in the embodiment of the present application alone performs the video generation method provided by the embodiment of the present application as an example for explanation. Figure 3 , Figure 3 This is a flow chart of the method for generating a video provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0084] In step 101 , the server determines the acquired initial image as the first frame image in a frame image sequence to be generated, and acquires description text of each frame image in the frame image sequence.

[0085] In actual implementation, the server receives a video generation request for the video to be generated, parses the video generation request, obtains the initial image, and a description text for each frame image of the video to be generated. The video to be generated can be represented by a frame image sequence to be generated comprising N consecutive frame images, wherein N is a positive integer ≥ 2, and N can be carried in the video generation request or read by the server from a pre-set configuration file. In addition, the initial image carried in the video generation request serves as the first frame image in the frame image sequence to be generated. In order to ensure consistency between the frame images in the frame image sequence to be generated, the video generation request can usually also carry a description text for each frame image in the frame image sequence to be generated. The description text associated with the frame image is used to characterize the image content of the frame image to be generated. The content of the description text can be empty.

[0086] In step 102, an i-th frame image is generated based on the description text of the i-th frame image in the frame image sequence, the frame images before the i-th frame image in the frame image sequence, and the description text of the frame images before the i-th frame image, where i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence.

[0087] In actual implementation, when the server generates the i-th (i is a positive integer ≥ 2) frame image in a frame image sequence, in order to maintain content consistency with the frame image before the i-th frame image, the frame images before the i-th frame image and the description text of the frame images before the i-th frame image can be used as constraint information when generating the i-th frame image, which is used to constrain the generation process of the i-th frame image, so that the generated i-th frame image is consistent with the content of the frame image before the i-th frame image. The content consistency can include the consistency of the characters, background and scene in the image. The judgment of content consistency is usually to calculate the similarity between the i-th frame image and the (i-1)-th frame image for a specific content type (character, background, scene, etc.). When the similarity reaches a preset similarity threshold, the i-th frame image is considered to be consistent with the (i-1)-th frame image.

[0088] The generation of the i-th frame image is described. In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the generation process of the i-th frame image provided in the embodiment of the present application, based on Figure 3 , step 102 can be implemented by steps 1021-1024, combined Figure 4 The steps shown are explained.

[0089] Step 1021 : The server generates text constraint information for the i-th frame image based on the description text of the i-th frame image in the frame image sequence and the description texts of the frame images before the i-th frame image.

[0090] In actual implementation, the constraint information constraining the generation process of the i-th frame image may include at least text constraint information, image-text constraint information, and image constraint information. The text constraint information is purely text-modal constraint information, the image constraint information is purely image-modal constraint information, and the image-text constraint information is cross-modal constraint information. The server may extract features from the descriptive text of the i-th frame image and the descriptive text of each frame image preceding the i-th frame image, and generate text constraint information based on the obtained text features.

[0091] In some embodiments, the text constraint information determination process can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1 The server 200 in the embodiment performs the above text constraint information determination process alone as an example for explanation, see Figure 5 , Figure 5 This is a schematic diagram of a method for determining text constraint information provided by an embodiment of the present application, combined with Figure 5 The steps shown are explained.

[0092] In step 201 , the server extracts features from the description text of the i-th frame image and the description texts of each frame image before the i-th frame image in the frame image sequence to obtain the description text features of each frame image.

[0093] In actual implementation, the server can implement a machine learning model for text feature extraction (such as CLIP) to extract text features from the description text of each frame. The CLIP model includes an image encoder and a text encoder. Through the text branch of CLIP, the description text of each frame is encoded by a text encoder (specifically, a transform) to obtain the corresponding text features of each description text (represented by vectors during calculation).

[0094] Step 202 : Acquire identification features for indicating text modality, fuse the identification features with descriptive text features, and use the fusion result as text constraint information of the i-th frame image.

[0095] For example, see Figure 6 , Figure 6 This is a schematic diagram of the text feature extraction process based on the CLIP model provided in an embodiment of the present application. For all descriptive texts associated with the frame images before the i-th frame image, text features are extracted through the CLIP text branch to obtain a 2x1x768 three-dimensional matrix, which is then fused with the flag bit (2x1x768) representing the pure text mode to obtain the text constraint information of the i-th frame image.

[0096] Step 1022 : Generate image-text constraint information for the i-th image frame based on the image frames preceding the i-th image frame and the description texts of the image frames. Both the text constraint information and the image-text constraint information are used to constrain the image content of the i-th image frame.

[0097] In actual implementation, the server can extract image features from each frame image located before the i-th frame image to obtain corresponding image features. At the same time, it can extract text features from the description texts of each frame image located before the i-th frame image to obtain corresponding text features. Then, through the fusion operation of text features and image features, the graphic and text constraint information of the i-th frame image is obtained.

[0098] In some embodiments, the process of determining the image and text constraint information can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1 The server 200 in the embodiment performs the above-mentioned determination process of the image and text constraint information separately as an example, see Figure 7 , Figure 7 This is a schematic diagram of a method for determining graphic and text constraint information provided in an embodiment of the present application, combined with Figure 7 The steps shown are explained.

[0099] In step 301 , the server extracts image features from each frame of image preceding the i-th frame of image to obtain image features of each frame of image.

[0100] In actual implementation, the server can be used to implement a machine learning model for cross-modal feature extraction (such as BLIP). On the one hand, the image features of each frame image before the i-th frame image are extracted. On the other hand, the text features of the descriptive text associated with each frame image before the i-th frame image are extracted using the image features as constraint information to obtain the graphic and text features of each frame image.

[0101] For example, see Figure 8 , Figure 8 This is a schematic diagram of the image and text feature extraction process provided by an embodiment of the present application. In the figure, the BLIP model is used to extract image and text features. In the specific implementation process, each frame image before the i-th frame image is input and the visual features, i.e., image features, are extracted through the ViT model to obtain the corresponding image features.

[0102] Step 302 : Using the image features of each frame as image constraints, extract text features from the description text associated with each frame before the i-th frame to obtain initial image and text features of each frame.

[0103] Among them, the initial image features are text features that carry image features.

[0104] Continuing from the above example, the description text associated with the frame image is input into BLIP, and the image features of the frame image extracted by ViT are used as constraint information to obtain text features carrying image constraints.

[0105] Step 303: Acquire identification features for indicating the image-text modality, fuse the identification features with the initial image-text features, and use the fusion result as the image-text constraint information of the i-th frame image.

[0106] Continuing with the above example, to facilitate calculations, the server also obtains an identification feature for indicating the image-text modality, and fuses the identification feature with the text feature carrying the image constraint to obtain the initial image-text feature, and uses the initial image-text feature as the image-text constraint information for the i-th frame image. The size of the text feature carrying the image constraint (1×3×768) is the same as the size of the flag bit of the image-text modality (1×3×768). In addition, the frame images in the frame image sequence have a temporal relationship. The server can also obtain time positioning information for characterizing the current frame image, and fuse the initial image-text feature with the time positioning information to obtain the target image-text feature as the image-text constraint information for the i-th frame image. The time positioning information can be used to indicate the playback time point of the frame image in the target video. The frame images in the frame image sequence are arranged according to the corresponding time positioning information.

[0107] Step 1023 , combining the text constraint information and the graphic constraint information to obtain target constraint information.

[0108] In actual implementation, the server obtains text constraint information for the i-th image frame using the aforementioned method for determining text constraint information, and then obtains image-text constraint information for the i-th image frame using the aforementioned method for determining image-text constraint information. The size of the text constraint information for the i-th image frame is the same as the size of the image-text constraint information for the i-th image frame. The text constraint information and the image-text constraint information are merged to obtain the target constraint information for the i-th image frame.

[0109] Step 1024 : Using the frames of image preceding the i-th frame of image as references and based on the target constraint information, constrain the generation process of the i-th frame of image to obtain the i-th frame of image.

[0110] In actual implementation, the server performs diffusion sampling on each frame image before the i-th frame image to obtain a corresponding noise image, and then denoises the noise image based on the target constraint information to generate the i-th frame image.

[0111] In some embodiments, the frame image generation process can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1The server 200 in the embodiment performs the above frame image generation process alone as an example for explanation, see Figure 9 , Figure 9 This is a flow chart of the frame image generation process provided by the embodiment of the present application, combined with Figure 9 The steps shown are explained.

[0112] In step 401 , the server extracts features from each frame of image preceding the i-th frame of image to obtain image features of each frame of image.

[0113] Step 402: perform feature fusion on the image features of each frame image to obtain fused features.

[0114] In actual implementation, when generating the i-th frame, to ensure consistency with the images preceding the i-th frame, the server extracts features from each frame preceding the i-th frame to obtain image features for each frame. Image feature extraction can also be implemented based on a machine learning model.

[0115] For example, see Figure 10 , Figure 10 This is a diagram of the i-th frame image generation process provided by an embodiment of the present application. In the diagram, the server first inputs the frame image before the i-th frame image into the encoder for encoding, that is, the frame image before the i-th frame image is feature extracted to generate the image feature z (represented by a vector) corresponding to the latent space. After diffusion sampling (which can be achieved through a diffusion model), the latent space representation of z at time T is obtained. After denoising at time T, the latent space representation at time T-1 is predicted. The representation predictions from time T-1 to time T-2 are continued in this order until the prediction at time T=0. The final prediction is passed through the decoder D to obtain the generated i-th frame image.

[0116] Step 403: Perform a first-order noise addition process on the fused features to obtain a noisy fused feature.

[0117] For example, see Figure 10The diffusion sampling process for image features in the latent space is actually a first number T (T is a positive integer greater than 1) of noise additions to the fused features (i.e., the image features fused from multiple frames) to obtain the noisy fused features. In practical applications, the types of noise added during noise addition include, but are not limited to, Gaussian noise, salt and pepper noise, and Poisson noise. The specific type of noise to be added can be determined based on actual needs. Gaussian noise adds noise that follows a Gaussian distribution to the image features. The degree of noise addition can be controlled by adjusting the standard deviation of the Gaussian distribution. Salt and pepper noise adds black and white noise to the image, with pepper representing black noise (0, 0, 0) and salt representing white noise (255, 255, 255). The ratio of noise addition is controlled by setting a parameter value. A larger value results in more noise addition, which results in more severe image damage. The first number here refers to the number of noise additions, i.e., the pre-set number of times the image is noised. Each noise addition produces a new noisy image. After completing n noise additions, the final noisy image is obtained and used as the noisy image for subsequent processing.

[0118] Step 404: Based on the target constraint information, perform a second denoising process on the noisy fusion feature to obtain a denoised fusion feature.

[0119] Continuing with the above example, after obtaining the corresponding noise-added fusion features, the denoising process is performed on the second number to obtain the denoised fusion features. In practical applications, the first number and the second number can be the same or different. When the first number and the second number are equal, the first number is set to T, and the noise-added fusion features at time T are obtained by performing the noise-added fusion process on the first number. Then, after the denoising process, the fusion features at time T are predicted (i.e., the denoised image corresponding to the noisy image at time T-1) for the noise-added fusion features at time T-1. The prediction of the denoised images at times T-1 to T-2 is continued by analogy until the prediction at time T=0, that is, the image features of the denoised image at time T=0 are obtained.

[0120] Step 405: Generate the i-th frame image based on the denoising and fusion features.

[0121] Continuing with the above example, the server will perform the denoising fusion feature obtained by the second denoising process to reconstruct the image and obtain the i-th frame image.

[0122] In some embodiments, the frame image generation process based on the machine learning model can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1 The server 200 in the embodiment performs the frame image generation process based on the machine learning model as an example, see Figure 11 , Figure 11The frame image generation method based on the machine learning model provided in the embodiment of the present application is combined with Figure 11 The steps shown are explained.

[0123] Step 501: The server obtains a plurality of serially connected frame image generation models, wherein the frame image generation models have a one-to-one correspondence with the frame images other than the initial image.

[0124] In actual implementation, the server can generate corresponding frame images based on a machine learning model, where the machine learning model can be called a frame image generation model. Since all frame images before the i-th frame image and the corresponding description text are required when generating the i-th frame image, the server uses multiple serial frame image generation models to generate corresponding frame images. That is to say, the number of frame image generation models and the other frame images in the frame image sequence to be generated except the initial frame are in a one-to-one correspondence. For example, if the number of frame images in the frame image sequence to be generated is N, then N-1 serial frame image generation models can be used to generate the remaining N-1 frame images except the first frame image.

[0125] For example, see Figure 12 , Figure 12 This is a flow chart of a serial frame image generation model provided by an embodiment of the present application. The figure shows two serial frame image generation models, with the input being an initial image and the output being a frame image sequence comprising three frame images. Image 1 is generated by predicting the initial image and description text 1 describing the image content of image 1 through frame image generation model 1. Image 2 is generated by predicting the initial image, generated image 1, description text 1 associated with image 1, and description text 2 associated with image 2 through frame image generation model 2. Finally, the initial image, image 1, and image 2 are synthesized into the target video.

[0126] Step 502: Using the (i-1)th frame image model, based on the description text of the i-th frame image in the frame image sequence, the frame images before the i-th frame image in the frame image sequence, and the description text of the frame images before the i-th frame image, the i-th frame image is predicted to obtain the i-th frame image.

[0127] Continuing with the above example, the description text of the i-th frame image in the frame image sequence, the frame images before the i-th frame image in the frame image sequence, and the description text of the frame images before the i-th frame image can be processed by the (i-1)-th frame image model to predict the i-th frame image and obtain the i-th frame image.

[0128] In some embodiments, the training process of the frame image generation model can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1 The server 200 in the embodiment alone performs the training process of the frame image generation model as an example for explanation, see Figure 13 , Figure 13 This is a schematic diagram of the training method of the frame image generation model provided in the embodiment of the present application, combined with Figure 13 The steps shown are explained.

[0129] In step 601 , the server obtains a plurality of serially connected initial frame image generation models and a reference frame image sequence including a plurality of continuous frame image samples, wherein the frame image samples carry description text for describing the next adjacent frame image sample.

[0130] For example, see Figure 12 In the example of multiple serially connected frame image generation models, the server first obtains an initial frame image generation model that has not yet been trained, and a reference frame image sequence used to train the initial frame image generation model. The reference frame image sequence includes multiple consecutive frame image samples, each of which carries descriptive text describing the next adjacent frame image sample. For example, if the first frame image sample is person A standing, the second frame image sample is person A in a running-ready pose, and the third frame image sample is person A running, the first frame image sample may carry the descriptive text "person A in a running-ready pose."

[0131] The method for obtaining the reference frame image sequence is described. In some embodiments, see Figure 14 , Figure 14 This is a schematic diagram of the video sample processing flow provided by the embodiment of the present application, combined with Figure 14 The steps shown are explained.

[0132] Step 6011: The server obtains a video sample and a frame sampling interval for the video sample.

[0133] In actual implementation, in order to be able to generate multiple frames of continuous images with the style of film and television works for the input image through a series of frame image generation models, thereby obtaining a corresponding video with the style of film and television works, video samples can be collected from massive film and television dramas, and the initial frame image generation model can be trained with video samples with the style of film and television dramas to make the characteristics of the input image close to the feature space of film and television dramas.

[0134] Step 6012: Based on the frame extraction interval, the video sample is subjected to frame extraction processing to obtain multiple frame images.

[0135] In actual implementation, in order to reduce the consumption of computing resources, the server first extracts frames for each video sample according to a preset frame extraction interval, such as 1 frame per second, or 1 frame every 2 seconds, etc. The frame extraction interval can be adjusted according to actual conditions.

[0136] Step 6013: cluster the multiple frame images to obtain multiple clusters.

[0137] In some embodiments, the clustering process for video samples can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1 The server 200 in the embodiment performs the above clustering process for the video sample separately as an example for explanation, see Figure 15 , Figure 15 This is a flow chart of the clustering process for video samples provided by the embodiment of the present application, combined with Figure 15 The steps shown are explained.

[0138] In step 701, the server divides each frame image into shots to obtain a plurality of storyboards, and the similarity between the frame images in each storyboard reaches a similarity threshold.

[0139] In actual implementation, the server divides the multiple frames obtained by frame extraction into multiple shots, where the content of the images in the same shot is almost the same (with only slight changes in lighting, action, and object perspective). In other words, the similarity between the frame images in each shot reaches a similarity threshold.

[0140] Step 702: Select a target frame image from the frame images of each storyboard. The target frame image can be used to identify the storyboard.

[0141] In actual implementation, since the similarity between frame images in each storyboard reaches a similarity threshold, in order to reduce the consumption of computing resources, a target frame image can be selected in each storyboard, and the target frame image can be used to identify the storyboard to which it belongs.

[0142] Step 703 : extract features from each target frame image to obtain multiple frame image features.

[0143] In actual implementation, there is a one-to-one correspondence between the frame image features and the target frame image.

[0144] Step 704: cluster the features of the multiple frame images to obtain multiple clusters.

[0145] In actual implementation, the server randomly selects a frame image from each storyboard to represent it. It then uses the image branch of the cross-modal CLIP model to extract features from each target frame image, obtaining multiple frame image features. K-means clustering is then performed on all frame image features to obtain the cluster centers of K clusters. The cluster center to which each frame image belongs is then determined.

[0146] Step 6014: Determine the Euclidean distance between each frame image in each cluster and the corresponding cluster center.

[0147] In actual implementation, the server calculates the Euclidean distance between the frame image in each cluster and the corresponding cluster center.

[0148] Step 6015: determine the distance threshold corresponding to each cluster, and for each frame image in each cluster, determine the frame image with a Euclidean distance less than the distance threshold as a frame image sample, and obtain a reference frame image sequence including multiple continuous frame image samples.

[0149] In actual implementation, the server obtains each corresponding distance threshold and, for each cluster, retains frame image samples whose Euclidean distance is less than the distance threshold, so that each of the K clusters obtained contains relatively clean images.

[0150] In some embodiments, the process of determining the distance threshold may be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Figure 1 The server 200 in the embodiment performs the above distance threshold determination process alone as an example for explanation, see Figure 16 , Figure 16 This is a schematic diagram of the process of determining the distance threshold provided in the embodiment of the present application, combined with Figure 16 The steps shown are explained.

[0151] In step 801 , the server determines, for each cluster, the Euclidean distance between each frame image in the cluster and the cluster center of the cluster.

[0152] In actual implementation, for each cluster, the Euclidean distance between each frame image in the cluster and the cluster center of the cluster is determined.

[0153] Step 802: Determine the maximum value of the Euclidean distance and the minimum value of the Euclidean distance from each Euclidean distance.

[0154] In actual implementation, the server determines the maximum value of the Euclidean distance in each cluster, as well as the minimum and maximum distances of the Euclidean distance, and records the threshold value thr = (max-min) / 2+min.

[0155] Step 803: Obtain the difference between the maximum value and the minimum value, and sum half of the difference with the minimum value, and use the summed result as the distance threshold corresponding to the cluster.

[0156] In actual implementation, the server determines the difference between the maximum and minimum values ​​(max-min), and sums half of the difference (max-min) / 2 with the minimum value min, (max-min) / 2+min. The resulting sum is then used as the distance threshold for the cluster, which can be denoted as thr. That is, thr = (max-min) / 2+min.

[0157] Step 602: Using the first frame image in the reference frame image sequence as the first frame image in the target frame image sequence, generating the i-th frame image in the target frame image sequence through the (i-1)th initial frame image generation model based on the description text of the i-th frame image sample in the target frame image sequence, each frame image sample located before the i-th frame image sample in the target frame image sequence, and the description text of each frame image sample located before the i-th frame image sample, generates the i-th frame image in the target frame image sequence; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame image samples in the target frame image sequence.

[0158] In actual implementation, the server uses the first frame image in the reference frame image sequence as the first frame image in the target frame image sequence, and generates the i-th frame image through the (i-1)th initial frame image generation model. In the process of generating the i-th frame image, based on the description text of the i-th frame image in the target frame image sequence, the frame images before the i-th frame image in the target frame image sequence, and the description text of the frame images before the i-th frame image, target constraint information for the i-th frame image in the target frame image sequence is generated, and based on the target constraint information, the generation process of the (i-1)th initial frame image generation model for the i-th frame image in the target frame image sequence is constrained.

[0159] Step 603: traverse i to predict each frame image in the target frame image sequence.

[0160] In actual implementation, the server traverses i, that is, it iteratively executes the above-mentioned generation process for the i-th frame image in the target frame image sequence for different values ​​of i, thereby predicting each frame image in the target frame image sequence.

[0161] Step 604 : Obtain the difference between the target frame image sequence and the reference frame image sequence, and based on the difference, update the model parameters of each initial frame image generation model to obtain a series of frame image generation models.

[0162] In actual implementation, for the other frame images in the target frame image sequence except the first frame image, the differences between the current frame image and the frame image sample at the current position in the reference frame image sequence are determined in turn. Based on the corresponding differences, the model parameters of the corresponding initial frame image generation model are updated to obtain multiple trained serial frame image generation models.

[0163] Exemplarily, the MSE loss can be used to calculate the difference between the current frame image and the frame image sample at the current position in the reference frame image sequence, that is, the mean square error of the pixel value at the same position of the current frame image in the target frame image sequence and the frame image sample at the corresponding position in the reference frame image sequence is used as the difference between the current frame image and the frame image sample at the current position in the reference frame image sequence.

[0164] Step 103: traverse i to obtain each frame image in the frame image sequence.

[0165] In actual implementation, the above-mentioned process of generating the i-th frame image is repeated to obtain each frame image in the sequence of frame images to be generated.

[0166] Step 104 , performing video synthesis on each frame image in the frame image sequence to obtain a target video.

[0167] During actual implementation, the server performs video synthesis on each frame image in the generated frame image sequence to obtain a target video.

[0168] In some embodiments, the server can also determine the attribute information of the target video in the following manner: the server obtains the frame rate of the target video, and the frame rate is used to indicate the number of continuous frame images played per second in the target video; determines the ratio of the number of frame images in the frame image sequence to the frame rate as the duration of the target video, and determines the duration as the attribute information of the target video.

[0169] In actual implementation, the specific video synthesis method can be that the server obtains a preset frame rate, and based on the frame rate, synthesizes each frame image in the frame image sequence into a target video with a target length, and the target length is equal to the ratio of the number of frame images in the frame image sequence to the preset frame rate.

[0170] For example, the number of frame images in the frame image sequence is 120, and the preset frame rate is 15fps (Frames Per Second), then the target duration of the generated target video is 120 / 15=8 seconds; if the preset frame rate is 20fps, then the target duration of the generated target video is 120 / 20=6 seconds.

[0171] By applying the above-mentioned embodiment of the present application, when generating each frame image in the frame image sequence to be generated, the frame images preceding the frame image and the descriptive text associated with the frame images preceding the frame image are used as target constraint information to constrain the generation process of each frame image. In this way, the generated frame image can maintain content consistency with the frame images preceding the frame image. In addition, by training a frame image generation model with video samples having a film and television style, a plurality of continuous frame images with a corresponding film and television style can be generated from an initial image, thereby generating a target video with a film and television style. In this way, it can be used to imitate and generate a film and television work for an image, thereby enriching the expressive content of the image. In addition, a method for predicting the current frame image based on the corresponding constraint information generated by the frame image preceding the current frame image can decompose the video generation process into the prediction task of the next frame image, thereby achieving efficient and fast generation of continuous frame images.

[0172] Next, we will continue to introduce the video generation method provided by the embodiment of the present application. Figure 17 This is another flow chart of the method for generating a video according to an embodiment of the present application. Here, a plurality of serial frame image generation models are run on a server as an example. Figure 17 The video generation method provided in the embodiment of the present application is implemented collaboratively by the terminal and the server.

[0173] In step 901 , the terminal sends a video generation request to the server in response to a video generation operation based on an initial image and a description text of each frame of the video to be generated.

[0174] Exemplarily, the video to be generated is the target video to be generated. The video to be generated is composed of a sequence of frame images to be generated. The video generation request carries an initial image which is the first frame image in the sequence of frame images to be generated, and a description text for each frame image in the sequence of frame images to be generated.

[0175] Step 902: The server parses the video generation request to obtain an initial image and a description text for each frame image in the frame image sequence to be generated.

[0176] In step 903 , the server inputs the initial image and the description text for each frame image in the frame image sequence to be generated into a plurality of serially connected frame image generation models.

[0177] For example, the frame image generation model has a one-to-one correspondence with the frame images other than the initial image. The server determines the obtained initial image as the first frame image in the frame image sequence to be generated, and obtains description text for each frame image in the frame image sequence. Before using multiple serially connected frame image generation models, the server can train the multiple serially connected frame image generation models in the following manner: obtain multiple serially connected initial frame image generation models and a reference frame image sequence including multiple continuous frame image samples, where the frame image samples carry description text for describing the next adjacent frame image sample; use the first frame image sample in the reference frame image sequence as the first frame image in the target frame image sequence, and generate the i-th frame image in the target frame image sequence through the (i-1)th initial frame image generation model based on the description text of the i-th frame image in the target frame image sequence, the frame images before the i-th frame image in the target frame image sequence, and the description text of the frame images before the i-th frame image; traverse i to predict each frame image in the target frame image sequence; obtain the difference between the target frame image sequence and the reference frame image sequence, and based on the difference, update the model parameters of each initial frame image generation model to obtain a serially connected frame image generation model.

[0178] In step 904, the server uses the (i-1)th frame image model to extract features of the description text of the i-th frame image in the frame image sequence and the description texts of the frame images before the i-th frame image to obtain the description text features of each frame image.

[0179] Step 905 : Acquire identification features for indicating text modality, fuse the identification features with the descriptive text features, and use the fusion result as text constraint information of the i-th frame image.

[0180] Step 906 : extract image features from each frame of image preceding the i-th frame of image to obtain image features of each frame of image.

[0181] Step 907 , using the image features of each frame as image constraints, extracting text features from the description text associated with each frame before the i-th frame to obtain initial image and text features of each frame.

[0182] Step 908 : Acquire identification features for indicating the image-text modality, fuse the identification features with the initial image-text features, and use the fusion result as the image-text constraint information of the i-th frame image.

[0183] Step 909 : Combine the text constraint information and the graphic constraint information to obtain target constraint information.

[0184] Step 910 : Taking each frame image before the i-th frame image as a reference and based on the target constraint information, constraining the generation process of the i-th frame image to obtain the i-th frame image.

[0185] Step 911: traverse i to obtain each frame image in the frame image sequence.

[0186] Step 912: Perform video synthesis on each frame image in the frame image sequence to obtain a target video, and send the target video to the terminal.

[0187] In step 913 , the terminal receives the target video returned by the server and performs a target operation on the target video. The target operation includes at least one of the following: play, download, share, etc.

[0188] By applying the embodiments of the present application, a method for predicting the current frame image is performed based on the corresponding constraint information generated by the frame image located before the current frame image. This method can not only decompose the video generation process into the prediction task of the next frame image, thereby achieving efficient and fast generation of continuous frame images, but also maintain content consistency between the generated frame image and the frame images before the current frame image.

[0189] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0190] In film and television scenarios, when users upload an image, they often want to use video technology to animate it into a film-style animation. This means animating the image in a way that evokes the same motion as in film and television, such as creating short videos showing a character's expression or changing scenery. While text-to-image technology works well for generating cartoons and artistic landscapes, it struggles with generating images of film-style scenes and characters. In addition, image-to-text technology cannot guarantee consistency between the output image and the input image due to random noise, often resulting in inconsistent results, such as inconsistent characters, backgrounds, or scenes.

[0191] Based on this, an embodiment of the present application provides a video generation method, which is based on the diffusion model latent space image generation technology, designs a continuous frame generation model for film and television works (also known as a sequence model, that is, the frame image generation model connected in series in the previous article), and converts video generation into continuous frame image generation, that is, by inputting an image to generate the next frame image consistent with it, completing the prediction task of the next frame image of the current image; at the same time, in order to improve the correlation between adjacent frame images, when generating the next frame image of the image, the process of generating the next frame image by the continuous frame generation model is constrained by referring to the historical frame information (that is, the relevant information of the frame image before the current frame image), so that the generated next frame image is consistent with the current image; at the same time, through the collection of training sets from massive film and television dramas, the latent space mapping module of the continuous frame generation model is trained with the training sets of film and television drama works so that the features of the input image are close to the feature space of the film and television drama.

[0192] The video generation method provided in the embodiment of the present application: not only collects training sets based on massive film and television dramas as supervision information for image generation; but also designs a continuous frame generation framework to generate the next frame image based on the input image, thereby forming a continuous frame generation model (i.e., the frame image generation model mentioned above); in addition, it also improves the scene and content relevance between adjacent frame images of the video through the accumulation input of historical frame image information, thereby making the generated video adjacent frame images more coordinated.

[0193] For example, see Figure 18A , Figure 18A This is a schematic diagram summarizing the flow of the video generation method provided in an embodiment of the present application. First, given an initial image, a continuous frame generation model is used to continuously generate multiple frames of subsequent frame images that conform to the data space of a film or TV series; then, the continuous frame images are used for video synthesis to obtain the target video. In actual implementation, the user uploads the initial image and prompt words related to the video he wants to generate (such as "cute", etc.). The electronic device (server or terminal) equipped with the continuous frame generation model selects a suitable continuous frame generation model according to the initial image uploaded by the user (such as uploading a person, then selecting a continuous frame generation model trained by the person; if uploading a landscape image, then selecting a continuous frame generation model trained by the landscape image); the continuous frame image is generated by the selected continuous frame generation model, and the generated continuous frame image and the initial image are merged into the target video and returned to the user. For example, see Figure 18B , Figure 18B This is a visual diagram of multi-frame image generation provided by an embodiment of the present application. In the figure, the user inputs an image and predicts 4 new frame images. The synthetic video includes 5 frames of images. Except for the first frame image, the other frame images are generated based on the other frame images before themselves and the corresponding descriptive text.

[0194] For an explanation of the generation process of continuous frame images, see Figure 12 The specific generation process is as follows: 1) Generate a result for the initial image and prompt word (i.e., description text 1) uploaded by the user, which contains multiple images; 2) Select one image from the multiple images as the current moment image (i.e., image 1); 3) Use the historical moment images, historical prompt words, and current moment prompt word (if the user does not enter a prompt word at the current moment, the current moment prompt word is set to empty) as model input to generate a result; 4) Repeat steps 2) to 3) until N images are generated. For example, the user uploads initial image 0 and description text 1, and the continuous frame generation model generates image 1. Then, the initial image 0, description text 1, image 1, and description text 2 are again input into the continuous frame generation model to generate image 2. This process is repeated to generate image 3, image 4, ..., image N, where N is the number of generated frame images. When N is known, different target video lengths can be synthesized based on different frame rates. For example, generating 75 frame images at a frame rate of 15fps can generate a video of 75 / 15 = 5 seconds.

[0195] This paper describes a continuous frame generation model for generating continuous frame images. Figure 19 , Figure 19This is a structural diagram of the continuous frame generation model provided by an embodiment of the present application. After the text features of the historical description text are extracted, and after the image features of the historical image are extracted, the historical text features are fused with the image features to obtain historical information. If the user's current input information is only an image and there is no description text, the description text is input into the continuous frame generation model using the default value (empty). After the text features of the description text at the current moment (such as text 3 in the figure) are extracted, they are spliced ​​with the historical information to finally form the constraint information generated by the continuous frame generation model at the current moment. The overall process of the continuous frame generation model is implemented using a generative model based on a latent space diffusion (SSD) algorithm. First, a 256x256 noisy image x is generated using Gaussian random noise as the overall input. This image is passed through the encoder to obtain a latent space representation. The latent space diffusion model samples the latent space representation of the noisy image x at time T (note that T is not related to the current time in the sequence frame generation process. The T time moments here can be understood as T restoration operations, and the overall video sequence generation can be understood as each frame in the video frame sequence being generated by a generative model with T steps). This latent space representation at time T is passed through a denoising U-Net module (which includes historical information constraints) to predict the latent space representation at time T-1. The representation at time T-1 is then fed into the U-Net input to predict the representation at time T-2. This process repeats for T steps, resulting in a predicted representation at time T = 0. This representation at time T = 0 is passed through the decoder to produce the generated image y. This completes the generation of a single frame, i.e., the process of generating image 1 from the initial image.

[0196] For an explanation of open source stable-diffusion, see Figure 20 , Figure 20This is a schematic diagram of a standard diffusion model provided in an embodiment of the present application. Assuming that each noise image is obtained through T steps of noise sampling, and performing T steps of denoising on this process, the original noise image can be predicted. The generation process of the model refers to the figure below: first randomly generate a noise image x, generate a latent space representation z through the encoder ε, obtain the latent space representation of z at time T through the diffusion model sampling, predict the latent space representation at time T-1 through the u-net module, and continue to predict the representations from time T-1 to time T-2, until the prediction at time T=0. The final prediction is obtained by the decoder D to obtain the generated image. During training, in order to ensure that the generated image is related to the text under the specified text, text encoding (or other such as images) is used as the attention feature in the denoising U-Net module, so that the denoising process is subject to text attention constraints. At the same time, the generated image uses the image in the image-text annotation sample pair as supervision information, thereby ensuring that the image generated by the model is related to the text under the condition of inputting the specified text. In some embodiments, a user image may be provided, and the text in the above process may be replaced with the image as a constraint, and the next image may be generated through the above process (also referred to as an img2img process).

[0197] For an explanation of the denoising U-Net module, see Figure 21A , Figure 21A This is a schematic diagram of the U-Net process without adding constraint information provided by the embodiment of this application. The latent space features at time T are passed through U-Net to obtain the representation prediction at time T-1. In the application scenario where historical image information is needed as constraint information to predict the new image, it can be Figure 21A Based on the U-Net structure shown in , new constraints are added to process historical information. Figure 21B , Figure 21B This is a schematic diagram of U-Net with added constraint information provided by an embodiment of the present application. In the figure, IGTE is a BLIP-based text extraction model. The image and text constraint information is determined by IGTE. For an explanation of the BLIP-based text extraction model, see Figure 22 , Figure 22 This is a structural diagram of a BLIP text extraction model provided in an embodiment of the present application. Figure 22 In the example, image 1 is extracted by ViT to obtain image features, and the corresponding description text is extracted by BLIP to extract text features. The image features extracted by ViT are combined to generate image constraint text. In actual implementation, IGTE directly uses the open source model weights, and there is no need to update the model parameters of the model later. Figure 21BFor all historical texts (texts 1 and 2), we extract text feature 0 (the descriptive text feature mentioned above) through the CLIP text branch. Text feature 0 is a matrix feature composed of multiple text vectors with a size of 2×1×768. Then, we add a flag bit 0 representing the text modality (using a vector representation of size 2×1×768) to obtain text feature 1. Text feature 1 is also a matrix feature composed of multiple text vectors with a size of 2×1×768. The flag bit represents a 2×1×768 three-dimensional matrix with all elements set to 0. For all historical and current image-text information, the image-text feature extraction is performed through the IGTE model to obtain the text feature with image constraint (i.e., the image-text feature in the previous article) 0. The text feature with image constraint is a matrix feature of size 1x3x768. In order to align the first dimension of text feature 0, the text feature 0 with image constraint is copied once along the first dimension to obtain the text feature 1 with image constraint. The size of the text feature 1 with image constraint is a 2x3x768 matrix. Then, the flag bit representing the image-text modality is added (the flag bit size is 2×3×768) to obtain the text feature 2 with image constraint. The text feature 2 with image constraint is a matrix feature of size 2×3×768. Then, the time series flag bit is combined to obtain the text feature 3 with image constraint (the matrix feature of size 2×3×768) (2x3x768). Finally, the text feature 1 and the text feature 3 with image constraint are spliced ​​to obtain the final historical constraint information (the matrix feature of size 2×4×768). The “+” represents vector addition and “C” represents concatenation vector. The historical constraint information is input into the denoising U-Net. Figure 1 、 2 The noise implicit representation is subjected to T denoising processes with constraint information to obtain the final noise prediction at time 0, that is, the noise representation minus the noise prediction at time 0 can be restored Figure 1 、 2 Hidden representation. In actual implementation, when extracting image or text features, the image branch of the cross-modal CLIP model is used to extract image features, and the text branch is used to extract text features. The open source model weights are directly used, and there is no need to update the model parameters of the CLIP model later.

[0198] The collection of training data (i.e., the reference frame image sequence obtained based on the video sample in the previous text) is described. In some embodiments, the sample data required for the training sequence model is as follows: Figure 23 , Figure 23This is a flow chart of training data collection provided by an embodiment of the present application. The collection of training data (i.e., the video samples mentioned above) includes at least the following four steps: 1) frame extraction of film and television works; 2) cross-modal model representation; 3) data clustering and cleaning; 4) labeling of image and text sample pairs. The specific implementation of frame extraction of film and television works is: extracting 1 frame per second for movies and TV series (the frame extraction interval can be adjusted as needed, such as extracting 1 frame every 2 seconds, etc.). The extracted frames are divided into shots to obtain multiple storyboards, each of which includes multiple frame images, see Figure 24 , Figure 24 This is a schematic diagram of the shot-by-shot operation for a video sample provided by an embodiment of the present application. In the figure, numbers 1 and 2 correspond to different adjacent shots, where the image content in the same shot is almost the same (only slight changes in lighting, action, and object perspective exist). At this time, the sequence images in each shot can be used as supervision information for the generation of sequence images. Supervision information such as the sequence shown in number 1 Figure 1 As a constraint input model, the model needs to generate a close sequence Figure 2 As a result, the sequence Figure 2 That is the sequence Figure 1 Generate supervision information.

[0199] In actual implementation, the realization method of cross-modal model representation is: for each storyboard, one image is randomly selected to represent the storyboard, and the image extracts cross-module feature representation. This article uses the image branch of the open source cross-modal CLIP model to extract video frame representation.

[0200] In practice, the data clustering and cleaning process involves: 1) performing k-means clustering on all representations (i.e., the frame image features mentioned above) to obtain K cluster centers (e.g., cluster 2); 2) finding the cluster center for each frame; 3) calculating the Euclidean distance between the frame and the center in each cluster, obtaining the minimum and maximum distances (max and min) within the cluster, and recording a threshold (thr = (max - min) / 2 + min); 4) for each cluster, only saving samples whose distance from the center is less than the threshold (thr). This results in K clusters, each containing a relatively clean image.

[0201] In practice, the specific implementation for labeling image-text pairs is as follows: For all clustered images, the image description is annotated. For example, if the image shows a person speaking, the corresponding description is annotated as "A man in a suit speaking, half-length portrait." This results in labeled image-text pairs. Since each image represents a storyboard, the sequence diagrams in the storyboard share the same text information as the image.

[0202] The training process of the continuous frame generation model is explained. Figure 20The stable-diffusion-based sequence model shown includes a CLIP model for extracting prompt text features, a VAE model for decoding and generating images, a diffusion model (diffusers) for generating random noise, and a U-Net for denoising. The VAE, CLIP, BLIP, and U-Net modules are all initialized with the weights of the open-source model. The VAE, CLIP, and BLIP models retain the weights of the open-source model and are not learned by this model; only the U-Net model is trained. The dataset (the video samples mentioned above) collects x images for each storyboard. Since different storyboards have different numbers of images, to avoid data volume differences, the model specifies every three images as a sequence data group. Therefore, we take 3 pictures for each storyboard. For storyboards with more than 3 pictures, we traverse all 3 consecutive pictures (for example, for 5 pictures: pictures 123, 234, 345 form 3 sequence data groups. For the N sequence data groups in the full training, each bs sequence data is a batch input into the model to learn and update the model parameters once, and a total of N / bs batches of data are learned as one round of model training. When learning each batch of data, follow the above process to Figure 21B The U-Net network is the parameter to be learned: 1) Perform a forward pass on the input data to obtain a noise prediction; 2) Calculate the mean square error (MSE) loss of the noise prediction; 3) Perform a backward pass to obtain the updated gradients of the model parameters; 4) Use the SGD gradient update method to update the model parameters, completing one batch of training. Repeat these steps until all (N / bs) batches of training are completed, ending one iteration. Initially, use a learning rate of 0.0005. After every 10 epochs, multiply the learning rate by 0.1. Continue training based on whether the loss decreases. If it stops, stop training. Alternatively, stop training after reaching a specified number of iterations, such as 100 epochs.

[0203] The prediction loss of the noise image during model training is explained. The MSE loss is calculated by the mean square error of the input noise image and the output noise prediction image. The calculation formula of the MSE loss is as follows: Among them, y is the pixel value of each point in the image, and p represents the pixel of the noise prediction map. The difference in the values ​​of pixels at the same position.

[0204] For an explanation of the inference process of the continuous frame generation model, see Figure 25 , Figure 25 This is a schematic diagram of the reasoning process of continuous frame images provided by the embodiment of the present application. First, adapt to various user inputs: 1) User input text (i.e., the description text in the previous text) 1, when there is no image 1, the process shown in the number 1 in the figure is generated Figure 1 ; 2) The user inputs an image 1 but no text 1, then set text 1 to be empty; 3) The user inputs Figure 1and text 1, no processing is required. Then, according to the number 2 shown in the figure Figure 1 、Text1>、< Figure 2 、Text2> form to organize input data. Among them, by default, if the user does not enter text, then Text2 is empty. When the user enters text at this moment, Text2 is the user-entered text. Figure 2 , where the noise image x0 needs to be the same as the input image in the process shown in number 1 in the figure. Then, follow the process shown in number 3 in the figure to generate Figure 3 Finally, Figure 1 、 2 , 3 are synthesized in sequence to obtain the target video. It should be noted that in order to simplify the process, the generation process of three consecutive frames during inference is introduced. It is also possible to continue generating more frames to form a longer video.

[0205] Applying the above embodiments of the present application has the following beneficial effects:

[0206] 1) It can generate film and television works imitating an image uploaded by the user, thereby enriching the expressive content of the user's image. For example, if a user uploads an image of a smile, a video of a gradually laughing person will be generated, making the information expressed by the image richer.

[0207] 2) An image can also be animated. For example, for an illustration in a book, the model can generate subsequent frame images to form a small video, so that the image can change dynamically and become a moving picture.

[0208] 3) Design a continuous frame generation system. This allows users to input an image and generate a new image in the style of a film or television work. By applying continuous frame constraints, the resulting video is a sequence of images, ensuring consistent information between frames. This system can generate a film or television-like image from a user's uploaded image, enriching the user's image's expressive content. For example, if a user uploads an image of a smile, a video of a gradual laugh will be generated, enriching the information conveyed by the image.

[0209] 4) Establish a next-frame prediction method based on historical frame constraints, including how to organize historical frame information and the data process during application, so as to effectively decompose video generation into the next-frame prediction task and achieve efficient and fast image sequence.

[0210] The following continues to describe the exemplary structure of the video generation device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the video generation device 555 of the memory 550 may include:

[0211] The acquisition module 5551 is used to determine the acquired initial image as the first frame image in the frame image sequence to be generated, and obtain the description text of each frame image in the frame image sequence;

[0212] a generating module 5552 configured to generate the i-th frame image based on the description text of the i-th frame image in the frame image sequence, the frame images preceding the i-th frame image in the frame image sequence, and the description text of the frame images preceding the i-th frame image; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence;

[0213] A traversal module 5553 is used to traverse the i to obtain each frame image in the frame image sequence;

[0214] The synthesis module 5554 is used to perform video synthesis on each frame image in the frame image sequence to obtain a target video.

[0215] In some embodiments, the generation module is further used to generate text constraint information of the i-th frame image based on the description text of the i-th frame image in the frame image sequence and the description text of each frame image located before the i-th frame image; generate graphic and text constraint information of the i-th frame image based on each frame image located before the i-th frame image and the description text of each frame image; wherein the text constraint information and the graphic and text constraint information are both used to constrain the image content of the i-th frame image; the text constraint information and the graphic and text constraint information are fused to obtain target constraint information; with the frame images located before the i-th frame image as a reference, the generation process of the i-th frame image is constrained based on the target constraint information to obtain the i-th frame image.

[0216] In some embodiments, the generation module is further used to perform feature extraction on the description text of the i-th frame image in the frame image sequence and the description text of each frame image located before the i-th frame image, respectively, to obtain the description text features of each frame image; obtain identification features for indicating the text modality, and fuse the identification features with the description text features, and use the fusion result as the text constraint information of the i-th frame image.

[0217] In some embodiments, the generation module is further used to extract image features of each frame image located before the i-th frame image to obtain the image features of each frame image; using the image features of each frame image as image constraints, perform text feature extraction on the descriptive text associated with each frame image located before the i-th frame image to obtain the initial image and text features of each frame image; obtain identification features for indicating the image and text modality, and fuse the identification features with the image and text features, and use the fusion result as the image and text constraint information of the i-th frame image.

[0218] In some embodiments, the generation module is further used to extract features from each frame image located before the i-th frame image to obtain image features of each frame image; perform feature fusion on the image features of each frame image to obtain fusion features; perform a first number of noise addition processing on the fusion features to obtain noisy fusion features; based on the target constraint information, perform a second number of denoising processing on the noisy fusion features to obtain denoised fusion features; and generate the i-th frame image based on the denoised fusion features.

[0219] In some embodiments, the generation module is also used to obtain multiple serially connected frame image generation models, wherein the frame image generation models have a one-to-one correspondence with other frame images except the initial image; through the (i-1)th frame image model, based on the description text of the i-th frame image in the frame image sequence, the frame images before the i-th frame image in the frame image sequence, and the description text of the frame images before the i-th frame image, the i-th frame image is predicted to obtain the i-th frame image.

[0220] In some embodiments, the generation module is further used to obtain multiple serially connected initial frame image generation models and a reference frame image sequence including multiple continuous frame image samples, wherein the frame image samples carry description text for describing the next adjacent frame image; using the first frame image sample in the reference frame image sequence as the first frame image in the target frame image sequence, and generating the i-th frame image in the target frame image sequence through the (i-1)th initial frame image generation model based on the description text of the i-th frame image in the target frame image sequence, each frame image in the target frame image sequence located before the i-th frame image, and the description text of each frame image located before the i-th frame image; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame image samples in the target frame image sequence; traversing i to predict each frame image in the target frame image sequence; obtaining the difference between the target frame image sequence and the reference frame image sequence, and updating the model parameters of each initial frame image generation model based on the difference to obtain the serially connected frame image generation model.

[0221] In some embodiments, the generation module is also used to obtain video samples and a frame sampling interval for the video samples; based on the frame sampling interval, the video samples are subjected to frame sampling processing to obtain multiple frame images; the multiple frame images are subjected to clustering processing to obtain multiple clusters; the Euclidean distance between each frame image in each cluster and the corresponding cluster center is determined; the distance threshold corresponding to each cluster is determined, and for each frame image in each cluster, the frame image whose Euclidean distance is less than the distance threshold is determined as a frame image sample, and a reference frame image sequence including multiple continuous frame image samples is obtained.

[0222] In some embodiments, the generation module is further used to divide each of the frame images into shots to obtain multiple storyboards, and the similarity between the frame images in each storyboard reaches a similarity threshold; select a target frame image from the frame images of each storyboard, and the target frame image can be used to identify the storyboard; perform feature extraction on each of the target frame images to obtain multiple frame image features; and perform clustering processing on the multiple frame image features to obtain the multiple clusters.

[0223] In some embodiments, the generation module is further used to determine, for each cluster, the Euclidean distance between each frame image in the cluster and the cluster center of the cluster; determine the maximum value of the Euclidean distance and the minimum value of the Euclidean distance from each of the Euclidean distances; obtain the difference between the maximum value and the minimum value, and sum half of the difference with the minimum value, and the summed result is used as the distance threshold corresponding to the cluster.

[0224] In some embodiments, the synthesis module is also used to obtain the frame rate of the target video, where the frame rate is used to indicate the number of consecutive frame images played per second in the target video; determine the ratio of the number of frame images in the frame image sequence to the frame rate as the duration of the target video, and determine the duration as the attribute information of the target video.

[0225] The present invention provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video generation method described in the present invention.

[0226] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the video generation method provided by the embodiment of the present application, for example, Figure 3 The method of generating the video is shown.

[0227] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0228] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0229] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0230] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0231] In summary, the embodiments of the present application have the following beneficial effects: when generating each frame image in the frame image sequence to be generated, the frame images before the frame image and the descriptive text associated with the frame images before the frame image are used as target constraint information to constrain the generation process of each frame image. In this way, the generated frame image can maintain content consistency with the frame images before the frame image. In addition, a frame image generation model is obtained by training video samples with film and television styles, and an initial image can be generated into multiple continuous frame images with corresponding film and television styles, thereby generating a target video with film and television styles. In this way, it can be used to imitate and generate film and television works for an image, thereby enriching the expression content of the image. In addition, a method for predicting the current frame image based on the corresponding constraint information generated by the frame image before the current frame image can decompose the video generation process into the prediction task of the next frame image, thereby achieving efficient and fast generation of continuous frame images.

[0232] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for generating a video, characterized in that: The method comprises: Determining the obtained initial image as the first frame image in the frame image sequence to be generated, and obtaining description text of each frame image in the frame image sequence; generating the i-th frame image based on the description text of the i-th frame image in the frame image sequence, the description texts of the frame images preceding the i-th frame image in the frame image sequence, and the description texts of the frame images preceding the i-th frame image; Wherein, i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence; Traversing the i to obtain each frame image in the frame image sequence; Perform video synthesis on each frame image in the frame image sequence to obtain a target video.

2. The method according to claim 1, wherein The generating of the i-th frame image based on the description text of the i-th frame image in the frame image sequence, the frame images preceding the i-th frame image in the frame image sequence, and the description text of the frame images preceding the i-th frame image includes: generating text constraint information of the i-th frame image based on the description text of the i-th frame image in the frame image sequence and the description texts of the frame images before the i-th frame image; Generate image-text constraint information for the i-th frame based on each frame image preceding the i-th frame image and the description text of each frame image; wherein both the text constraint information and the image-text constraint information are used to constrain the image content of the i-th frame image; splicing the text constraint information and the graphic constraint information to obtain target constraint information; The i-th frame image is obtained by constraining the generation process of the i-th frame image based on the target constraint information and taking the frame images preceding the i-th frame image as references.

3. The method according to claim 2, wherein The generating of text constraint information of the i-th frame image based on the description text of the i-th frame image in the frame image sequence and the description texts of each frame image located before the i-th frame image includes: Extracting features of the description text of the i-th frame image and the description texts of the frame images before the i-th frame image in the frame image sequence to obtain description text features of the frame images; An identification feature for indicating a text modality is obtained, the identification feature is fused with the descriptive text feature, and the fusion result is used as text constraint information of the i-th frame image.

4. The method according to claim 2, wherein The step of generating the image-text constraint information of the i-th frame image based on the frame images preceding the i-th frame image and the description texts of the frame images includes: Extracting image features from each frame of image preceding the i-th frame of image to obtain image features of each frame of image; Taking the image features of each frame of image as image constraints, extracting text features of the description text associated with each frame of image located before the i-th frame of image to obtain initial image-text features of each frame of image; An identification feature for indicating a graphic-text modality is acquired, the identification feature is fused with the initial graphic-text feature, and the fusion result is used as the graphic-text constraint information of the i-th frame image.

5. The method according to claim 2, wherein The step of constraining the generation process of the i-th frame image based on the target constraint information and taking each frame image before the i-th frame image as a reference to obtain the i-th frame image includes: Perform feature extraction on each frame image before the i-th frame image to obtain the image features of each frame image; Performing feature fusion on the image features of each frame image to obtain fused features; Performing a first-order noise addition process on the fused features to obtain a noisy fused feature; Based on the target constraint information, performing a second denoising process on the noise-added fusion feature to obtain a denoised fusion feature; Based on the denoising and fusion features, the i-th frame image is generated.

6. The method according to claim 1, wherein The generating of the i-th frame image based on the description text of the i-th frame image in the frame image sequence, the frame images preceding the i-th frame image in the frame image sequence, and the description text of the frame images preceding the i-th frame image includes: Acquire a plurality of serially connected frame image generation models, wherein the frame image generation models have a one-to-one correspondence with other frame images except the initial image; The (i-1)th frame image generation model is used to predict the i-th frame image based on the description text of the i-th frame image in the frame image sequence, the frame images before the i-th frame image in the frame image sequence, and the description text of the frame images before the i-th frame image to obtain the i-th frame image.

7. The method according to claim 6, wherein Before acquiring the multiple series-connected frame image generation models, the method further includes: Acquire multiple serially connected initial frame image generation models and a reference frame image sequence comprising multiple continuous frame image samples, wherein the frame image samples carry description text for describing the next adjacent frame image sample; Using the first frame image sample in the reference frame image sequence as the first frame image in the target frame image sequence, generating the i-th frame image in the target frame image sequence based on the description text of the i-th frame image in the target frame image sequence, the frame images preceding the i-th frame image in the target frame image sequence, and the description text of the frame images preceding the i-th frame image by using the (i-1)th initial frame image generation model; Wherein, i is a positive integer greater than 1, and i is less than or equal to the number of frame image samples in the target frame image sequence; Traversing the i to predict each frame image in the target frame image sequence; The difference between the target frame image sequence and the reference frame image sequence is obtained, and based on the difference, the model parameters of each of the initial frame image generation models are updated to obtain the series frame image generation model.

8. The method according to claim 7, wherein The obtaining of a reference frame image sequence comprising a plurality of continuous frame image samples comprises: Obtaining a video sample and a frame sampling interval for the video sample; Based on the frame extraction interval, the video sample is subjected to frame extraction processing to obtain a plurality of frame images; performing clustering processing on the plurality of frame images to obtain a plurality of clusters; Determining the Euclidean distance between each frame image in each cluster and the corresponding cluster center; A distance threshold corresponding to each cluster is determined, and for each frame image in each cluster, frame images whose Euclidean distance is less than the distance threshold are determined as frame image samples, and a reference frame image sequence including multiple continuous frame image samples is obtained.

9. The method according to claim 8, wherein The clustering process is performed on the plurality of frame images to obtain a plurality of clusters, including: Performing shot division on each of the frame images to obtain a plurality of split shots, wherein the similarity between the frame images in each split shot reaches a similarity threshold; Selecting a target frame image from the frame images of each storyboard, wherein the target frame image can be used to identify the storyboard; Extracting features from each of the target frame images to obtain multiple frame image features; Clustering is performed on the multiple frame image features to obtain the multiple clusters.

10. The method according to claim 8, wherein Determining the distance threshold corresponding to each cluster includes: For each cluster, respectively determining the Euclidean distance between each frame image in the cluster and the cluster center of the cluster; Determine the maximum value of the Euclidean distance and the minimum value of the Euclidean distance from the Euclidean distances; The difference between the maximum value and the minimum value is obtained, and half of the difference is summed with the minimum value, and the obtained sum is used as the distance threshold corresponding to the cluster.

11. The method according to claim 1, wherein After performing video synthesis on each frame image in the frame image sequence to obtain a target video, the method further includes: Obtaining a frame rate of the target video, where the frame rate indicates the number of consecutive frame images played per second in the target video; A ratio of the number of frame images in the frame image sequence to the frame rate is determined as the duration of the target video, and the duration is determined as attribute information of the target video.

12. A video generation device, characterized in that: The device comprises: an acquisition module, configured to determine the acquired initial image as the first frame image in a frame image sequence to be generated, and to acquire description text of each frame image in the frame image sequence; a generating module, configured to generate the i-th frame image based on a description text of the i-th frame image in the frame image sequence, each frame image in the frame image sequence that is located before the i-th frame image, and each frame image that is located before the i-th frame image; wherein i is a positive integer greater than 1, and i is less than or equal to the number of frame images in the frame image sequence; A traversal module, configured to traverse the i to obtain each frame image in the frame image sequence; The synthesis module is used to perform video synthesis on each frame image in the frame image sequence to obtain a target video.

13. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the video generation method according to any one of claims 1 to 11 when executing the computer-executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer-executable instructions are executed by a processor, the video generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions, characterized in that When the computer-executable instructions are executed by a processor, the video generation method according to any one of claims 1 to 11 is implemented.

Images