Method and device for generating video from text, computer device and storage medium

Abstract

The application discloses a method for generating a video based on text information. The method comprises the following steps: searching a video library based on the text information to obtain a candidate picture set matched with the text information; generating an image feature sequence with time sequence based on the text information, and searching a picture sequence matched with the image feature sequence from the candidate picture set based on the image feature sequence, wherein, the pictures in the picture sequence have a time sequence relationship. The application can improve the matching degree of the video and the text.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a method, apparatus, computer device, and storage medium for generating video from text. Background Technology

[0002] With the rise of video sharing platforms, more and more video creators are enthusiastically using their own written summaries to create videos. In professional video creation, the first and crucial step is translating the written summary into a video storyboard. A video storyboard is a series of images, shot by shot, that visualizes the key plot points in the script. However, creating a high-quality video storyboard is a challenge for amateurs. It requires not only organizing the relevant scenes, characters, and actions in the video, but also arranging keyframes according to film shooting requirements, such as smooth transitions between shots. Therefore, there is a high demand for applications that help amateurs create more professional video storyboards from their written summaries.

[0003] While significant progress has been made in text-to-video retrieval and even text-to-video generation, these technologies remain limited in creating storyboards from text. For example, text-to-video work can retrieve or generate videos, but the resulting videos often contain highly redundant images that fail to meet the coherence requirements of video storyboard keyframes. Summary of the Invention

[0004] In view of this, a method, apparatus, computer device, and computer-readable storage medium for generating video from text are provided to solve the problems of highly redundant images and insufficient continuity between video frames in existing technologies that are generated from text summaries.

[0005] This application provides a method for generating video from text, including: Obtain text information, which is a summary description of the video storyboard; The video library is searched based on the text information to obtain a set of candidate images that match the text information; A temporally ordered series of image features is generated based on the text information, and an image sequence matching the image feature sequence is retrieved from the candidate image set based on the image feature sequence, wherein there is a temporal relationship between the images in the image sequence.

[0006] Optionally, the step of searching the video library based on the text information to obtain a candidate image set matching the text information includes: The text information is input into a pre-trained text-image model, and the text feature extraction layer in the text-image model extracts text features to obtain a text feature vector. Image features are extracted from all video frames in the video library through the image feature extraction layer in the text-image model to obtain multiple image feature vectors; The similarity between the text feature vector and each image feature vector is calculated, and the video frames corresponding to the image feature vectors that meet the preset similarity conditions are used as candidate images in the candidate image set that match the text information.

[0007] Optionally, the method further includes: Acquire multiple training sample data, each training sample data including a text sample and the corresponding image sample of the text sample; The text-image model is obtained by fine-tuning the pre-trained model based on multiple training sample data, wherein the pre-trained model is the model obtained by training the CLIP model with a dataset in the corpus.

[0008] Optionally, generating a temporally ordered series of image features based on the text information includes: The text information is input into a pre-trained encoder-decoder module. The encoder module encodes the text information to obtain an encoded feature vector, and the encoded feature vector is then input into the decoder module. The decoder module predicts image features based on the encoded feature vector to generate a temporal sequence of image features.

[0009] Optionally, the decoder module includes a cross-modal attention mechanism module, wherein the step of predicting image features based on the encoded feature vector by the decoder module to generate a temporal series of image features includes: The decoder module uses the cross-modal attention features obtained from the cross-modal attention mechanism module to predict image features based on the encoded feature vector, thereby generating a temporal sequence of image features.

[0010] Optionally, the method further includes: The weight parameters of the decoder module in the untrained encoder-decoder module are initialized using the weight parameters of the pre-trained decoder module; Acquire multiple training sample data, each training sample data including a text sample and a sequence of sample images corresponding to the text sample; The encoder-decoder module, after initialization, is trained multiple times based on multiple training sample data to obtain a trained encoder-decoder module.

[0011] Optionally, retrieving an image sequence matching the image feature sequence from the candidate image set based on the image feature sequence includes: Calculate the similarity between each image feature vector in the image feature sequence and the image feature vector corresponding to each candidate image in the candidate image set, and take the candidate image with the highest similarity value as the candidate image that matches each of the image feature vectors.

[0012] Optionally, the method further includes: The image sequence is synthesized into a video according to the temporal relationship between the images in the image sequence.

[0013] This application also provides a text-to-video device, comprising: The acquisition module is used to acquire text information, which is a summary description of the video storyboard; The retrieval module is used to retrieve the video library based on the text information and obtain a set of candidate images that match the text information; The generation module is used to generate a temporally ordered series of image features based on the text information, and to retrieve an image sequence that matches the image feature sequence from the candidate image set based on the image feature sequence, wherein there is a temporal relationship between the images in the image sequence.

[0014] This application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0015] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] In this embodiment, text information is acquired, which is a summary description of the video storyboard. A video library is searched based on the text information to obtain a set of candidate images matching the text information. A temporally ordered series of image features is generated based on the text information, and an image sequence matching the image feature sequence is retrieved from the candidate image set based on the image feature sequence. The images in the image sequence have a temporal relationship. Using this text-to-video method, video frames matching the text information can be extracted from different videos, resulting in no redundant images in the generated image sequence, strong coherence between the images, and a high degree of matching between the resulting image sequence and the text information. Attached Figure Description

[0017] Figure 1 An environmental schematic diagram of an embodiment of the text-to-video method of this application; Figure 2 A flowchart of one embodiment of the text-to-video method described in this application; Figure 3 This is a detailed flowchart illustrating the steps of retrieving a video library based on the text information to obtain a set of candidate images that match the text information, according to an embodiment of this application. Figure 4 A flowchart of another embodiment of the method for generating video from text described in this application; Figure 5 This is a schematic diagram of the encoder-decoder structure according to an embodiment of this application; Figure 6 This is a detailed flowchart illustrating the steps of generating a temporally ordered series of image features based on the text information according to an embodiment of this application. Figure 7 A flowchart of another embodiment of the method for generating video from text described in this application; Figure 8 A program block diagram of one embodiment of the text-to-video generation apparatus described in this application; Figure 9 This is a schematic diagram of the hardware structure of a computer device for performing a method of generating video from text, as provided in an embodiment of this application. Detailed Implementation

[0018] The advantages of this application are further illustrated below with reference to the accompanying drawings and specific embodiments.

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0020] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0023] The following provides an exemplary application environment for this application. Figure 1 The illustration shows an environmental application diagram of the text-to-video method according to an embodiment of this application.

[0024] In an exemplary embodiment, the system of this application environment may include a computer device 10 and a server 20. The computer device 10 and the server 20 are connected via a wireless or wired network. The computer device 10 may be, for example, a smartphone, tablet, laptop computer, smart TV, or in-vehicle terminal. The server 20 may be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers). The network may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, and / or proxy devices. The network may also include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, and combinations thereof and / or the like.

[0025] Below, several embodiments will be provided in the above exemplary application environment to illustrate the text-to-video conversion scheme of this application.

[0026] See Figure 2 This is a flowchart illustrating a method for generating video from text according to an embodiment of this application. It is understood that the flowchart in this method embodiment is not intended to limit the order of execution steps. As can be seen from the figure, the method for generating video from text provided in this embodiment includes: Step S21: Obtain text information, which is a summary description of the video storyboard.

[0027] Specifically, the text information is a summary description of the video storyboard. A video storyboard is a series of images shot by shot; that is, a video storyboard is essentially a sequence of images. In this embodiment, when a user needs to generate a sequence of images, the user can first provide a summary description of the generated video storyboard using text. Then, the user can use the method of generating a video from the text to generate the image sequence based on the summary description.

[0028] In one embodiment, the text information can be entered by providing a text input interface. In another embodiment, the text information can be imported by providing a file import interface.

[0029] It should be noted that this sequence of images can serve as a storyboard. A storyboard, also known as a panel, refers to a visual representation of the composition of an image in various media such as film, animation, television series, commercials, and music videos, before actual shooting or drawing. It breaks down a continuous sequence of shots into multiple images, with each shot representing a single movement.

[0030] Step S21: Search the video library based on the text information to obtain a set of candidate images that match the text information.

[0031] Specifically, the video library stores multiple videos, which can be movies, TV series, variety shows, etc.

[0032] In this embodiment, when searching the video library based on text information, the matching degree between the text information and the video frames in each video in the video library can be compared sequentially, and the video frame with a good matching degree can be used as a candidate image in the candidate image set that matches the text information.

[0033] In one implementation, a pre-trained text-image model can be used to retrieve candidate image datasets.

[0034] The text-image model can perform image retrieval on all video frames in the video library to obtain a candidate image set that matches the text information. In this embodiment, the text-image model can be a CLIP (Contrastive Language-Image Pre-training) model. CLIP is a pre-training method or model based on contrastive text-image pairs. It is a multimodal model based on contrastive learning. Unlike some contrastive learning methods in computer vision, such as MoCo and SimCLR, CLIP's training data consists of text-image pairs (containing an image and its corresponding text description). Through contrastive learning, the CLIP model can learn the matching relationship between text-image pairs. The CLIP model includes two models: a Text Encoder and an Image Encoder. The Text Encoder extracts text features and can use a Transformer model commonly used in NLP; while the Image Encoder extracts image features and can use a common CNN model or Vision Transformer.

[0035] In one exemplary implementation, see [reference] Figure 3 The step of retrieving a video library based on the text information to obtain a candidate image set matching the text information includes: Step S30, inputting the text information into a pre-trained text-image model, and extracting text features from the text information through the text feature extraction layer in the text-image model to obtain a text feature vector; Step S31, extracting image features from all video frames in the video library through the image feature extraction layer in the text-image model to obtain multiple image feature vectors; Step S32, calculating the similarity between the text feature vector and each image feature vector, and using the video frames corresponding to the image feature vectors that meet the preset similarity conditions as candidate images in the candidate image set matching the text information.

[0036] Specifically, the text feature extraction layer in the text-image model is a text encoder, and the image feature extraction layer in the text-image model is an image encoder. In this embodiment, when text information is input into the text-image model, the text encoder extracts features from the text information, thereby converting the text information into a text feature vector. Simultaneously, to achieve matching between text information and images, the image encoder extracts image features from all video frames in the video library, thereby converting each video frame into a corresponding image feature vector.

[0037] In this embodiment, after obtaining the text feature vector and the image feature vectors of all video frames, the similarity between the text feature vector and each image feature vector can be calculated. For example, there are a total of 100 video frames, namely the 1st video frame, the 2nd video frame, ..., the 99th video frame, and the 100th video frame. After obtaining the text feature vector and the image feature vectors corresponding to these 100 video frames, it is necessary to calculate the similarity between the text feature vector and the image feature vector corresponding to the 1st video frame, the text feature vector and the image feature vector corresponding to the 2nd video frame, the text feature vector and the image feature vector corresponding to the 3rd video frame, ..., the text feature vector and the image feature vector corresponding to the 99th video frame, and the text feature vector and the image feature vector corresponding to the 100th video frame. After calculating 100 similarity values, it can be determined whether these 100 similarity values ​​meet the preset similarity conditions. If the current similarity value meets the similarity conditions, it can be determined that the video frame corresponding to the image feature vector associated with the similarity value is a candidate image in the candidate image set that matches the text information.

[0038] In one embodiment, the preset similarity condition can be to select the top K elements with the highest similarity, such as selecting the 20 most similar images. In another embodiment, the preset similarity condition can also be to select elements whose similarity values ​​exceed a similarity threshold; for example, if the similarity threshold is 90%, then all images with a similarity greater than 90% can be selected.

[0039] In one exemplary implementation, see [reference] Figure 4 The method further includes: step S40, acquiring multiple training sample data, each training sample data including a text sample and an image sample corresponding to the text sample; step S41, fine-tuning the pre-trained model based on the multiple training sample data to obtain the text-image model, wherein the pre-trained model is a model obtained by training the CLIP model using a dataset in a corpus.

[0040] Specifically, the plurality of training sample data includes positive training sample data and negative training sample data. Positive training sample data refers to a text-image pair consisting of a video frame extracted from a video generated based on target text information and the target text information, while negative training sample data refers to a text-image pair consisting of a video frame extracted from other videos and the target text information.

[0041] The pre-trained model is a model obtained by training the CLIP model using a dataset from a corpus, such as the CLIP-VIT-B / 32 model.

[0042] The dataset in the corpus can be obtained in the following way: First, multiple pairs of text summary paragraphs and movie clips were obtained from the dataset. The obtained text summary paragraphs consisted of an average of 8 sentences, and each clip contained 95 shots.

[0043] Secondly, a paragraph is divided into several sentences, and then each sentence is aligned with the minimum number of keyframes in the movie clip.

[0044] Finally, the annotators selected sentences to form a text summary, and then selected the minimum number of images as the visual text, thus obtaining the dataset.

[0045] In this embodiment, during the fine-tuning of the pre-trained model, it can be trained multiple times based on multiple training sample data. During training, a contrastive learning method is used, employing Info-NCE loss to maximize the similarity between sample text and positive sample images (image samples in the positive training sample data), while minimizing the similarity between sample text and negative sample images (negative training sample data). When the loss function meets the set requirements, the fine-tuning of the pre-trained model is completed, resulting in the text-image model.

[0046] As an example, during fine-tuning training, the model randomly selects one frame from the real keyframe sequence corresponding to the text information as a positive text-image pair, and then randomly selects one frame from other keyframe sequences in the batch as a negative image sample. Then, a contrastive learning loss can be used to maximize the similarity between the matched image and text in the positive examples, while minimizing the similarity between the mismatched image and text. The loss function is shown below:

[0047]

[0048] in and It is the size of The first batch The image and the first The normalized features of the sentence, and It is the temperature coefficient, the overall text-image alignment loss. yes and The average.

[0049] Step S22: Generate a temporally ordered image feature series based on the text information, and retrieve an image sequence matching the image feature sequence from the candidate image set based on the image feature sequence, wherein there is a temporal relationship between the images in the image sequence.

[0050] Specifically, when generating image feature sequences based on text information, an encoder-decoder architecture can be used to convert text information into image sequences.

[0051] As an example, see Figure 5 , can be Figure 5 The encoder-decoder model shown generates a temporal sequence of image features based on the text information. This encoder-decoder model consists of a Transformer encoder for text encoding. With a decoder for image feature prediction composition. A cross-attention mechanism can be used to perform autoregressive predictions of image features based on textual information. These autoregressive predicted image features can be used to retrieve images from a candidate image set using cosine similarity. The model is optimized with NCE loss in each prediction step and randomly samples negative examples from a mini-batch. The NCE loss function is shown below: , It is from the batch from the first The sequence of the first... The normalized features of an image These are the normalized features of negative example images obtained from batch sampling. The encoder-decoder module can retrieve and sort image feature sequences.

[0052] In one exemplary implementation, see [reference] Figure 6 The step of generating a temporally ordered image feature series based on the text information includes: step S60, inputting the text information into a pre-trained encoder-decoder module, encoding the text information through the encoder module to obtain an encoded feature vector, and inputting the encoded feature vector into the decoder module; step S61, using the decoder module to predict image features based on the encoded feature vector to generate a temporally ordered image feature series.

[0053] Specifically, when encoding text information, the encoder module can encode the text information into encoded feature vectors by querying a pre-established word vector lookup table, where the lookup table contains the word vectors corresponding to each word. Specifically, the encoder first converts the text information into word vectors, and then further maps the word vectors in semantic space using structures such as Transformer to obtain the encoded feature vectors of the text.

[0054] After obtaining the encoded feature vector, the decoder module uses autoregressive prediction to decode the encoded feature vector, thereby predicting image features and generating a time-series image feature sequence of variable length.

[0055] In this embodiment, the multiple image feature vectors contained in the image feature sequence have a temporal relationship, which is used to determine the order of the various video frames that make up the video in the video when the video is generated.

[0056] In this embodiment, the image feature sequence is generated by the encoder-decoder module, which can accurately sort the individual image feature sequences in the image feature sequence, and at the same time, can avoid the generated image feature sequence being too long.

[0057] In one exemplary embodiment, to improve the accuracy of the decoding process, the decoder module includes a cross-modal attention mechanism module. The step of predicting image features based on the encoded feature vector by the decoder module to generate a temporal image feature series includes: predicting image features based on the encoded feature vector by the decoder module using the cross-modal attention features obtained by the cross-modal attention mechanism module to generate a temporal image feature series.

[0058] Specifically, after the encoder module encodes the encoded feature vector, the cross-modal attention mechanism module performs feature fusion processing on the encoded feature vector and the image features obtained during the decoding process to obtain cross-modal attention features. These cross-modal attention features can provide both textual and image information for the image decoding process. In this way, when the decoder module predicts image features based on the encoded feature vector, it can obtain a more accurate image feature sequence.

[0059] In one exemplary embodiment, to improve the training efficiency of the encoder-decoder module, see [reference needed]. Figure 7The method further includes: step S70, initializing the weight parameters of the decoder module in the untrained encoder-decoder module using the weight parameters of the pre-trained decoder module; step S71, acquiring multiple training sample data, each training sample data including a text sample and a sequence of sample images corresponding to the text sample; step S72, training the initialized encoder-decoder module multiple times based on the multiple training sample data to obtain a trained encoder-decoder module.

[0060] Specifically, the network structure of the pre-trained decoder module is basically the same as that of the decoder module in the final trained encoder-decoder module. The only difference is that the pre-trained decoder module does not contain a cross-modal attention mechanism module, while the decoder module in the final trained encoder-decoder module does contain a cross-modal attention mechanism module.

[0061] In this embodiment, the pre-trained decoder module can be pre-trained using a large-scale movie dataset to learn the temporal relationships between adjacent video frames in film shot language. After completing the pre-training of the decoder module, the weight parameters of the pre-trained decoder module can be used to initialize the weight parameters of the decoder module in the untrained encoder-decoder module, thereby eliminating the need to retrain the encoder-decoder module from scratch and improving training efficiency.

[0062] In this embodiment, by pre-training the decoder module with a large number of movie frame sequences without using text annotations, the training scale can be easily expanded, allowing the pre-trained decoder module to better learn the temporal relationships between images. At the same time, large-scale pre-training can also improve the performance of the model.

[0063] In one embodiment, the following method can also be used to sort the images in the candidate image set: Method 1, CLIP-Naive: We use CLIP to calculate the similarity between text summary features and corresponding keyframe features, and then rank the keyframes based on the similarity.

[0064] Method 2, CLIP-Sliding: We first segment the sentence into several segments, treating them as a set of queries, where the number of segments equals the number of their corresponding keyframes. Then, we use a sliding window to sequentially retrieve the most similar keyframes for each sentence segment. Once a keyframe is selected, it is removed from the candidate queue.

[0065] Method 3, CLIP-Cumulative: We first segment the sentence into several segments, treating them as a set of queries. However, during the retrieval process, we concatenate the preceding segments before searching for the most similar keyframes. This approach takes into account more context. For example, to retrieve the second keyframe, we use the result of concatenating the first two segments as the query. We also remove keyframes from the candidate queue after they are selected.

[0066] In one exemplary embodiment, retrieving an image sequence matching the image feature sequence from the candidate image set based on the image feature sequence includes: calculating the similarity between each image feature vector contained in the image feature sequence and the image feature vector corresponding to each candidate image in the candidate image set, and taking the candidate image with the highest similarity value as the candidate image matching each of the image feature vectors.

[0067] Specifically, in the process of acquiring candidate images, it is first necessary to calculate the similarity between the current image feature vector and the image feature vector corresponding to each candidate image in each candidate image set. Then, the calculated similarity values ​​are sorted so that the largest similarity value can be selected. After obtaining the largest similarity value, the candidate image associated with the largest similarity value can be used as the candidate image that matches the current image feature vector.

[0068] Similarly, after obtaining the first candidate image whose feature vector matches the first image, the same method is used to obtain the second candidate image whose feature vector matches the first image, until all candidate images whose feature vectors match the first image are obtained.

[0069] In one exemplary embodiment, the method further includes: synthesizing a video storyboard from the image sequence according to the temporal relationship between the images in the image sequence.

[0070] Specifically, if an image in the image sequence is the first in the sequence, it will be used as the first frame of the synthesized video storyboard. If an image in the image sequence is the second in the sequence, it will be used as the second frame. If an image in the image sequence is the Nth in the sequence, it will be used as the Nth frame of the synthesized video storyboard.

[0071] In this embodiment, text information is acquired, which is a summary description of the video storyboard. A video library is searched based on the text information to obtain a set of candidate images matching the text information. A temporally ordered series of image features is generated based on the text information, and an image sequence matching the image feature sequence is retrieved from the candidate image set based on the image feature sequence. The images in the image sequence have a temporal relationship. Using this text-to-video method, video frames matching the text information can be extracted from different videos, resulting in no redundant images in the generated image sequence, strong coherence between the images, and a high degree of matching between the resulting image sequence and the text information.

[0072] See Figure 8 The diagram shown is a program block diagram of an embodiment of the text-to-video device 80 of this application.

[0073] In this embodiment, the text-to-video device 80 includes a series of computer program instructions stored in a memory. When these computer program instructions are executed by a processor, the shooting functions of the various embodiments of this application can be implemented. In some embodiments, based on the specific operations implemented by each part of the computer program instructions, the text-to-video device 80 can be divided into one or more modules, specifically as follows: The acquisition module 81 is used to acquire text information, which is a summary description of the video storyboard; The retrieval module 82 is used to retrieve the video library based on the text information and obtain a set of candidate images that match the text information; The generation module 83 is used to generate a temporally ordered series of image features based on the text information, and to retrieve an image sequence that matches the image feature sequence from the candidate image set based on the image feature sequence, wherein there is a temporal relationship between the images in the image sequence.

[0074] In an exemplary embodiment, the retrieval module 82 is further configured to input the text information into a pre-trained text-image model, extract text features from the text information through the text feature extraction layer in the text-image model to obtain a text feature vector; extract image features from all video frames in the video library through the image feature extraction layer in the text-image model to obtain multiple image feature vectors; calculate the similarity between the text feature vector and each image feature vector, and use the video frames corresponding to the image feature vectors that meet the preset similarity conditions as candidate images in the candidate image set that match the text information.

[0075] In one exemplary embodiment, the text-to-video apparatus 80 further includes an adjustment module.

[0076] The acquisition module also acquires multiple training sample data, each training sample data including a text sample and an image sample corresponding to the text sample.

[0077] The adjustment module is used to fine-tune the pre-trained model based on multiple training sample data to obtain the text-image model, wherein the pre-trained model is the model obtained by training the CLIP model with a dataset in the corpus.

[0078] In an exemplary embodiment, the generation module 83 is further configured to input the text information into a pre-trained encoder-decoder module, encode the text information through the encoder module to obtain an encoded feature vector, and input the encoded feature vector into the decoder module; the decoder module predicts image features based on the encoded feature vector to generate a temporal image feature series.

[0079] In one exemplary embodiment, the decoder module includes a cross-modal attention mechanism module and a generation module 83, which is further configured to predict image features based on the encoded feature vector by the decoder module using the cross-modal attention features obtained by the cross-modal attention mechanism module, thereby generating a temporal sequence of image features.

[0080] In one exemplary embodiment, the text-to-video device 80 further includes an initialization module and a training module.

[0081] The initialization module is used to initialize the weight parameters of the decoder module in the untrained encoder-decoder module using the weight parameters of the pre-trained decoder module. The acquisition module 81 is also used to acquire multiple training sample data, each training sample data including a text sample and a sequence of sample images corresponding to the text sample; The training module is used to train the initialized encoder-decoder module multiple times based on multiple training sample data to obtain a trained encoder-decoder module.

[0082] In an exemplary embodiment, the generation module 83 is further configured to calculate the similarity between each image feature vector contained in the image feature sequence and the image feature vector corresponding to each candidate image in the candidate image set, and to take the candidate image with the largest similarity value as the candidate image that matches each of the image feature vectors.

[0083] In one exemplary embodiment, the generation module 83 is further configured to synthesize a video storyboard by combining the image sequence according to the temporal relationship between the images in the image sequence.

[0084] Figure 9 This illustration schematically depicts the hardware architecture of a computer device 9 suitable for implementing a method of text-to-video generation according to an embodiment of this application. In this embodiment, the computer device 9 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it may be a tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 9 As shown, the computer device 9 includes, but is not limited to, at least: a memory 120, a processor 121, and a network interface 122 that can communicate with each other via a system bus. Wherein: The memory 120 includes at least one type of computer-readable storage medium, which can be volatile or non-volatile. Specifically, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 120 can be an internal storage module of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 120 can also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 120 can also include both the internal storage module and the external storage device of the computer device 9. In this embodiment, the memory 120 is typically used to store the operating system and various application software installed on the computer device 9, such as program code for a method of generating video from text. In addition, the memory 120 can also be used to temporarily store various types of data that have been output or will be output.

[0085] In some embodiments, processor 121 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other imaging chip. This processor 121 is typically used to control the overall operation of the computer device 9, such as performing control and processing related to data interaction or communication with the computer device 9. In this embodiment, processor 121 is used to run program code stored in memory 120 or process data.

[0086] Network interface 122 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 9 and other computer devices. For example, network interface 122 is used to connect computer device 9 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 9 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0087] It should be pointed out that, Figure 9 Only computer devices with components 120 to 122 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0088] In this embodiment, the method for generating video from text stored in memory 120 can be divided into one or more program modules and executed by one or more processors (processor 121 in this embodiment) to complete this application.

[0089] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the text-to-video method described in the embodiment.

[0090] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the text-to-video method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across at least two network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of this application. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating video from text, characterized in that, include: Obtain text information, which is a summary description of the video storyboard; The video library is searched based on the text information to obtain a set of candidate images that match the text information; A temporally ordered image feature sequence is generated based on the text information, and an image sequence matching the image feature sequence is retrieved from the candidate image set based on the image feature sequence, wherein there is a temporal relationship between the images in the image sequence; The step of generating a temporally ordered image feature sequence based on the text information includes: The text information is input into a pre-trained encoder-decoder module. The encoder module encodes the text information to obtain an encoded feature vector, and the encoded feature vector is then input into the decoder module. The decoder module predicts image features based on the encoded feature vector to generate a temporal sequence of image features. The decoder module employs an autoregressive prediction method to decode the encoded feature vector in order to predict and generate the time-series image feature sequence of variable length.

2. The method for generating video from text according to claim 1, characterized in that, The step of searching the video library based on the text information to obtain a candidate image set that matches the text information includes: The text information is input into a pre-trained text-image model, and the text feature extraction layer in the text-image model extracts text features to obtain a text feature vector. Image features are extracted from all video frames in the video library through the image feature extraction layer in the text-image model to obtain multiple image feature vectors; The similarity between the text feature vector and each image feature vector is calculated, and the video frames corresponding to the image feature vectors that meet the preset similarity conditions are used as candidate images in the candidate image set that match the text information.

3. The method for generating video from text according to claim 1, characterized in that, The method further includes: Acquire multiple training sample data, each training sample data including a text sample and the corresponding image sample of the text sample; The pre-trained model is fine-tuned based on multiple training sample data to obtain a text-image model. The pre-trained model is the model obtained by training the CLIP model using a dataset from a corpus.

4. The method for generating video from text according to claim 1, characterized in that, The decoder module includes a cross-modal attention mechanism module. The step of predicting image features based on the encoded feature vector using the decoder module to generate a temporal sequence of image features includes: The decoder module uses the cross-modal attention features obtained from the cross-modal attention mechanism module to predict image features based on the encoded feature vector, generating a temporal image feature sequence.

5. The method for generating video from text according to claim 4, characterized in that, The method further includes: The weight parameters of the decoder module in the untrained encoder-decoder module are initialized using the weight parameters of the pre-trained decoder module; Acquire multiple training sample data, each training sample data including a text sample and a sequence of sample images corresponding to the text sample; The encoder-decoder module, after initialization, is trained multiple times based on multiple training sample data to obtain a trained encoder-decoder module.

6. The method for generating video from text according to claim 1, characterized in that, The step of retrieving an image sequence that matches the image feature sequence from the candidate image set includes: Calculate the similarity between each image feature vector in the image feature sequence and the image feature vector corresponding to each candidate image in the candidate image set, and take the candidate image with the highest similarity value as the candidate image that matches each of the image feature vectors.

7. The method for generating video from text according to claim 1, characterized in that, The method further includes: The image sequence is synthesized into a video storyboard according to the temporal relationship between the images in the image sequence.

8. An apparatus for generating video from text, characterized in that, The device includes: The acquisition module is used to acquire text information, which is a summary description of the video storyboard; The retrieval module is used to retrieve the video library based on the text information and obtain a set of candidate images that match the text information; The generation module is used to generate a temporally ordered image feature sequence based on the text information, and to retrieve an image sequence that matches the image feature sequence from the candidate image set based on the image feature sequence, wherein there is a temporal relationship between the images in the image sequence; The step of generating a temporally ordered image feature sequence based on the text information includes: The text information is input into a pre-trained encoder-decoder module. The encoder module encodes the text information to obtain an encoded feature vector, and the encoded feature vector is then input into the decoder module. The decoder module predicts image features based on the encoded feature vector to generate a temporal sequence of image features. The decoder module employs an autoregressive prediction method to decode the encoded feature vector in order to predict and generate the time-series image feature sequence of variable length.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 7.

Images