Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology and artificial intelligence. The method comprises the following steps: performing video frame prediction on a word segmentation text sequence through a bidirectional attention layer to obtain an initial video frame; performing word segmentation on the initial video frame to obtain a word segmentation video sequence; performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through an autoregression layer to obtain a target video sequence; and generating a target video according to the target video sequence. According to the method, firstly, video frame prediction is performed on the word segmentation text sequence, so that the correlation of video generation contents is improved; secondly, the initial video frame is subjected to word segmentation, and rich information representation is provided for subsequent video prediction; and finally, video sequence prediction is performed on the word segmentation sequence, and a target video is generated, so that the video can be constructed step by step through multiple iterations, the consistency of the generated video in time is ensured, and the accuracy of video generation is effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of financial technology and artificial intelligence technology, and in particular to a video generation method and device, an electronic device and a storage medium. Background Art

[0002] At present, the input video data is usually encoded into a low-dimensional latent space representation through a variational autoencoder, and then the target video is generated by decoding the latent space representation. For example, in the financial technology scenario, if the vehicle claim is caused by a traffic accident due to slippery roads in rainy days, the key features of driving in rainy days (such as road reflections, blurred vision, and vehicle slippage) are obtained, and a variational autoencoder is used to generate a rainy driving simulation video containing the moment of the accident when the slippery roads occurred. However, due to the randomness of the latent space representation, this method makes the generated video content lack stability, and it is easy to have incoherent video content or degraded quality, resulting in low accuracy of video generation. Therefore, how to improve the accuracy of video generation has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a video generation method and device, an electronic device and a storage medium, aiming to improve the accuracy of video generation.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a video generation method, the method comprising:

[0005] Acquire target indication information; wherein the target indication information includes target text indication information;

[0006] Performing word segmentation processing on the target text indication information to obtain a word segmentation text sequence;

[0007] Obtain a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer;

[0008] Performing video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame;

[0009] Using a pre-trained video segmentation model to segment the initial video frame to obtain a segmentation video sequence;

[0010] Performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence;

[0011] A target video is generated according to the target video sequence.

[0012] In some embodiments, performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence includes:

[0013] Performing a first frame video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a first frame video sequence;

[0014] Iterative video sequence prediction is performed on the first frame video sequence, the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer until a preset prediction end mark appears, thereby obtaining the target video sequence.

[0015] In some embodiments, performing video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame includes:

[0016] Encoding the word segmentation text sequence through the bidirectional attention layer to obtain an encoded text sequence;

[0017] Video frame prediction is performed on the encoded text sequence to obtain the initial video frame.

[0018] In some embodiments, the using a pre-trained video word segmentation model to segment the initial video frame to obtain a word segmentation video sequence includes:

[0019] Using the video word segmentation model to extract features from the initial video frame to obtain video frame features;

[0020] The video frame features are subjected to quantized word segmentation processing to obtain the word segmentation video sequence.

[0021] In some embodiments, the step of performing quantized word segmentation processing on the video frame features to obtain the word segmentation video sequence includes:

[0022] Obtaining a preset codebook; the codebook includes codebook elements;

[0023] Constructing a codebook index for the video frame feature, and determining a target codebook element according to the codebook index and the codebook element;

[0024] The video frame features are mapped to the target codebook elements to obtain the word segmentation video sequence.

[0025] In some embodiments, generating a target video according to the target video sequence includes:

[0026] Converting the target video sequence into a target video frame;

[0027] The target video frames are combined in sequence according to a preset time sequence to obtain the target video.

[0028] In some embodiments, obtaining a trained video generation model includes:

[0029] Obtaining a training data set; the training data set includes training text and training video;

[0030] Performing word segmentation processing on the training text to obtain a word segmentation training text sequence;

[0031] Performing word segmentation processing on the training video to obtain a word segmentation training video sequence;

[0032] Performing video prediction on the training text and the training video using a pre-trained video generation model to obtain a predicted video;

[0033] Using a preset loss function to calculate the loss of the predicted video and the training video to obtain a loss value;

[0034] The parameters in the pre-trained video generation model are adjusted according to the loss value until the loss value meets a preset threshold, thereby obtaining the trained video generation model.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video generation device, the device comprising:

[0036] An indication information acquisition module, used to acquire target indication information; wherein the target indication information includes target text indication information;

[0037] A text segmentation module, used for performing segmentation processing on the target text indication information to obtain a segmented text sequence;

[0038] A model acquisition module is used to acquire a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer;

[0039] An initial video frame prediction module, used to perform video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame;

[0040] A video word segmentation module, used to segment the initial video frame using a pre-trained video word segmentation model to obtain a word segmentation video sequence;

[0041] A video sequence prediction module, used for performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence;

[0042] The video generation module is used to generate a target video according to the target video sequence.

[0043] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] The video generation method and device, electronic device and storage medium proposed in the present application obtain target indication information; wherein the target indication information includes target text indication information; segmentation processing is performed on the target text indication information to obtain a segmentation text sequence; a trained video generation model is obtained; the video generation model includes a bidirectional attention layer and an autoregressive layer; video frame prediction is performed on the segmentation text sequence through the bidirectional attention layer to obtain an initial video frame; the initial video frame is segmented using a pre-trained video segmentation model to obtain a segmentation video sequence; video sequence prediction is performed on the segmentation text sequence and the segmentation video sequence through the autoregressive layer to obtain a target video sequence; and a target video is generated according to the target video sequence. The present application firstly predicts video frames of the segmented text sequence through a bidirectional attention layer, and can generate video frames related to the indicative information, thereby improving the relevance of the video generation content; secondly, the initial video frame is segmented by a pre-trained video segmentation model, which provides a richer information representation for subsequent video sequence prediction, thereby facilitating the subsequent generation of more coherent and high-quality video content; finally, the video sequence is predicted for the segmented text sequence and the segmented video sequence through an autoregressive layer, and a target video is generated according to the predicted video sequence, which can realize the gradual construction of video content through multiple iterations, ensure the temporal coherence of the generated video, and always keep consistent with the input text information, thereby effectively improving the accuracy of video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a video generation method provided in an embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0048] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0049] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0050] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0051] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0052] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0053] Figure 8 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0054] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] First, some nouns involved in this application are analyzed:

[0059] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] Based on this, the embodiments of the present application provide a video generation method and device, an electronic device and a storage medium, aiming to improve the accuracy of video generation.

[0061] The video generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video generation method in the embodiments of the present application is described.

[0062] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0064] The video generation method provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The video generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video generation method, etc., but is not limited to the above forms.

[0065] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0066] Figure 1 is an optional flowchart of the video generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0067] Step S101, obtaining target indication information; wherein the target indication information includes target text indication information.

[0068] Step S102, performing word segmentation processing on the target text indication information to obtain a word segmentation text sequence.

[0069] Step S103, obtaining a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer.

[0070] Step S104, performing video frame prediction on the word segmentation text sequence through a bidirectional attention layer to obtain an initial video frame.

[0071] Step S105, using a pre-trained video segmentation model to segment the initial video frame to obtain a segmented video sequence.

[0072] Step S106, performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through an autoregressive layer to obtain a target video sequence.

[0073] Step S107, generating a target video according to the target video sequence.

[0074] Steps S101 to S107 shown in the embodiment of the present application are performed by acquiring target indication information; wherein the target indication information includes target text indication information; performing word segmentation processing on the target text indication information to obtain a word segmentation text sequence; acquiring a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer; performing video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame; performing word segmentation on the initial video frame using a pre-trained video segmentation model to obtain a word segmentation video sequence; performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence; and generating a target video according to the target video sequence. The present application firstly predicts video frames of the segmented text sequence through a bidirectional attention layer, and can generate video frames related to the indicative information, thereby improving the relevance of the video generation content; secondly, the initial video frame is segmented by a pre-trained video segmentation model, which provides a richer information representation for subsequent video sequence prediction, thereby facilitating the subsequent generation of more coherent and high-quality video content; finally, the video sequence is predicted for the segmented text sequence and the segmented video sequence through an autoregressive layer, and a target video is generated according to the predicted video sequence, which can realize the gradual construction of video content through multiple iterations, ensure the temporal coherence of the generated video, and always keep consistent with the input text information, thereby effectively improving the accuracy of video generation.

[0075] In step S101 of some embodiments, specifically, the target indication information includes target text indication information, wherein the target text indication information is used to indicate the video generation model to generate a corresponding video of the text indicator.

[0076] For example, in a fintech scenario, the target text indicates that a car collided with a truck at an intersection. The video generation model uses the target indication information as a prefix to learn the association between the text and related video data and generate a video that reproduces the accident scene.

[0077] Specifically, the target indicator information may also include text indication information and image indication information.

[0078] For example, in the fintech scenario, the text indication information may be a car skidding and hitting a bicycle on a rainy day, and the image indication information may be an image of a car driving on a rainy day and an image of a car hitting a bicycle, etc. The video generation model is used to take the text indication information and the image indication information as prefixes, learn the text and image information at the same time, and learn the association between the video data related to the text and image information, to reproduce the accident scene video of a car hitting an automatic car on a rainy day.

[0079] In step S102 of some embodiments, specifically, the segmented text sequence refers to a segmented text token sequence obtained after the target text indication information is segmented.

[0080] Specifically, the target text indication information can be segmented by a tokenizer text segmenter to obtain segmented text tokens, and each segmented text token is concatenated to obtain a segmented text sequence.

[0081] For example, in the FinTech scenario, the target text indication information is "a car insurance customer had a car accident while driving in the rain". The target text indication information is segmented into a series of text tokens through the tokenizer, namely "car insurance", "customer", "in", "rainy day", "driving", "when", "occurred", "car accident".

[0082] See also Figure 2 In some embodiments, step S103 includes but is not limited to steps S201 to S206:

[0083] Step S201, obtaining a training data set; the training data set includes training text and training video.

[0084] Step S202, performing word segmentation processing on the training text to obtain a word segmentation training text sequence.

[0085] Step S203, performing word segmentation processing on the training video to obtain a word segmentation training video sequence.

[0086] Step S204: Perform video prediction on the training text and the training video using a pre-trained video generation model to obtain a predicted video.

[0087] Step S205, using a preset loss function to calculate the loss of the predicted video and the training video to obtain a loss value.

[0088] Step S206, adjusting the parameters in the pre-trained video generation model according to the loss value until the loss value meets a preset threshold, thereby obtaining a trained video generation model.

[0089] In step S201 of some embodiments, the training data set may be a webvid-10m data set, which includes training text and training videos, and a training text description and a corresponding training video constitute a training data set. The training video has a duration of several seconds to more than ten seconds.

[0090] For example, in a fintech scenario, if the training text is a car and a truck colliding at an intersection, the corresponding training video can be a ten-second short video of the car and truck colliding recorded by a road monitor.

[0091] In step S202 of some embodiments, specifically, the word segmentation training text sequence refers to a training text token sequence after word segmentation of the training text.

[0092] Specifically, the training text is still segmented by the tokenizer text segmenter to obtain the segmentation training text token, and each segmentation training text token is concatenated to obtain the segmentation training text sequence.

[0093] In step S203 of some embodiments, the training video may be segmented using a token vocabulary of a video segmentation model (such as MAGVIT-v2) to obtain a segmentation training video token sequence, wherein the token vocabulary includes tokens that can represent different visual element information in the training video content.

[0094] Specifically, the training video is framed to obtain training video frames, features are extracted from the training video frames to obtain training video frame features, the extracted video frames are encoded into training video word segmentation tokens through MAGVIT-v2, and the encoded tokens are serialized to form an ordered word segmentation training video token sequence.

[0095] In step S204 of some embodiments, specifically, the pre-trained video generation model can be a Prefix LLM (Prefix Large Language Model), and the pre-trained video generation model includes a bidirectional attention layer and an autoregressive layer. Among them, the bidirectional attention layer is used to predict video frames for the word segmentation text sequence, and can simultaneously consider the prefix information and the generated video token sequence through the bidirectional attention mechanism, so as to realize the context learning of the word segmentation text sequence and improve the content relevance of the video generation; the autoregressive layer is used to predict the video sequence for the word segmentation text sequence and the word segmentation video sequence, and can predict the next video token according to the generated video token sequence and the text prefix, so as to realize the gradual construction of the video content through multiple iterations, ensure the temporal coherence of the generated video, and always keep consistent with the input text information, thereby improving the accuracy of the video generation.

[0096] Specifically, in the Prefix LLM model, the bidirectional attention layer shares the same Transformer structure, which means that the same neural network layer is used when processing input and output data.

[0097] In step S205 of some embodiments, specifically, the loss function may be a cross entropy loss function, which is used to measure the degree of error between the training video and the predicted video.

[0098] Specifically, the predicted video is compared with the training video frame by frame through the cross entropy loss function to calculate the loss value, which reflects the degree of difference between the predicted video and the training video.

[0099] In step S206 of some embodiments, specifically, the preset threshold may be determined based on an actual scenario.

[0100] Specifically, in car insurance claims, it is necessary to identify the liability of the insured vehicle. If the close-up shot of the vehicle in the predicted video of the insured vehicle is significantly different from the corresponding vehicle shot in the training video, the gradient of the parameters of the pre-trained video generation model can be calculated through the back-propagation process, and the parameters of the model can be updated based on these gradients.

[0101] For example, if the gradient indicates that the weight of the bidirectional attention layer needs to be increased, the value of the bidirectional attention layer weight is increased accordingly.

[0102] See also Figure 3 In some embodiments, step S104 includes but is not limited to steps S301 to S302:

[0103] Step S301, encode the word segmentation text sequence through a bidirectional attention layer to obtain an encoded text sequence.

[0104] Step S302, performing video frame prediction on the encoded text sequence to obtain an initial video frame.

[0105] In step S301 of some embodiments, specifically, the bidirectional attention mechanism of the bidirectional attention layer enables each word segmentation text token to see other word segmentation text tokens in the sequence and their positions, so that each word segmentation text token can capture global context information.

[0106] Specifically, the bidirectional attention layer can convert the word segmentation text token sequence into an encoded text sequence represented by high-dimensional features. Since the bidirectional attention mechanism enables each word segmentation text token to see other word segmentation text tokens in the sequence and their positions, the representation of each word segmentation text token can be realized during encoding not only depending on the previous word segmentation text token, but also on the following word segmentation text token, so as to capture the complete semantic information of the word segmentation text sequence.

[0107] In this embodiment, the word segmentation text sequence is encoded through a bidirectional attention layer, so that each encoded word segmentation text token contains both the semantic information of the word segmentation text token and the context information of the word segmentation text token, which is convenient for capturing the complete semantic information of the word segmentation text sequence and improving the accuracy of subsequent video generation.

[0108] In step S302 of some embodiments, specifically, the initial video frame is a static image represented by a high-dimensional space.

[0109] Specifically, video frame prediction can be achieved through linear transformation, that is, by using the encoded text sequence output by the encoder to predict the generation of the initial video frame.

[0110] Specifically, the encoded token (encoded text sequence) is gradually predicted by the video frame through linear transformation, and a series of initial video frames can be obtained.

[0111] In an optional embodiment of the present application, if the target indication information also includes target image indication information, the target image indication information is encoded through a bidirectional attention layer to generate a coded image sequence, the coded image sequence and the coded text sequence are spliced, and then the initial video frame prediction is performed. This can generate an initial video frame that contains both image and text information, thereby effectively improving the accuracy of video frame prediction.

[0112] For example, in the fintech scenario, the target image indication information may be a scene image of a car and a truck colliding while driving in the rain. The scene image of the car and the truck colliding while driving in the rain is converted into an image token, and the image token and the word segmentation text token are encoded through a bidirectional attention layer to generate an encoding token including the image token and the word segmentation text token. The encoding token is then used to predict the video frame to obtain the initially generated accident video of the car and truck collision while driving in the rain.

[0113] See also Figure 4 In some embodiments, step S105 includes but is not limited to steps S401 to S402:

[0114] Step S401: extract features from the initial video frame using a video word segmentation model to obtain video frame features.

[0115] Step S402, performing quantization and word segmentation processing on the video frame features to obtain a word segmentation video sequence.

[0116] In step S401 of some embodiments, specifically, the video word segmentation model may be a MAGVIT-v2 model. The video frame feature is a video frame token feature represented by a high-dimensional space.

[0117] Specifically, the features of the initial video frame can be extracted through the three-dimensional convolutional layer of the video segmentation model to obtain the token features of the video frame represented in a high-dimensional space.

[0118] See also Figure 5 In some embodiments, step S402 includes but is not limited to steps S501 to S503:

[0119] Step S501, obtaining a preset codebook; the codebook includes codebook elements.

[0120] Step S502: construct a codebook index for the video frame feature, and determine a target codebook element according to the codebook index and the codebook element.

[0121] Step S503: Map the video frame features to target codebook elements to obtain quantized video frame features.

[0122] In step S501 of some embodiments, specifically, the codebook elements are codewords, and the codebook is a low-dimensional discrete vector space, which is used to map the continuous video frame feature high-dimensional space to a low-dimensional discrete token space, wherein each codeword can be regarded as a low-dimensional discrete token space.

[0123] In step S502 of some embodiments, specifically, the codebook can be represented by an integer set C, the codebook represents K elements, and each codeword can be represented by binary (-1, 1), that is, each codeword consists of only 1 or -1, and a token index of the video frame feature is further constructed, and the corresponding target codebook element is found from the codeword according to the token index.

[0124] Specifically, the codebook index may be determined by the following formula:

[0125]

[0126] Among them, Index(y) represents the codebook index of y video frame features; y represents the video frame feature; y i >0 is an indicator function, which is used to indicate that the dimension of the video frame feature is positive or negative, and if and only if y u >0, the value of the indicator function is 1, otherwise it is 0; K represents the number of codebook elements, and log2K refers to the number of codebook elements required to represent the characteristics of a video frame.

[0127] For example, if the codebook size is 8, then log28=3, a 3-bit codebook element is required to represent a video frame feature, and if there is a video frame feature y=[2.5,-1.3,0.8], the contribution of each codebook dimension to the codebook index is calculated; for i=1, y1=2.5>0, so the indicator function is 1, then Index(y1)=2 0 x1=1; for i=2, y2=-1.3<0, so the indicator function is 0, then Index(y2)=2 1 x0=0; for i=3, y3=0.8>0, so the indicator function is 1, then Index(y3)=2 2 x1=4, Index(y1)+Index(y2)+Index(y3) are summed to obtain the final Index(y)=5, indicating that the target codebook element corresponding to the codebook index is located at the 5th position of the codebook.

[0128] In this embodiment, by constructing a codebook index for video frame features and determining a target codebook element based on the codebook index and the codebook element, the target codebook element corresponding to the codebook index can be quickly located among multiple codebook elements by constructing the codebook index, which helps to improve the efficiency of subsequent video segmentation.

[0129] In step S503 of some embodiments, specifically, segmenting the video sequence refers to mapping video frame features to features in the corresponding token vector space.

[0130] For example, the codebook size is 8, and the corresponding codeword is {-1,-1,-1,-1,1,1,1,1}. If the target codebook element is the 5th bit, the video frame feature is mapped to the 5th bit codeword, and the value of the codeword is 1.

[0131] In this embodiment, by mapping the video frame features to the target codebook elements, quantized video frame features are obtained, and the video frame features can be converted into token sequences that can be recognized by the video generation model. By mapping them to the codebook elements, the main information of the video frame features can also be represented by the codebook elements, thereby reducing redundant information in the video frame features, which will help to further improve the efficiency of video segmentation in the future.

[0132] See also Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S602:

[0133] Step S601, predicting the first frame video sequence of the word segmentation text sequence and the word segmentation video sequence through an autoregressive layer to obtain the first frame video sequence.

[0134] Step S602, iterative video sequence prediction is performed on the first frame video sequence, the word segmentation text sequence and the word segmentation video sequence through an autoregressive layer until a preset prediction end mark appears, thereby obtaining a target video sequence.

[0135] In step S601 of some embodiments, specifically, the first frame video sequence refers to the first frame video token generated by the video generation model.

[0136] Specifically, the unidirectional attention mechanism of the autoregressive layer can see all the context-containing encoded text sequences generated by the bidirectional attention layer and the generated video tokens when predicting the first frame of the video sequence, but cannot see the future video tokens that have not yet been generated.

[0137] Specifically, the Attention Mask mechanism ensures that when the autoregressive layer performs video sequence prediction, each video token to be predicted can only see all the previous segmentation text tokens and segmentation video tokens. That is, the attention mask can be implemented by adjusting the weights in the attention matrix so that the video token to be predicted cannot see tokens at other positions.

[0138] For example, in a fintech scenario, if the segmented text token is a car driving in the rain, and the segmented video token is a video scene of a car driving in the rain, then the autoregressive layer can be used to predict the first frame of the video sequence as the video token of the first frame of the car driving in the rain.

[0139] In an optional embodiment of the present application, if the target indication information also includes target image indication information, the encoded image token can be used together with the segmentation text token and the segmentation video token as prefix information for predicting the first frame of video to perform first frame video prediction. By combining the target image, the indication information can be enriched, and when predicting the video sequence, the prediction accuracy of the video sequence can be effectively improved.

[0140] In step S602 of some embodiments, specifically, the target video sequence refers to all video tokens generated by the video generation model according to the target indication information.

[0141] Specifically, the autoregressive mechanism of the autoregressive layer is used to take the generated video frame token, word segmentation text token and word segmentation video token as the prefix part to gradually predict the video token of the next frame. Among them, each video sequence to be generated depends on the previously generated video frame token, word segmentation text token and word segmentation video token.

[0142] Specifically, the prediction end marker can be a specific video frame (such as indicating the end of a scene or action) or a threshold reaching the maximum number of generated video frames (such as the maximum number of video frames is 25 frames, then the 25th video frame is the prediction end marker).

[0143] Specifically, when a predicted video frame appears, the video generation model stops generating video frames and outputs a complete predicted video sequence as a target video sequence.

[0144] For example, in the FinTech scenario, if a vehicle claim is caused by a traffic accident due to slippery roads in rainy days, it is necessary to identify the responsibility of the insured vehicle through the traffic accident video. The autoregressive layer predicts a series of actions based on the prefix information, such as the insured vehicle driving normally in the rain, starting to slip, and finally colliding with another vehicle. When the video generation model predicts a video token indicating that the insured vehicle has collided with another vehicle, or when the number of generated video frames reaches 25 frames, the video generation model will stop generating new video frames.

[0145] In this embodiment, an autoregressive layer is used to iteratively predict the video sequence of the first frame video sequence, the word segmentation text sequence and the word segmentation video sequence until a preset prediction end mark appears. The target video sequence can be gradually constructed through multiple iterations to ensure the temporal coherence of the generated video and always keep it consistent with the input text information, thereby effectively improving the accuracy of video generation.

[0146] See also Figure 7 In some embodiments, step S107 includes but is not limited to steps S701 to S702:

[0147] Step S701: converting a target video sequence into a target video frame.

[0148] Step S702: Combine target video frames in sequence according to a preset time sequence to obtain a target video.

[0149] In step S701 of some embodiments, specifically, the target video frames are static images, and together constitute the entire content of the target video.

[0150] Specifically, the target video token is embedded to obtain the embedding vector of the target video token, and the embedding vector is converted into the target pixel value through the sigmoid activation function to map each target video token back to the original pixel representation.

[0151] In step S702 of some embodiments, specifically, the preset time sequence is consistent with the sequence of original frames in the target video sequence.

[0152] For example, in a fintech scenario, if a vehicle claim is caused by a traffic accident due to slippery roads in rainy days, it is necessary to determine the liability of the insured vehicle by viewing the traffic accident video of the insured vehicle. The target video frame is a static image of the insured vehicle driving normally in the rain, starting to slip, and finally colliding with another vehicle. These static images of the vehicle accident are combined in sequence according to the order in which the accidents occurred to obtain a live reproduction video of the traffic accident of the insured vehicle.

[0153] In this embodiment, the target video frames are combined in sequence according to a preset time sequence to obtain the target video, which can ensure the time continuity of the generated video and ensure that it is always consistent with the target indication information, thereby effectively improving the accuracy of video generation.

[0154] The embodiment of the present application obtains target indication information; wherein the target indication information includes target text indication information; performs word segmentation processing on the target text indication information to obtain a word segmentation text sequence; obtains a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer; performs video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame; performs word segmentation on the initial video frame using a pre-trained video segmentation model to obtain a word segmentation video sequence; performs video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through an autoregressive layer to obtain a target video sequence; and generates a target video according to the target video sequence. The present application firstly predicts video frames of the segmented text sequence through a bidirectional attention layer, and can generate video frames related to the indicative information, thereby improving the relevance of the video generation content; secondly, the initial video frame is segmented by a pre-trained video segmentation model, which provides a richer information representation for subsequent video sequence prediction, thereby facilitating the subsequent generation of more coherent and high-quality video content; finally, the video sequence is predicted for the segmented text sequence and the segmented video sequence through an autoregressive layer, and a target video is generated according to the predicted video sequence, which can realize the gradual construction of video content through multiple iterations, ensure the temporal coherence of the generated video, and always keep consistent with the input text information, thereby effectively improving the accuracy of video generation.

[0155] See also Figure 8 The embodiment of the present application further provides a video generating device, which can implement the above-mentioned video generating method, and the device includes:

[0156] An indication information acquisition module, used to acquire target indication information; wherein the target indication information includes target text indication information;

[0157] A text segmentation module is used to perform segmentation processing on the target text indication information to obtain a segmentation text sequence;

[0158] The model acquisition module is used to obtain the trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer;

[0159] The initial video frame prediction module is used to predict the video frame of the word segmentation text sequence through a bidirectional attention layer to obtain the initial video frame;

[0160] A video word segmentation module is used to segment the initial video frame using a pre-trained video word segmentation model to obtain a word segmentation video sequence;

[0161] The video sequence prediction module is used to perform video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through an autoregressive layer to obtain a target video sequence;

[0162] The video generation module is used to generate a target video according to a target video sequence.

[0163] The specific implementation of the video generating device is substantially the same as the specific implementation of the above-mentioned video generating method, and will not be described in detail here.

[0164] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above video generation method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0165] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0166] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0167] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store processing systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 calls and executes the video generation method of the embodiment of this application;

[0168] Input / output interface 903, used to implement information input and output;

[0169] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0170] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0171] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0172] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0173] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The video generation method, video generation device, electronic device and storage medium provided by the embodiments of the present application obtain target indication information; wherein the target indication information includes target text indication information; segmentation processing is performed on the target text indication information to obtain a segmentation text sequence; a trained video generation model is obtained; the video generation model includes a bidirectional attention layer and an autoregressive layer; video frame prediction is performed on the segmentation text sequence through the bidirectional attention layer to obtain an initial video frame; the initial video frame is segmented using a pre-trained video segmentation model to obtain a segmentation video sequence; video sequence prediction is performed on the segmentation text sequence and the segmentation video sequence through the autoregressive layer to obtain a target video sequence; and a target video is generated according to the target video sequence. The present application firstly predicts video frames of the segmented text sequence through a bidirectional attention layer, and can generate video frames related to the indicative information, thereby improving the relevance of the video generation content; secondly, the initial video frame is segmented by a pre-trained video segmentation model, which provides a richer information representation for subsequent video sequence prediction, thereby facilitating the subsequent generation of more coherent and high-quality video content; finally, the video sequence is predicted for the segmented text sequence and the segmented video sequence through an autoregressive layer, and a target video is generated according to the predicted video sequence, which can realize the gradual construction of video content through multiple iterations, ensure the temporal coherence of the generated video, and always keep consistent with the input text information, thereby effectively improving the accuracy of video generation.

[0175] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0176] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0177] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0179] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0180] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0185] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire target indication information; wherein the target indication information includes target text indication information; Performing word segmentation processing on the target text indication information to obtain a word segmentation text sequence; Obtain a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer; Performing video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame; Using a pre-trained video segmentation model to segment the initial video frame to obtain a segmentation video sequence; Performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence; A target video is generated according to the target video sequence.

2. The method according to claim 1, characterized in that The step of performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence includes: Performing a first frame video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a first frame video sequence; Iterative video sequence prediction is performed on the first frame video sequence, the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer until a preset prediction end mark appears, thereby obtaining the target video sequence.

3. The method according to claim 1, characterized in that The step of performing video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame includes: Encoding the word segmentation text sequence through the bidirectional attention layer to obtain an encoded text sequence; Video frame prediction is performed on the encoded text sequence to obtain the initial video frame.

4. The method according to claim 1, characterized in that The method of using a pre-trained video word segmentation model to segment the initial video frame to obtain a word segmentation video sequence includes: Using the video word segmentation model to extract features from the initial video frame to obtain video frame features; The video frame features are subjected to quantized word segmentation processing to obtain the word segmentation video sequence.

5. The method according to claim 4, characterized in that The step of performing quantization and word segmentation processing on the video frame features to obtain the word segmentation video sequence includes: Obtaining a preset codebook; the codebook includes codebook elements; Constructing a codebook index for the video frame feature, and determining a target codebook element according to the codebook index and the codebook element; The video frame features are mapped to the target codebook elements to obtain the word segmentation video sequence.

6. The method according to any one of claims 1 to 5, characterized in that: The generating a target video according to the target video sequence comprises: Converting the target video sequence into a target video frame; The target video frames are combined in sequence according to a preset time sequence to obtain the target video.

7. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining the trained video generation model includes: Obtaining a training data set; the training data set includes training text and training video; Performing word segmentation processing on the training text to obtain a word segmentation training text sequence; Performing word segmentation processing on the training video to obtain a word segmentation training video sequence; Performing video prediction on the training text and the training video using a pre-trained video generation model to obtain a predicted video; Using a preset loss function to calculate the loss of the predicted video and the training video to obtain a loss value; The parameters in the pre-trained video generation model are adjusted according to the loss value until the loss value meets a preset threshold, thereby obtaining the trained video generation model.

8. A video generating device, characterized in that: The device comprises: An indication information acquisition module, used to acquire target indication information; wherein the target indication information includes target text indication information; A text segmentation module, used for performing segmentation processing on the target text indication information to obtain a segmented text sequence; A model acquisition module, used to acquire a trained video generation model; the video generation model includes a bidirectional attention layer and an autoregressive layer; An initial video frame prediction module, used to perform video frame prediction on the word segmentation text sequence through the bidirectional attention layer to obtain an initial video frame; A video word segmentation module, used to segment the initial video frame using a pre-trained video word segmentation model to obtain a word segmentation video sequence; A video sequence prediction module, used for performing video sequence prediction on the word segmentation text sequence and the word segmentation video sequence through the autoregressive layer to obtain a target video sequence; The video generation module is used to generate a target video according to the target video sequence.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.

Images