A video generation method, apparatus, device, and medium

Abstract

The application relates to the field of video generation and discloses a video generation method, device, equipment and medium, which comprises the following steps: processing original video data to obtain a plurality of initial frame images; dividing and remodeling each initial frame image into a two-dimensional tile sequence and performing embedding processing to generate a tile embedding sequence; inputting the tile embedding sequence into a video generation model with a test-time learning layer, wherein the hidden state is a convolutional neural network; performing multi-head attention mechanism processing on the tile embedding sequence to obtain an output result of the multi-head attention mechanism; performing feature extraction and residual connection on the output result of the multi-head attention mechanism, generating an image representation according to the output result of the residual connection; decoding the image representation to generate new image frames; and splicing all the new image frames to obtain a target video. The application realizes high-quality conversion from an input video to new video content, can capture complex space-time relationships in the video, and generates coherent and realistic video content.

Description

Technical Field

[0001] The present invention relates to the fields of video generation technology and financial technology, and in particular to a video generation method, device, equipment and medium. Background Art

[0002] With the continuous development of technology and people's growing demand for faster information acquisition, short videos, as a new form of communication, are rapidly gaining widespread application in various fields. Fintech, as a fusion of finance and technology, has also begun to explore the use of short videos to transform traditional financial business models, enhance user experience, and bring new development opportunities to the financial industry. For example, financial companies can use videos to popularize and educate financial knowledge, promote financial products, report financial news, and introduce financial services.

[0003] Early video generation primarily combined convolutional and recurrent neural networks to predict consecutive video frames. Recent efforts have attempted to apply the Transformer architecture to video generation. By discretizing visual blocks, methods such as the Latent Video Transformer (LVT) and VideoGPT have been proposed. While these methods enable open-domain video generation by scaling up the model size, they are still limited by the Transformer architecture's unidirectional bias, error accumulation, and high computational cost. This impacts the generalization and accuracy of the video generation model, thereby reducing the quality of the generated video. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a video generation method, device, equipment and medium that can be applied to financial technology or other related fields. Its main purpose is to better adapt to different data and improve the compatibility and generation quality of video generation.

[0005] The technical solutions of the present invention are as follows:

[0006] A first aspect of the present invention provides a video generation method, comprising:

[0007] Acquire original video data, perform preprocessing and data enhancement processing on the original video data, and obtain a plurality of initial frame images;

[0008] Each initial frame image is segmented and reshaped into a corresponding two-dimensional block sequence, and the two-dimensional block sequence is embedded to generate a corresponding block embedding sequence;

[0009] Inputting the tile embedding sequence into a video generation model with a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network;

[0010] Processing the tile embedding sequence by a multi-head attention mechanism through the test-time learning layer to obtain an output result of the multi-head attention mechanism;

[0011] Using the convolutional neural network to perform feature extraction and residual connection on the output results of the multi-head attention mechanism, and generating a corresponding image representation according to the output results of the residual connection;

[0012] The image representation corresponding to each initial frame image is decoded to generate a new image frame, and all the new image frames are spliced ​​together to obtain the target video.

[0013] A second aspect of the present invention provides a video generation device, comprising:

[0014] An acquisition module is used to acquire original video data, perform preprocessing and data enhancement processing on the original video data, and obtain a plurality of initial frame images;

[0015] a reshaping and embedding module, configured to segment and reshape each initial frame image into a corresponding two-dimensional image block sequence, and perform embedding processing on the two-dimensional image block sequence to generate a corresponding image block embedding sequence;

[0016] An input module for inputting the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network;

[0017] an attention processing module, configured to perform a multi-head attention mechanism on the tile embedding sequence through the test-time learning layer to obtain an output result of the multi-head attention mechanism;

[0018] A convolution processing module, configured to perform feature extraction and residual connection on the output results of the multi-head attention mechanism using the convolutional neural network, and generate a corresponding image representation based on the output results of the residual connection;

[0019] The decoding generation module is used to decode the image representation corresponding to each initial frame image to generate a new image frame, and to obtain the target video after splicing all the new image frames.

[0020] A third aspect of the present invention provides a computer device comprising at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned video generation method.

[0023] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned video generation method.

[0024] Beneficial effects: The present invention discloses a video generation method, apparatus, device and medium. Compared with the prior art, the embodiment of the present invention obtains original video data, performs preprocessing and data enhancement processing on the original video data, and obtains several initial frame images; each initial frame image is divided and reshaped into a corresponding two-dimensional tile sequence, and the two-dimensional tile sequence is embedded to generate a corresponding tile embedding sequence; the tile embedding sequence is input into a video generation model with a test-time learning layer, and the hidden state of the test-time learning layer is a convolutional neural network; the tile embedding sequence is processed by a multi-head attention mechanism through the test-time learning layer to obtain an output result of the multi-head attention mechanism; the convolutional neural network is used to perform feature extraction and residual connection on the output result of the multi-head attention mechanism, and a corresponding image representation is generated according to the output result of the residual connection; the image representation corresponding to each initial frame image is decoded to generate a new image frame, and all the new image frames are spliced ​​to obtain the target video. By reshaping each frame of raw video data into a sequence of two-dimensional tiles for embedding, a video generation model with test-time learning layers, combined with a multi-head attention mechanism and a convolutional neural network, is used to perform in-depth analysis of the tile sequences, extract features, and generate image representations. Finally, these representations are decoded to generate new image frames, which are then spliced ​​into the target video. This achieves high-quality conversion from the input video to new video content, capturing the complex spatiotemporal relationships in the video and generating coherent and realistic video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the solutions in the present invention, a brief introduction is given below to the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A schematic diagram of an application environment of the video generation method provided by an embodiment of the present invention;

[0027] Figure 2 A flow chart of a video generation method provided by an embodiment of the present invention;

[0028] Figure 3 A flowchart of step S201 in the video generation method provided in an embodiment of the present invention;

[0029] Figure 4 A flowchart of step S202 in the video generation method provided in an embodiment of the present invention;

[0030] Figure 5 A flow chart of step S204 in the video generation method provided in an embodiment of the present invention;

[0031] Figure 6 A flow chart of step S205 in the video generation method provided in an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the functional modules of a video generation device provided by an embodiment of the present invention;

[0033] Figure 8 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and effects of the present invention more clear and distinct, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below with reference to the accompanying drawings.

[0035] The video generation method provided by the embodiment of the present invention can be applied in the following Figure 1 In an application environment, the system includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0037] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a backend server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The backend server may analyze and process the received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0039] It should be noted that the video generation method provided in the embodiments of the present application can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the video generation apparatus provided in the embodiments of the present invention can also be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103. Alternatively, the video generation method provided in the embodiments of the present invention can generally be executed by the server 105. Accordingly, the video generation apparatus provided in the embodiments of the present invention can generally be set in the server 105.

[0040] It should be understood that the numbers of the above terminal devices, networks and servers are merely illustrative and any number of terminal devices, networks and servers may be provided as required.

[0041] like Figure 2 As shown, the video generation method provided by the embodiment of the present invention specifically includes the following steps:

[0042] S201 : Acquire original video data, perform preprocessing and data enhancement processing on the original video data, and obtain a plurality of initial frame images.

[0043] In this embodiment, raw video data is obtained through material collection as the basis for video generation. For example, in a short video marketing scenario in the financial field, a financial product promotion video needs to be generated. Relevant video materials in the financial field can be collected, such as product introductions, market performance charts, customer evaluation videos, and other video data. The raw video data is preprocessed, including deduplication, deletion of invalid samples, resizing, and normalization to ensure the quality of the video data. At the same time, data enhancement processing, such as random cropping and horizontal flipping, is also performed to increase the diversity of the video data and improve the generalization ability of the model. By obtaining several initial frame images through preprocessing and data enhancement, the quality and diversity of the video data can be improved, providing richer materials for subsequent video generation.

[0044] S202 : Segment and reshape each initial frame image into a corresponding two-dimensional image block sequence, and perform embedding processing on the two-dimensional image block sequence to generate a corresponding image block embedding sequence.

[0045] In this embodiment, each initial frame image is segmented and reshaped. Each pre-processed initial frame image is segmented into tiles of a fixed size, which are then sequentially spliced ​​and reshaped into a two-dimensional tile sequence. Each sequence contains a certain number of tiles. Each tile is embedded, for example, linearly embedded and positionally embedded, to unify the dimension of the sequence and capture the content and position information of the tile, thereby converting the two-dimensional tile sequence into a format suitable for processing by a video generation model. At the same time, important information of the tile is retained, providing the model with the content and position information of the tile, which helps to generate coherent video content and ensure the quality of video generation.

[0046] S203: Input the tile embedding sequence into a video generation model having a test-time learning layer, where the hidden state of the test-time learning layer is a convolutional neural network.

[0047] In this embodiment, the tile embedding sequence is input into a video generation model with a test-time learning layer, and the hidden state of the test-time learning layer is a convolutional neural network. Specifically, the video generation model is a video generation model that combines test-time learning (TTT), Transformer architecture and convolutional neural network (CNN). Test-time learning, i.e., TTT, is a technique for fine-tuning the model during the test phase. In the traditional training-test paradigm, the model learns during the training phase and then evaluates fixed parameters during the test phase. TTT allows the model to quickly adapt and fine-tune each new input sample or task during the test phase, thereby improving the performance of the model on specific tasks. In video generation, the TTT layer can fine-tune the video generation model directly on the test sequence, so that the model can be optimized for a specific video sequence. As the hidden state of the TTT layer, the convolutional neural network can extract spatial features in the video frame and enhance the model's ability to capture local details. By combining these components, efficient feature extraction, capture of long-distance dependencies, and rapid adaptation to specific tasks can be achieved in video generation tasks. Inputting the tile embedding sequence into the video generation model can be better applied to application scenarios that require generating high-quality and coherent video content, such as video reports in the financial field and market analysis videos.

[0048] S204. Perform a multi-head attention mechanism on the tile embedding sequence through the test-time learning layer to obtain an output result of the multi-head attention mechanism.

[0049] In this embodiment, the learning layer during testing, i.e., the TTT layer, is an autoregressive mapping from one sequence to another sequence, and its hidden state is a convolutional neural network model. The TTT layer trained during testing is equivalent to the self-attention mechanism, i.e., the TTT layer uses the self-attention mechanism to process sequence data, specifically, it can be a multi-head attention mechanism to process the tile embedding sequence. The multi-head attention mechanism obtains the attention distribution of different subspaces of the input sequence by running multiple independent attention mechanisms in parallel, thereby more comprehensively capturing the potential multiple semantic associations in the sequence. When the TTT layer processes the tile embedding sequence with the multi-head attention mechanism, it captures the relationship between frames by processing multiple attention heads in parallel. Each head focuses on different information, such as local features, edge information, global context, etc. The multi-head attention mechanism can capture features of different scales, enhance the model's understanding of the video content, and make the generated video richer and more expressive.

[0050] S205. Use the convolutional neural network to perform feature extraction and residual connection on the output results of the multi-head attention mechanism, and generate a corresponding image representation based on the output results of the residual connection.

[0051] In this embodiment, in the TTT layer, the convolutional neural network (CNN) serves as a hidden state, so the output of the multi-head self-attention mechanism is further processed by the convolutional neural network to extract more complex feature representations, allowing the model to not only capture the dependencies of the sequence data, but also utilize the spatial feature extraction capabilities of the convolutional neural network. The output results of the multi-head attention mechanism are subjected to feature extraction by the convolutional neural network to extract key features, and the residual connection is applied to add the feature extraction results to the original tile embedding sequence, and the corresponding image representation is generated based on the output results of the residual connection. The image representation may refer to the internal activation or feature vector obtained after processing through a series of network layers (such as convolutional layers, pooling layers, fully connected layers or self-attention layers). The obtained image representation can be used for a variety of tasks, including classification, detection, generation, and the like. Feature extraction and residual connection help prevent the gradient vanishing problem, improve the model's ability to capture the details of the video frame, and ensure the clarity and quality of the generated video.

[0052] S206 , decoding the image representation corresponding to each initial frame image to generate a new image frame, and splicing all the new image frames to obtain the target video.

[0053] In this embodiment, because the image representation is obtained by applying a multi-head self-attention mechanism to all tiles of the entire image, the image representation generated based on each initial frame image contains rich visual content and captures global information of the input image, such as content, style, and texture. In video generation tasks, this image representation can be used to generate new video frames. Specifically, the image representation is input into a decoder, which is responsible for converting the feature vector of the image representation into a new visual image frame. This decoding process may include a transposed convolution layer (also called a deconvolution layer), an activation function, and possible upsampling operations to reconstruct the pixel values ​​of the image. The image representation of each initial frame image is decoded to generate a new image frame, and all new image frames are spliced ​​together in chronological order to form the final target video. For example, through the decoding and splicing process, a coherent, high-quality promotional video is generated to effectively showcase the characteristics and advantages of financial products.

[0054] In the above embodiment, the present invention discloses a video generation method, which obtains raw video data, preprocesses and data augments the raw video data to obtain a plurality of initial frame images; segments and reshapes each initial frame image into a corresponding two-dimensional tile sequence, and embeds the two-dimensional tile sequence to generate a corresponding tile embedding sequence; inputs the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network; processes the tile embedding sequence through a multi-head attention mechanism by the test-time learning layer to obtain an output result of the multi-head attention mechanism; uses the convolutional neural network to perform feature extraction and residual connection on the output result of the multi-head attention mechanism, and generates a corresponding image representation based on the output result of the residual connection; decodes the image representation corresponding to each initial frame image to generate a new image frame, and splices all the new image frames to obtain a target video. By reshaping each frame of the raw video data into a two-dimensional tile sequence for embedding processing, the video generation model having a test-time learning layer, combined with the multi-head attention mechanism and the convolutional neural network, deeply analyzes the tile sequence, extracts features, and generates an image representation. Finally, by decoding these representations to generate new image frames and splicing them into the target video, high-quality conversion from the input video to the new video content is achieved, which can capture the complex spatiotemporal relationships in the video and generate coherent and realistic video content.

[0055] In one embodiment, Figure 3 As shown, step S201 includes:

[0056] S301, obtaining original video data, and extracting frame images from the original video data;

[0057] S302, removing duplicate frames and invalid frames from all extracted frame images to obtain frame images to be enhanced;

[0058] S303 : performing horizontal rotation and random cropping on the frame image to be enhanced in sequence to obtain a plurality of initial frame images for data enhancement.

[0059] In this embodiment, when obtaining the initial frame image, the corresponding original video data is first obtained. For example, to generate a financial product promotion video, the original video data about the product introduction, market performance analysis, customer evaluation, etc. can be obtained first, and all frame images in the original video data can be extracted through video processing tools or libraries. These frame images are the basic units of video content and provide materials for subsequent data enhancement and video generation.

[0060] In the extracted frame images, there may be some duplicate frames (for example, static slide pages in a video) or invalid frames caused by recording problems (such as blur, overexposure, etc.). Therefore, these duplicate frames are identified and removed through image processing algorithms such as feature matching and histogram comparison. At the same time, invalid frames are identified and eliminated through visual quality assessment algorithms. Removing duplicate and invalid frames can improve the quality of the video material and obtain an effective set of frame images to be enhanced, ensuring that the generated video content is coherent and free of redundant information, thereby improving the audience's viewing experience.

[0061] For the frame image to be enhanced, further horizontal operation and random cropping data enhancement processing are performed, where horizontal rotation can be to rotate each frame image 90 degrees, 180 degrees or 270 degrees to simulate different shooting angles and increase the diversity of video content. Specifically, horizontal rotation (Horizontal Rotation). For an image of size width×height, the center point coordinates are (width / 2, height / 2), and the rotation angle is recorded as θ°. Rotate around an axis perpendicular to the image plane, that is, rotate around the longitudinal axis of the image. For each point (x, y) in the image, the new coordinates (x′, y′) after rotation can be calculated by the following formula:

[0062] x′=x·cos(θ)-y·sin(θ)

[0063] y′=x·sin(θ)+y·cos(θ)

[0064] Random cropping can be to randomly select an area (such as the center area, the upper left corner area, etc.) from each frame image and crop a sub-image of a certain size. This cropping can simulate different compositions and perspectives, further increasing the diversity of video content. Specifically, assuming the size of the cropped area is width c ×height c , the width and height of the original image are width image and height image Randomly determine the coordinates of the upper left corner of the cropping area (x start ,y start ), randint(a,b) is a random integer generator function that returns a random integer in the interval [a,b]. start and y start It should be within the original image dimensions, so the following conditions need to be met:

[0065] 0≤x start <width image -width c

[0066] 0≤ystart <height image -height c

[0067] x start and y start is selected by a random number generator, according to x start and y start And the size of the cropping area width c ×height c , crop the corresponding sub-region from the original image.

[0068] x start =randint(0,width image -width c )

[0069] y start =randint(0,height image -height c )

[0070] By performing data augmentation on the extracted valid frame images through horizontal rotation and random cropping, the diversity of video materials can be significantly increased, the generalization ability of the model can be improved, and the generated video content can be made richer and more interesting. It can also help the model learn more visual features and patterns, and improve the quality and realism of the generated video.

[0071] In one embodiment, Figure 4 As shown, step S202 includes:

[0072] S401, determining the number of segmented blocks and the block resolution according to the resolution and the number of color channels of the initial frame image, and segmenting and reshaping the initial frame image into corresponding two-dimensional block sequences according to the number of segmented blocks and the block resolution;

[0073] S402: linearly embed each image block in the two-dimensional image block sequence using a linear transformation matrix to perform dimensionality conversion on each image block;

[0074] S403, stitching the linear embedding results of all tiles in order, and adding the category embedding vector at the beginning of the stitching result to obtain a stitching sequence;

[0075] S404, performing position encoding according to the position information of each tile in the two-dimensional tile sequence to obtain a position embedding matrix;

[0076] S405 : Add the splicing sequence and the position embedding matrix to obtain a corresponding tile embedding sequence.

[0077] In this embodiment, the number of segmented blocks and the block resolution are determined according to the resolution and color channel number of the initial frame image. C represents the number of color channels in each frame; H represents the vertical size of each frame, that is, the height of the frame (in pixels); W represents the horizontal size of each frame, that is, the width of the frame (in pixels). Reshape into a 2D block sequence Where (H×W) is the resolution of the original image, C is the number of channels, (P,P) is the resolution of each image segment, and N = HW / P 2 is the number of segments generated, that is, by segmentation and reshaping, each initial frame image is reshaped into a sequence with a length of N and a dimension of O for each block 2 A two-dimensional tile sequence of C.

[0078] For each tile in the two-dimensional tile sequence, a linear embedding is performed through a learnable linear transformation matrix E, and the dimension of the linear transformation matrix E is (P 2 C) × D, thereby reducing the dimension of each tile from P 2 C is mapped to the embedding dimension D of the model, and each tile is transformed to fit the embedding space of the model, converting the high-dimensional image data into a low-dimensional representation that is more suitable for processing while retaining important visual information. The linear embedding results of all tiles are sequentially concatenated to form a matrix of dimension N×D, where N is the number of tiles, and a learnable category embedding vector is added at the beginning of the concatenation result. The dimension of this vector is also D category embedding vector, and the concatenated sequence is obtained. Similar to the [class] tag in the BERT model, the category embedding vector is in the output state of the Transformer encoder. It can be used as an image representation y, and the length of the spliced ​​sequence after splicing becomes N+1.

[0079] In addition, in order to retain the position information of each tile, position encoding is performed based on the position information of each tile in the two-dimensional tile sequence to obtain a position embedding matrix, which can be, for example, one-dimensional position embedding, two-dimensional position embedding, relative position embedding, etc., to capture the position information of the tile. The dimension of the obtained position embedding matrix Epos is (N+1)×D. Among them, one-dimensional position embedding means that the input data is regarded as a sequence, and a unique embedding vector (a set of numerical values) is learned for each position in the sequence. These embedding vectors capture the relative position information of the tile in the sequence. They are usually combined with the content embedding of the tile for the model to use in subsequent processing. Two-dimensional position embedding means that the input data is regarded as a grid, and the position of the tile can be represented by a two-dimensional coordinate (h, w), where h is the height coordinate and w is the width coordinate. The height coordinate embedding matrix E h , width coordinate embedding matrix E w , two-dimensional position embedding p h,w =[E h [h]; E w [w]]. Relative position embedding refers to taking into account the relative positions between tiles when calculating the attention weights. It considers the relative distance between tiles rather than their absolute positions in the sequence, thereby better capturing the structural information in the sequence data. The position embedding matrix is ​​used to add position information to each element in the spliced ​​sequence (including the category embedding vector and the linear embedding of each tile). The position embedding vector is added to the corresponding category embedding vector and the linear embedding of each tile, that is, the spliced ​​sequence and the position embedding matrix are added element by element to obtain the final tile embedding sequence z0, whose dimension is (N+1)×D. This tile embedding sequence z0 will be fed into the adaptive generation model for subsequent processing.

[0080] In specific implementation, the two-dimensional block sequence is calculated by the following formula: Embedding processing:

[0081]

[0082] Among them, the linear transformation matrix Position embedding matrix x class is the category embedding vector, N is the length of the two-dimensional block sequence, and z0 is the block embedding sequence, which is also the initial input sequence of the video generation model. The summation operation combines the content information and position information of the block, providing the model with a rich input representation that encodes the spatial location.

[0083] In one embodiment, Figure 5 As shown, step S204 includes:

[0084] S501, performing layer normalization processing on the tile embedding sequence to obtain a normalized sequence;

[0085] S502, performing multi-head attention processing on the normalized sequence through the test-time learning layer to obtain a multi-head attention result;

[0086] S503: Perform a residual connection on the tile embedding sequence and the multi-head attention result to obtain the output result of the multi-head attention mechanism.

[0087] In this embodiment, for the tile embedding sequence, layer normalization is first performed. Layer normalization is a technique in deep learning, which is used to normalize the features of each sample so that the output has unit variance and zero mean, thereby improving the robustness of the model to changes in the input data distribution. The layer-normalized sequence is input into the test-time learning layer, namely the TTT layer, which contains a multi-head attention mechanism. The multi-head attention mechanism processes multiple attention heads in parallel, and each head focuses on different parts of the sequence. For example, one head may focus on local features, while another head may focus on the global context. When the multi-head attention mechanism is executed in the TTT layer, each head has its own independent query, key, and value matrix. The standard attention operation is performed on each head, and the outputs of all heads are spliced ​​together, and then fused through a linear layer to obtain the final attention output vector.

[0088] In specific implementation, for the input tile embedding sequence z∈R N+1×D With learnable weight matrix Multiply and project the input sequence z into the representation of queries, keys, and values, generating query (Q), key (K), and value (V) projections, that is, [Q, K, V] = zW QKV , where D is the dimension of the sequence z, Dh is the dimension of each attention head, and W QKV =[θ Q θ K θ V ],θ Q ,θ K ,θ V are the parameters of the TTT layer, which serve as the weight matrices of query (Q), key (K) and value (V); is the connection operation. For the input tile embedding sequence z∈R N+1×D Each element of will calculate the weighted sum of all v in the sequence to get the standard attention mechanism score:

[0089] SA(z)=Av

[0090]

[0091] Among them, A ij It is based on two elements in the sequence and their respective query q j and key k j Represents the pairwise similarity between is the square root of the dimension, the attention weight matrix

[0092] The multi-head attention mechanism runs k standard self-attention SA calculations in parallel, concatenates the outputs of all heads together, and then fuses them through a linear layer to obtain the multi-head attention output vector:

[0093] MSA(z)=[SA1(z);SA2(z);...;SA k (z)]W msa

[0094] Among them, the multi-head attention weight matrix k is the number of heads. To maintain consistency in computation and parameter count when changing the number of heads k, the dimension of each attention head can usually be set to D / k. The multi-head attention mechanism allows the model to understand the data from different perspectives, capture richer information, and improve the quality and diversity of video generation.

[0095] Afterwards, the multi-head attention result is residually connected with the original tile embedding sequence. The residual connection obtains the final output result of the multi-head attention mechanism by adding the original sequence and the multi-head attention result, that is, the multi-head attention output vector. This ensures that the model can retain the original information while incorporating the additional information extracted by the attention mechanism, which helps alleviate the gradient vanishing problem in deep models, enables the model to learn more effectively, and improves the quality and accuracy of generated videos.

[0096] In specific implementation, the multi-head attention mechanism is processed on the tile embedding sequence through the following formula:

[0097]

[0098] in, For the The output of the layer, For the The output of the multi-head attention mechanism of the layer, L is the number of model layers, LN is layer normalization, and MSA is the multi-head attention mechanism.

[0099] In one embodiment, step S501 includes:

[0100] Calculating the mean and variance of the tile embedding sequence according to the feature value of each sample in the tile embedding sequence;

[0101] The characteristic value of each sample is normalized and calculated according to the mean and variance of the tile embedding sequence, and the normalized characteristic value of each sample is obtained to form the normalized sequence.

[0102] In this embodiment, when performing layer normalization processing on the tile embedding sequence, the mean and variance of each sample in the tile embedding sequence are first calculated. i Represents the i-th element in the tile embedding sequence z. Based on each element of the tile embedding sequence, calculate the mean μ and variance σ of all elements 2 ,

[0103]

[0104] The feature value of each sample is normalized based on the mean and variance of the tile embedding sequence:

[0105]

[0106] LN(z i ) is the normalized value of the i-th sample. The normalized feature values ​​of each sample are obtained to form the normalized sequence. The normalized sequence is used as the input of the multi-head attention mechanism in the next step. The normalization process eliminates the differences in feature value scales, ensures the comparability of feature values ​​between different samples, and improves the quality of video generation.

[0107] In one embodiment, Figure 6 As shown, step S205 includes:

[0108] S601, performing layer normalization processing on the output result of the multi-head attention mechanism to obtain a normalized output result;

[0109] S602: performing convolution, pooling, and full connection operations on the normalized output results in sequence, extracting and integrating features in the normalized output results to obtain a feature extraction result;

[0110] S603. Perform a residual connection on the output result of the multi-head attention mechanism and the feature extraction result to obtain an output result of the residual connection.

[0111] In this embodiment, in the TTT layer, the convolutional neural network as its hidden state will further process the output of the self-attention mechanism to extract more complex feature representations. conv(x) consists of a convolution layer, a pooling layer, and a fully connected layer. Specifically, the output of the multi-head attention mechanism is a set of feature vectors that contain the contextual information of each block in the video frame. These output results are layer-normalized, that is, the mean and variance of each feature vector are calculated, and these statistics are used to adjust and scale the feature values ​​to make the output feature distribution more stable. The normalized output results are sent to the convolutional neural network for feature extraction, wherein the normalized output results are convolved by the convolution layer to capture local spatial features such as edges and textures; then they are input to the pooling layer for pooling operations, such as maximum pooling or mean pooling, to reduce the spatial dimension of the features while retaining the most important information; then they are input to the fully connected layer for fully connected operations to integrate features at different levels and extract more abstract and global feature representations. Thus, the features in the normalized output results are extracted and integrated to obtain feature extraction results.

[0112] Furthermore, a residual connection is performed between the original output of the multi-head attention mechanism and the feature extraction results after convolutional neural network processing. This is done by adding the feature extraction results to the original output to obtain the residual connection output. This residual connection allows the model to retain the original information while incorporating the advanced features extracted and integrated by the convolutional neural network, thereby helping to generate more accurate and high-quality video frames.

[0113] In specific implementation, the output results of the multi-head attention mechanism are processed by the following formula:

[0114]

[0115] in, For the The output of the residual connection of the layer, For the The output of the multi-head attention mechanism of the layer, L is the number of model layers, LN is layer normalization, f conv It is a convolutional neural network.

[0116] In one embodiment, step S205 further includes:

[0117] Obtaining an output state value of the category embedding vector at an output end from an output result of the residual connection;

[0118] The output state value of the category embedding at the output end is subjected to layer normalization processing to generate a corresponding image representation.

[0119] In this embodiment, in the output result of the residual connection, the category embedding vector It is passed and updated in multiple layers of the network, and the output state value of the category embedding vector in the last layer of the network is extracted from the output result of the residual connection. This output state value contains the contextual information of the entire video sequence and the prior knowledge encoded by the category embedding vector. By obtaining the output state value of the category embedding vector at the output end, the model can use the category information to guide the generation of video frames, improving the relevance and accuracy of the generated video. The output state value of the category embedding vector at the output end is layer-normalized to obtain the corresponding image representation y, that is, The normalized output state value can be used as an image representation to guide the generation of the final video frame. By generating image representations that contain rich contextual information and category information, decoding these image representations to generate new image frames, and splicing them into the target video, high-quality conversion from input video to new video content is achieved. It can capture the complex spatiotemporal relationships in the video, generate coherent and realistic video content, better adapt to different data, and improve the compatibility and generation quality of video generation.

[0120] It should be noted that there is not necessarily a certain order between the above steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or may be executed interchangeably, etc.

[0121] Further references Figure 7 , as a response to the above Figure 2 The present invention provides an embodiment of a video generation device, which is similar to the embodiment of the video generation device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 7 As shown, the video generating device 70 described in this embodiment includes:

[0123] An acquisition module 701 is used to acquire original video data, perform preprocessing and data enhancement processing on the original video data, and obtain a plurality of initial frame images;

[0124] A reshaping and embedding module 702 is configured to segment and reshape each initial frame image into a corresponding two-dimensional image block sequence, and perform embedding processing on the two-dimensional image block sequence to generate a corresponding image block embedding sequence;

[0125] An input module 703 is configured to input the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network;

[0126] an attention processing module 704 for performing a multi-head attention mechanism on the tile embedding sequence through the test-time learning layer to obtain an output result of the multi-head attention mechanism;

[0127] A convolution processing module 705 is configured to perform feature extraction and residual connection on the output of the multi-head attention mechanism using the convolutional neural network, and generate a corresponding image representation based on the output of the residual connection;

[0128] Decoding and generation module 706 is configured to decode the image representation corresponding to each initial frame image to generate a new image frame, and then stitch all the new image frames together to produce the target video. A module, as used herein, refers to a series of computer program instruction segments that perform a specific function and is more suitable for describing the video generation process than a program. The specific implementation of each module is described in the corresponding method embodiment above and will not be further elaborated here.

[0129] In one embodiment, the acquisition module 701 includes:

[0130] An extraction unit, configured to obtain original video data and extract frame images from the original video data;

[0131] A pre-processing unit, configured to remove duplicate frames and invalid frames from all extracted frame images to obtain frame images to be enhanced;

[0132] The data enhancement unit is used to horizontally rotate and randomly crop the frame image to be enhanced in sequence to obtain a plurality of data-enhanced initial frame images.

[0133] In one embodiment, the remodeling embedding module 702 includes:

[0134] a segmentation and reshaping unit, configured to determine the number of segmented image blocks and the image block resolution according to the resolution and the number of color channels of the initial frame image, and segment and reshape the initial frame image into corresponding two-dimensional image block sequences according to the number of segmented image blocks and the image block resolution;

[0135] a linear embedding unit, configured to linearly embed each image block in the two-dimensional image block sequence through a linear transformation matrix, so as to perform dimensionality conversion on each image block;

[0136] The splicing unit is used to splice the linear embedding results of all tiles in sequence and add the category embedding vector at the beginning of the splicing result to obtain a splicing sequence;

[0137] a position embedding unit, configured to perform position encoding according to the position information of each image block in the two-dimensional image block sequence to obtain a position embedding matrix;

[0138] A summing unit is configured to sum the concatenated sequence and the position embedding matrix to obtain a corresponding tile embedding sequence.

[0139] In one embodiment, the attention processing module 704 includes:

[0140] a first normalization unit, configured to perform layer normalization processing on the tile embedding sequence to obtain a normalized sequence;

[0141] a multi-head attention unit, configured to perform multi-head attention processing on the normalized sequence through the test-time learning layer to obtain a multi-head attention result;

[0142] The first residual unit is used to perform a residual connection between the tile embedding sequence and the multi-head attention result to obtain the output result of the multi-head attention mechanism.

[0143] In one embodiment, the first normalization unit includes:

[0144] a mean-variance calculation unit, configured to calculate a mean and a variance of the image block embedding sequence according to a feature value of each sample in the image block embedding sequence;

[0145] A normalization calculation unit is used to perform normalization calculation on the eigenvalue of each sample according to the mean and variance of the tile embedding sequence, and obtain the normalized eigenvalue of each sample to form the normalized sequence.

[0146] In one embodiment, the convolution processing module 705 includes:

[0147] A second normalization unit, configured to perform layer normalization processing on the output result of the multi-head attention mechanism to obtain a normalized output result;

[0148] A convolution unit is used to perform convolution operations, pooling operations, and full connection operations on the normalized output results in sequence, extract and integrate features in the normalized output results, and obtain a feature extraction result;

[0149] The second residual unit is used to perform a residual connection on the output result of the multi-head attention mechanism and the feature extraction result to obtain an output result of the residual connection.

[0150] In one embodiment, the convolution processing module 705 further includes:

[0151] a state acquisition unit, configured to acquire an output state value of the category embedding vector at an output end from an output result of the residual connection;

[0152] The third normalization unit is used to perform layer normalization processing on the output state value of the category embedding at the output end to generate a corresponding image representation.

[0153] In the above embodiment, the present invention discloses a video generation device, which obtains raw video data, preprocesses and data augments the raw video data to obtain a plurality of initial frame images; segments and reshapes each initial frame image into a corresponding two-dimensional tile sequence, and embeds the two-dimensional tile sequence to generate a corresponding tile embedding sequence; inputs the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network; processes the tile embedding sequence using a multi-head attention mechanism by the test-time learning layer to obtain an output result of the multi-head attention mechanism; uses the convolutional neural network to perform feature extraction and residual connection on the output result of the multi-head attention mechanism, and generates a corresponding image representation based on the output result of the residual connection; decodes the image representation corresponding to each initial frame image to generate a new image frame, and splices all the new image frames to obtain a target video. By reshaping each frame of the raw video data into a two-dimensional tile sequence for embedding processing, the video generation model having a test-time learning layer, combined with the multi-head attention mechanism and the convolutional neural network, deeply analyzes the tile sequence, extracts features, and generates an image representation. Finally, by decoding these representations to generate new image frames and splicing them into the target video, high-quality conversion from the input video to the new video content is achieved, which can capture the complex spatiotemporal relationships in the video and generate coherent and realistic video content.

[0154] Another embodiment of the present invention provides a computer device, such as Figure 8 As shown, the computer device 80 includes:

[0155] One or more processors 801 and memory 802, Figure 8 In the description, a processor 801 is used as an example. The processor 801 and the memory 802 can be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0156] The processor 801 is used to complete various control logics of the computer device 80. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. In addition, the processor 801 can also be any traditional processor, microprocessor, or state machine. The processor 801 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP, and / or any other such configuration.

[0157] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the video generation method in the embodiments of the present invention. Processor 801 executes the non-volatile software programs, instructions, and modules stored in memory 802 to execute various functional applications and data processing of computer device 80, thereby implementing the video generation method in the above-mentioned method embodiments.

[0158] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the computer device 80, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely located relative to the processor 801, and these remote memories may be connected to the computer device 80 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. One or more units are stored in the memory 802, and when executed by one or more processors 801, the steps of the video generation method in any of the above-mentioned method embodiments are executed.

[0159] In the above embodiment, the present invention discloses a computer device that obtains raw video data, preprocesses and data augments the raw video data to obtain a plurality of initial frame images; segments and reshapes each initial frame image into a corresponding two-dimensional tile sequence, and embeds the two-dimensional tile sequence to generate a corresponding tile embedding sequence; inputs the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network; processes the tile embedding sequence using a multi-head attention mechanism by the test-time learning layer to obtain an output result of the multi-head attention mechanism; uses the convolutional neural network to perform feature extraction and residual connection on the output result of the multi-head attention mechanism, and generates a corresponding image representation based on the output result of the residual connection; decodes the image representation corresponding to each initial frame image to generate a new image frame, and splices all the new image frames to obtain a target video. By reshaping each frame of the raw video data into a two-dimensional tile sequence for embedding, the video generation model having a test-time learning layer, combined with the multi-head attention mechanism and the convolutional neural network, performs in-depth analysis on the tile sequence, extracts features, and generates an image representation. Finally, by decoding these representations to generate new image frames and splicing them into the target video, high-quality conversion from the input video to the new video content is achieved, which can capture the complex spatiotemporal relationships in the video and generate coherent and realistic video content.

[0160] An embodiment of the present invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the video generation method in any of the above method embodiments are performed.

[0161] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains a plurality of initial frame images by obtaining raw video data, preprocessing and data augmenting the raw video data, and dividing and reshaping each initial frame image into a corresponding two-dimensional tile sequence, and embedding the two-dimensional tile sequence to generate a corresponding tile embedding sequence; inputting the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network; processing the tile embedding sequence with a multi-head attention mechanism by the test-time learning layer to obtain an output result of the multi-head attention mechanism; performing feature extraction and residual connection on the output result of the multi-head attention mechanism using the convolutional neural network, and generating a corresponding image representation based on the output result of the residual connection; decoding the image representation corresponding to each initial frame image to generate a new image frame, and splicing all the new image frames to obtain a target video. By reshaping each frame of the raw video data into a two-dimensional tile sequence for embedding, the video generation model having a test-time learning layer, combined with the multi-head attention mechanism and the convolutional neural network, performs in-depth analysis on the tile sequence, extracts features, and generates an image representation. Finally, by decoding these representations to generate new image frames and splicing them into the target video, high-quality conversion from the input video to the new video content is achieved, which can capture the complex spatiotemporal relationships in the video and generate coherent and realistic video content.

[0162] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0163] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0164] In summary, the present invention discloses a video generation method, device, equipment and medium, the method including: obtaining original video data, preprocessing and data enhancement processing on the original video data to obtain several initial frame images; dividing each initial frame image into a corresponding two-dimensional tile sequence, and embedding the two-dimensional tile sequence to generate a corresponding tile embedding sequence; inputting the tile embedding sequence into a video generation model with a test-time learning layer, the hidden state of the test-time learning layer is a convolutional neural network; performing a multi-head attention mechanism on the tile embedding sequence through the test-time learning layer to obtain the output result of the multi-head attention mechanism; using the convolutional neural network to perform feature extraction and residual connection on the output result of the multi-head attention mechanism, and generating a corresponding image representation according to the output result of the residual connection; decoding the image representation corresponding to each initial frame image to generate a new image frame, and splicing all the new image frames to obtain the target video. By reshaping each frame of raw video data into a sequence of two-dimensional tiles for embedding, a video generation model with test-time learning layers, combined with a multi-head attention mechanism and a convolutional neural network, is used to perform in-depth analysis of the tile sequences, extract features, and generate image representations. Finally, these representations are decoded to generate new image frames, which are then spliced ​​into the target video. This achieves high-quality conversion from the input video to new video content, capturing the complex spatiotemporal relationships in the video and generating coherent and realistic video content.

[0165] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. The storage medium can be a memory, a magnetic disk, a floppy disk, a flash memory, an optical storage device, etc.

[0166] It should be noted that if any software tools or components not developed by our company appear in the examples of this application, they are for illustration purposes only and do not represent actual use. It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications shall fall within the scope of protection of the appended claims.

Claims

1. A video generation method, characterized in that: include: Acquire original video data, perform preprocessing and data enhancement processing on the original video data, and obtain a plurality of initial frame images; Each initial frame image is segmented and reshaped into a corresponding two-dimensional block sequence, and the two-dimensional block sequence is embedded to generate a corresponding block embedding sequence; Inputting the tile embedding sequence into a video generation model with a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network; Processing the tile embedding sequence by a multi-head attention mechanism through the test-time learning layer to obtain an output result of the multi-head attention mechanism; Using the convolutional neural network to perform feature extraction and residual connection on the output results of the multi-head attention mechanism, and generating a corresponding image representation according to the output results of the residual connection; The image representation corresponding to each initial frame image is decoded to generate a new image frame, and all the new image frames are spliced ​​together to obtain the target video.

2. The video generation method according to claim 1, wherein: The obtaining of the original video data, performing pre-processing and data enhancement processing on the original video data, and obtaining a plurality of initial frame images, including: Acquire original video data, and extract frame images from the original video data; Removing duplicate frames and invalid frames from all extracted frame images to obtain the frame image to be enhanced; The frame image to be enhanced is horizontally rotated and randomly cropped in sequence to obtain a plurality of initial frame images for data enhancement.

3. The video generation method according to claim 1, wherein: The step of segmenting and reshaping each initial frame image into a corresponding two-dimensional image block sequence, and embedding the two-dimensional image block sequence to generate a corresponding image block embedding sequence includes: Determining the number of segmented blocks and the block resolution according to the resolution and the number of color channels of the initial frame image, and segmenting and reshaping the initial frame image into a corresponding two-dimensional block sequence according to the number of segmented blocks and the block resolution; Linearly embedding each tile in the two-dimensional tile sequence through a linear transformation matrix to perform dimension conversion on each tile; Concatenate the linear embedding results of all tiles in sequence and add the category embedding vector at the beginning of the concatenation result to obtain a concatenated sequence. Performing position encoding according to the position information of each tile in the two-dimensional tile sequence to obtain a position embedding matrix; The concatenated sequence and the position embedding matrix are summed to obtain a corresponding tile embedding sequence.

4. The video generation method according to claim 1, wherein: The test-time learning layer processes the tile embedding sequence through a multi-head attention mechanism to obtain an output result of the multi-head attention mechanism, including: Performing layer normalization on the tile embedding sequence to obtain a normalized sequence; Performing multi-head attention processing on the normalized sequence through the test-time learning layer to obtain a multi-head attention result; Perform a residual connection between the tile embedding sequence and the multi-head attention result to obtain the output result of the multi-head attention mechanism.

5. The video generation method according to claim 4, characterized in that: The performing layer normalization on the tile embedding sequence to obtain a normalized sequence includes: Calculating the mean and variance of the tile embedding sequence according to the feature value of each sample in the tile embedding sequence; The characteristic value of each sample is normalized and calculated according to the mean and variance of the tile embedding sequence, and the normalized characteristic value of each sample is obtained to form the normalized sequence.

6. The video generation method according to claim 1, wherein: The using the convolutional neural network to perform feature extraction and residual connection on the output result of the multi-head attention mechanism includes: Performing layer normalization on the output result of the multi-head attention mechanism to obtain a normalized output result; Performing convolution, pooling, and full connection operations on the normalized output results in sequence, extracting and integrating features in the normalized output results to obtain a feature extraction result; Perform a residual connection on the output result of the multi-head attention mechanism and the feature extraction result to obtain an output result of the residual connection.

7. The video generation method according to claim 3, characterized in that: Generating a corresponding image representation according to the output result of the residual connection includes: Obtaining an output state value of the category embedding vector at an output end from an output result of the residual connection; The output state value of the category embedding at the output end is subjected to layer normalization processing to generate a corresponding image representation.

8. A video generating device, characterized in that: include: An acquisition module is used to acquire original video data, perform preprocessing and data enhancement processing on the original video data, and obtain a plurality of initial frame images; a reshaping and embedding module, configured to segment and reshape each initial frame image into a corresponding two-dimensional image block sequence, and perform embedding processing on the two-dimensional image block sequence to generate a corresponding image block embedding sequence; An input module for inputting the tile embedding sequence into a video generation model having a test-time learning layer, wherein the hidden state of the test-time learning layer is a convolutional neural network; an attention processing module, configured to perform a multi-head attention mechanism on the tile embedding sequence through the test-time learning layer to obtain an output result of the multi-head attention mechanism; A convolution processing module, configured to perform feature extraction and residual connection on the output results of the multi-head attention mechanism using the convolutional neural network, and generate a corresponding image representation based on the output results of the residual connection; The decoding generation module is used to decode the image representation corresponding to each initial frame image to generate a new image frame, and to obtain the target video after splicing all the new image frames.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video generation method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the video generation method according to any one of claims 1 to 7.

Images