Video generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a video generation method and device, equipment, a storage medium and a program product, and belongs to the field of video processing. The method comprises the steps that N * M image block vectors are acquired, the N * M image block vectors are used for representing M image blocks in each video frame in N existing video frames, both N and M are positive integers, and M is not smaller than one; determining M sequences, wherein each sequence in the M sequences comprises N image block vectors with the same indicated spatial positions in the N * M image block vectors; respectively predicting the M sequences in the time dimension to obtain the (N + 1) th image block vector in each sequence; a first video is generated based on the image block vectors within each sequence. According to the method, the N * M image block vectors (corresponding to the N * M image blocks in a one-to-one manner) are split into the M sequences according to the spatial positions, and then prediction is performed on the M sequences, so that the video generation speed can be increased.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to a video generation method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of video generation technology, it is common to see videos generated using artificial intelligence models.

[0003] Among related technologies, techniques for generating videos using autoregressive models have emerged. When applying an autoregressive model to generate video, it is necessary to arrange the existing k image patch vectors (each image patch vector corresponds to an image patch in a video frame) into a one-dimensional sequence, thereby transforming the video generation task into the task of predicting the (k+1)th image patch vector in the sequence, where the image patch vector refers to the data representation of the image patch during actual computation.

[0004] However, when using an autoregressive model to generate video, the generation of each image patch vector depends on all the preceding image patch vectors, which greatly limits the video generation speed. Summary of the Invention

[0005] This application provides a video generation method, apparatus, device, storage medium, and program product. This application can speed up video generation by splitting N*M image block vectors (which correspond one-to-one with N*M image blocks) into M sequences according to their spatial positions, and then performing prediction on the M sequences.

[0006] According to one aspect of this application, a video generation method is provided, the method comprising: Obtain N*M image block vectors. The N*M image block vectors are used to represent M image blocks in each of the N video frames. N and M are both positive integers and M is not less than one. Determine M sequences, where each sequence comprises N image block vectors that indicate the same spatial location from N*M image block vectors; For each of the M sequences, prediction is performed along the time dimension to obtain the (N+1)th image patch vector within each sequence; The first video is generated based on the image patch vectors within each sequence.

[0007] According to another aspect of this application, a video generation method is provided, the method comprising: Obtain P*Q image block vectors. The P*Q image block vectors are used to represent the Q image blocks that are present in each of the P video frames. Each video frame is used to accommodate M image blocks. P and Q are both positive integers, and P is not less than one and Q is less than M. Determine P sequences, each of the P sequences comprising Q image block vectors that indicate the same time point in P*Q image block vectors; For each of the P sequences, a prediction is made in the spatial dimension to obtain the Q+1th image patch vector in each sequence; A second video is generated based on the image patch vectors within each sequence.

[0008] According to another aspect of this application, a video generation apparatus is provided, the apparatus comprising: The acquisition module is used to acquire N*M image block vectors. The N*M image block vectors are used to represent M image blocks in each of the N video frames that already exist. N and M are both positive integers and M is not less than one. The determination module is used to determine M sequences, where each of the M sequences includes N image block vectors that indicate the same spatial location in N*M image block vectors; The prediction module is used to predict M sequences in the time dimension to obtain the N+1th image patch vector in each sequence; The generation module is used to generate the first video based on the image patch vectors within each sequence.

[0009] According to another aspect of this application, a video generation apparatus is provided, the apparatus comprising: The acquisition module is used to acquire P*Q image block vectors. The P*Q image block vectors are used to represent the Q image blocks that are present in each of the P video frames. Each video frame is used to accommodate M image blocks. P and Q are both positive integers, and P is not less than one and Q is less than M. The determination module is used to determine P sequences, each of which includes Q image block vectors that indicate the same time point in P*Q image block vectors; The prediction module is used to predict the spatial dimensions of P sequences respectively, and obtain the Q+1th image patch vector in each sequence; The generation module is used to generate a second video based on the image patch vectors within each sequence.

[0010] According to one aspect of this application, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the video generation method described above.

[0011] According to one aspect of this application, a computer-readable storage medium is provided, which stores a computer program that is loaded and executed by a processor to implement the video generation method described above.

[0012] According to one aspect of this application, a computer program product is provided, which stores a computer program that is loaded and executed by a processor to implement the video generation method described above.

[0013] The beneficial effects of the technical solutions provided in this application include at least the following:

[0014] This application divides N*M image block vectors into M sequences and then performs predictions on the M sequences. Compared with related technologies that flatten N*M image block vectors into a one-dimensional sequence and perform predictions on the one-dimensional sequence in a strictly serialized manner, this application can perform predictions in parallel in spatial location, which can make full use of the parallel computing power of the graphics processor, thereby significantly improving computational efficiency and video generation speed.

[0015] Furthermore, compared to related techniques that require relying on all preceding image block vectors when predicting each image block vector, the method provided in this application only requires relying on preceding image block vectors at the same spatial location, reducing redundant computation and lowering the consumption of computational resources. The method in this application is more suitable for high-resolution and long-time sequence video generation tasks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a video generation method provided in one embodiment of this application.

[0018] Figure 2 This is a flowchart of a video generation method provided in one embodiment of this application.

[0019] Figure 3 This is a schematic diagram of a video generation method provided in one embodiment of this application.

[0020] Figure 4 This is a flowchart of a video generation method provided in one embodiment of this application.

[0021] Figure 5 This is a flowchart of a video generation method provided in one embodiment of this application.

[0022] Figure 6 This is a flowchart of a video generation method provided in one embodiment of this application.

[0023] Figure 7This is a schematic diagram of a video generation method provided in one embodiment of this application.

[0024] Figure 8 This is a flowchart of a video generation method provided in one embodiment of this application.

[0025] Figure 9 This is a schematic diagram of a video generation method provided in one embodiment of this application.

[0026] Figure 10 This is a flowchart of a video generation method provided in one embodiment of this application.

[0027] Figure 11 This is a flowchart of a video generation method provided in one embodiment of this application.

[0028] Figure 12 This is a structural block diagram of a video generation apparatus provided in one embodiment of this application.

[0029] Figure 13 This is a structural block diagram of a video generation apparatus provided in one embodiment of this application.

[0030] Figure 14 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] First, some terms used in the embodiments of this application will be introduced: Video generation technology refers to the technology of automatically generating videos using artificial intelligence (AI) models. It is widely used in film and television production, game development, advertising design, education, and other fields. Among related technologies, generative adversarial networks (GANs), diffusion models, or autoregressive models are commonly used for video generation.

[0033] Autoregressive models are artificial intelligence models that predict future data based on sequential data. Their core idea is to predict future data based on past data. In machine learning and deep learning, autoregressive models are widely used to generate continuous data (such as text, audio, and video). When using an autoregressive model to generate video, the existing k image patch vectors (each corresponding to one of the k image patches) need to be arranged into a one-dimensional sequence, thus transforming the video generation task into predicting the (k+1)th image patch vector in the sequence. However, when using an autoregressive model to generate video, the generation of each image patch vector depends on all preceding image patch vectors, resulting in a strictly serialized generation process. This greatly limits the speed of video generation, especially in high-resolution and long-term video generation tasks, where the computational overhead and time cost are particularly prominent.

[0034] Figure 1 A schematic diagram of a video generation method provided in an exemplary embodiment of this application is shown.

[0035] To illustrate, during the video generation process, N video frames 101 are acquired. These N video frames 101 are existing video frames (for example, they are already generated video frames, or they are video frames acquired through other channels). Each of the N video frames 101 is divided into M image blocks and quantized to obtain N*M image block vectors 102. These N*M image block vectors 102 correspond one-to-one with the N*M image blocks. To illustrate, Figure 1 In this process, image block vectors belonging to the same video frame are marked with the same number (such as "1", "2", "3"), and image block vectors belonging to the same spatial location are marked with the same letter (such as A, B, C). Therefore, each image block vector has a unique number, such as "A1", "B1", "C1", etc. Figure 1 The diagram shows how each of the three video frames is divided into four image blocks and each image block is vectorized to obtain twelve image block vectors.

[0036] Optional, image patch vector, refers to the data representation of image patches during the operation.

[0037] For N*M image block vectors 102, the N image block vectors indicating the same spatial location are identified as the same sequence, resulting in M ​​sequences 103. Spatial location refers to the position of an image block within its respective video frame. Since each image block vector corresponds to one image block, each image block vector can indicate a spatial location. (Illustratively, in...) Figure 1In the N*M image block vectors 102, the three image block vectors indicating spatial position "A" are determined to be in the same sequence, the three image block vectors indicating spatial position "B" are determined to be in the same sequence, the three image block vectors indicating spatial position "C" are determined to be in the same sequence, and the three image block vectors indicating spatial position "D" are determined to be in the same sequence.

[0038] For each of the M sequences 103, the next image patch vector is predicted along the time dimension, resulting in the (N+1)th image patch vector within each sequence. For a given sequence, there are N image patch vectors corresponding to N time points (N video frames). Based on these N image patch vectors, the (N+1)th image patch vector corresponding to the next time point (the (N+1)th time point) can be predicted. By combining the (N+1)th image patch vectors from each of the M sequences 103, the video frame vector corresponding to the (N+1)th video frame can be obtained. Furthermore, by continuously predicting the video frame vectors of the next frame, the final generated video can be obtained.

[0039] In summary, the video generation method provided in this application divides N*M image patch vectors into M sequences (each of the M sequences is a dimension of the M-dimensional sequence), and then performs parallel prediction on the M sequences. Compared with related technologies that flatten N*M image patch vectors into a one-dimensional sequence and perform prediction on the one-dimensional sequence in a strictly serialized manner, this application can perform prediction in parallel at spatial locations, thereby making full use of the parallel computing capabilities of the graphics processor and significantly improving computational efficiency and video generation speed.

[0040] In some embodiments, the above Figure 1 The provided video generation method can be executed by computer equipment, which includes at least one of terminal equipment and a server. That is, as described above. Figure 1 The provided video generation method can be executed by a terminal device (optionally, the terminal device has a video generation model deployed locally, and the video generation model applies the video generation method provided in this application), or by a server, or by the terminal device and the server working together.

[0041] In one scenario, an artificial intelligence program is installed on the terminal device. In response to a user inputting a text description through the artificial intelligence program, such as "a video with flowers" or "a video with many vehicles on the road," the server applies the video generation method provided in this application to generate a video that matches the text description.

[0042] In another scenario, a video production software is installed on the terminal device. The video production software provides a video generation plugin. In response to the user inputting a text description and / or sketch through the video generation plugin, the server applies the video generation method provided in this application to generate a video that conforms to the text description and / or sketch.

[0043] Optionally, the device type of the terminal device includes at least one of the following: smartphone, smartwatch, in-vehicle terminal, wearable device, smart TV, tablet computer, e-book reader, MP3 player, MP4 player, laptop computer, and desktop computer.

[0044] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0045] Figure 2 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, the method being performed by a computer device, the method comprising: Step 220: Obtain N*M image block vectors. The N*M image block vectors are used to represent M image blocks in each of the N video frames. N and M are both positive integers and M is not less than one. In one embodiment, N video frames are acquired, and these N video frames are divided into image blocks according to a uniform partitioning method, resulting in M ​​image blocks within each video frame. Since each video frame uses the same partitioning method, any two video frames after partitioning have identical image block sizes, number of image blocks, and image block positions. For example, if one video frame is divided into four 4*4 image blocks and one 8*8 image block, then any other video frame will also be divided into four 4*4 image blocks and one 8*8 image block. By partitioning the N video frames into image blocks according to a uniform partitioning method, it is beneficial to subsequently divide the N*M image block vectors into M sequences based on their spatial positions.

[0046] Optionally, the M image blocks divided within a video frame include at least one size, such as 4*4, 8*8, 16*16, etc., with units in pixels. Illustratively, the M image blocks divided within a video frame may only include image blocks of size 4*4. Alternatively, the M image blocks divided within a video frame may include both image blocks of size 4*4 and image blocks of size 8*8.

[0047] Optionally, if the M image blocks divided within a video frame do not overlap, the video frame can be reconstructed from the M image blocks using a non-overlapping stitching method. Optionally, if at least two of the M image blocks divided within a video frame overlap, the video frame can be reconstructed from the M image blocks using an overlapping stitching method. It is understood that this application does not limit the image block division method within a video frame; it only requires ensuring that the image block division method is consistent between any two video frames.

[0048] In one embodiment, the N*M image blocks obtained from the division are vectorized to obtain N*M image block vectors, and there is a one-to-one correspondence between the N*M image block vectors and the N*M image blocks. The image block vector refers to the data representation of the image block during the operation. The image block is represented as an image block vector and then participates in the operation process.

[0049] Optionally, the image patch vector is an image patch token, which is the basic computational unit in the video generation process. Optionally, each image patch is mapped to a discrete image patch token using techniques such as VQ-VAE (Vector Quantized Variational Autoencoder) and Patch-based Transformer.

[0050] Step 240: Determine M sequences, where each of the M sequences includes N image block vectors that indicate the same spatial location in N*M image block vectors; M sequences can be understood as an M-dimensional sequence, where each sequence represents one dimension of the M-dimensional sequence. In one embodiment, N image patch vectors indicating the same spatial location from N*M image patch vectors are identified as the same sequence, resulting in M ​​sequences, each corresponding to a spatial location. The spatial location refers to the position of the image patch within its respective video frame; for example, a spatial location like (1, 1) or (2, 2). Since each image patch vector corresponds to one image patch, each image patch vector can indicate the spatial location of the corresponding image patch within its respective video frame. (Refer to the reference...) Figure 1 Any two video frames are divided into image blocks that occupy the same spatial position, and each video frame is divided into image blocks that occupy four positions: upper left, lower left, upper right, and lower right.

[0051] In one embodiment, N image patch vectors in a sequence are ordered from furthest to nearest according to the indicated time. The N image patch vectors correspond one-to-one with N video frames, and in a sequence, the N image patch vectors are arranged in order from furthest to nearest according to the N video frames.

[0052] Step 260: Predict the M sequences in the time dimension to obtain the N+1th image patch vector in each sequence.

[0053] Each of the M sequences comprises N image patch vectors arranged in chronological order. In one embodiment, at least two sequences among the M sequences undergo prediction simultaneously at a single time point. For example, prediction is performed simultaneously on all M sequences at each time point; for example, prediction of sequence 2 begins before the prediction of sequence 1 ends, prediction of sequence 3 begins before the prediction of sequence 2 ends, and so on; for example, prediction is performed simultaneously on M1 sequences, and prediction of the remaining M2 sequences begins only after the prediction of M1 sequences is completed. In one embodiment, the M sequences are predicted in parallel along the time dimension to obtain the (N+1)th image patch vector in each sequence. Optionally, the M sequences are predicted in parallel using the same prediction tool to obtain the (N+1)th image patch vector in each sequence. Optionally, the M sequences are predicted in parallel using different prediction tools to obtain the (N+1)th image patch vector in each sequence.

[0054] In one embodiment, at least one autoregressive model is used to predict M sequences in the time dimension to obtain the N+1th image patch vector in each sequence.

[0055] Optionally, M autoregressive models are used to predict the N+1th image patch vector in each of the M sequences along the time dimension. For example, for the first sequence, the first autoregressive model is used to predict the N+1th image patch vector in the time dimension. For the second sequence, the second autoregressive model is used to predict the N+1th image patch vector in the time dimension, and so on. Optionally, any two sequences among the M sequences may use the same or different autoregressive models. For example, the first and second autoregressive models mentioned above can be the same or different autoregressive models. Optionally, the autoregressive model includes any one of the following: a Transformer-based model, a causal convolutional model, or a recurrent neural network-based model.

[0056] Optionally, using M image generation channels in an autoregressive model, predictions are made on M sequences along the time dimension to obtain the (N+1)th image patch vector in each sequence. Optionally, for a sequence, the (N+1)th image patch vector is predicted based on the N image patch vectors using one image generation channel in the autoregressive model. Optionally, the M image generation channels are independent of each other; alternatively, the M image generation channels are correlated with each other.

[0057] For example, when predicting the first sequence using the first image generation channel, information generated by other image generation channels during the prediction process can be fused to enhance the correlation between image patches at different spatial locations in the final generated sequence. For instance, in a deep neural network autoregressive model, during parallel prediction, the M hidden features generated by the M image generation channels in the target hidden layer are fused into global features. These global features are then fused with the hidden features originally generated by the first image generation channel in the target hidden layer to obtain fused features. These fused features then replace the hidden features generated by the first image generation channel and are input into the next network layer.

[0058] Step 280: Generate the first video based on the image patch vectors within each sequence.

[0059] In each of the M sequences, prediction is performed along the time dimension to obtain the next image patch vector. In one embodiment, when the number of image patch vectors in each sequence reaches a video frame count threshold, the generation of the next image patch vector stops, and the first video is generated based on the existing image patch vectors in each of the M sequences. Optionally, the video frame count threshold is a pre-set threshold. Optionally, based on the existing image patch vectors in each of the M sequences, the image patches corresponding to all image patch vectors are reconstructed, and all image patches are stitched together in temporal and spatial order to obtain the first video.

[0060] In one embodiment, whenever M sequences predict image patch vectors corresponding to the same time point, the M image patch vectors corresponding to the same time point are extracted, and M image patches are reconstructed. These M image patches are then stitched together according to their spatial relationships to obtain a video frame from the first video. This process is repeated, reconstructing and stitching together a video frame each time an image patch vector corresponding to the same time point is generated, avoiding the need to wait until all image patch vectors have been generated before reconstructing and stitching together to obtain the first video. Optionally, the time point at which each sequence stops predicting the next image patch vector is when the number of generated image patch vectors reaches a threshold for the number of video frames. In summary, the video generation method provided in this application divides N*M image block vectors (used to represent M image blocks in each of the N video frames) into M sequences (each of the M sequences is a dimension of the M-dimensional sequence), and then performs parallel prediction on the M sequences. Compared with related technologies that flatten the N*M image block vectors into a one-dimensional sequence and perform prediction on the one-dimensional sequence according to strict serialization, this application can perform prediction in parallel in spatial location, making full use of the parallel computing power of the graphics processor, thereby significantly improving computational efficiency and video generation speed.

[0061] Understandably, through Figure 2The illustrated video generation method can generate the (N+1)th video frame based on the existing N video frames, and then generate the (N+2)th video frame based on the N+1 video frames, and so on, to generate the entire video. Optionally, the length of the final generated video is a preset value, such as a preset value to generate a 30-second video or a 5-minute video.

[0062] Optionally, when N is one, that is, the very first video frame in the video generation process, it can be an image generated by other image generation methods, such as an image generated by a generative adversarial network, a diffusion model, or a reference image specified by the user for video generation, or an existing image obtained from an image library. This application does not limit the method of obtaining the first video frame.

[0063] Figure 3 This is a schematic diagram of a video generation process provided by another exemplary embodiment of this application.

[0064] To illustrate, during video generation, N video frames 301 are acquired. These N video frames 101 are existing video frames. Each of the N video frames 301 is divided into M image blocks and quantized to obtain N*M image block vectors 302. Each of the N*M image block vectors 302 corresponds one-to-one with one of the N*M image blocks. To illustrate, Figure 3 In this process, image block vectors belonging to the same video frame are marked with the same number (such as "1", "2", "3"), and image block vectors belonging to the same spatial location are marked with the same letter (such as A, B, C). Therefore, each image block vector has a unique number, such as "A1", "B1", "C1", etc. Figure 3 The diagram shows how to divide each of the three video frames into four image blocks and vectorize each image block to obtain twelve image block vectors.

[0065] Optional, image patch vector, refers to the data representation of image patches during the operation.

[0066] For N*M image block vectors 302, global spatial information is calculated based on the M image block vectors indicating the same time point, resulting in N global spatial information pieces 303. These N global spatial information pieces 303 correspond one-to-one with N video frames 301. For example... Figure 3As shown, for video frame 1, a global spatial information will be generated based on the four image block vectors (image block vectors A1, B1, C1, D1) corresponding to video frame 1; for video frame 2, a global spatial information will be generated based on the four image block vectors (image block vectors A2, B2, C2, D2) corresponding to video frame 2; and for video frame 3, a global spatial information will be generated based on the four image block vectors (image block vectors A3, B3, C3, D3) corresponding to video frame 3.

[0067] Global spatial information is used to characterize the spatial information of a video frame. Optionally, global spatial information can be used to characterize the overall image information of a video frame.

[0068] Perform a copy operation on N global space information 303 to obtain N*M global space information.

[0069] Based on the temporal correspondence, each global spatial information element is combined with its corresponding image patch vector, resulting in N*M combined vectors (304). The temporal correspondence refers to the relationship to the same video frame. Figure 3 In this process, three global spatial information units are copied to obtain twelve global spatial information units. Then, each global spatial information unit is concatenated and combined with its corresponding image patch vector to obtain twelve combined vectors. (Illustrative example) Figure 3 The global spatial information of video frame 1 is combined into image block vectors A1, B1, C1 and D1, the global spatial information of video frame 2 is combined into image block vectors A2, B2, C2 and D2, and the global spatial information of video frame 3 is combined into image block vectors A3, B3, C3 and D3.

[0070] The N*M combination vectors 304 are sorted to obtain M sequences 305. Each of the M sequences 305 includes N combination vectors with the same spatial location indicated by the N*M combination vectors 304. Each combination vector includes a global spatial information and an image patch vector.

[0071] For each of the M sequences 305, the next image patch vector is predicted along the time dimension, resulting in the (N+1)th image patch vector within each sequence. For a given sequence, there are N combined vectors corresponding to N time points (N video frames). Based on these N combined vectors, the (N+1)th image patch vector corresponding to the next time point (the (N+1)th time point) can be predicted. By combining the (N+1)th image patch vectors of the M sequences 103, the video frame vector corresponding to the (N+1)th video frame can be obtained. Furthermore, by continuously predicting the video frame vectors of the next frame, the final generated video can be obtained.

[0072] Understandably, in Figure 1In the illustrated embodiment, M sequences corresponding to M spatial locations are obtained, each sequence including N image patch vectors. Parallel prediction is then performed on the M sequences. While this method effectively speeds up image generation, it loses the spatial information of the video frames. Based on this, Figure 3 The illustrated embodiment compensates for the lack of spatial information by generating global spatial information for each video frame and combining it with the corresponding image block vector of the video frame. This not only effectively speeds up image generation but also further improves the quality of the generated video frames.

[0073] Figure 4 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, with the method being executed by a computer device. Optionally, Figure 4 Steps 420 to 460 are located in Figure 2 Prior to step 240 shown, the method includes: Step 420: Obtain N*M global spatial information. The N*M global spatial information includes N types of global spatial information. The N types of global spatial information correspond one-to-one with N video frames. Each type of global spatial information is used to characterize the spatial information of a video frame. Global spatial information is used to characterize the spatial information of a video frame; specifically, it characterizes the overall image information of a video frame. Optionally, global spatial information can also characterize the relationships between M image patches within a video frame. For example, if image patch 1 contains key feature points of a face, the global spatial information implies that image patch 1 is the most important image patch among the M image patches. Or, if image patch 2 is a background image patch in the video frame, the global spatial information implies that image patch 2 is a less important image patch among the M image patches. Optionally, global spatial information can be represented in vector form; it can also be called a global spatial token.

[0074] In one embodiment, for N time points indicated by N*M image block vectors, global spatial information is calculated based on the M image block vectors corresponding to each time point to obtain N global spatial information. The global spatial information is used to characterize the spatial information of a video frame. The N global spatial information is copied M times to obtain N*M global spatial information.

[0075] For N*M image patch vectors, since each image patch vector corresponds to one image patch, and one image patch comes from its corresponding video frame, and N*M image patches come from N video frames, the N*M image patch vectors can indicate N time points corresponding to the N video frames. Global spatial information is calculated for the M image patch vectors corresponding to each time point, resulting in N global spatial information pieces, which correspond one-to-one with the N time points. It can be understood that in this embodiment, one global spatial information piece is calculated for each video frame; therefore, a total of N global spatial information pieces are calculated for the N video frames.

[0076] Optionally, for N time points indicated by N*M image patch vectors, a self-attention mechanism is used to calculate global spatial information based on the M image patch vectors corresponding to each time point, resulting in N global spatial information pieces. The self-attention mechanism calculates the correlation between each image patch vector and other image patch vectors in the M image patch vectors to obtain attention weights. Then, a weighted image patch vector is generated for each image patch vector using these attention weights. In this embodiment, the global spatial information includes all the weighted image patch vectors. It can be understood that the global spatial information obtained through the self-attention mechanism carries the correlation between any two image patch vectors among the M image patch vectors.

[0077] Understandably, by calculating global spatial information through the self-attention mechanism, not only can we obtain the image information of the video frame, but we can also further obtain the distribution of image blocks within the video frame, know which image blocks are key image blocks (such as image blocks carrying facial features), and know which image blocks are non-key image blocks (such as background image blocks), thus improving the accuracy of subsequent video frames generated using image block vectors.

[0078] Optionally, for N time points indicated by N*M image patch vectors, global spatial information is calculated using a weighted summation method based on the M image patch vectors corresponding to each time point, resulting in N global spatial information pieces. Illustratively, the average of the M image patch vectors is taken to obtain one global spatial information piece, the size of which is consistent with the size of an image patch vector.

[0079] In another embodiment, for N time points indicated by N*M image block vectors, global spatial information is calculated based on at least two image block vectors corresponding to each time point to obtain N global spatial information. The global spatial information is used to characterize the spatial information of a video frame. Each of the N global spatial information is copied M times to obtain N*M global spatial information.

[0080] Optionally, each time point corresponds to M image patch vectors. At least two image patch vectors that satisfy preset conditions are selected from these M vectors, and global spatial information is calculated based on these at least two image patch vectors. For example, image patch vectors corresponding to image patches located at the image edge and image center are selected from the M vectors to obtain at least two image patch vectors. Alternatively, image patch vectors corresponding to image patches located in the foreground region of the image are selected from the M vectors to obtain at least two image patch vectors. Or, image patch vectors corresponding to image patches containing entity contours are selected from the M vectors to obtain at least two image patch vectors.

[0081] Optionally, for N time points, based on at least two image patch vectors corresponding to each time point, a self-attention mechanism is used to calculate global spatial information, resulting in N global spatial information pieces. The self-attention mechanism calculates the correlation between each image patch vector and other image patch vectors within the at least two image patch vectors to obtain attention weights. These attention weights are then used to generate a weighted image patch vector for each of the at least two image patch vectors. In this embodiment, the global spatial information includes all the weighted image patch vectors. It can be understood that the global spatial information obtained through the self-attention mechanism carries the correlation between any two image patch vectors within the at least two image patch vectors.

[0082] In one embodiment, each of the N global spatial information pieces is copied M-1 times to obtain N*(M-1) global spatial information pieces; combined with the original sample, a total of N*M global spatial information pieces are obtained. It can be understood that this application copies the global spatial information of each video frame to obtain M copies of the global spatial information, therefore, for N video frames, a total of N*M global spatial information pieces will be generated.

[0083] Step 440: According to the time correspondence, combine the N*M global spatial information with the N*M image patch vectors one by one to obtain N*M combined vectors; Since N*M global spatial information pieces correspond to N video frames, these N*M global spatial information pieces correspond to N time points, and the N*M image patch vectors also correspond to N time points. Therefore, through the time correspondence, the N*M global spatial information pieces and the N*M image patch vectors can be combined one by one to obtain N*M combined vectors. It can be understood that M image patch vectors from the same video frame will be combined with the global spatial information of the same video frame. Optionally, the combination method includes either concatenation or summation.

[0084] Step 460: Sort the N*M combination vectors to obtain M sequences. Each of the M sequences includes N combination vectors that have the same spatial position indicated by the N*M combination vectors.

[0085] In this embodiment, the N*M combination vectors correspond one-to-one with the N*M image block vectors. Since each image block vector corresponds to one image block, each image block vector can indicate the spatial position of the corresponding image block in the video frame. Therefore, each combination vector can indicate the spatial position of the corresponding image block in the video frame.

[0086] In one embodiment, N binding vectors in a sequence are ordered from furthest to nearest according to the indicated time. Each of the N binding vectors corresponds one-to-one with a N image patch vector, and each of the N image patch vectors corresponds one-to-one with a N video frame. Therefore, the N binding vectors can be arranged in order of the N video frames from furthest to nearest.

[0087] In one embodiment, prediction is performed on M sequences along the time dimension to obtain the (N+1)th image patch vector within each sequence. Optionally, for a sequence, the (N+1)th image patch vector is predicted based on N combined vectors using an autoregressive model. Optionally, for a sequence, the (N+1)th image patch vector is predicted based on N combined vectors using an image generation channel within the autoregressive model; for detailed explanation, please refer to step 260 above.

[0088] In conclusion, Figure 2 In the illustrated embodiment, M sequences corresponding to M spatial locations are obtained, each sequence including N image block vectors. Parallel prediction is then performed on the M sequences. While this method can effectively speed up image generation, it loses the spatial information of the video frames. Based on this, the above embodiment generates global spatial information for each video frame and combines it with the image block vector corresponding to that video frame, thus compensating for the lack of spatial information. This further improves the quality of the generated video frames while effectively speeding up image generation.

[0089] Application Scenario Examples In some embodiments, Figure 5 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, the method being performed by a computer device, the method comprising: Step 520: Obtain N video frames, including the start frame and end frame of the target video; Optionally, the N video frames include a start frames and b end frames of the target video (the sum of a and b equals N). For example, the N video frames include the first three frames and the last three frames of the target video, and the video generation method provided in this application is used to generate video frames between the first three frames and the last three frames.

[0090] Step 540: Divide each of the N video frames into M image blocks and quantize them to obtain N*M image block vectors; For details regarding the splitting method, please refer to step 220 above; it will not be repeated here.

[0091] Step 560: Determine M sequences, where each of the M sequences includes N image block vectors that indicate the same spatial location in N*M image block vectors; In one embodiment, N image block vectors that indicate the same spatial location among N*M image block vectors are determined to be the same sequence, resulting in M ​​sequences, each sequence corresponding to a spatial location.

[0092] Optionally, when the N video frames include a start frames and b end frames of the target video, the N image patch vectors in a sequence are arranged from furthest to nearest according to the indicated time. For example, in a sequence, the a image patch vectors corresponding to the a start frames are located in the first a positions of the sequence, and the b image patch vectors corresponding to the b end frames are located in the last b positions of the sequence.

[0093] Step 580: Predict the M sequences in the time dimension to obtain the N+1th image patch vector in each sequence.

[0094] Optionally, following the sorting method described above, for a sequence, prediction is performed in the time dimension to obtain the image patch vector ordered at position a+1 within the sequence. Alternatively, for a sequence, prediction is performed in the time dimension to obtain the image patch vector ordered at position b+1 from the end of the sequence.

[0095] The above embodiments provide a scheme for predicting intermediate frames of a given target video, given the start and end frames. By performing parallel prediction at M spatial locations, the generation speed of intermediate frames is improved.

[0096] In some embodiments, Figure 6 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, the method being performed by a computer device, the method comprising: Step 620: Obtain N video frames, which are the video frames of the first video; Optionally, the first video is a video generated using other video generation methods. Optionally, the first video is a video obtained from a video library. Optionally, the first video has a first duration. This embodiment aims to extend the first video to obtain a second video, which has a second duration greater than the first duration.

[0097] Step 640: Divide each of the N video frames into M image blocks and quantize them to obtain N*M image block vectors. The N*M image block vectors are used to represent the M image blocks in each of the N video frames. For details regarding the splitting method, please refer to step 220 above; it will not be repeated here.

[0098] Step 660: Determine M sequences, where each of the M sequences includes N image block vectors that indicate the same spatial location in N*M image block vectors; For details regarding step 660, please refer to the description of step 240 above.

[0099] Step 680: Predict the M sequences in the time dimension to obtain the N+1th image patch vector in each sequence.

[0100] Optionally, for a sequence containing N image patch vectors, the sequence is predicted along the time dimension to obtain the (N+1)th image patch vector within the sequence. Furthermore, based on the (N+1)th image patch vector, the sequence can be further used to predict the (N+2)th image patch vector, and so on, until a second video is generated. Optionally, the duration of the second video is preset; for example, the first video is 60 seconds long, and the second video is 70 seconds long. For M sequences, by continuously generating image patch vectors for the next video frame until the preset duration is reached, the M sequences will eventually obtain all the image patch vectors for the second video, thus generating the second video.

[0101] The above embodiments provide a scheme for extending a given first video to obtain a second video. By performing parallel predictions at M spatial locations, the speed of video extension is improved.

[0102] The previous section introduced a scheme for predicting multiple spatial locations in parallel along the temporal dimension. The following section will introduce a scheme for predicting multiple video frames in parallel along the spatial dimension.

[0103] Figure 7 A schematic diagram of a video generation process provided in an exemplary embodiment of this application is shown.

[0104] To illustrate, during the video generation process, P video frames 701 are acquired, where each video frame already contains Q image blocks, such as... Figure 7 As shown, each of the three video frames already contains three image blocks. The method provided in this embodiment aims to complete the image blocks in each video frame to obtain the final video.

[0105] Each of the P video frames 701 is divided into image blocks, resulting in Q image blocks within each video frame. Then, each image block is vectorized to obtain P*Q image block vectors 702. (Illustrative example) Figure 7 In this process, image block vectors belonging to the same video frame are marked with the same number (such as "1", "2", "3"), and image block vectors belonging to the same spatial location are marked with the same letter (such as A, B, C). Therefore, each image block vector has a unique number, such as "A1", "B1", "C1", etc.

[0106] For P*Q image block vectors 702, image block vectors indicating the same time point are grouped into the same sequence, resulting in P sequences, each corresponding to a video frame (one time point). (Illustratively, in...) Figure 7 In the P*Q image block vectors 702, the three image block vectors indicating video frame "1" are determined to be the same sequence, the three image block vectors indicating video frame "2" are determined to be the same sequence, and the three image block vectors indicating video frame "3" are determined to be the same sequence.

[0107] For each of the P sequences, prediction is performed in the spatial dimension to obtain the (Q+1)th image patch vector within each sequence. This can be understood as follows: predicting a sequence involves using the Q image patch vectors from the same video frame to predict the (Q+1)th image patch vector of that video frame, thus achieving spatial prediction. For each sequence, after generating the (Q+1)th image patch vector, the (Q+2)th image patch vector can be predicted based on the existing Q+1 image patch vectors, and so on, until the entire video frame is completed. For example, if a video frame has M spatial locations, the image patch at the next spatial location is predicted continuously for each video frame until the M image patches for each video frame are obtained.

[0108] In summary, the above embodiments split P*Q image block vectors into P sequences (each of the P sequences being a dimension of a P-dimensional sequence), and then perform parallel prediction on the P sequences. Compared to related technologies where P*Q image block vectors are expanded into a one-dimensional sequence and prediction is strictly performed serially on the one-dimensional sequence, this application can perform parallel prediction on P video frames, which can fully utilize the parallel computing power of the graphics processor, thereby significantly improving computational efficiency and video generation speed.

[0109] In some embodiments, the above Figure 7 The provided video generation method can be executed by computer equipment, which includes at least one of terminal equipment and server, i.e., the above. Figure 7The provided video generation method can be executed by a terminal device (optionally, the terminal device has a video generation model deployed locally, and the video generation model applies the video generation method provided in this application), or by a server, or by the terminal device and the server working together.

[0110] In an optional embodiment, Figure 7 The provided video generation method can be applied to video restoration scenarios. The server acquires a damaged video, which consists of P video frames. Within these P video frames, at least one image block in each frame is blurred, contaminated, or otherwise damaged. This is then processed... Figure 7 The method provided aims to fully recover image blocks within each video frame in the case of a corrupted video.

[0111] Figure 8 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, the method being performed by a computer device, the method comprising: Step 820: Obtain P*Q image block vectors. The P*Q image block vectors are used to represent the Q image blocks that are present in each of the P video frames. Each video frame has M spatial locations. P and Q are both positive integers, and P is not less than one and Q is less than M. In one embodiment, P video frames are obtained, which represent all the video frames in the final generated video. Each of the P video frames has M spatial locations. The P video frames are incomplete; this embodiment aims to complete the image blocks within each video frame. The P video frames are divided into image blocks, resulting in Q image blocks within each video frame. Optionally, any two video frames in the P video frames may use the same image block division method. For example, each of the P video frames may be divided into image blocks at the upper left, lower left, and upper right positions.

[0112] In one embodiment, the P*Q image blocks obtained from the partitioning are vectorized to obtain P*Q image block vectors, with each of the P*Q image block vectors corresponding one-to-one with a P*Q image block. The image block vector refers to the data representation of the image block during the computation process; the image block is represented as an image block vector and thus participates in the computation.

[0113] Optionally, the image patch vector is an image patch token, which is the basic computational unit in the video generation process. Optionally, each image patch is mapped to a discrete image patch token using techniques such as VQ-VAE (Vector Quantized Variational Autoencoder) and Patch-based Transformer.

[0114] Step 840: Determine P sequences, where each of the P sequences includes Q image block vectors that indicate the same time point in P*Q image block vectors; P sequences can be understood as a P-dimensional sequence, where each sequence represents a dimension of the P-dimensional sequence. In one embodiment, Q image patch vectors indicating the same time point from P*Q image patch vectors are identified as the same sequence, resulting in P sequences, each corresponding to a video frame. The time point refers to the time corresponding to the video frame; for example, the time point corresponding to the t-th video frame is t, and the time point corresponding to the (t+1)-th video frame is t+1. Since each image patch vector corresponds to an image patch, and each image patch comes from its respective video frame, each image patch vector can indicate the time point of the corresponding video frame. (Refer to the reference...) Figure 7 Each sequence of image block vectors comes from the same video frame.

[0115] In one embodiment, the Q image patch vectors in a sequence are ordered according to their indicated spatial locations, for example, from left to right and from top to bottom.

[0116] Step 860: Predict the spatial dimensions of each of the P sequences to obtain the Q+1th image patch vector in each sequence; Each of the P sequences includes Q image patch vectors arranged spatially. In one embodiment, at least two sequences in the P sequences undergo prediction simultaneously at a single time point. For example, prediction is performed simultaneously on all P sequences at each time point; for example, prediction of sequence 2 begins before the prediction of sequence 1 ends, prediction of sequence 3 begins before the prediction of sequence 2 ends, and so on; for example, prediction is performed simultaneously on P1 sequences out of the P sequences, and prediction of the remaining P2 sequences begins only after the prediction of P1 sequences is completed. In one embodiment, the P sequences are predicted spatially in parallel to obtain the (Q+1)th image patch vector in each sequence. Optionally, the same prediction tool is used for parallel prediction of the P sequences to obtain the (Q+1)th image patch vector in each sequence. Optionally, different prediction tools are used for parallel prediction of the P sequences to obtain the (N+1)th image patch vector in each sequence.

[0117] In one embodiment, at least one autoregressive model is used to predict P sequences in the spatial dimension to obtain the Q+1th image patch vector in each sequence.

[0118] Optionally, P autoregressive models are used to predict the spatial dimension of each of the P sequences, resulting in the (Q+1)th image patch vector within each sequence. For example, for the first sequence, a first autoregressive model is used to predict the spatial dimension, yielding the (Q+1)th image patch vector. For the second sequence, a second autoregressive model is used to predict the spatial dimension, yielding the (Q+1)th image patch vector, and so on. Optionally, for a sequence, an autoregressive model is used to predict the (Q+1)th image patch vector based on the Q image patch vectors. Optionally, any two sequences among the P sequences may use the same or different autoregressive models. For example, the first and second autoregressive models mentioned above can be the same or different autoregressive models. Optionally, the autoregressive model includes any one of the following: a model based on a Transformer architecture, a model based on a causal convolutional architecture, and a model based on a recurrent neural network.

[0119] Optionally, using P image generation channels in an autoregressive model, the spatial dimensions of the P sequences are predicted to obtain the (Q+1)th image patch vector in each sequence. Optionally, for a sequence, the (Q+1)th image patch vector is predicted based on the Q image patch vectors using one image generation channel in the autoregressive model. Optionally, the P image generation channels are independent of each other; optionally, the P image generation channels are correlated with each other.

[0120] For example, when predicting the first sequence using the first image generation channel, information generated by other image generation channels during the prediction process can be fused to enhance the correlation between image patches in the final generated video frames. For instance, in a deep neural network autoregressive model, during parallel prediction, the P hidden features generated by the P image generation channels in the target hidden layer are fused into global features. These global features are then fused with the hidden features originally generated by the first image generation channel in the target hidden layer to obtain fused features. These fused features then replace the hidden features generated by the first image generation channel and are input into the next network layer.

[0121] Step 880: Generate a second video based on the image patch vectors within each sequence.

[0122] In each of the P sequences, prediction is performed in the spatial dimension to obtain the next image patch vector. In one embodiment, when the number of image patch vectors in each sequence reaches a video frame size threshold, the generation of the next image patch vector stops, and a second video is generated based on the existing image patch vectors in each of the P sequences. Optionally, the video frame size threshold is a pre-set threshold, such as M image patches. Optionally, based on the existing image patch vectors in each of the P sequences, the image patches corresponding to all image patch vectors are reconstructed, and all image patches are stitched together in temporal and spatial order to obtain the second video.

[0123] In summary, the above embodiments split P*Q image block vectors into P sequences (each of the P sequences being a dimension of a P-dimensional sequence), and then perform parallel prediction on the P sequences. Compared to related technologies where P*Q image block vectors are expanded into a one-dimensional sequence and prediction is strictly performed serially on the one-dimensional sequence, this application can perform parallel prediction on P video frames, which can fully utilize the parallel computing power of the graphics processor, thereby significantly improving computational efficiency and video generation speed.

[0124] Furthermore, compared to related techniques that require relying on all preceding image block vectors when predicting each image block vector, the method provided in this application only requires relying on the image block vectors of the same preceding video frame, reducing redundant calculations and lowering the consumption of computing resources. The method in this application is more suitable for high-resolution and long-time sequence video generation tasks.

[0125] Understandably, through Figure 8 The illustrated video generation method can predict the (Q+1)th image block based on the Q image blocks already present in each of P video frames. Then, based on the (Q+1)th image block, the (Q+2)th image block can be generated, and this process can be repeated to generate the entire video. Optionally, the video generation process ends when all image blocks (M image blocks) within each video frame have been recovered.

[0126] Figure 9 A schematic diagram of a video generation process provided by another exemplary embodiment of this application is shown.

[0127] To illustrate, during the video generation process, P video frames 901 are acquired, where each video frame already contains Q image blocks, such as... Figure 9 As shown, each of the three video frames already contains three image blocks. The method provided in this embodiment aims to complete the image blocks in each video frame to obtain the final video.

[0128] Each of the P video frames 901 is divided into image blocks, resulting in Q image blocks within each video frame. Then, each image block is vectorized to obtain P*Q image block vectors 902. (Illustrative example) Figure 9 In this process, image block vectors belonging to the same video frame are marked with the same number (such as "1", "2", "3"), and image block vectors belonging to the same spatial location are marked with the same letter (such as A, B, C). Therefore, each image block vector has a unique number, such as "A1", "B1", "C1", etc.

[0129] For the Q spatial locations indicated by P*Q image patch vectors 902, global time information is calculated based on the P image patch vectors corresponding to each spatial location, resulting in Q global time information 903. For example... Figure 9 As shown, for spatial location A, a global time information will be generated based on the three image patch vectors (image patch vectors A1, A2, A3) corresponding to spatial location A; for spatial location B, a global time information will be generated based on the three image patch vectors (image patch vectors B1, B2, B3) corresponding to spatial location B; and for spatial location C, a global time information will be generated based on the three image patch vectors (image patch vectors C1, C2, C3) corresponding to spatial location C. Figure 9 The CCP has three spatial locations. Global time information is calculated for each spatial location, resulting in a total of three global time information.

[0130] Global temporal information is used to characterize the temporal information of a spatial location across P video frames. Optionally, global temporal information is used to characterize the overall image information of a spatial location across P video frames.

[0131] A copy operation is performed on the Q global time information units 903 to obtain P*Q global time information units. According to spatial correspondence, the P*Q global time information units are combined one-to-one with the P*Q image patch vectors to obtain P*Q combined vectors 904. Figure 9 In this process, three global time information entries are copied to obtain nine global time information entries. Then, each global time information entry is concatenated and combined with its corresponding image patch vector to obtain nine combined vectors. (Illustrative example) Figure 9 The global time information of spatial location A is combined into image patch vectors A1, A2 and A3, the global time information of spatial location B is combined into image patch vectors B1, B2 and B3, and the global time information of spatial location C is combined into image patch vectors C1, C2 and C3.

[0132] The P*Q combination vectors 904 are sorted to obtain P sequences 905. Each of the P sequences 905 includes Q combination vectors with the same time point indicated by the P*Q combination vectors 904. Each combination vector includes a global time information and an image patch vector.

[0133] For each of the P sequences 905, predict the next image patch vector in the spatial dimension to obtain the (Q+1)th image patch vector in each sequence. For a sequence, there are Q combined vectors corresponding to Q spatial locations. Based on the Q combined vectors, the (Q+1)th image patch vector corresponding to the next spatial location can be predicted. Furthermore, the (Q+2)th image patch vector corresponding to the next spatial location can be predicted based on the (Q+1)th image patch vector, until all M spatial locations in each video frame have been predicted.

[0134] Understandably, in Figure 7 In the illustrated embodiment, P sequences corresponding to P video frames are obtained, each sequence including Q image block vectors. Parallel prediction is then performed on the P sequences. While this method effectively speeds up image generation, it loses the temporal information of the video. Based on this, Figure 9 The illustrated embodiment compensates for the lack of temporal information by generating global temporal information for each spatial location and combining it with the image patch vector corresponding to that spatial location. This not only effectively speeds up image generation but also further improves the quality of the generated video frames.

[0135] Figure 10 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, with the method being performed by a computer device. Figure 10 Steps 1020 to 1080 shown are located in Figure 8 Before step 840 shown.

[0136] Step 1020: Obtain P*Q global time information. The P*Q global time information includes Q types of global time information. Each type of global time information corresponds one-to-one with a spatial location. Each type of global time information is used to characterize the time information of a spatial location in P video frames. Global temporal information is used to characterize the temporal information of a spatial location across P video frames. It also characterizes the overall image information of a spatial location across the P video frames. Optionally, global temporal information can also characterize the relationships between the image content presented in the P video frames at the same spatial location. For example, if the image content presented in video frame t at the same spatial location changes significantly compared to previous video frames, then the image content presented in video frame t is considered key image content. Optionally, global temporal information can be represented in vector form; it can also be called a global temporal token.

[0137] In one embodiment, for Q spatial locations indicated by P*Q image block vectors, global time information is calculated based on the P image block vectors corresponding to each spatial location to obtain Q global time information. The global time information is used to characterize the time information of a spatial location in P video frames. The Q global time information is copied P times to obtain P*Q global time information.

[0138] For P*Q image block vectors, since each of the P*Q image block vectors corresponds one-to-one with a P*Q image block, and the P*Q image blocks occupy Q spatial locations in P video frames, the P*Q image block vectors can indicate Q spatial locations. Global time information is calculated for the P image block vectors corresponding to each spatial location, resulting in Q global time information entries, which correspond one-to-one with the Q spatial locations. It can be understood that in this embodiment, one global time information entry is calculated for each spatial location; therefore, a total of Q global time information entries are calculated for the Q spatial locations.

[0139] Optionally, for the Q spatial locations indicated by P*Q image patch vectors, a self-attention mechanism is used to calculate global temporal information based on the P image patch vectors corresponding to each spatial location, resulting in Q global temporal information. The self-attention mechanism calculates the correlation between each image patch vector and other image patch vectors in the P image patch vectors to obtain attention weights. These attention weights are then used to generate a weighted image patch vector for each image patch vector. In this application, the global temporal information includes all the weighted image patch vectors. It can be understood that the global temporal information obtained through the self-attention mechanism carries the correlation between any two image patch vectors in the P image patch vectors.

[0140] Understandably, by calculating global temporal information through a self-attention mechanism, we can not only obtain the image information of the same spatial location across P video frames, but also further obtain the distribution of image blocks at the same spatial location across P video frames. This allows us to identify which image blocks are key image blocks (such as image blocks with abrupt changes in content) and which image blocks are non-key image blocks (such as image blocks whose image content remains almost unchanged compared to previous and subsequent time steps), thereby improving the accuracy of the subsequently generated Q+1th image block.

[0141] Optionally, for the Q spatial locations indicated by P*Q image patch vectors, global temporal information is calculated using a weighted summation method based on the P image patch vectors corresponding to each spatial location, resulting in Q global temporal information pieces. Illustratively, the average of the P image patch vectors is taken to obtain a single global temporal information piece, the size of which is consistent with the size of a single image patch vector.

[0142] In another embodiment, for the Q spatial locations indicated by P*Q image block vectors, global time information is calculated based on at least two image block vectors corresponding to each spatial location to obtain Q global time information. The global time information is used to characterize the time information of a spatial location in P video frames. Each of the Q global time information is copied P times to obtain P*Q global time information.

[0143] Optionally, each spatial location corresponds to P image patch vectors. At least two image patch vectors that satisfy preset conditions are selected from the P image patch vectors, and global temporal information is calculated based on these at least two image patch vectors. For example, image patch vectors at key time points are selected from the P image patch vectors to obtain at least two image patch vectors. Alternatively, image patch vectors with significant differences in content between consecutive frames are selected from the P image patch vectors to obtain at least two image patch vectors; or, one image patch vector is selected from the P image patch vectors at preset time intervals to obtain at least two image patch vectors.

[0144] Optionally, for the Q spatial locations indicated by P*Q image patch vectors, a self-attention mechanism is used to calculate global temporal information based on at least two image patch vectors corresponding to each spatial location, resulting in Q global temporal information pieces. The self-attention mechanism calculates the correlation between each image patch vector and other image patch vectors within the at least two image patch vectors to obtain attention weights. These attention weights are then used to generate a weighted image patch vector for each image patch vector. In this application, the global temporal information includes all the weighted image patch vectors. It can be understood that the global temporal information obtained through the self-attention mechanism carries the correlation between any two image patch vectors within the at least two image patch vectors.

[0145] In one embodiment, each of the Q global spatial information pieces is copied P-1 times to obtain N*(P-1) global time information pieces; combined with the original sample, a total of P*Q global time information pieces are obtained. It can be understood that this application will copy the global time information corresponding to each spatial location to obtain P copies of global time information; therefore, for Q spatial locations, a total of P*Q global time information pieces will be generated.

[0146] Step 1040: According to the spatial location correspondence, combine the P*Q global time information with the P*Q image block vectors one by one to obtain P*Q combined vectors; Since P*Q global temporal information points correspond to Q spatial locations, and P*Q image patch vectors also correspond to Q spatial locations, the P*Q global temporal information points and P*Q image patch vectors can be combined one-to-one through the spatial location correspondence to obtain P*Q combined vectors. It can be understood that P image patch vectors at the same spatial location will be combined with the same global temporal information. Optionally, the combination method includes either concatenation or summation.

[0147] Step 1060: Sort the P*Q combination vectors to obtain P sequences. Each of the P sequences includes Q combination vectors that indicate the same time point as the P*Q combination vectors.

[0148] In this embodiment, since the P*Q combination vectors correspond one-to-one with the P*Q image block vectors, and each image block vector comes from a certain video frame, the P*Q image block vectors come from P video frames. Therefore, the P*Q combination vectors can indicate P time points. By placing the Q combination vectors that indicate the same time points in the P*Q combination vectors into a sequence, a total of P sequences are obtained.

[0149] In one embodiment, the Q associative vectors in a sequence are ordered according to their indicated spatial positions. For example, they are arranged in order from left to right and from top to bottom.

[0150] In conclusion, Figure 8 In the illustrated embodiment, P sequences corresponding to P video frames are obtained, each sequence including Q image block vectors. Parallel prediction is then performed on the P sequences. While this method effectively speeds up image generation, it loses the temporal information of the video frames. Based on this... Figure 10 The illustrated embodiment compensates for the lack of temporal information by generating global temporal information for each spatial location and combining it with the image patch vector corresponding to that spatial location. This not only effectively speeds up image generation but also further improves the quality of the generated video frames.

[0151] Application Scenario Examples In some embodiments, Figure 11 A flowchart of a video generation method provided in an exemplary embodiment of this application is shown, illustrated by way of example, the method being performed by a computer device, the method comprising: Step 1120: Obtain P video frames, where the P video frames are the video frames of the third video; A third video is obtained, which consists of P video frames. The resolution of the third video is the first resolution. This embodiment aims to improve the quality of the third video to obtain a fourth video. The resolution of the fourth video is the second resolution, which is greater than the first resolution.

[0152] Step 1140: Divide each of the P video frames into Q image blocks and quantize them to obtain P*Q image block vectors; For information on the splitting method, please refer to the relevant introduction in step 820 above.

[0153] Step 1160: Determine P sequences, where each of the P sequences includes Q image block vectors that indicate the same time point in P*Q image block vectors; For information on step 1160, please refer to the relevant description of step 840 above.

[0154] Step 1180: Predict the spatial dimensions of each of the P sequences to obtain the Q+1th image block vector in each sequence.

[0155] Optionally, for a sequence containing Q image patch vectors, the sequence is predicted spatially to obtain the (Q+1)th image patch vector within the sequence. Furthermore, based on the (Q+1)th image patch vector, the sequence can be further used to predict the (Q+2)th image patch vector, and so on, until a fourth video is generated. Optionally, the resolution of the fourth video is preset. For example, if the resolution of the third video is 16*9 and the resolution of the fourth video is 16*12, for P sequences, by continuously generating image patch vectors at the next spatial location until the preset resolution is reached, the P sequences will eventually yield all the image patch vectors of the fourth video, thus generating the fourth video.

[0156] The above embodiments provide a scheme for obtaining a fourth video by increasing the resolution of a given third video. By performing parallel prediction on P video frames, the video generation speed is improved.

[0157] Figure 12 This application shows a structural block diagram of a video generation apparatus provided in an exemplary embodiment, the apparatus comprising: The acquisition module 1201 is used to acquire N*M image block vectors. The N*M image block vectors are used to represent M image blocks in each of the N video frames that already exist. N and M are both positive integers and M is not less than one. The determination module 1202 is used to determine M sequences, each of the M sequences including N image block vectors that indicate the same spatial location in N*M image block vectors; The prediction module 1203 is used to predict M sequences in the time dimension to obtain the N+1th image patch vector in each sequence; The generation module 1204 is used to generate the first video based on the image block vectors within each sequence.

[0158] In an optional embodiment, the prediction module 1203 is further configured to predict the M sequences in the time dimension using at least one autoregressive model to obtain the N+1th image patch vector in each sequence.

[0159] In an optional embodiment, any two sequences among the M sequences may use the same or different autoregressive models; Autoregressive models include any one of the following: models based on the Transformer architecture, models based on the causal convolutional architecture, and models based on recurrent neural networks.

[0160] In an optional embodiment, the device further includes a calculation module 1205. The calculation module 1205 is configured to calculate global spatial information based on the M image block vectors corresponding to each of the N*M image block vectors indicating N time points, thereby obtaining N global spatial information pieces. These global spatial information pieces are used to characterize the spatial information of a video frame. Perform a copy operation on N global spatial information to obtain N*M global spatial information; According to the time correspondence, the N*M global spatial information and the N*M image block vectors are combined one by one to obtain N*M combined vectors; The N*M associative vectors are sorted to obtain M sequences. Each of the M sequences includes N associative vectors that have the same spatial position as the N*M associative vectors.

[0161] In an optional embodiment, the calculation module 1205 is further configured to calculate global spatial information based on the M image block vectors corresponding to each of the N*M image block vectors at N time points, using a self-attention mechanism, to obtain N global spatial information.

[0162] In summary, this application divides N*M image block vectors into M sequences and then performs predictions on the M sequences. Compared with related technologies that flatten N*M image block vectors into a one-dimensional sequence and perform predictions on the one-dimensional sequence in a strictly serialized manner, this application can perform predictions in parallel in spatial location, which can make full use of the parallel computing capabilities of the graphics processor, thereby significantly improving computational efficiency and video generation speed.

[0163] Furthermore, compared to related techniques that require relying on all preceding image block vectors when predicting each image block vector, the method provided in this application only requires relying on preceding image block vectors at the same spatial location, reducing redundant computation and lowering the consumption of computational resources. The method in this application is more suitable for high-resolution and long-time sequence video generation tasks.

[0164] Figure 13 This application shows a structural block diagram of a video generation apparatus provided in an exemplary embodiment, the apparatus comprising: The acquisition module 1301 is used to acquire P*Q image block vectors. The P*Q image block vectors are used to represent the Q image blocks that are present in each of the P video frames. Each video frame is used to accommodate M image blocks. P and Q are both positive integers and P is not less than one and Q is less than M. The determination module 1302 is used to determine P sequences, each of the P sequences including Q image block vectors that indicate the same time point in P*Q image block vectors; The prediction module 1303 is used to predict P sequences in the spatial dimension to obtain the Q+1th image patch vector in each sequence; The generation module 1304 is used to generate a second video based on the image patch vectors within each sequence.

[0165] In an optional embodiment, the prediction module 1303 is used to predict the P sequences in the spatial dimension using at least one autoregressive model to obtain the Q+1th image patch vector in each sequence.

[0166] In an optional embodiment, any two sequences among the P sequences may use the same or different autoregressive models; Autoregressive models include any one of the following: models based on the Transformer architecture, models based on the causal convolutional architecture, and models based on recurrent neural networks.

[0167] In an optional embodiment, the device further includes a calculation module 1305. The calculation module 1305 is configured to calculate global temporal information based on the P image block vectors corresponding to each spatial location, for the Q spatial locations indicated by P*Q image block vectors, to obtain Q global temporal information, which characterizes the temporal information of a spatial location within P video frames; Perform a copy operation on Q global time information to obtain P*Q global time information; According to the spatial correspondence, P*Q global time information is combined with P*Q image block vectors one by one to obtain P*Q combined vectors; The P*Q associative vectors are sorted to obtain P sequences. Each of the P sequences includes Q associative vectors that indicate the same time point as the P*Q associative vectors.

[0168] In an optional embodiment, the calculation module 1305 is further configured to calculate global time information based on the P image block vectors corresponding to each spatial location, using a self-attention mechanism, to obtain Q global time information for the Q spatial locations indicated by the P*Q image block vectors.

[0169] In summary, this application splits P*Q image block vectors (representing the Q image blocks present in each of the P video frames) into P sequences, and then performs parallel prediction on the P sequences. Compared to related technologies where the P*Q image block vectors are expanded into a one-dimensional sequence and prediction is strictly performed serially on the one-dimensional sequence, this application can perform parallel prediction on the P video frames, which can make full use of the parallel computing power of the graphics processor, thereby significantly improving computational efficiency and video generation speed.

[0170] Furthermore, compared to related techniques that require relying on all preceding image block vectors when predicting each image block vector, the method provided in this application only requires relying on the image block vectors of the same preceding video frame, reducing redundant calculations and lowering the consumption of computing resources. The method in this application is more suitable for high-resolution and long-time sequence video generation tasks.

[0171] Figure 14This is a schematic diagram of a computer device according to an exemplary embodiment. Optionally, the computer device is used to execute the video generation method provided in this application. The computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including random access memory (RAM) 1402 and read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the central processing unit 1401. The computer device 1400 also includes a basic input / output system (I / O system) 1406 to facilitate the transmission of information between various devices within the computer device, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0172] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0173] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer device readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer device readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0174] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0175] According to various embodiments of this disclosure, the computer device 1400 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1411 via a network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer device systems (not shown).

[0176] The memory also includes one or more programs stored in the memory, and the central processing unit 1401 executes the one or more programs to implement all or part of the steps applied to the video generation method.

[0177] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the video generation method provided in the above method embodiments.

[0178] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video generation method provided in the above-described method embodiments.

Claims

1. A video generation method, characterized in that, The method includes: Obtain N*M image block vectors, which are used to represent M image blocks in each of the N video frames, where N and M are both positive integers and M is not less than one; M sequences are determined, each of the M sequences comprising N image block vectors that indicate the same spatial location in the N*M image block vectors; Predict the M sequences in the time dimension to obtain the (N+1)th image patch vector in each sequence; A first video is generated based on the image block vectors within each sequence.

2. The method according to claim 1, characterized in that, The step of predicting the M sequences in the time dimension to obtain the (N+1)th image patch vector in each sequence includes: By using at least one autoregressive model, the M sequences are predicted in the time dimension to obtain the N+1th image patch vector in each sequence.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain N*M global spatial information, which includes N types of global spatial information. The N types of global spatial information correspond one-to-one with N video frames. Each type of global spatial information is used to characterize the spatial information of a video frame. According to the time correspondence, the N*M global spatial information and the N*M image block vectors are combined one by one to obtain N*M combined vectors; The N*M combined vectors are sorted to obtain the M sequences, and each of the M sequences includes N combined vectors that have the same spatial position indicated by the N*M combined vectors.

4. The method according to claim 3, characterized in that, The acquisition of N*M global space information includes: For the N time points indicated by the N*M image block vectors, global spatial information is calculated based on the M image block vectors corresponding to each time point to obtain N global spatial information. Each of the N global spatial information is copied M times to obtain the N*M global spatial information.

5. The method according to claim 4, characterized in that, For the N*M image patch vectors indicating N time points, global spatial information is calculated based on the M image patch vectors corresponding to each time point to obtain N global spatial information, including: For the N time points indicated by the N*M image patch vectors, the global spatial information is calculated using a self-attention mechanism based on the M image patch vectors corresponding to each time point, thus obtaining the N global spatial information.

6. A video generation method, characterized in that, The method includes: Obtain P*Q image block vectors, which are used to represent the Q image blocks that are present in each of the P video frames. Each video frame has M spatial locations, where P and Q are both positive integers and P is not less than one and Q is less than M. Determine P sequences, each of the P sequences comprising Q image block vectors that indicate the same time point in the P*Q image block vectors; Predict the P sequences in the spatial dimension to obtain the Q+1th image patch vector in each sequence; A second video is generated based on the image block vectors within each sequence.

7. The method according to claim 6, characterized in that, The method further includes: P*Q global time information are obtained. The P*Q global time information includes Q types of global time information. The Q types of global time information correspond one-to-one with Q spatial locations. Each type of global time information is used to characterize the time information of a spatial location in the P video frames. According to the spatial correspondence, the P*Q global time information is combined with the P*Q image block vectors one by one to obtain P*Q combined vectors; The P*Q binding vectors are sorted to obtain the P sequences, and each of the P sequences includes Q binding vectors that indicate the same time point as the P*Q binding vectors.

8. The method according to claim 7, characterized in that, The acquisition of P*Q global time information includes: For the Q spatial locations indicated by the P*Q image block vectors, global time information is calculated based on the P image block vectors corresponding to each spatial location to obtain Q global time information. Each of the Q global time information is copied P times to obtain the P*Q global time information.

9. The method according to claim 8, characterized in that, For the Q spatial locations indicated by the P*Q image block vectors, global time information is calculated based on the P image block vectors corresponding to each spatial location to obtain Q global time information, including: For the Q spatial locations indicated by the P*Q image block vectors, the global time information is calculated using a self-attention mechanism based on the P image block vectors corresponding to each spatial location, thus obtaining the Q global time information.

10. A video generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire N*M image block vectors, wherein the N*M image block vectors are used to represent M image blocks in each of the N video frames that already exist, where N and M are both positive integers and M is not less than one; A determination module is used to determine M sequences, wherein each of the M sequences includes N image block vectors that indicate the same spatial location in the N*M image block vectors; The prediction module is used to predict the M sequences in the time dimension to obtain the N+1th image patch vector in each sequence; A generation module is used to generate a first video based on the image block vectors within each sequence.

11. A video generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire P*Q image block vectors, wherein the P*Q image block vectors are used to represent the Q image blocks that are present in each of the P video frames, and each video frame is used to accommodate M image blocks, where P and Q are both positive integers and P is not less than one and Q is less than M; A determination module is used to determine P sequences, each of the P sequences including Q image block vectors that indicate the same time point in the P*Q image block vectors; The prediction module is used to predict the P sequences in the spatial dimension to obtain the Q+1th image block vector in each sequence; A generation module is used to generate a second video based on the image block vectors within each sequence.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the video generation method as described in any one of claims 1 to 5 or 6 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the video generation method as described in any one of claims 1 to 5 or 6 to 9.

14. A computer program product, characterized in that, The computer program product stores a computer program that is loaded and executed by a processor to implement the video generation method as described in any one of claims 1 to 5 or 6 to 9.

Images