Video generation method based on visual identifier

Abstract

The invention discloses a video generation method based on a visual identifier, and belongs to the technical field of video generation. The method comprises the following steps: acquiring input video data, and extracting initial spatio-temporal characteristics; initializing a learnable base vector as a visual identifier, and modulating the learnable base vector and the initial spatial-temporal characteristics respectively; and splicing the modulated visual identifier and the spatio-temporal feature to form an extended feature sequence, optimizing the extended feature sequence, generating a video frame or a video sequence, splicing the video frame or the video sequence with video data to form new video data, continuously iterating and repeating, and finally outputting a high-quality video sequence. According to the method, the visual identifier is introduced, so that the global information of the video can be effectively captured and utilized in the video generation process, and the spatial-temporal characteristics are accurately adjusted, so that the generated video has better coherence and consistency in space and time dimensions. According to the method, the video generation quality can be improved, modeling of spatial-temporal characteristics is optimized, and the development cost of a video generation model is reduced.

Description

Technical Field

[0001] This invention belongs to the field of video generation, and in particular relates to a video generation method based on visual identifiers. Background Technology

[0002] Video generation technology has experienced rapid development in recent years, especially driven by Artificial Intelligence Generated Content (AIGC). Video, as an important form of visual content, creates a sense of motion by continuously presenting a series of images or frames, providing rich dynamic expression and information delivery methods. Compared to still image generation, video generation faces more challenging problems, requiring not only high quality for each generated frame but also ensuring spatiotemporal consistency to guarantee the continuity and natural transitions of the video.

[0003] In video generation tasks, spatiotemporal consistency is a critical challenge. Spatiotemporal consistency comprises two main aspects: spatial consistency and temporal consistency. Spatial consistency focuses on the details and elements within each frame (such as color, shape, and object position), ensuring these features remain consistent across frames to avoid spatial distortion and video flickering. Temporal consistency, on the other hand, focuses on changes between adjacent frames; changes in elements such as objects, scenes, lighting, and motion should be smooth and coherent, avoiding abrupt jumps or unnatural changes.

[0004] To maintain spatiotemporal consistency, recent research has proposed various methods, including improved generative adversarial networks (GANs), recursive models, autoregressive models, and diffusion models, each with its own advantages and applicable scenarios. However, current research on spatiotemporal consistency still has some shortcomings, especially in how to simultaneously ensure spatial and temporal consistency during the generation process. Although some methods have achieved good results, many challenges remain in maintaining the continuity, detail, and quality of long videos. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and to provide a video generation method based on visual identifiers.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a video generation method based on visual identifiers, which includes the following steps:

[0008] S1: Obtain the input video data, use block embedding to transform the video data into initial spatiotemporal features, obtain global video information from the video data, obtain time steps and generate their corresponding time step embeddings;

[0009] S2: Initialize learnable basis vectors as visual identifiers, and modulate the visual identifiers and initial spatiotemporal features respectively;

[0010] S3: The modulated visual identifiers and spatiotemporal features are concatenated to form an extended feature sequence. A multi-head self-attention mechanism is used to extract the spatiotemporal dependencies between frames of video data, thereby optimizing the extended feature sequence. Video frames or video sequences are generated from the optimized extended feature sequence.

[0011] S4: Concatenate the video data in S1 with the video frames or video sequences output by S3 to form new video data. Replace the video data in S1 with the new video data. Repeat S1 to S3 until the generated video data reaches the specified length. Output the final generated video data to complete the video generation based on visual identifiers.

[0012] Based on the above scheme, each step can be implemented in the following preferred manner.

[0013] As a preferred embodiment of the first aspect above, the specific sub-steps of step S1 are as follows:

[0014] S11: Acquire video data, decompose the input video data into multiple small blocks, and convert each decomposed small block into high-dimensional features through three-dimensional convolution;

[0015] S12: Apply position embedding to each small block of video data to obtain positional encoding. Add the positional encoding to the high-dimensional features to obtain the initial spatiotemporal features x. ori ;

[0016] S13: Extract video features from video data through 3D convolution, thereby transforming the video data from pixel dimension to feature vector dimension, and embedding the obtained features as global video information;

[0017] S14: Obtain the time step, concatenate the sine frequency embedding and cosine frequency embedding of the time step and input them into the multilayer perceptron (MLP) to transform the time step into a high-dimensional feature vector and obtain the time step embedding.

[0018] As a preferred embodiment of the first aspect above, the specific sub-steps of step S2 are as follows:

[0019] S21: Create a learnable basis vector as a visual identifier VI token And initialize it randomly;

[0020] S22: The time step is embedded to generate modulation parameters of sub-features through a multilayer perceptron, and the modulation parameters of sub-features are split into a first scaling modulation parameter and a first offset modulation parameter. The first scaling modulation parameter is used to control the scaling of high-dimensional features in the attention mechanism and MLP layer, and the first offset modulation parameter is used to control the offset of high-dimensional features in the attention mechanism and MLP layer.

[0021] S23: Use the first scaling modulation parameter X scale and the first offset modulation parameter X shift The initial spatiotemporal features are modulated using the following formula to obtain the final spatiotemporal features;

[0022] S24: The global video information is used to generate modulation parameters for visual identifiers through a multilayer perceptron, and the modulation parameters of the visual identifiers are split into second scaling modulation parameters and second offset modulation parameters, so as to better aggregate global information;

[0023] S25: According to the second scaling modulation parameter VI scale Second offset modulation parameter VI shift Modulate the visual identifiers.

[0024] Furthermore, in step S21, the random initialization method of the visual identifier is as follows:

[0025]

[0026] Where resolution represents the resolution of the video data; hiddensize represents the feature dimension of the input; and randn represents the function that generates random numbers or matrices that follow a standard normal distribution.

[0027] Furthermore, the final spatiotemporal feature x final The modulation method with visual identifiers is as follows:

[0028] x final =x ori *(1+X scale )+X shift

[0029] VI token ←VI token *(1+VI scale )+VI shift .

[0030] As a preferred embodiment of the first aspect above, the specific sub-steps of step S3 are as follows:

[0031] S31: The modulated visual identifier and the final spatiotemporal features are concatenated in the time dimension to form an extended feature sequence;

[0032] S32: In each Transformer block, the extended feature sequence is optimized through multi-head self-attention computation to extract the spatiotemporal dependencies between frames of video data; wherein, the query, key, and value in multi-head self-attention are all generated by linear projection of the extended feature sequence;

[0033] S33: The optimized extended feature sequence is processed iteratively through multiple Transformer blocks, and then an optimized high-dimensional feature sequence is output. The pre-trained VAE decoder decodes the optimized high-dimensional feature sequence and outputs video frames or video sequences.

[0034] Furthermore, during training, the multi-layer Transformer blocks incorporate the spatially aware loss function L... S Time consistency loss function L T and mean squared error loss function L MSE The weighted summation is used as the total loss function, and the parameters of the multi-layer Transformer block are updated based on minimizing the total loss function.

[0035] The mean square error loss function L MSE The spatial perception loss function L S and the time consistency loss function L T The function forms are as follows:

[0036]

[0037]

[0038] in, w is the feature extraction function of the l-th layer of the pre-trained network VGG; l Let be the weights of the l-th layer of the pre-trained VGG network; l is the number of layers in the pre-trained VGG network. This is the (i+1)th frame for generating video data; This represents the i-th frame of the generated video data; Represents the i-th frame of the actual video data; This represents the square of the L2 norm.

[0039] In a second aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the video generation method based on visual identifiers as described in any of the solutions of the first aspect above.

[0040] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video generation method based on visual identifiers as described in any of the solutions of the first aspect above.

[0041] Fourthly, the present invention provides a computer electronic device, including a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is configured to, when executing the computer program, implement the video generation method based on visual identifiers as described in any of the first aspects above.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1) This invention proposes a video generation method based on visual identifiers. This method can precisely adjust spatiotemporal features according to the spatiotemporal global information of the video sequence, resulting in generated videos with better coherence and consistency in both spatial and temporal dimensions. The entire process extracts information from the generated video frames and integrates it into the global information, enabling the method to generate long videos or even infinitely.

[0046] 2) This invention is based on the fact that visual identifiers are learnable basis vectors, so it has scalability. In the future, various different control conditions (such as text, audio or other external features) can be flexibly added to the generation process through modulation.

[0047] 3) The backbone of this invention is a Transformer, which ensures high quality of the generated video and provides a solid foundation for achieving complex video generation tasks. Attached Figure Description

[0048] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating the video generation results on the Minecraft dataset using the method of this invention;

[0050] Figure 3 This is a schematic diagram of the video generation results on the DMLab dataset using the method of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0052] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0053] This invention provides a video generation method based on visual identifiers for a fundamental task in machine vision: generating video sequences. In this task, given a known video sequence and the required video frame length, the method learns and extracts global information from the known video sequence using visual identifiers, continuously optimizing the process to generate the video. The specific implementation process of this invention is described in detail below.

[0054] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned video generation method based on visual identifiers includes the following steps S1 to S4:

[0055] S1: Obtain the input video data, use block embedding to transform the video data into initial spatiotemporal features, obtain global video information from the video data, obtain time steps and generate their corresponding time step embeddings.

[0056] S2: Initialize learnable basis vectors as visual identifiers, and modulate the visual identifiers and initial spatiotemporal features respectively.

[0057] S3: The modulated visual identifiers and spatiotemporal features are concatenated to form an extended feature sequence. A multi-head self-attention mechanism is used to extract the spatiotemporal dependencies between frames of video data, thereby optimizing the extended feature sequence. Video frames or video sequences are generated from the optimized extended feature sequence.

[0058] S4: Concatenate the video data in S1 with the video frames or video sequences output by S3 to form new video data. Replace the video data in S1 with the new video data. Repeat S1 to S3 until the generated video data reaches the specified length. Output the final generated video data to complete the video generation based on visual identifiers.

[0059] The core of the above method is to enable visual identifiers to effectively capture and utilize global information of the video during video generation, thereby improving the precise adjustment of spatiotemporal features. S1 to S3 can be implemented in various ways; their specific implementation process in this embodiment is described below.

[0060] In this embodiment, the specific sub-steps for implementing S1 are as follows:

[0061] S11: Obtain video data x∈R B×C×T×H×WThe input video data is decomposed into S patches, and each patch is converted into a high-dimensional feature x′=Conv3d(x)∈R through 3D convolution. B×T×S×D .in:

[0062]

[0063] T = P T *F

[0064] In the formula: H and W represent the length and width of the video data, respectively; D represents the feature dimension of each block; B represents the batch number; C represents the number of channels in the video data; T represents the number of frames in the video data; S represents the number of blocks obtained from the decomposition; P H P W and P T These represent the size of the chunk in the length, width, and time dimensions, respectively; F represents the number of frames for each chunk.

[0065] S12: Apply position embedding to each small block of video data to obtain the positional encoding emb. post The initial spatiotemporal feature x is obtained by adding the positional encoding to the high-dimensional feature. ori =x′+emb post .

[0066] S13: Extract video features from video data through 3D convolution, thereby transforming the video data from pixel dimension to feature vector dimension, and embedding the obtained features into the emb. fea As global_feature, this global information is continuously optimized as video frames are generated.

[0067] S14: Obtain the time step t, concatenate the sinusoidal frequency embedding sin(t) and the cosine frequency embedding cos(t) of the time step, and input them into a multilayer perceptron (MLP) to transform the time step into a high-dimensional feature vector, thus obtaining the time step embedding emb. t =MLP(sin(t)||cos(t)), where || represents the concatenation operation of sine frequency embedding and cosine frequency embedding.

[0068] Thus, through the steps S11 to S14 above, global information of the input video data is obtained, ensuring that the model can effectively model the spatiotemporal relationship of the video.

[0069] In this embodiment, the specific sub-steps for implementing S2 are as follows:

[0070] S21: Create a learnable basis vector as a visual identifier VI token Initialize the visual identifiers randomly as follows:

[0071]

[0072] Where resolution represents the resolution of the video data; hiddensize represents the feature dimension of the input; and randn represents the function that generates random numbers or matrices that follow a standard normal distribution.

[0073] It should be noted that in this invention, the aforementioned feature dimensions are numerically consistent with the feature dimensions of each small block; while the resolution of the video data varies according to the resolution of the videos in the dataset used during training.

[0074] S22: Embed time steps into emb t Modulation parameters mpl are generated for sub-features using a multilayer perceptron. t The modulation parameters of the sub-features are then split into the first scaling modulation parameter X. scale and the first offset modulation parameter X shift The first scaling modulation parameter is used to control the scaling of high-dimensional features in the attention mechanism and MLP layer, and the first offset modulation parameter is used to control the offset of high-dimensional features in the attention mechanism and MLP layer.

[0075] S23: Use the first scaling modulation parameter X scale and the first offset modulation parameter X shift The initial spatiotemporal features are modulated according to the following formula to obtain the final spatiotemporal features x. final :

[0076] x final =x ori *(1+X scale )+X shift

[0077] S24: Transfer video global information (emb) fea Generate visual identifier VI using a multilayer perceptron. token modulation parameters mpl fea and the visual identifier VI token The modulation parameters are split into the second scaling modulation parameters VI. scale Second offset modulation parameter VI shift This allows for better aggregation of global information.

[0078] In this embodiment, the second scaling modulation parameter and the second offset modulation parameter are dynamically adjusted according to the global video information.

[0079] S25: According to the second scaling modulation parameter VI scale Second offset modulation parameter VI shift Visual identifier VI is defined by the following formula. token Modulation:

[0080] VI token ←VI token *(1+VI scale )+VI shift

[0081] Therefore, the steps S21 to S25 described above are for the initialization and modulation of the visual identifier. The modulation is performed in the Transformer module. After the visual identifier is dynamically adjusted, the modulated visual identifier can serve as a global information aggregator for video data, providing support for the subsequent attention mechanism.

[0082] In this embodiment, the specific sub-steps for implementing S3 are as follows:

[0083] S31: Modulate the visual identifier VI token and the final spatiotemporal feature x final Concatenate the sequences along the time dimension to form an extended feature sequence x. extended =concat(VI token ,x final ); where concat(·) represents the concatenation operation;

[0084] S32: In each Transformer block, the extended feature sequence is optimized through multi-head self-attention calculation to extract the spatiotemporal dependencies between frames of video data; wherein, the query Q, key K and value V in multi-head self-attention are all generated by linear projection of the extended feature sequence;

[0085]

[0086] Where, x attn The expression represents the result of multi-head self-attention calculation; Attention(·) represents multi-head self-attention; softmax represents the softmax function; T represents transpose; d k For each attention head dimension.

[0087] S33: The optimized extended feature sequence is processed iteratively through multiple Transformer blocks, and then an optimized high-dimensional feature sequence is output. The pre-trained VAE decoder decodes the optimized high-dimensional feature sequence and outputs video frames or video sequences.

[0088] It should be noted that during training, the multi-layer Transformer block uses a weighted sum of the spatial awareness loss function, the temporal consistency loss function, and the mean squared error loss function as the total loss function, and updates the parameters of the multi-layer Transformer block based on minimizing the total loss function.

[0089] In this embodiment, S1 to S4 constitute the framework of a video generation method based on visual identifiers. When this framework is actually used for video generation, the visual identifiers need to be optimized accordingly. The optimization objectives include two categories: the first is to make the generated video data as realistic as possible and consistent with the distribution of real video data. The optimization of visual identifiers is achieved by minimizing the difference between the generated video data and the real video data. Therefore, the mean squared error loss function L between the video data and the real video data... MSE for:

[0090]

[0091] in, This represents the i-th frame of the generated video data; Represents the i-th frame of the actual video data; This represents the square of the L2 norm.

[0092] The second type aims to maintain temporal and spatial coherence in the generated video data, as well as consistency between adjacent frames; that is, the optimization of visual identifiers needs to consider inter-frame consistency. Therefore, the spatial perception loss function L... S and time consistency loss function L T They are respectively:

[0093]

[0094] in, w is the feature extraction function of the l-th layer of the pre-trained network VGG; l Let be the weights of the l-th layer of the pre-trained VGG network; l is the number of layers in the pre-trained VGG network. This is for generating the (i+1)th frame of the video data. Therefore, the overall optimization objective set during the training process in this embodiment is as follows:

[0095] L=αL MSE +βL S +γL T

[0096] Where α, β, and γ all represent hyperparameters, and in this embodiment, α, β, and γ are all 1.

[0097] In this embodiment, by combining the spatiotemporal video generation method based on visual identifiers given in S1 to S4, videos with better coherence and consistency can be generated.

[0098] The present invention will now demonstrate the application effect of the video generation method based on visual identifiers described in S1 to S4 of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0099] Example

[0100] The specific implementation process of the video generation method based on visual identifiers used in this embodiment is as described above and will not be repeated here.

[0101] To demonstrate the technical effectiveness of this embodiment, the invention was validated on the Minecraft and DMLab datasets. The video evaluation metric FVD was used, which is used to evaluate the quality of generated videos. Quality is quantified by calculating the difference in depth feature distribution between the generated video and the real video; a lower value indicates better generation quality. Figure 2 The video generation results using the method of this invention on the Minecraft dataset are shown. Figure 3 The video generation results using the method of this invention on the DMLab dataset are shown in Table 1. The video generation quality on both datasets is shown in Table 1.

[0102] Table 1. FVD scores of the present invention on two video generation datasets.

[0103] Dataset This embodiment VDT Minecraft 385.19 510.76 DMLab 682.69 983.72

[0104] The results above demonstrate that the video quality generated by this invention is significantly superior to the comparative model VDT in Minecraft and DMLab video generation tasks. Particularly on the DMLab dataset, the videos generated by this invention more closely resemble the real-world distribution, showcasing its advantages in spatiotemporal modeling.

[0105] It is understood that the video generation method based on visual identifiers described in S1 to S4 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the video generation method based on visual identifiers provided in the above embodiments, which includes a computer program / instructions that, when executed by a processor, can implement the video generation method based on visual identifiers as described in the above embodiments.

[0106] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the video generation method based on visual identifiers provided in the above embodiments, which includes a memory and a processor;

[0107] The memory is used to store computer programs;

[0108] The processor is configured to implement the video generation method based on visual identifiers in the above embodiments when executing the computer program.

[0109] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0110] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the video generation method based on visual identifiers provided in the above embodiments. The storage medium stores a computer program that, when executed by a processor, can implement the video generation method based on visual identifiers in the above embodiments.

[0111] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0112] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0113] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A video generation method based on visual identifiers, characterized in that, Includes the following steps: S1: Obtain the input video data, use block embedding to transform the video data into initial spatiotemporal features, obtain global video information from the video data, obtain time steps and generate their corresponding time step embeddings; S2: Initialize learnable basis vectors as visual identifiers, and modulate the visual identifiers and initial spatiotemporal features respectively; S3: The modulated visual identifiers and spatiotemporal features are concatenated to form an extended feature sequence. A multi-head self-attention mechanism is used to extract the spatiotemporal dependencies between frames of video data, thereby optimizing the extended feature sequence. Video frames or video sequences are generated from the optimized extended feature sequence. S4: Concatenate the video data in S1 with the video frames or video sequences output by S3 to form new video data. Replace the video data in S1 with the new video data. Repeat S1 to S3 until the generated video data reaches the specified length. Output the final generated video data to complete the video generation based on visual identifiers.

2. The video generation method based on visual identifiers as described in claim 1, characterized in that, The specific sub-steps of step S1 are as follows: S11: Acquire video data, decompose the input video data into multiple small blocks, and convert each decomposed small block into high-dimensional features through three-dimensional convolution; S12: Apply position embedding to each small block of video data to obtain positional encoding. Add the positional encoding to the high-dimensional features to obtain the initial spatiotemporal features x. ori ; S13: Extract video features from video data through 3D convolution, thereby transforming the video data from pixel dimension to feature vector dimension, and embedding the obtained features as global video information; S14: Obtain the time step, concatenate the sine frequency embedding and cosine frequency embedding of the time step and input them into the multilayer perceptron (MLP) to transform the time step into a high-dimensional feature vector and obtain the time step embedding.

3. The video generation method based on visual identifiers as described in claim 2, characterized in that, The specific sub-steps of step S2 are as follows: S21: Create a learnable basis vector as a visual identifier VI token And initialize it randomly; S22: The time step is embedded to generate modulation parameters of sub-features through a multilayer perceptron, and the modulation parameters of sub-features are split into a first scaling modulation parameter and a first offset modulation parameter. The first scaling modulation parameter is used to control the scaling of high-dimensional features in the attention mechanism and MLP layer, and the first offset modulation parameter is used to control the offset of high-dimensional features in the attention mechanism and MLP layer. S23: Use the first scaling modulation parameter X scale and the first offset modulation parameter X shift The initial spatiotemporal features are modulated using the following formula to obtain the final spatiotemporal features; S24: The global video information is used to generate modulation parameters for visual identifiers through a multilayer perceptron, and the modulation parameters of the visual identifiers are split into second scaling modulation parameters and second offset modulation parameters, so as to better aggregate global information; S25: According to the second scaling modulation parameter VI scale Second offset modulation parameter VI shift Modulate the visual identifiers.

4. The video generation method based on visual identifiers as described in claim 3, characterized in that, In step S21, the visual identifier is randomly initialized in the following way: Where resolution represents the resolution of the video data; hiddensize represents the feature dimension of the input; and randn represents the function that generates random numbers or matrices that follow a standard normal distribution.

5. The video generation method based on visual identifiers as described in claim 3, characterized in that, Final spatiotemporal features x final The modulation method with visual identifiers is as follows: x final =x ori *(1+X scale )+X shift WE token ←WE token *(1+VI scale )+VI shift 。 6. The video generation method based on visual identifiers as described in claim 3, characterized in that, The specific sub-steps of step S3 are as follows: S31: The modulated visual identifier and the final spatiotemporal features are concatenated in the time dimension to form an extended feature sequence; S32: In each Transformer block, the extended feature sequence is optimized through multi-head self-attention computation to extract the spatiotemporal dependencies between frames of video data; wherein, the query, key, and value in multi-head self-attention are all generated by linear projection of the extended feature sequence; S33: The optimized extended feature sequence is processed iteratively through multiple Transformer blocks, and then an optimized high-dimensional feature sequence is output. The pre-trained VAE decoder decodes the optimized high-dimensional feature sequence and outputs video frames or video sequences.

7. The video generation method based on visual identifiers as described in claim 6, characterized in that, During training, the multi-layer Transformer block uses a weighted sum of the spatial awareness loss function, the temporal consistency loss function, and the mean squared error loss function as the total loss function, and updates the parameters of the multi-layer Transformer block based on minimizing the total loss function.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the video generation method based on visual identifiers as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the video generation method based on visual identifiers as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the video generation method based on visual identifiers as described in any one of claims 1 to 7 when executing the computer program.

Images