Video generation method and system

Abstract

The application provides a video generation method and system. The method comprises the following steps: constructing an initial picture-to-video model; performing augmentation processing on each preset training picture to obtain a training video frame sequence corresponding to the training picture; performing model training on the initial picture-to-video model through the training picture and the training video frame sequence corresponding to the training picture to obtain a target picture-to-video model after training; obtaining text description information; inputting the text description information into a preset target text-to-picture model to obtain a first picture; inputting the first picture into the target picture-to-video model to obtain a video corresponding to the text description information. The application guarantees the quality of the generated first picture by using the target text-to-picture model, and uses the inter-frame transformation logic learned by the target picture-to-video model to make the generated video picture natural and coherent, thereby improving the video generation quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of video technology, and in particular to video generation methods and systems. Background Technology

[0002] In video generation technology, large-scale video generation models can automatically generate entire videos based on user-provided textual descriptions. The generation process typically involves concurrently generating a large number of video frames, each generated independently, modifying only the original textual descriptions, and then simply combining the frame sequences into a video. Because of the lack of temporal dependencies and collaborative constraints between frames, this independent parallel generation method easily leads to problems such as jumpy frames, disjointed motion, and inconsistent logic in the final video, affecting viewing smoothness. Furthermore, the concurrent generation of numerous frames consumes significant GPU and other computing resources, resulting in low training and inference efficiency. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a video generation method and system that can improve the quality and efficiency of video generation.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A video generation method, characterized by comprising: Build an initial image-to-video model; Each preset training image is augmented to obtain a training video frame sequence corresponding to the training image. The initial image-to-video model is trained using the training images and the corresponding training video frame sequences to obtain the trained target image-to-video model. Obtain text description information; The text description information is input into a preset target text-to-image model to obtain the first frame image; The first frame image is input into the target image-to-video model to obtain a video corresponding to the text description information.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A video generation system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the various steps of any of the video generation methods described above.

[0006] The beneficial effects of this invention are as follows: In the training phase of the image-to-video model, a training video frame sequence with strict temporal correlation is generated by amplifying a single training image. The image-to-video model is then trained using the training image and its corresponding training video frame sequence to obtain the trained target image-to-video model. This dataset construction method enables the image-to-video model to learn basic motion patterns from static images, improving the model's prediction accuracy. A high-quality first frame image is generated based on user-inputted text description information using a text-to-image model. This first frame image is then input into the target image-to-video model, which autoregressively generates all subsequent video frames, resulting in a video corresponding to the text description information. This ensures a causal dependency between the generated video frames, making the motion logic more stable and the transitions smoother. This solves the problem of discontinuous and jumpy content in videos caused by each frame being generated independently, only gradually changing based on the original text description information, and then simply combining the frame sequence into a video. Furthermore, since it is not necessary to repeatedly encode text semantics during each frame generation, nor to generate a large number of redundant frames, the consumption of GPU and other computing resources is reduced, improving video generation efficiency and quality. Attached Figure Description

[0007] Figure 1 This is a flowchart of a video generation method according to an embodiment of the present invention; Figure 2 This is another flowchart of a video generation method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a video generation system according to an embodiment of the present invention. Detailed Implementation

[0008] Definitions:

[0009] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0010] In existing technologies, many large-scale video generation models can automatically generate the entire video based on user-provided text and other descriptive information. However, the process involves generating numerous video frames, typically concurrently generating multiple closely related frames and then combining them to form the final video data. Because each frame is generated independently and concurrently, with frame data progressively modified based on the original text description, the final integrated video data may result in a less smooth or disjointed video. Furthermore, this automatic video generation consumes significant GPU and other computing resources.

[0011] To address at least some of the aforementioned issues, an initial image-to-video model is first built. Then, multi-dimensional augmentation processing is performed on pre-set training images to generate a sequence of training video frames with temporal correlation. Next, the model is iteratively trained using the training images and their corresponding training video frame sequences to obtain a target image-to-video model capable of stably generating dynamic images. In the application phase, the user-input text description is first obtained. The pre-trained target text-to-image model generates the first frame image, which is then input into the target image-to-video model. Finally, a complete video matching the text description is output. This approach improves both the efficiency and quality of video generation.

[0012] The following describes a video generation method of the present invention in detail. Please refer to [link / reference]. Figure 1 The method 100 includes steps 110 to 160.

[0013] Step 110: Construct the initial image-to-video model.

[0014] In one alternative implementation, an initial image-to-video model is constructed using a Transformer model or a diffusion model based on a spatiotemporal attention mechanism. This model possesses the ability to learn temporal features and render spatial images. It can receive a single image as input, autonomously learn the spatiotemporal transformation rules between image frames, and output a coherent sequence of video frames.

[0015] Step 120: Amplify each preset training image to obtain a training video frame sequence corresponding to the training images.

[0016] In one optional implementation, a batch of pre-selected large-scale, high-quality, multi-scene training images are acquired. Based on a single training image, multi-dimensional augmentation and parameter expansion operations are performed sequentially, such as scaling, displacement, and parameter adjustment, to generate a frame sequence with continuous motion correlation and rich parameters. The frames are then organized and stored in strict temporal order to finally form a training video frame sequence that corresponds one-to-one with each training image.

[0017] Step 130: Train the initial image-to-video model using the training images and the training video frame sequence corresponding to the training images to obtain the trained target image-to-video model.

[0018] In one optional implementation, the prepared training images and the corresponding temporally sequenced training video frame sequences are combined to form training sample pairs. The initial image-to-video model is iteratively trained. The difference between video frames is calculated through a loss function, the optimizer updates the model parameters, and the model convergence training is completed by combining a preset loss threshold or number of iterations. Finally, a target image-to-video model with stable inter-frame motion video generation capability is obtained.

[0019] Step 140: Obtain text description information.

[0020] In one optional implementation, the user's input natural language text description information is collected in real time, and the natural language text description information is preprocessed, such as invalid character filtering and semantic error correction, to form standardized model input text description information.

[0021] Step 150: Input the text description information into the preset target text-to-image model to obtain the first frame image.

[0022] In one optional implementation, the preprocessed standardized text description information is input into a pre-trained target text-to-image model. The model parses the text semantics and renders a single high-definition first frame image by using the correspondence between text description information and real images in the training dataset, which serves as the reference image for the generation of subsequent video frames.

[0023] Step 160: Input the first frame image into the target image-to-video model to obtain the video corresponding to the text description information.

[0024] In one optional implementation, the generated first frame image is input into the trained target image to video model. The model learns the spatiotemporal motion rules and inter-frame transformation logic through the correspondence between text description information and real images in the training dataset. Taking the first frame image as the initial reference, it generates subsequent video frames one by one with coherent timing, reasonable motion, and stable image structure. The first frame image and all subsequent video frames are spliced ​​and integrated in time sequence to finally output a complete video that highly matches the user's text description information and has smooth and coherent visuals.

[0025] As described above, an initial image-to-video model is first constructed. Then, training images are augmented to generate a sequence of training video frames with temporal correlation. The initial image-to-video model is iteratively trained using the training images and their corresponding training video frame sequences as training samples. This allows the model to learn the continuous motion patterns of the images, resulting in a target image-to-video model. This improves the accuracy of the model in generating a continuous sequence of video frames based on the first frame. Alternatively, by utilizing a target text-to-image model, the first frame image is generated from the text description information. This first frame image is then used as input to drive the target image-to-video model to generate a sequence of subsequent video frames, resulting in a video corresponding to the text description information. Compared to directly generating all video frames based on text, this approach ensures the quality of the generated first frame using the target text-to-image model. The inter-frame transformation logic learned by the target image-to-video model ensures natural transitions and smooth motion in the generated video, effectively improving video stuttering and content gaps. It also reduces the computational cost of repetitive semantic parsing, thereby improving both the quality and efficiency of video generation.

[0026] In one alternative implementation, step 120 includes steps 121 to 123.

[0027] Step 121: Using the training images as a reference, perform the first amplification process and the second amplification process to obtain the initial video frame sequence.

[0028] In one optional implementation, based on a single original training image, amplification operations are performed individually or progressively overlaid using various amplification methods across different dimensions to generate multiple frames with basic motion change characteristics and inter-frame correlation, which are then combined to form an initial video frame sequence. The amplification process includes spatial transformation operations such as scaling, left shifting, right shifting, up shifting, and down shifting. The order in which various amplification operations are performed on the training image can be adjusted according to actual needs; that is, the first and second amplification processes are merely a general description and do not limit the types of amplification processes. For example, scaling, left shifting, right shifting, up shifting, and down shifting operations can be performed on the training image to obtain the initial video frame sequence.

[0029] Step 122: Based on the image parameters of each video frame in the initial video frame sequence, perform an expansion operation on each video frame to obtain an expanded image corresponding to each video frame.

[0030] In one optional implementation, the original resolution, screen size, color parameters, contrast, saturation, and other image parameters of each video frame in the initial video frame sequence are expanded and adjusted in multiple dimensions. The parameters of a single video frame image are adjusted to reconstruct new images in a diversified manner. While retaining the original inter-frame motion correlation logic, an expanded image with parameters different from that of a single video frame is generated to enrich the diversity of sample video frames and improve the generalization ability of the model.

[0031] Step 123: Store each video frame in the initial video frame sequence and the corresponding extended image in the temporal order of the initial video frame sequence to obtain the training video frame sequence.

[0032] In one optional implementation, following the chronological order of the initial video frame sequence, the original video frame at each temporal position is paired and associated with the corresponding generated extended image for storage, and a unified temporal index label is used to ensure that the motion logic and temporal order of all frame data are not disrupted, thus forming a temporally ordered and completely associated training video frame sequence.

[0033] As described above, the process begins by using the original training images as a baseline. Various augmentation operations are then performed to generate an initial video frame sequence with continuous motion characteristics, simulating the spatial transformation process of real-world images. Next, image parameters such as resolution, color, and size of each frame within the initial video frame sequence are expanded to obtain multiple extended images corresponding to the video frames, enriching the sample diversity. Finally, all video frames and their corresponding extended images are stored sequentially according to the temporal order of the initial video frame sequence to obtain the training video frame sequence. This allows the image-to-video model to fully learn the inter-frame correlation patterns under different motion states, improving its adaptability to different image qualities and styles. Ultimately, the trained target image-to-video model can generate smoothly connected and richly formatted videos.

[0034] In one alternative implementation, step 122 includes either step 1221 or step 1222.

[0035] Step 1221: Using the training images as a reference, perform the first amplification process and the second amplification process respectively to obtain the initial video frame sequence.

[0036] In one optional implementation, the training image is continuously scaled according to a preset scaling ratio to obtain a first video frame sequence; the training image is continuously shifted left according to a preset first displacement step to obtain a second video frame sequence; the training image is continuously shifted right according to a preset second displacement step to obtain a third video frame sequence; the training image is continuously shifted upward according to a preset third displacement step to obtain a fourth video frame sequence; the training image is continuously shifted downward according to a preset fourth displacement step to obtain a fifth video frame sequence; and the initial video frame sequence is constructed based on at least two of the first, second, third, fourth, and fifth video frame sequences. It can be seen that if there are more than two amplification processes, more than two video frame sequences will be generated. Using the training image as a reference, after performing all amplification processes, the initial video frame sequence is constructed based on all the obtained video frame sequences.

[0037] Step 1222: Perform the first amplification process based on the training image, and then perform the second amplification process based on the result of the first amplification process to obtain the initial video frame sequence.

[0038] In one optional implementation, the training images are continuously scaled according to a preset scaling ratio to obtain a first video frame sequence; the last frame of the first video frame sequence is continuously shifted left by a preset first displacement step to obtain a second video frame sequence; the last frame of the second video frame sequence is continuously shifted right by a preset second displacement step to obtain a third video frame sequence; the last frame of the third video frame sequence is continuously shifted upward by a preset third displacement step to obtain a fourth video frame sequence; the last frame of the fourth video frame sequence is continuously shifted downward by a preset fourth displacement step to obtain a fifth video frame sequence; and the initial video frame sequence is constructed based on at least two of the first, second, third, fourth, and fifth video frame sequences. It can be seen that if there are more than two amplification processes, the result of each amplification process is used as the basis for the next amplification process in a preset order, with the first amplification process using the training images as the basis.

[0039] As described above, setting multiple amplification methods can enrich the types of basic motion samples, enable the initial video frame sequence to contain diverse motion features, and allow the image-to-video model to fully learn the inter-frame evolution relationship under different motion logics during training. This can effectively improve the image-to-video model's ability to fit and generate various dynamic changes in images.

[0040] In one alternative implementation, step 1221 includes steps 12211 to 12213.

[0041] Step 12211: Perform the first amplification process on the training images to obtain the first video frame sequence.

[0042] In one optional implementation, the first augmentation process is image scaling augmentation. A uniform gradient scaling ratio is preset. The original training image is used as the starting frame, and continuous gradual scaling operations of enlarging and shrinking are performed in sequence. Each scaling generates a new video frame. All gradually scaled frames are arranged in the order of operation to form a first video frame sequence with dynamic scale change characteristics.

[0043] Step 12212: Perform the second amplification process on the training images to obtain the second video frame sequence.

[0044] In one optional implementation, the second amplification process is image position translation amplification. A fixed and uniform displacement step size is preset. Based on the original training image, continuous left, right, up, and down translation operations are performed to gradually change the position of the image on the canvas, generating multiple frames of images with gradually changing positions. These frames are arranged in the order of the translation operations to form a second video frame sequence with dynamic position change characteristics.

[0045] Step 12213: Construct the initial video frame sequence based on the first video frame sequence and the second video frame sequence.

[0046] In one optional implementation, the first video frame sequence and the second video frame sequence are fused and spliced ​​together to unify the dynamic change logic of the frame sequence, remove duplicate frames and invalid frames, and integrate to obtain the initial video frame sequence.

[0047] As described above, by using the training images as a unified benchmark, the first amplification operation and the second amplification operation are performed respectively to generate the corresponding first video frame sequence and second video frame sequence. The two video frame sequences are then integrated to construct the initial video frame sequence, which can construct single motion samples respectively, so that the initial video frame sequence contains different types of dynamic features of the scene.

[0048] In one alternative implementation, step 1222 includes steps 12221 to 12223.

[0049] Step 12221: Perform the first amplification process on the training images to obtain the first video frame sequence.

[0050] In one alternative implementation, based on the original training images, a first augmentation process of continuous gradient scaling is performed to generate a series of video frames with gradually changing scales through multiple uniform scaling operations, which are then arranged in chronological order to form a first video frame sequence.

[0051] Step 12222: Perform the second amplification process on the first frame image of the first video frame sequence to obtain the third video frame sequence.

[0052] In one optional implementation, the starting reference frame of the first video frame sequence is selected as a new operation object, and the first frame image is subjected to continuous multi-directional position translation second amplification processing. Through step-by-step translation, multiple video frames with gradually changing positions are generated and arranged in time sequence to form a third video frame sequence with composite dynamic characteristics.

[0053] Step 12223: Construct the initial video frame sequence based on the first video frame sequence and the third video frame sequence.

[0054] In one alternative implementation, the first video frame sequence and the third video frame sequence are integrated, the dynamic transformation timing of all frames is regularized, abnormal frame data is removed, and an initial video frame sequence is constructed.

[0055] As described above, the first amplification operation is performed on the training images to obtain the first video frame sequence. Then, the first frame of the first video frame sequence is selected to perform the second amplification operation to generate the third video frame sequence. At least two video frame sequences are fused to construct the initial video frame sequence. This can simulate the combined motion pattern of the scene performing the first amplification operation and then the second amplification operation, thus enriching the dynamic types of the initial video frame samples.

[0056] In one alternative implementation, step 130 includes steps 131 to 133.

[0057] Step 131: Input the training image into the initial image-to-video model for iterative model training to obtain the predicted video frame sequence.

[0058] In one alternative implementation, training images are input one by one into the initial image-to-video model. Based on its own spatiotemporal feature learning ability and relying on the initial weight parameters, the model autonomously infers and generates a predicted video frame sequence corresponding to the training images. Each round of training iteration outputs a set of frame sequence prediction results.

[0059] Step 132: During the model iterative training process, a first loss value is calculated between the predicted video frames in the predicted video frame sequence and the corresponding video frames in the training video frame sequence using a preset first loss function.

[0060] In one optional implementation, a first loss function adapted to the temporal sequence of video frames is used. For each training image, the differences between the predicted video frames in the predicted video frame sequence and the video frames in the training video frame sequence are quantified and calculated, such as pixel differences, semantic differences, motion trajectory differences, etc. The error values ​​of all frames are summarized to obtain the first loss value of this iteration. The first loss value is used to reflect the prediction accuracy of the image-to-video model.

[0061] Step 133: Using a preset first optimizer, iteratively update the model parameters of the initial image-to-video model until the first loss value is lower than a preset first threshold or the number of model iterations reaches a preset first iteration number, to obtain the trained target image-to-video model.

[0062] In one optional implementation, a first optimizer with an adaptive learning rate is used. Backpropagation is performed based on the calculated first loss value to update the network weights and bias parameters of the initial image-to-video model layer by layer, continuously reducing the error between the predicted frame and the standard frame. The model iterative training process is executed repeatedly. When the loss value converges to below a preset first threshold, or when the number of iterations reaches a set maximum value, training is terminated, the optimal model parameters are saved, and the target image-to-video model is obtained based on these optimal parameters.

[0063] As described above, the initial image-to-video model takes the training image as input and outputs a sequence of predicted video frames. The deviation between the predicted video frames in the sequence and the corresponding video frames in the training sequence is quantified by the first loss function. Then, the model parameters are updated iteratively by the first optimizer. The termination of model training is controlled by the loss threshold or the number of iterations. This can continuously correct the learning bias of the initial image-to-video model in terms of inter-frame motion and image changes, allowing the model to fully grasp the evolution law from the reference image to continuous video frames. Finally, a target image-to-video model with high prediction accuracy and stable dynamic generation effect is obtained, ensuring the ability to generate smooth and coherent videos based on a single first frame image.

[0064] In an alternative implementation, steps 170 to 190 are included before step 150.

[0065] Step 170: Construct a training dataset based on the collected training text description information and the real images corresponding to the training text description information.

[0066] In one optional implementation, natural language text description information with multiple scenes, styles, and semantics is collected, and high-definition real images corresponding to the text description information are collected simultaneously to complete image-text data pairing and construct a large-scale, high-quality, and diverse training dataset for image-text pairing.

[0067] Step 180: Construct an initial text-to-image model using a diffusion model or an autoregressive model.

[0068] In one alternative implementation, a diffusion model or an autoregressive model is selected as the basic architecture to build an initial text-to-image model. The initial model parameters of the initial text-to-image model are set so that the initial text-to-image model has basic text semantic parsing and image generation capabilities.

[0069] Step 190: Train the initial text-to-image model using the training dataset to obtain the trained target text-to-image model.

[0070] In one alternative implementation, the initial text-to-image model is trained in batches using a training dataset with image-text pairing. The semantic-image mapping capability of the model is continuously optimized through multiple rounds of iterative training. The difference between the generated image and the real image is calculated using a loss function until the model converges. Finally, a target text-to-image model that can generate high-definition, accurate, and stable first-frame images based on any standardized text description is obtained.

[0071] As described above, a standardized training dataset is first constructed by collecting paired text descriptions and real images, providing sufficient samples to support the initial text-to-image model in learning the mapping relationship between text and images. An initial network structure is built using a diffusion model or autoregressive model adapted for text-image generation. The model is then trained using the dataset, and its parameters are continuously optimized to ensure the model accurately grasps the semantic-to-image conversion logic. Ultimately, a high-quality text-to-image model with good content matching is obtained, capable of outputting a high-quality first frame based on the text description information.

[0072] In one alternative implementation, step 190 includes steps 191 to 193.

[0073] Step 191: Input the training text description information in the training dataset into the initial text-to-image model for iterative training to obtain a set of predicted images.

[0074] In one alternative implementation, the training dataset is divided into batches, and standardized training text description information is input into the initial text-to-image model in batches. The model parses the core features of the text through semantic encoding, decodes and renders the corresponding predicted images, and after a single iteration completes the inference of all batches of data, the predicted image set for this round is generated.

[0075] Step 192: During the model iterative training process, a second loss function is used to calculate the second loss value between the predicted images in the predicted image set and the corresponding real images in the training dataset that correspond to the training text description information.

[0076] In one alternative implementation, a second loss function is invoked to quantify the error between the predicted image and the real image for each training text description information. This error includes factors such as pixel error, texture error, color error, and semantic content error. The error data of all training text description information is integrated to calculate the second loss value for this iteration. The second loss value is used to evaluate the accuracy of image generation by the initial text-to-image model.

[0077] Step 193: Using a preset second optimizer, iteratively update the model parameters of the initial text-to-image model until the second loss value is lower than a preset second threshold or the number of model iterations reaches a preset second iteration number, thereby obtaining the trained target text-to-image model.

[0078] In one optional implementation, a second optimizer performs backpropagation calculations based on a second loss value, adaptively adjusting the model parameters of the initial text-to-image model to continuously optimize the model's text-image matching accuracy and image rendering effect. The iterative training, loss calculation, and parameter update process is repeatedly executed. When the second loss value converges to a preset second threshold, or the number of iterations reaches the upper limit, model training stops, the optimal model parameters are saved, and the target text-to-image model is obtained based on these optimal parameters.

[0079] As described above, by inputting the training text description information into the initial text-to-image model to generate predicted images, quantifying the difference between the predicted images and the real images through the second loss function, and continuously updating the model parameters through the second optimizer, and terminating the model training by combining the loss threshold or the maximum number of iterations, the initial text-to-video model can deeply learn the conversion rules from text semantics to image content, and finally output a target text-to-image model with accurate imaging and stable image quality. Using the target text-to-image model, the first frame image that meets the requirements can be reliably generated based on the text description information.

[0080] Figure 2 This is another flowchart illustrating a video generation method according to an embodiment of the present invention. (Refer to...) Figure 2 This embodiment includes steps 201 to 207.

[0081] Step 201: Construct a training dataset for image-text pairing.

[0082] In one optional implementation, publicly available high-definition image-text pairing data from the internet is collected in batches to construct a text description information-real image pairing sample set. The text description information in the text description information-real image pairing sample set is cleaned, deduplicated, and semantically standardized. The real images in the text description information-real image pairing sample set are subjected to resolution unification, defect repair, and color normalization processing to construct a standardized image-text pairing training dataset.

[0083] Step 202: Construct the initial text-to-image model.

[0084] In one alternative implementation, the Stable Diffusion model in the diffusion model architecture is used as the base model. Publicly pre-trained weights are loaded to initialize the model parameters, and an initial text-to-image model including a text encoding module, a latent space diffusion module, and an image decoding and rendering module is built, giving it basic text semantic parsing and image generation capabilities.

[0085] Step 203: Iteratively train the initial text-to-image model to obtain the target text-to-image model.

[0086] In one optional implementation, the training dataset is divided into batches, for example, with a batch size of 32. The textual descriptions from each batch of training dataset are input into the initial text-to-image model, which then generates a corresponding set of predicted images. The MSE loss function is used as the second loss function, and the pixel, color, and semantic errors between the predicted image and the corresponding real image are calculated frame by frame to obtain the second loss value for each iteration. The Adam optimizer is used as the second optimizer, with an initial learning rate set. The model parameters are updated via backpropagation based on the second loss value. For example, the preset second loss threshold is 0.01, and the maximum number of second iterations is 10,000. Training is continuously iterated until the second loss value falls below 0.01 or the number of iterations reaches 10,000. At this point, model training is stopped, the optimal model parameters are saved, and these optimal parameters are used to obtain the model parameters for the target text-to-image model.

[0087] Step 204: Construct the initial image-to-video model.

[0088] In one alternative implementation, a video diffusion model architecture based on a spatiotemporal attention mechanism is adopted to build an initial image-to-video model. The model has built-in spatial attention and temporal attention modules, which can learn the multi-dimensional spatiotemporal transformation rules of a single training image and support using a single frame image as input to output a continuous video frame sequence.

[0089] Step 205: Perform training image augmentation processing to construct a training video frame sequence.

[0090] In one optional implementation, a single high-resolution training image from various scenes is selected as a reference image, and augmentation processing is performed. For example, using the training image as a reference, scaling, left-shifting, right-shifting, up-shifting, and down-shifting operations are performed sequentially to obtain a scaled video frame sequence, a left-shifted video frame sequence, a right-shifted video frame sequence, an up-shifted video frame sequence, and a down-shifted video frame sequence. The scaled video frame sequence, the left-shifted video frame sequence, the right-shifted video frame sequence, the up-shifted video frame sequence, and the down-shifted video frame sequence are then integrated according to the motion sequence to generate an initial training video frame sequence.

[0091] For each individual frame in the initial video frame sequence, image parameters such as resolution, image size, saturation, contrast, and brightness are adjusted to generate diverse extended images. Following the order of image augmentation and transformation, each individual frame and its corresponding extended image in the initial video frame sequence are uniformly labeled with a temporal index. All frame data is stored in an orderly manner to prevent temporal discrepancies, resulting in a standardized training video frame sequence with continuous motion correlation and abundant samples.

[0092] Step 206: Iteratively train the initial image-to-video model to obtain the target image-to-video model.

[0093] In one optional implementation, training images are input into an initial image-to-video model, and the model infers and outputs a sequence of predicted video frames. A temporal gradient matching loss function is used as the first loss function. For each training image, the motion trajectory error and pixel difference between the predicted video frames in the predicted video frame sequence and the video frames in the training video frame sequence are calculated frame by frame to obtain a first loss value. An SGD optimizer is used as the first optimizer to iteratively update the model parameters. For example, a preset first loss threshold of 0.008 and a maximum first iteration count of 80,000 are set. Training stops when the loss value reaches the threshold or the maximum number of iterations is reached, and the optimal model parameters are saved. These optimal model parameters are then used as the model parameters for the target image-to-video model.

[0094] Step 207: Using the target text-to-image model and the target image-to-video model, the user-input text description information is used to generate the corresponding video.

[0095] In one optional implementation, user input information is acquired and preprocessed. The text description information input by the user is collected, and invalid characters are filtered, semantic errors are corrected, and keywords are extracted to obtain standardized text description information.

[0096] Generate the first frame of the video. Input the standardized text description into the trained target text-to-image model. The model parses the semantics of the text and renders a clear, accurately composed first frame image that matches the text description.

[0097] Generate consecutive video frames and synthesize the video. The generated first frame image is input into the target image-to-video model. Based on the spatiotemporal motion patterns learned during training, the model uses the first frame as a baseline to autoregressively generate consecutive and coherent subsequent video frames. The first frame image is then stitched together with all subsequent video frames in chronological order to obtain a complete video.

[0098] Please refer to Figure 3The present invention also provides a video generation system 300, including a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302. When the processor executes the computer program, it implements the various steps of the video generation method described above.

[0099] In summary, the video generation method and system provided in this application construct an initial image-to-video model using a Transformer model or a diffusion model based on a spatiotemporal attention mechanism. Then, based on training images, it constructs an initial video frame sequence with motion features through various augmentation operations such as scaling and translation. Furthermore, it expands the resolution, color, size, and other parameters of each frame in the sequence. Simultaneously, it stores all frames and corresponding extended images in strict temporal order, forming a training video frame sequence with rich samples and standardized temporal order. This allows the initial image-to-video model to comprehensively learn the inter-frame correlation rules under various motions, improving its adaptability and dynamic generation capabilities for different image qualities and styles.

[0100] The training images are input into the initial image-to-video model to obtain the predicted video frame sequence. The inter-frame deviation is calculated using the first loss function, and the model parameters of the initial image-to-video model are iteratively updated through the first optimizer. Combined with the loss threshold or the termination training condition of the number of iterations, the model learning deviation is corrected, and an accurate target image-to-video model can be obtained.

[0101] A training dataset is constructed by collecting paired data of text description information and real images. An initial text-to-image model is built using a diffusion model or an autoregressive model. The training text is input into the initial text-to-image model to generate predicted images. The imaging differences are evaluated using a second loss function and iterative training is completed with the second optimizer. This enables the initial text-to-image model to accurately grasp the mapping relationship between text and images, and to produce a target text-to-image model with high imaging quality and good matching degree.

[0102] In practical applications, the target text-to-image model first generates a high-quality first-frame image based on the user-input text description. This first-frame image is then input into the target image-to-video model to generate subsequent video frames and synthesize a complete video. The text-to-image model ensures the quality of the first-frame image, while the image-to-video model's learned inter-frame transformation logic guarantees natural transitions and smooth motion in the generated video, effectively solving problems such as video stuttering and content gaps. Furthermore, it reduces the computational overhead of repetitive semantic parsing, improving both the quality and efficiency of video generation.

[0103] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A video generation method, characterized in that, include: Build an initial image-to-video model; Each preset training image is augmented to obtain a training video frame sequence corresponding to the training image. The initial image-to-video model is trained using the training images and the training video frame sequence corresponding to the training images to obtain the trained target image-to-video model. Obtain text description information; The text description information is input into a preset target text-to-image model to obtain the first frame image; The first frame image is input into the target image-to-video model to obtain a video corresponding to the text description information.

2. The method according to claim 1, characterized in that, The step of amplifying each preset training image to obtain a training video frame sequence corresponding to the training images includes: Based on the training images, perform the first amplification process and the second amplification process to obtain the initial video frame sequence; Based on the image parameters of each video frame in the initial video frame sequence, an expansion operation is performed on each video frame to obtain an expanded image corresponding to each video frame. Each video frame in the initial video frame sequence and its corresponding extended image are stored in the temporal order of the initial video frame sequence to obtain the training video frame sequence.

3. The method according to claim 2, characterized in that, The step of performing a first amplification process and a second amplification process based on the training images to obtain an initial video frame sequence includes: Using the training images as a reference, the first amplification process and the second amplification process are performed respectively to obtain the initial video frame sequence; Alternatively, the first amplification process can be performed based on the training images, and then the second amplification process can be performed based on the result of the first amplification process to obtain the initial video frame sequence.

4. The method according to claim 3, characterized in that, The step of performing the first amplification process and the second amplification process respectively, based on the training images, to obtain the initial video frame sequence includes: The first amplification process is performed on the training images to obtain a first video frame sequence; The training images are subjected to the second amplification process to obtain a second video frame sequence; The initial video frame sequence is constructed based on the first video frame sequence and the second video frame sequence.

5. The method according to claim 3, characterized in that, The step of performing the first amplification process based on the training images, and then performing the second amplification process based on the result of the first amplification process to obtain the initial video frame sequence includes: The first amplification process is performed on the training images to obtain a first video frame sequence; The first frame of the first video frame sequence is subjected to the second amplification process to obtain the third video frame sequence; The initial video frame sequence is constructed based on the first video frame sequence and the third video frame sequence.

6. The method according to claim 1, characterized in that, The step of training the initial image-to-video model using the training images and the corresponding training video frame sequences to obtain the trained target image-to-video model includes: The training images are input into the initial image-to-video model for iterative training to obtain a predicted video frame sequence. During the model iterative training process, a first loss function is used to calculate the first loss value between the predicted video frames in the predicted video frame sequence and the corresponding video frames in the training video frame sequence. Using a preset first optimizer, the model parameters of the initial image-to-video model are iteratively updated until the first loss value is lower than a preset first threshold or the number of model iterations reaches a preset first iteration number, thus obtaining the trained target image-to-video model.

7. The method according to claim 1, characterized in that, Also includes: A training dataset is constructed based on the collected training text description information and the real images corresponding to the training text description information; An initial text-to-image model is constructed using either a diffusion model or an autoregressive model. The initial text-to-image model is trained using the training dataset to obtain the trained target text-to-image model.

8. The method according to claim 7, characterized in that, The step of training the initial text-to-image model using the training dataset to obtain the trained target text-to-image model includes: The training text description information in the training dataset is input into the initial text-to-image model for iterative training to obtain a set of predicted images. During the model iterative training process, a second loss function is used to calculate the second loss value between the predicted images in the predicted image set and the real images in the corresponding training dataset that correspond to the training text description information. Using a preset second optimizer, the model parameters of the initial text-to-image model are iteratively updated until the second loss value is lower than a preset second threshold or the number of model iterations reaches a preset second iteration number, thus obtaining the trained target text-to-image model.

9. The method according to claim 1, characterized in that, The construction of the initial image-to-video model includes: An initial image-to-video model is constructed using either the Transformer model or a diffusion model based on spatiotemporal attention.

10. A video generation system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the video generation method according to any one of claims 1 to 9.

Images