A method and system for generating transition video

Abstract

The present application discloses a method and system for generating a transition video, and belongs to the field of image generation technology. The present application method inputs the first and last frames of images into a variational autoencoder respectively to extract their potential features, and introduces a preset number of frames of noise between the potential features of the positive order and the reverse order. Then, the first and last frames of images are described, the corresponding text features are extracted using a text feature extractor, and the image description features corresponding to the number of frames are obtained by spherical linear interpolation. Subsequently, these descriptions and potential features are jointly input into a pre-trained text and image-guided video generation diffusion model for denoising. In the denoising process, a noise feature containing intermediate frame information is generated, and a temporal constraint is imposed on the positive order and reverse order noise features by a Gaussian process. Next, high and low frequency separation and weighted fusion are performed, and denoising is performed again. Finally, a coherent and smooth transition video between the first and last frames is generated using the decoder of the video generation model.

Description

Technical Field

[0001] The present application belongs to the field of image generation technology, and in particular, relates to a transition video generation method and system. Background Art

[0002] Transition Video Generation is an advanced video processing technology that aims to create natural and smooth transition effects between two videos or image sequences. It is widely used in film and television production, advertising, video editing and other fields. Its main goal is to enhance the visual continuity and viewing experience of the video. Traditional transition effects (such as fade in and fade out, switching, etc.) usually rely on predefined patterns or simple special effects operations. Although they can achieve basic transitions, they are often difficult to fully express the dynamic changes of the picture in complex scenes, and cannot ensure the semantic consistency of the image content.

[0003] With the rapid development of video generation technology, especially the video generation method based on denoising model has gradually matured. This type of technology can generate high-quality video content at a low computational cost by using text or images as input, and has been widely used in video content production. Given the first and last frame images, especially when the content is similar, the generation model can automatically generate intermediate transition frames with semantic consistency and temporal rationality based on the image content, thereby achieving a smooth transition effect.

[0004] However, when the content of the first and last frames differ greatly, the existing video generation models still face certain challenges in terms of temporal consistency, and often have problems such as sudden timing changes and unnatural transitions, resulting in a decrease in video coherence. Therefore, it is urgent to further study and optimize the timing constraints in the transition video generation process to ensure that even when the content of the first and last frames differs greatly, a transition video with reasonable timing and smooth visuals can be generated. Summary of the invention

[0005] The purpose of the present application is to: in order to overcome the problems of the prior art, disclose a method and system for generating transition videos, and solve the problems of sudden changes in timing and unnatural transitions in the current video fusion methods in generating transition videos.

[0006] On the one hand, the purpose of this application is achieved through the following technical solutions:

[0007] A method for generating a transition video, the method comprising the following steps:

[0008] S100: Inputting the first frame and the last frame used to generate the transition video into a variational autoencoder to extract image features, and then inserting a preset number of frames of noise between the extracted feature frames to generate a positive order and a reverse order noise feature sequence respectively;

[0009] S200: describing the first and last two frames of images used to generate the transition video respectively, extracting corresponding text features, and then using spherical linear interpolation to obtain text description features corresponding to the number of frames;

[0010] S300: The noise features in the positive order and the reverse order obtained in S100 are combined with the text description features obtained in S200 and input into the pre-trained text and image guided video generation diffusion model for denoising, and the denoised intermediate features are extracted;

[0011] S400: for the intermediate features of the positive sequence and the reverse sequence extracted in S300, a Gaussian process is used to establish timing constraints for the noise features of the positive sequence and the reverse sequence respectively;

[0012] S500: Separate the high frequency and low frequency of the positive and reverse order Gaussian constrained noise features obtained in S400, assign different weights to the two, add them together, and interpolate to restore to the original feature size;

[0013] S600: weighted fusion of the weighted fused forward and reverse feature frames in frame order, and repeat S300 to S500 until the denoising feature reaches a convergence state;

[0014] S700: Inputting the feature frame that has been denoised and has a stable noise level into a video generation decoder to generate a transition video.

[0015] According to a preferred embodiment, step S100 includes:

[0016] S110: using a variational autoencoder to respectively extract first frame and last frame image features for generating transition content;

[0017] S120: Generate a specified number of random Gaussian noise frames, where the size of each frame of Gaussian noise is the same as the size of the input first and last frame features;

[0018] S130: inserting the generated Gaussian noise with the first frame and the last frame as the positive order, and then swapping the order of the first and last frames to insert the Gaussian noise, to obtain the positive order noise feature and the reverse order noise feature.

[0019] According to a preferred embodiment, step S200 includes:

[0020] S210: Analyze the input first frame and last frame images, and use a text description generation model or manually generate respective text descriptions;

[0021] S220: Process the generated text description using a text feature extractor to extract its corresponding text feature vector;

[0022] S230: uniformly generating a set of weight values ​​according to a preset number of frames, representing a transition from a first frame to a last frame, wherein the weight value changes from 0 to 1, the weight of the first frame gradually decreases, and the weight of the last frame gradually increases;

[0023] S240: By setting the generated weights, spherical linear interpolation is performed on the text features of the first and last frames to obtain a number of intermediate text features corresponding to a preset number of frames.

[0024] According to a preferred embodiment, step S300 includes:

[0025] S310: Fusing the positive order and reverse order features with the corresponding intermediate text features respectively, using a cross attention mechanism or a feature accumulation and superposition form, to obtain positive order noise features and reverse order noise features containing text information;

[0026] S320: Input the obtained positive sequence noise features and reverse sequence noise features into a video denoising model;

[0027] S330: Using the denoising features extracted from the video denoising model as intermediate features.

[0028] According to a preferred embodiment, step S400 includes:

[0029] S410: Taking the first frame of the positive sequence intermediate feature as the training input and the last frame as the target output, a positive sequence Gaussian process model is constructed;

[0030] S420: Using the first frame of the reverse intermediate feature as training input and the last frame as target output, to construct a reverse Gaussian process model;

[0031] S430: Select a covariance function, calculate the correlation between the input and output of the positive order Gaussian process model and the reverse order Gaussian process model and the corresponding intermediate frame noise features, and construct a joint Gaussian distribution;

[0032] S440: Calculate the prediction means corresponding to the forward and reverse intermediate frames respectively according to the joint Gaussian distribution;

[0033] S450: Weight the predicted means corresponding to the forward and reverse intermediate frames respectively and add them to the corresponding original intermediate frames.

[0034] According to a preferred embodiment, step S500 includes:

[0035] S510: Use maximum pooling processing on the positive order and reverse order Gaussian constrained noise features to capture high-frequency information in the features;

[0036] S520: using mean pooling processing for the positive order and reverse order Gaussian constrained noise features respectively to capture low-frequency information in the features;

[0037] S530: Assign weights to the high-frequency features extracted in step S510 , assign weights to the low-frequency features extracted in step S520 ,in The coefficient for controlling the balance of high and low frequency characteristics, with a value range between 0 and 1;

[0038] S540: adding the weighted positive-sequence high-frequency features to the low-frequency features to generate fused positive-sequence features; at the same time, adding the weighted reverse-sequence high-frequency features to the low-frequency features to generate fused reverse-sequence features;

[0039] S550: interpolating the positive-order features and the reverse-order features obtained in step S540 to restore them to the same size as the original Gaussian constrained noise features.

[0040] According to a preferred embodiment, step S600 includes:

[0041] S610: Add the first frame of the weighted fusion of the forward sequence features and the last frame of the reverse sequence features, and add the second frame of the forward sequence and the second to last frame of the reverse sequence, until all the forward sequence features and the reverse sequence features are added; and in the process of feature addition, the forward sequence features are given weights , weights are assigned to features in reverse order ,in The value range of is 0 to 0.5;

[0042] S620: Input the fused features into the pre-trained diffusion model frame by frame, use the self-attention mechanism to analyze the relationship between the features and perform feature reorganization to remove noise;

[0043] S630: In the diffusion model, the self-attention mechanism is used to process the features layer by layer. The weight of the features is adjusted in each layer according to the context information, so that the information of interest is enhanced and the noise is suppressed, thereby extracting new intermediate features and completing the intermediate feature update;

[0044] S640: Repeat steps S300 to S500 to repeatedly perform Gaussian process constraint features, then perform high and low frequency decomposition and feature fusion, and then perform denoising, and continuously optimize the intermediate features; and evaluate the noise level of the feature after each processing, and compare it with the preset low noise standard until the denoising feature reaches a convergence state.

[0045] According to a preferred embodiment, step S700 includes:

[0046] S710: The decoder receives the denoised feature input and processes it through a neural network; each layer of the neural network can perform feature mapping based on the spatial and temporal information of the feature to extract various details of the video frame, including texture, color, and lighting;

[0047] S720: During the decoding process, the decoder uses an upsampling method to gradually restore the spatial resolution of the video frame, thereby generating each video frame image;

[0048] S730: After the generated transition video frames are batch processed, image enhancement processing is performed, including contrast adjustment, sharpening processing, and color correction, so as to improve image quality;

[0049] S740: After all video frames are generated and processed, the video frames are combined into a complete transition video in a correct order.

[0050] On the other hand, the present application also discloses:

[0051] A transition video generation system, wherein the transition video generation system adopts the aforementioned transition video generation method to perform video processing, and the transition video generation system comprises:

[0052] Input module: used to receive the first and last two frames of images and their corresponding text descriptions;

[0053] Feature extraction unit: A variational autoencoder is used to extract the features of the input image, and a text feature extractor is used to generate the corresponding text feature vector;

[0054] Interpolation module: Based on the extracted text features, several intermediate text features are generated through spherical linear interpolation method to achieve smooth transition;

[0055] Noise generation unit: Generates Gaussian noise of the same size as the feature and inserts it between the potential features in positive and negative order;

[0056] Fusion module: combines noise features with intermediate text features to form new features and input them into the diffusion model;

[0057] Denoising unit: Input the fused features into the pre-trained video generation diffusion model, extract the denoised intermediate features, and apply Gaussian process constraints to ensure temporal consistency;

[0058] High-frequency and low-frequency separation module: separates the high-frequency and low-frequency features of the denoising features, and assigns different weights to the two for weighted fusion;

[0059] Decoder: Input the denoised features into the video generation decoder to generate the final transition video frame;

[0060] Output module: used to store or play the generated transition video in real time to ensure that users have a high-quality visual experience.

[0061] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0062] Beneficial effects of this application:

[0063] This application combines text conditions with first and last frame images to guide video generation, has strong controllability and flexibility, and can generate high-quality transition videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flow chart of a method for generating a transition video according to an embodiment of the present application is shown.

[0065] Figure 2 A schematic diagram showing an example result of generating a transition video according to an embodiment of the present application is shown.

[0066] Figure 3 A schematic diagram of the process of generating a transition video proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0067] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0068] Transition video generation is an advanced video processing technology that aims to achieve a natural and smooth transition between two videos or image sequences. Existing technologies still face the problem of insufficient timing consistency when processing large differences in the content of the first and last frames, which often leads to sudden timing changes and unnatural transitions in the video, affecting the overall coherence.

[0069] In order to solve the above technical problems, the embodiment of the present application proposes a method and system for generating transition videos, aiming to improve the temporal consistency of transition video generation and improve the generation effect. A video generation model is adopted, a random Gaussian process is introduced as a temporal constraint, and the generation quality is improved by combining forward and reverse order generation, which makes full use of the creativity of the video generation model and has strong controllability, and can be widely used in video generation, which will be explained below.

[0070] Example 1

[0071] Please refer to Figure 1 , Figure 1 A schematic diagram of a process of generating a transition video according to an embodiment of the present application is shown, and the method includes:

[0072] Step S100: input the first frame and the last frame into the variational autoencoder to extract potential features, and then insert a preset number of frames of noise between the extracted feature frames to generate positive and reverse noise feature sequences respectively.

[0073] Step S100 includes:

[0074] Step S110: using a variational autoencoder to respectively extract the image features of the first frame and the last frame for generating transition content.

[0075] Variational autoencoder: A variational autoencoder corresponding to a pre-trained video generation model can be used to encode input images. The encoding may include 3D encoding or 2D encoding. 3D encoding may involve copying a specified number of frames before encoding.

[0076] Step S120: Generate a specified number of frames of random Gaussian noise, where the size of each frame of Gaussian noise is the same as the size of the input first and last frame features.

[0077] Step S130: insert the generated Gaussian noise with the first frame and the last frame as the positive sequence, and insert the Gaussian noise after swapping the first and last frames to obtain the positive sequence noise feature and the reverse sequence noise feature.

[0078] Step S200: respectively describe the first and last two frames of images, extract corresponding text features, and then use spherical linear interpolation to obtain text description features corresponding to the number of frames.

[0079] Step S200 includes:

[0080] Step S210: Analyze the input first frame and last frame images, and generate respective text descriptions using a text description generation model or manual means. These descriptions should be able to effectively capture the key content and context information of the image.

[0081] Step S220: Use a text feature extractor to process the generated text description and extract its corresponding text feature vector.

[0082] Step S230: According to the preset number of frames, a set of weight values ​​is uniformly generated to represent the transition from the first frame to the last frame. The weight value changes from 0 to 1, the weight of the first frame gradually decreases, and the weight of the last frame gradually increases.

[0083] Step S240: by setting the generated weights, spherical linear interpolation is performed on the text features of the first and last frames to obtain a plurality of intermediate text features corresponding to a preset number of frames.

[0084] Spherical linear interpolation: is a method for smooth interpolation in high-dimensional space, which can generate a smooth transition between two directional vectors. Specifically, the spherical linear interpolation method is used to calculate the intermediate frame image description corresponding to the preset number of frames, and the steps to ensure that the generated description is semantically coherent and smooth include:

[0085] According to the preset number of frames, a set of weight values ​​is evenly distributed to represent the proportion of the text features in the first frame. The mathematical expression is:

[0086]

[0087] Among them, the length of the weight array is the preset number of frames , the weight of the corresponding tail frame text feature is ;

[0088] Use the spherical linear interpolation method to calculate the intermediate text feature vector based on the generated weights :

[0089]

[0090] in, and are the text feature vectors of the first frame and the last frame respectively;

[0091] Each intermediate text feature vector The input to the generative model is considered to generate the corresponding intermediate frame image description to ensure that the generated description is semantically coherent and visually smooth.

[0092] Step S300: The noise features in the positive order and the reverse order are combined with the text description features and input into the pre-trained text and image guided video generation diffusion model for denoising, and the denoised intermediate features are extracted.

[0093] Step S300 includes:

[0094] Step S310: Fusing the forward and reverse order features with the corresponding intermediate text features respectively, using a cross-attention module or feature accumulation, superposition, etc., to obtain new forward and reverse order noise features containing text information.

[0095] Step S320: inputting the new positive-sequence noise features and the reverse-sequence noise features into the video denoising model.

[0096] Step S330: using the denoising features extracted from the Transformer module included in the video denoising model as intermediate features.

[0097] Among them, the positive order noise feature and the reverse order noise feature: the positive order noise is combined with the text feature, the reverse order noise feature is combined with the text feature, the two combined features form a batch data, and are input into the video denoising model together. The positive order intermediate features and the reverse order intermediate features can be obtained in parallel, which can enhance the processing efficiency.

[0098] Intermediate features: Intermediate features play a key role in the entire process. They are not just simple denoising results, but a representation that contains rich temporal and semantic information. These features will be further used to enhance temporal constraints, that is, to maintain the coherence between frames during the generation process. They are further input into the subsequent denoising module to enhance the continuity of the generated video during the denoising process.

[0099] Input to the subsequent denoising module: The intermediate features that have been processed with enhanced temporal constraints will be used as input and passed to the subsequent similar denoising module. In this module, new intermediate features will be extracted and further processed with temporal constraints. This process is repeated until the denoising result is stable. In each iteration, the optimized intermediate features are used for deeper denoising and feature optimization, and finally a high-quality, coherent transition video is generated.

[0100] Step S400: for extracting the intermediate features of the positive sequence and the reverse sequence, a Gaussian process is used to establish timing constraints for the noise features of the positive sequence and the reverse sequence respectively.

[0101] Step S400 includes:

[0102] Step S410: Use the first frame of the positive sequence intermediate features as training input and the last frame as target output to construct a positive sequence Gaussian process model.

[0103] Step S420: Use the first frame of the reverse intermediate features as training input and the last frame as target output to construct a reverse Gaussian process model.

[0104] Step S430: Select a covariance function, calculate the correlation between the input and output of the positive order Gaussian process model and the reverse order Gaussian process model, and the corresponding intermediate frame noise features, and construct a joint Gaussian distribution.

[0105] Step S440: Calculate the prediction means corresponding to the middle frames in the positive order and the middle frames in the reverse order respectively according to the joint Gaussian distribution.

[0106] Step S450: weighting the corresponding means respectively and adding them to the corresponding original intermediate frames, thereby improving the smoothness of the forward and reverse features in their respective time series.

[0107] Gaussian process model: It is a random process defined on the input space, in which each input point corresponds to a Gaussian distribution. For any finite number of input points, the joint distribution of its output values ​​is a multivariate Gaussian distribution. It is a non-parametric Bayesian inference tool. The transition process of the transition video can be assumed to be a Gaussian random process.

[0108] Covariance function: It is the core part of the Gaussian process and is used to define the correlation between input points. Choosing a suitable covariance function can affect the flexibility and fitting ability of the model. Common covariance functions include square exponential kernel, linear kernel, etc. Depending on the effect of the transition video, you may need to choose a different covariance function.

[0109] Video transitions usually involve a smooth transition from one scene to another. In this process, the intermediate frames need to maintain a certain degree of coherence and natural smoothness. This is consistent with the characteristics of the Gaussian process, because the Gaussian process can ensure the temporal consistency of the generated intermediate frames by modeling the timing constraints between the first and last frames. This constraint can enhance the smoothness of the generation process and avoid sudden jumps or unnatural changes.

[0110] Step S500: Separate the high frequency and low frequency of the Gaussian constrained noise features in the positive order and the reverse order, assign different weights to the two, add them together, and interpolate to restore them to the original feature size.

[0111] The main reason for separating high and low frequencies is that the human eye has different perceptions of features at different frequencies. The high-frequency content in the features usually corresponds to detailed information and rapidly changing features in the video, while the low-frequency content mainly includes the basic structure and overall brightness of the image. This step aims to adjust the importance of high-frequency and low-frequency information in the final video generation to better conform to the observation habits of the human eye.

[0112] Step S500 includes:

[0113] Step S510: Use maximum pooling operations on the positive order features and the reverse order features respectively to capture high-frequency information in the features.

[0114] Step S520: Use mean pooling operations on the positive order features and the reverse order features respectively to capture low-frequency information in the features.

[0115] Step S530: assign weights to the high-frequency features of the positive and reverse order extracted in step S510 , weight the positive and reverse low-frequency features extracted in step S520 ,in The coefficient for controlling the balance of high and low frequency characteristics is between 0 and 1. The value can adjust the impact of high-frequency and low-frequency features on the final result, so that the generated video achieves a balance between details and overall perception.

[0116] Step S540: Add the weighted positive-sequence high-frequency features to the low-frequency features to generate fused positive-sequence features. At the same time, add the weighted reverse-sequence high-frequency features to the low-frequency features to generate fused reverse-sequence features.

[0117] Step S550: interpolate the positive and negative features obtained in step S540 to restore them to the same size as the original Gaussian constrained noise features. The interpolation process should ensure that the generated features retain the key information of the original noise features and achieve effective fusion of high-frequency and low-frequency features.

[0118] Step S560: The restored forward and reverse features are input into a subsequent feature processing module, combined with other possible processing steps, such as feature fusion or feature enhancement, to further optimize the generated results and ensure that the details and overall perception of the video output meet the expected quality standards.

[0119] Step S600: weighted fusion of the weighted fused forward and reverse feature frames in frame order, and input into the next stage denoising module.

[0120] Step S600 includes:

[0121] Step S610: Add the first frame of the positive sequence feature to the last frame of the reverse sequence feature, add the second frame of the positive sequence to the second frame of the reverse sequence, and so on. In this process, the positive sequence feature is assigned a weight , weights are assigned to features in reverse order ,in The value range is 0 to 0.5; when the positive order features are fused to the reverse order features, the weights will be exchanged.

[0122] Step S620: Input the fused features frame by frame into the next Transformer module in the pre-trained diffusion model. This module uses the self-attention mechanism to analyze the relationship between features and perform feature reorganization, effectively removing noise and improving the quality of features.

[0123] Step S630: In the Transformer module, the features are processed layer by layer using a multi-layer self-attention mechanism. Each layer adjusts the weight of the feature according to the context information, so that important information is enhanced and unnecessary noise is suppressed, thereby extracting new intermediate features.

[0124] Step S640: Repeat steps S300 to S500 to repeatedly perform Gaussian process constraint features, then perform high and low frequency decomposition and feature fusion, and then perform denoising to continuously optimize intermediate features.

[0125] After each processing, the noise level of the feature is evaluated and compared with the preset low-noise standard until the denoising feature reaches a convergence state, ensuring that the final output feature achieves an ideal visual effect.

[0126] Step S700: basic feature denoising is completed, the noise level is stable, the input video is generated into a decoder, and a transition video is generated.

[0127] Figure 2 The example result diagram of the first frame image generation proposed in the embodiment of the present application is shown. The successful implementation of this step marks the completion of the entire video generation process, ensuring that the generated transition video not only meets the technical standards, but also meets the audience's expectations in terms of artistic effect.

[0128] Step S700 includes:

[0129] Step S710: The decoder receives the denoised feature input and processes it through a multi-layer neural network structure. Each layer of the network performs feature mapping based on the spatial and temporal information of the feature to extract various details of the video frame, including texture, color, and lighting.

[0130] Step S720: During the decoding process, the decoder gradually converts the high-dimensional features into low-dimensional image data. In this process, the decoder uses upsampling technology to gradually restore the spatial resolution of the video frame, thereby generating an image with higher visual quality.

[0131] Step S730: After the generated transition video frames are batch processed, they can be post-processed to improve the overall quality of the video, including applying image enhancement techniques such as contrast adjustment, sharpening, and color correction to ensure that the final output video meets high standards in visual effects.

[0132] Step S740: After all video frames are generated and processed, the system will combine these frames into a complete transition video in the correct order.

[0133] Step S750: Finally, the generated transition video will be stored or played in real time through the output module to ensure that the user can obtain a high-quality and smooth visual experience and complete a smooth transition from forward order to reverse order.

[0134] Figure 3The schematic flow chart of the transition video generation proposed in the embodiment of the present application is shown. First, the first and last two frames of images are input into the system, which represent the starting and ending parts of the video respectively, and the corresponding text description is input. The first and last frame images input are processed, and their potential features are extracted using a variational autoencoder, and these features will be used in the subsequent denoising step. At the same time, the text feature vectors corresponding to the first and last frame images are extracted by a text feature extractor. The spherical linear interpolation method is used to interpolate the text features of the first and last frames according to the generated weights to obtain a series of intermediate text features. Noise is added between the potential features in the positive order and the reverse order to generate noise features. After combining these noise features with the intermediate text features, they are input into a pre-trained video generation diffusion model for denoising. During the denoising process, the model extracts the denoised intermediate features and uses Gaussian process constraints to ensure the temporal consistency of the intermediate features. In addition, the high-frequency and low-frequency noise features are separated, and weighted addition is used to generate fused forward and reverse features. Finally, they are interpolated and restored to their original size and input into the subsequent feature processing module for denoising again. After iteration until the noise is stable, they are input into the video generation decoder to generate a transition video.

[0135] In one possible implementation, the video denoising model adopts a U-shaped network structure, which is divided into two parts: encoding and decoding, and consists of a three-dimensional convolution module, a cross-attention module, a spatial self-attention module, and a temporal self-attention module.

[0136] The intermediate features obtained in the denoising process also include the features of the Transformer module output provided by the spatial self-attention module and the temporal self-attention module in the U-shaped network structure.

[0137] In one possible implementation, the Gaussian process constraint process is to first construct a Gaussian process model, use the intermediate features in the forward and reverse order for training, and select an appropriate covariance function to calculate the correlation, thereby establishing a joint Gaussian distribution and calculating the predicted mean. Then, these means are weighted and added to the original intermediate frame features to enhance the smoothness of the features in their respective time series.

[0138] Example 2

[0139] A possible implementation of a transition video generation system is provided below, which is used to execute the various execution steps and corresponding technical effects of the transition video generation method shown in the above embodiment 1 and possible implementations. The transition video generation system includes:

[0140] Input module: used to receive the first and last two frames of images and their corresponding text descriptions for subsequent processing.

[0141] Feature extraction unit: A variational autoencoder is used to extract the potential features of the input image, and a text feature extractor is used to generate the corresponding text feature vector.

[0142] Interpolation module: Based on the extracted text features, a series of intermediate text features are generated through the spherical linear interpolation method to achieve smooth transition.

[0143] Noise generation unit: Generates Gaussian noise of the same size as the features and inserts it between the positive and negative latent features.

[0144] Fusion module: Combines noise features with intermediate text features to form new features that are input into the diffusion model.

[0145] Denoising unit: Input the fused features into the pre-trained video generation diffusion model, extract the denoised intermediate features, and apply Gaussian process constraints to ensure temporal consistency.

[0146] High-frequency and low-frequency separation module: separates the high-frequency and low-frequency features of the denoising features, and assigns different weights to the two for weighted fusion.

[0147] Decoder: The denoised features are input into the video generation decoder to generate the final transition video frame.

[0148] Output module: used to store or play the generated transition video in real time to ensure that users have a high-quality visual experience.

[0149] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a transition video, characterized in that: The transition video generation method comprises the following steps: S100: Inputting the first frame and the last frame used to generate the transition video into a variational autoencoder to extract image features, and then inserting a preset number of frames of noise between the extracted feature frames to generate a positive order and a reverse order noise feature sequence respectively; S200: describing the first and last two frames of images used to generate the transition video respectively, extracting corresponding text features, and then using spherical linear interpolation to obtain text description features corresponding to the number of frames; S300: The noise features in the positive order and the reverse order obtained in S100 are combined with the text description features obtained in S200 and input into the pre-trained text and image guided video generation diffusion model for denoising, and the denoised intermediate features are extracted; Step S300 includes: S310: Fusing the positive order and reverse order features with the corresponding intermediate text features respectively, using a cross attention mechanism or a feature accumulation and superposition form, to obtain positive order noise features and reverse order noise features containing text information; S320: Input the obtained positive sequence noise features and reverse sequence noise features into a video denoising model; S330: using the denoising features extracted from the video denoising model as intermediate features; S400: for the intermediate features of the positive sequence and the reverse sequence extracted in S300, a Gaussian process is used to establish timing constraints for the noise features of the positive sequence and the reverse sequence respectively; S500: Separate the high frequency and low frequency of the positive and reverse order Gaussian constrained noise features obtained in S400, assign different weights to the two, add them together, and interpolate to restore to the original feature size; S600: weighted fusion of the weighted fused forward and reverse feature frames in frame order, and repeat S300 to S500 until the denoising feature reaches a convergence state; S700: Inputting the feature frame that has been denoised and has a stable noise level into a video generation decoder to generate a transition video.

2. The method for generating a transition video according to claim 1, wherein: Step S100 includes: S110: using a variational autoencoder to respectively extract first frame and last frame image features for generating transition content; S120: Generate a specified number of random Gaussian noise frames, where the size of each frame of Gaussian noise is the same as the size of the input first and last frame features; S130: inserting the generated Gaussian noise with the first frame and the last frame as the positive order, and then swapping the order of the first and last frames to insert the Gaussian noise, to obtain the positive order noise feature and the reverse order noise feature.

3. The method for generating a transition video according to claim 1, wherein: Step S200 includes: S210: Analyze the input first frame and last frame images, and use a text description generation model or manually generate respective text descriptions; S220: Process the generated text description using a text feature extractor to extract its corresponding text feature vector; S230: uniformly generating a set of weight values ​​according to a preset number of frames, representing a transition from a first frame to a last frame, wherein the weight value changes from 0 to 1, the weight of the first frame gradually decreases, and the weight of the last frame gradually increases; S240: By setting the generated weights, spherical linear interpolation is performed on the text features of the first and last frames to obtain a number of intermediate text features corresponding to a preset number of frames.

4. The method for generating a transition video according to claim 1, wherein: Step S400 includes: S410: Taking the first frame of the positive sequence intermediate feature as the training input and the last frame as the target output, a positive sequence Gaussian process model is constructed; S420: Using the first frame of the reverse intermediate feature as training input and the last frame as target output, to construct a reverse Gaussian process model; S430: Select a covariance function, calculate the correlation between the input and output of the positive order Gaussian process model and the reverse order Gaussian process model and the corresponding intermediate frame noise features, and construct a joint Gaussian distribution; S440: Calculate the prediction means corresponding to the forward and reverse intermediate frames respectively according to the joint Gaussian distribution; S450: Weight the predicted means corresponding to the forward and reverse intermediate frames respectively and add them to the corresponding original intermediate frames.

5. The method for generating a transition video according to claim 1, wherein: Step S500 includes: S510: Use maximum pooling processing on the positive order and reverse order Gaussian constrained noise features respectively to capture high-frequency information in the features; S520: using mean pooling processing for the positive order and reverse order Gaussian constrained noise features respectively to capture low-frequency information in the features; S530: Assign weights to the high-frequency features extracted in step S510 , assign weights to the low-frequency features extracted in step S520 ,in The coefficient for controlling the balance of high and low frequency characteristics, with a value range between 0 and 1; S540: adding the weighted positive-sequence high-frequency features to the low-frequency features to generate fused positive-sequence features; at the same time, adding the weighted reverse-sequence high-frequency features to the low-frequency features to generate fused reverse-sequence features; S550: interpolating the positive-order features and the negative-order features obtained in step S540 to restore them to the same size as the original Gaussian constrained noise features.

6. The method for generating a transition video according to claim 1, wherein: Step S600 includes: S610: Add the first frame of the weighted fusion of the forward sequence features and the last frame of the reverse sequence features, and add the second frame of the forward sequence and the second to last frame of the reverse sequence, until all the forward sequence features and the reverse sequence features are added; and in the process of feature addition, the forward sequence features are given weights , weights are assigned to features in reverse order ,in The value range of is 0 to 0.5; S620: Input the fused features into the pre-trained diffusion model frame by frame, use the self-attention mechanism to analyze the relationship between the features and perform feature reorganization to remove noise; S630: In the diffusion model, the self-attention mechanism is used to process the features layer by layer. The weight of the features is adjusted in each layer according to the context information, so that the information of interest is enhanced and the noise is suppressed, thereby extracting new intermediate features and completing the intermediate feature update; S640: Repeat steps S300 to S500 to repeatedly perform Gaussian process constraint features, then perform high and low frequency decomposition and feature fusion, and then perform denoising, and continuously optimize the intermediate features; and evaluate the noise level of the feature after each processing, and compare it with the preset low noise standard until the denoising feature reaches a convergence state.

7. The method for generating a transition video according to claim 1, wherein: Step S700 includes: S710: The decoder receives the denoised feature input and processes it through a neural network; each layer of the neural network can perform feature mapping based on the spatial and temporal information of the feature to extract various details of the video frame, including texture, color, and lighting; S720: During the decoding process, the decoder uses an upsampling method to gradually restore the spatial resolution of the video frame, thereby generating each video frame image; S730: After the generated transition video frames are batch processed, image enhancement processing is performed, including contrast adjustment, sharpening processing, and color correction, so as to improve image quality; S740: After all video frames are generated and processed, the video frames are combined into a complete transition video in a correct order.

8. A transition video generation system, characterized in that: The transition video generation system uses the transition video generation method according to any one of claims 1 to 7 to perform video processing, and the transition video generation system comprises: Input module: used to receive the first and last two frames of images and their corresponding text descriptions; Feature extraction unit: A variational autoencoder is used to extract the features of the input image, and a text feature extractor is used to generate the corresponding text feature vector; Interpolation module: Based on the extracted text features, several intermediate text features are generated through spherical linear interpolation method to achieve smooth transition; Noise generation unit: Generates Gaussian noise of the same size as the feature and inserts it between the potential features in positive and negative order; Fusion module: combines noise features with intermediate text features to form new features and input them into the diffusion model; Denoising unit: Input the fused features into the pre-trained video generation diffusion model, extract the denoised intermediate features, and apply Gaussian process constraints to ensure temporal consistency; High-frequency and low-frequency separation module: separates the high-frequency and low-frequency features of the denoising features, and assigns different weights to the two for weighted fusion; Decoder: Input the denoised features into the video generation decoder to generate the final transition video frame; Output module: used to store or play the generated transition video in real time to ensure that users have a high-quality visual experience.

Images