Video super-division method and system based on single-step diffusion acceleration, terminal and medium

Abstract

The invention provides a video super-division method and system based on single-step diffusion acceleration, and the method comprises the steps: carrying out the size amplification processing of a to-be-processed low-resolution video, and then mapping the amplified video to a hidden space, and obtaining an input video; performing video definition improvement processing on the input video by using a video processing model based on single-step diffusion to obtain a high-resolution restored video; and converting the restored video back to a video space to complete video super-division processing. According to the method, a brand-new single-step diffusion acceleration video super-resolution model is constructed based on a video generation basic model, so that the reasoning efficiency of a video super-resolution technology can be greatly improved, and video super-resolution restoration with higher quality can be realized. According to the method, the applicability of video super-division in a practical application scene is remarkably enhanced, and when a real video super-division task is processed, a relatively high reasoning speed can be maintained, and feasible support is provided for deployment of a video super-division technology on terminal equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of video super-resolution, and in particular to a video super-resolution method, system, terminal and medium based on single-step diffusion acceleration. Background Art

[0002] In the field of artificial intelligence research, video super-resolution is a key research direction, and numerous technical solutions have been explored in this area. Some solutions use traditional convolutional neural networks and recurrent neural networks to handle video super-resolution. However, the degradation patterns of their training datasets are mostly simple synthetic degradation, which differs from the degradation in real scenes. This leads to insufficient performance of the models when processing real videos in real life. Other methods, although trained on real datasets, are limited by the model architecture and the details of the restored videos are less than ideal. With the development of diffusion models, many diffusion models that contain rich prior knowledge have been applied to the field of video super-resolution. Although these methods perform well in detail restoration, they generally use a multi-step denoising process and require the introduction of a large number of additional modules to maintain the temporal coherence of the generated videos. Therefore, they still have major problems in terms of training efficiency and inference time.

[0003] Currently, no description or report of similar technology to the present invention has been found, and similar information at home and abroad has not been collected. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a video super-resolution method, system, terminal and medium based on single-step diffusion acceleration.

[0005] According to one aspect of the present invention, a video super-resolution method based on single-step diffusion acceleration is provided, comprising:

[0006] The low-resolution video to be processed is enlarged, and then the enlarged video is mapped to the latent space to obtain the input video;

[0007] Performing video definition enhancement processing on the input video using a video processing model based on single-step diffusion to obtain a high-resolution restored video;

[0008] The restored video is converted back to the video space to complete the video super-resolution processing.

[0009] According to another aspect of the present invention, a video super-resolution system based on single-step diffusion acceleration is provided, comprising:

[0010] The input module is used to upscale the low-resolution video to be processed and then map the upscaled video to the latent space to obtain the input video;

[0011] a processing module, which uses a video processing model based on single-step diffusion to perform video definition enhancement processing on the input video to obtain a high-resolution restored video;

[0012] The output module is used to convert the restored video back into the video space to complete the video super-resolution processing.

[0013] According to a third aspect of the present invention, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the terminal can be used to execute the method described above in the present invention, or to execute the system described above in the present invention.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can be used to execute the method described above in the present invention, or to run the system described above in the present invention.

[0015] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0016] The video super-resolution method, system, terminal and medium based on single-step diffusion acceleration provided by the present invention adopt single-step diffusion acceleration technology to solve the technical problem of slow inference speed of current multi-step diffusion video super-resolution methods, and greatly improve the inference speed of video super-resolution.

[0017] The video super-resolution method, system, terminal and medium based on single-step diffusion acceleration provided by the present invention adopt retraining technology based on pre-trained diffusion model to solve the technical problem that existing methods are insufficient in the ability to restore video details, achieve the technical goal of restoring rich video details, better restore degraded real video details, and ensure their authenticity.

[0018] The video super-resolution method, system, terminal and medium based on single-step diffusion acceleration provided by the present invention adopt a single-step diffusion video super-resolution model to quickly and accurately complete the purpose of video super-resolution, overcoming the problems of slow inference speed and poor restoration effect in the prior art.

[0019] The video super-resolution method, system, terminal and medium based on single-step diffusion acceleration provided by the present invention implement video super-resolution technology by using a more powerful basic model and applying model priors. This can not only greatly improve the inference efficiency of video super-resolution technology, but also achieve higher-quality video super-resolution restoration, significantly enhancing the applicability of video super-resolution in actual application scenarios. When processing real video super-resolution tasks, it can maintain a high inference speed and provide feasibility support for the deployment of video super-resolution technology on terminal devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0021] Figure 1 This is a workflow diagram of a video super-resolution method based on single-step diffusion acceleration in a preferred embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the components of a video super-resolution system based on single-step diffusion acceleration in a preferred embodiment of the present invention.

[0023] Figure 3 FIG. 1 is a working architecture diagram of a video super-resolution system based on single-step diffusion acceleration in a preferred embodiment of the present invention.

[0024] Figure 4 (a) to (f) are implementation effect diagrams of a video super-resolution method and system based on single-step diffusion acceleration in a specific application example of the present invention compared with other existing technologies. DETAILED DESCRIPTION

[0025] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention.

[0026] In response to the problems of slow inference speed and poor restoration effect in the existing technology, an embodiment of the present invention provides a video super-resolution method based on single-step diffusion acceleration. This method is based on the basic model of video generation and constructs a new single-step diffusion accelerated video super-resolution model to replace the traditional multi-step diffusion video super-resolution model. Through the above improvements, not only the inference efficiency of the video super-resolution technology can be greatly improved, but also higher-quality video super-resolution restoration can be achieved. This method significantly enhances the applicability of video super-resolution in actual application scenarios. When processing real video super-resolution tasks, it can not only maintain a high inference speed, but also provide feasibility support for the deployment of video super-resolution technology on terminal devices. Compared with multi-step diffusion that uses Gaussian noise map as the diffusion starting point, the present invention adopts a single-step diffusion model, and takes the result of a given low-resolution video after bilinear interpolation amplification and encoder processing as the diffusion starting point. The simulation formula (in Represents the cumulative scaling factor obtained by multiplying the single-step scaling factor, α s represents the single-step scaling factor, α s =1-β s , represents the complementary relationship between the single-step scaling coefficient and the single-step noise coefficient in any step, β srepresents the single-step noise coefficient) t In the formula, ∈ is Gaussian noise, and z is a "clean" potential sample, which corresponds to the latent space representation of the high-resolution video after denoising. The super-resolution task is actually the inverse process of the above process. The model needs to learn "how to restore the unclear image to a clear result" to obtain z. In view of the slow processing problem of multi-step diffusion, the present invention innovatively completes the diffusion process directly in a single step, realizing the transition from low-quality data with a small amount of effective information to high-quality data: Then, the Transformer architecture is used to predict v θ , and directly get the high-definition sample z sr The formula is expressed as The meaning of each parameter is: lr is the potential representation of low-resolution data (features of low-resolution video before encoding); z sr It is a high-quality potential representation after denoising (subsequent decoding can obtain super-resolution results); v θ is the denoising model (learning “noise prediction methods” with the help of Transformer); c is the additional condition (semantic information); t is the optional number of denoising steps.

[0027] Specifically, if Figure 1 As shown, the video super-resolution method based on single-step diffusion acceleration provided by this embodiment may include:

[0028] S1, enlarges the size of the low-resolution video to be processed, and then maps the enlarged video to the latent space to obtain the input video;

[0029] S2, using a video processing model based on single-step diffusion to improve the video clarity of the input video to obtain a high-resolution restored video;

[0030] S3 converts the restored video back to the video space and completes the video super-resolution processing.

[0031] In some preferred implementations, the above S1, the low-resolution video to be processed, may further include: a degraded synthetic video and a low-resolution real video.

[0032] In some preferred embodiments, the above S1, which upscales the low-resolution video to be processed and then maps the upscaled video to the latent space, may further include:

[0033] The video size is enlarged using a bilinear interpolation algorithm, and then the enlarged video is mapped to the latent space. Further preferably:

[0034] The bilinear interpolation algorithm is used to enlarge the video size, which can further include: for the video sequence, the bilinear interpolation algorithm is used frame by frame: the original image (size W×H) is enlarged by the scaling factor s w =W′ / W and s h =H′ / H is mapped to the target size W′×H′. For the target pixel coordinate (x′, y′), calculate its floating-point coordinate (x, y) in the original image = (x′W / W′, y′H / H′) and take the integer part And the decimal part a=xi、b=yj, calculate the bilinear interpolation I′(x′,y′)=(1-a)(1-b)I(i,j)+a(1-b)I(i+1,j)+(1-a)bI(i,j+1)+abI(i+1,j+1) through the neighborhood pixels (i,j), (i+1,j), (i,j+1), (i+1,j+1), and achieve four times magnification. Among them, I′ is the pixel value of the target image after magnification, and I is the pixel value of the original image;

[0035] Mapping the enlarged video to the latent space may further include:

[0036] This process is completed using the mapping of the 3D variational encoder E, which can be expressed as: lr =E(x lr ), where x lr represents a low-resolution video sequence, z lr represents the low-dimensional latent space representation.

[0037] Furthermore, the 3D variational encoder E preferably adopts a hierarchical architecture: first, 3D convolutional layers are used to extract spatiotemporal features while maintaining the spatial continuity of the feature maps. Subsequently, 3D downsampling is used to gradually reduce the spatial resolution, forming a pyramidal feature representation. A residual block structure is inserted between each convolution layer to alleviate the vanishing gradient problem and retain key information from the original input. This step significantly reduces the data dimension and improves processing efficiency.

[0038] In some preferred embodiments, the above S2, performing video definition enhancement processing on the input video using a video processing model based on a single-step diffusion model, may further include:

[0039] S21, providing a Transformer-based single-step diffusion model, pre-training the Transformer-based single-step diffusion model to obtain a video processing model. Further preferably, the method includes:

[0040] This is the core innovation of the entire single-step diffusion acceleration method, which performs targeted fine-tuning based on the pre-trained CogVideoX model parameters. Further preferably, it includes:

[0041] First, we load the 3D causal VAE (including encoder and decoder) and Transformer denoiser weights pre-trained on a large-scale video dataset to establish the basic architecture of the video processing model. We then use transfer learning to preserve the model's basic representation capabilities for video spatiotemporal features. The transfer learning process includes:

[0042] In order to avoid destroying the visual structure prior obtained by pre-training, the VAE decoder parameters are strategically frozen, and only the Transformer denoiser is adaptively optimized. The fine-tuning process adopts mixed training of video and image datasets, and minimizes the feature difference between LR (low-resolution video) and HR (high-resolution video) in the latent space through a two-stage strategy: in the first stage, the low-resolution video x is converted into the HR image by the encoder in the latent space. lr With high-resolution reference video x hr Mapped to z lr With z hr , with MSE loss Constraining the Transformer denoiser to learn z lr →z sr The model then uses a direct mapping of the image to the LR image, and then combines the MSE loss, DISTS perceptual loss, and frame difference loss in the pixel space to optimize detail and temporal consistency. Ultimately, the model achieves efficient super-resolution from LR video to HR video in a single-step inference, and the final video processing model is trained. This model is 28 times faster than traditional multi-step methods while maintaining restoration quality. sr Latent space representation of denoised high-resolution video.

[0043] S22, inputting the input video into the video processing model, restoring the representation of the input video in the latent space, and completing the task of improving the video clarity in the latent space. Further preferably, it includes:

[0044] The input video after quadrupling by bilinear interpolation is input into the fine-tuned video processing model. The model is based on the 3D causal VAE encoder of CogVideoX with a hierarchical architecture. As shown above, the input video is mapped from pixel space to latent space to obtain the potential representation z lr . Then z lr As the diffusion starting point, the simulation formula (in α t =1-β s ) t , and use the pre-trained Transformer denoiser v θ Perform a single-step denoising operation, the formula is as follows: This process will lr Convert to zsr This high-quality potential representation after denoising improves the video clarity in the latent space and lays the foundation for subsequent decoding to obtain high-definition video. represents the cumulative scaling factor, α s represents the single-step scaling factor; z t represents the latent space representation of the tth step in the diffusion process, and z represents the latent space sample after enhancement and clarity, i.e. z sr ;∈ represents Gaussian noise; c represents additional conditions; t represents the number of denoising steps.

[0045] In the above steps, transfer learning is used to retain the basic representation capabilities of the video processing model for spatiotemporal features. The basic architecture of the video processing model is to integrate the pre-trained 3D causal VAE responsible for encoding and decoding with the Transformer denoiser responsible for optimizing the latent representation to form an end-to-end video processing pipeline: the input video is encoded into a latent space representation using the 3D causal VAE, and then the latent representation output by the VAE is denoised and optimized using the Transformer denoiser to improve feature purity. Finally, the video is decoded and restored to obtain a clear video result. When considering the issue of initializing model parameters, transfer learning is used to impart the basic representation capabilities of spatiotemporal features of videos learned by the pre-trained model on large-scale data to the video processing model. During transfer, the parameters of some pre-trained layers of the 3D causal VAE are frozen, and only the Transformer denoiser is fine-tuned, so that the model retains these common features and avoids the loss of basic capabilities due to retraining.

[0046] In some preferred embodiments, the above S22 may further include:

[0047] The number of denoising steps is set to 399. This setting fully utilizes the denoising characteristics of the single-step diffusion model to complete the task of improving video clarity in the latent space. Further preferred:

[0048] Unlike the method that sets the number of diffusion steps to 999, this step innovatively selects a smaller t value. This is based on the observation that the early diffusion steps focus on global structure, while the later steps refine details. Since low-resolution (LR) videos already contain sufficient structural information, using low-resolution videos as a starting point only requires the later refinement of details. Using the formula: Directly predict noise and refine details, where the time step t=399 is empirically set to balance global structure and detail restoration.

[0049] In some preferred embodiments, the above S3, converting the restored video back to the video space, may further include:

[0050] Based on the decoder, the restored video representation in the latent space is decoded and the restored video is converted back to the video space. Further preferably, it includes:

[0051] Based on the video processing model, the 3D causal VAE decoder based on CogVideoX represents the restored video in the latent space z sr During decoding, the CogVideoX pre-trained 3D causal VAE decoder architecture is used. This architecture gradually restores low-dimensional latent features to high-resolution pixel values ​​through cascaded 3D deconvolution modules and sub-pixel upsampling operations.

[0052] Furthermore, the 3D causal VAE decoder based on CogVideoX is preferably implemented with a residual block after each level of deconvolution to preserve detail information. At the same time, a channel attention mechanism is introduced in the spatiotemporal dimension, and a weight matrix is ​​generated by global average pooling to achieve adaptive enhancement of key feature channels. Through multi-layer deconvolution and upsampling operations, low-dimensional latent features are gradually restored to high-resolution pixel values. During the decoding process, the temporal consistency between consecutive frames is ensured, and the latent space representation z is finally converted to the original image. sr Convert back to video space to generate detailed and temporally coherent HD restored video Complete the precise mapping from latent space to pixel space.

[0053] Based on the same inventive concept, an embodiment of the present invention further provides a video super-resolution system based on single-step diffusion acceleration.

[0054] Specifically, if Figure 2 As shown, the video super-resolution system based on single-step diffusion acceleration provided by this embodiment may include:

[0055] The input module is used to upscale the low-resolution video to be processed and then map the upscaled video to the latent space to obtain the input video;

[0056] A processing module, which uses a video processing model based on single-step diffusion to improve the video clarity of the input video to obtain a high-resolution restored video;

[0057] The output module is used to convert the restored video back to the video space and complete the video super-resolution processing.

[0058] The specific implementation of each functional module constituting the system provided by the above embodiment of the present invention is further described in detail below in conjunction with a preferred implementation manner.

[0059] The working architecture of the video super-resolution system based on single-step diffusion acceleration provided in the preferred embodiment is as follows: Figure 3 As shown. Among them:

[0060] The input module, primarily composed of an encoder, receives degraded synthetic video or low-resolution real-world video. It first uses a bilinear interpolation algorithm to quadruple the video size, then converts it into a format that the core module of the model can process—mapping it into a latent space, thereby achieving efficient and accurate representation of the video data.

[0061] The processing module, based on the Transformer module, processes the latent space representation of the input video. Unlike other diffusion models, this system uses low-definition video as input rather than noise. Based on experience, the number of steps is set to 399, leveraging the denoising properties of the single-step diffusion model to improve video clarity within the latent space.

[0062] The output module uses a decoder as its core component. It decodes the restored video representation in the latent space and converts it back to the video space, fully realizing the processing flow of quadrupling the low-definition video and improving the clarity.

[0063] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, and will not be elaborated here.

[0064] The technical solution and technical effects provided by the above embodiment of the present invention are further described in detail below with reference to a specific application example.

[0065] like Figure 3 As shown in the specific application example, based on the above technical solution, as long as the input video is input into the video processing model, the video super-resolution process of single-step diffusion acceleration can be completed. Figure 4(a) to (f) show the comparison results of video super-resolution between the technical solution of the present invention and other methods on the commonly used real-world video dataset MVSR4x. The closer the results of the methods are to HR, the better. Among them, (a) HR stands for High-Resolution video; (b) LR stands for Low-Resolution video; (c) Comparison Method 1 is the method in the paper Upscale-a-video: Temporal-consistentdiffusion model for real-world video super-resolution; (d) Comparison Method 2 is the method in the paper Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution; (e) Comparison Method 3 is the method in the paper STAR: Spatial-TemporalAugmentation with Text-to-Video Models for Real-World Video Super-Resolution; (f) Ours is the technical solution of the present invention. In the comparison of image restoration effects, the technical solution of the present invention (this technical solution) shows significant advantages. Compared to other solutions, this technology can more accurately capture architectural details such as building outlines, window structures, and roof decorations when processing low-resolution (LR) images, making them clearly visible in the restored image. This is due to its unique algorithmic architecture, which deeply integrates advanced image feature extraction and reconstruction technologies to effectively reduce blur and distortion, retaining and restoring the image's high-frequency detail information to the greatest extent possible. This achieves high-quality conversion from low-resolution (LR) to high-resolution (HR) images, presenting visual effects that approach or even exceed those of traditional high-resolution images.

[0066] An embodiment of the present invention further provides a computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor can be used to execute any one of the methods described in the foregoing embodiments of the present invention, or to execute any one of the systems described in the foregoing embodiments of the present invention.

[0067] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0068] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0069] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0070] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0071] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0072] The video super-resolution method, system, terminal, and medium based on single-step diffusion acceleration provided by the above-mentioned embodiments of the present invention use single-step diffusion acceleration technology to solve the technical problem of slow inference speed of current multi-step diffusion video super-resolution methods, greatly improving the inference speed of video super-resolution. The retraining technology based on pre-trained diffusion models is used to solve the technical problem of insufficient video detail restoration ability of existing methods, achieve the technical goal of restoring rich video details, better restore degraded real video details, and ensure their authenticity. The video super-resolution model using single-step diffusion can quickly and accurately complete the purpose of video super-resolution, overcoming the problems of slow inference speed and poor restoration effect in existing technologies. By using a more powerful basic model and applying model priors to implement video super-resolution technology, it can not only significantly improve the inference efficiency of video super-resolution technology, but also achieve higher-quality video super-resolution restoration, significantly enhancing the applicability of video super-resolution in actual application scenarios. When processing real video super-resolution tasks, it can maintain a high inference speed and provide feasibility support for the deployment of video super-resolution technology on terminal devices.

[0073] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0074] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A video super-resolution method based on single-step diffusion acceleration, characterized in that: include: The low-resolution video to be processed is enlarged, and then the enlarged video is mapped to the latent space to obtain the input video; Performing video definition enhancement processing on the input video using a video processing model based on single-step diffusion to obtain a high-resolution restored video; The restored video is converted back to the video space to complete the video super-resolution processing.

2. The video super-resolution method based on single-step diffusion acceleration according to claim 1, characterized in that: The low-resolution video to be processed includes: a degraded synthetic video and a low-resolution real video.

3. The video super-resolution method based on single-step diffusion acceleration according to claim 1, characterized in that: The step of enlarging the size of the low-resolution video to be processed includes: For video sequences, bilinear interpolation algorithm is used to perform video magnification processing frame by frame, including: The original image of size W×H is scaled by the factor s w =W′ / W and s h =H′ / H mapped to the target size W′×H′, calculate the floating-point coordinates (x, y) of the target pixel coordinates (x′, y′) in the original image = (x′W / W′, y′H / H′), and take the integer part And the decimal part a=xi、b=yj, calculate the bilinear interpolation I′(x′,y′)=(1-a)(1-b)I(i,j)+a(1-b)I(i+1,j)+(1-a)bI(i,j+1)+abI(i+1,j+1) through the neighborhood pixels (i,j), (i+1,j), (i,j+1), (i+1,j+1), and enlarge the size of the original image; where I′ is the pixel value of the enlarged target image, and I is the pixel value of the original image; Mapping the enlarged video to the latent space includes: The 3D variational encoder E is used to map the enlarged video to the latent space, which is expressed as: z lr =E(x rr ) Where x lr is a low-resolution video sequence, z lr is a low-dimensional latent space representation.

4. The video super-resolution method based on single-step diffusion acceleration according to claim 1, characterized in that: The step of performing video definition enhancement processing on the input video by using a video processing model based on single-step diffusion includes: A single-step diffusion model based on a Transformer architecture is provided, and the single-step diffusion model is pre-trained to obtain a video processing model, including: Obtain the CogVideoX-based 3D causal VAE and Transformer denoiser weights pre-trained on the video dataset to establish the basic architecture of the video processing model; Freeze the VAE decoder parameters, use the video image dataset to perform hybrid training for the Transformer denoiser, and perform adaptive fine-tuning optimization on the Transformer denoiser; through a two-stage strategy, first minimize the feature difference between low-resolution and high-resolution videos in the latent space, and then combine the MSE loss, DISTS perceptual loss and frame difference loss in the pixel space to optimize the details and time consistency, and finally enable the model to achieve super-resolution from low-resolution to high-resolution videos in a single-step reasoning, and train the final video processing model; wherein: the two-stage strategy includes: in the first stage, in the latent space, the encoder converts the low-resolution video x lr With high-resolution reference video x hr Mapped to z lr With z hr , the second stage uses MSE loss Constraining the Transformer denoiser to learn z lr →z sr Direct mapping of z sr Latent space representation for denoised high-resolution video; Inputting the input video into the video processing model, restoring the representation of the input video in the latent space, and completing the video definition improvement task in the latent space, including: Input the input video into the video processing model, and use the video processing model to map the input video from pixel space to latent space based on CogVideoX’s 3D causal VAE encoder to obtain the latent space representation z lr ; Then the latent space representation z lr As the diffusion starting point, the simulation formula z in t , and use the pre-trained Transformer denoiser v θ Perform a single-step denoising operation, expressed as: Represent the latent space z lr Converted to the latent space representation z of the denoised high-resolution video sr , that is, to obtain the representation of the high-resolution restored video and improve the video clarity in the latent space; is the cumulative scaling factor, which indicates the retention ratio of the original signal during the process of adding noise in the diffusion process. α s represents the single-step scaling factor; z t represents the potential space representation of the tth step in the diffusion process, and z represents the potential space sample after the enhancement, that is, corresponding to z sr ;∈ represents Gaussian noise; c represents additional conditions; t represents the number of denoising steps.

5. The video super-resolution method based on single-step diffusion acceleration according to claim 4, characterized in that: Also includes: The number of denoising steps is set to 399.

6. The video super-resolution method based on single-step diffusion acceleration according to claim 1, characterized in that: Converting the restored video back to the video space includes: Based on the video processing model, the 3D causal VAE decoder based on CogVideoX is used to represent the restored video in the latent space. sr Decoding is performed to gradually restore the low-dimensional latent features to high-resolution pixel values, completing the precise mapping from latent space to pixel space, and converting the restored video back to video space.

7. A video super-resolution system based on single-step diffusion acceleration, characterized in that: include: The input module is used to upscale the low-resolution video to be processed and then map the upscaled video to the latent space to obtain the input video; a processing module, which uses a video processing model based on single-step diffusion to perform video definition enhancement processing on the input video to obtain a high-resolution restored video; The output module is used to convert the restored video back into the video space to complete the video super-resolution processing.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the computer program, the processor can be used to perform the method according to any one of claims 1 to 6, or run the system according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 6, or to run the system according to claim 7.

Images