An arbitrary scale video super-resolution method based on diffusion model

Abstract

The application discloses an arbitrary scale video super-resolution method based on a diffusion model and belongs to the technical field. The arbitrary scale super-resolution method of the application can effectively solve the deficiency of the existing video super-resolution method in the aspect of arbitrary scale through space-time feature fusion and arbitrary scale implicit neural up-sampling, can not only solve the integer multiple and single multiple limitation of the existing method, can generate high-resolution video sequences of arbitrary size, can also guarantee the consistency between the front and back of the video sequences, enriches the application scene of the video super-resolution, and expands the technical boundary of the field.

Description

Technical Field

[0001] This invention relates to the field of video super-resolution technology, and more specifically to an arbitrary-scale video super-resolution method based on a diffusion model. Background Technology

[0002] In recent years, diffusion models, as an emerging generative model, have achieved significant breakthroughs in image generation and inpainting tasks. The core idea of ​​diffusion models is to gradually transform data from a structured distribution into noise, and then recover the original data through a reverse process. This approach has a natural advantage in image super-resolution because it can finely model the detail restoration process while maintaining the global consistency of the image.

[0003] In video super-resolution tasks, the potential of diffusion models is increasingly recognized. However, the randomness of diffusion models makes it impossible to control the continuity between video frames when applied directly. Upscale-A-Video inserts a temporal layer into a pre-trained image diffusion model to enhance the modeling of dependencies between video frames and optimize local consistency within short segments. It introduces spatiotemporal 3D residual blocks and a Spatial Feature Transform (SFT) layer into the VAE decoder to reduce flicker artifacts and color shifts during decoding. Furthermore, it employs stream-guided cyclic latent code propagation, requiring no additional training. It achieves forward and backward propagation and fusion of latent space features through optical flow estimation (such as RAFT), selecting only regions with small optical flow consistency errors for propagation, thereby enhancing the overall coherence of long videos. By fine-tuning the U-Net network and VAEDecoder, an additional temporal layer is introduced into the U-Net network to achieve local consistency constraints within video segments, and an additional spatiotemporal 3D residual block is introduced into the VAEDecoder to enhance low-level consistency. Additionally, a training-free stream-guided cyclic propagation module is introduced into the latent space to ensure global spatiotemporal consistency of long video sequences. SATeCo proposes a novel framework that calibrates the denoising and reconstruction process of high-resolution videos by learning the spatial-temporal features of low-resolution videos. Its core modules include a Spatial Feature Adaptation (SFA) module and a Temporal Feature Alignment (TFA) module. SFA adaptively estimates the affine parameters of each pixel to adjust frame features, ensuring pixel-level guidance for high-resolution frame synthesis. TFA utilizes self-attention and cross-attention mechanisms to perform feature interactions within a 3D local window (tubelet) and aligns it with its low-resolution counterpart to maintain temporal consistency between frames. SATeCo freezes all parameters of the pre-trained UNet and VAE during training, optimizing only the SFA and TFA modules, thereby reducing computational cost and improving efficiency. The Spatial Adaptation module in the diffusion model learns the spatial adaptability between high-resolution video frames and their corresponding low-resolution video frames, resulting in high-resolution videos with rich detail that matches the corresponding content of the input low-resolution video. Following the Spatial Adaptation module, the Temporal Alignment module further learns the temporal coherence between high-resolution video frames, ensuring consistency in image detail across different frames of the generated high-resolution video.

[0004] The methods described above have achieved good performance, but they still have some shortcomings in arbitrary-scale super-resolution:

[0005] (1) Difficulty adapting to arbitrary magnification: Current video super-resolution technologies are typically trained and optimized at fixed magnifications (e.g., ×2, ×4), but the generated results are difficult to maintain consistently at any scale. In particular, when the magnification does not match the training magnification, the generated video often appears blurry, loses details, or has image distortion, making it difficult to achieve high-quality dynamic enhancement effects. This dependence on a fixed magnification limits the flexibility of super-resolution models, making it difficult for existing technologies to meet the video magnification requirements of different scenarios.

[0006] (2) High-frequency detail loss and artifact issues: In video super-resolution upscaling, especially at high magnification, existing methods are insufficient in restoring high-frequency details, easily leading to detail loss or artifacts, resulting in blurred or unrealistic images. In multi-scale changing scenes, super-resolution models struggle to accurately restore minute details and complex textures in videos, thus affecting the overall visual effect and realism of the video.

[0007] The above problems urgently need to be solved. To address this, an arbitrary-scale video super-resolution method based on a diffusion model is proposed. Summary of the Invention

[0008] The technical problem to be solved by this invention is: how to solve the problems of difficulty in adapting to arbitrary magnification, loss of high-frequency details and artifacts in the existing technology, and to provide an arbitrary scale video super-resolution method based on a diffusion model.

[0009] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0010] S1: Low-resolution video frame input

[0011] Obtain the input low-resolution video sequence L and the preset magnification factor s or target resolution;

[0012] S2: Interpolation and Optical Flow Information Calculation

[0013] Interpolation is performed on the low-resolution video sequence L to bring it up to the target resolution, resulting in a coarse high-resolution video frame sequence. An optical flow network of pretrained optical flow is used to calculate the optical flow information between adjacent frames and generate an optical flow map O.

[0014] S3: Latent Feature Representation Generation

[0015] The coarse high-resolution video frames are encoded to generate latent feature representations z;

[0016] S4: Initial Input Generation

[0017] Based on the encoded latent feature representation z, generate a noise of the same size, and initialize an initial input x based on the latent feature representation z and the noise. T The latent feature representation z is fed into a guided conditional network for processing to generate conditional inputs; then the conditional inputs and the initial input x are combined. T They are fed together into the U-Net network of the diffusion model;

[0018] S5: Spatiotemporal Information Fusion

[0019] In the encoder part of the U-Net network, the input features in step S4 are processed according to the original structure to extract multi-scale noisy features; in the decoder part of the U-Net network, the multi-scale noisy features are processed by the residual block and the Transformer block and then sent to the implicit neural upsampling module, where spatiotemporal information fusion processing is performed.

[0020] S6: Implicit Neural Upsampling Processing

[0021] Based on the multi-scale noisy feature sizes of the encoder part of the U-Net network, a coordinate grid (coord) of corresponding size is generated. The size of both the spatiotemporally fused features and the coordinate grid (coord) is scaled to the range of -1 to 1. For each point in the coordinate grid, the spatiotemporally fused feature map (f) is located. new The corresponding position is selected and the feature value is queried. Then, the image is scaled back to its original size, and the resulting feature map f is displayed. query The output f is obtained by feeding the data into a hidden neural function layer parameterized by a multilayer perceptron. out Then, it is connected to the multi-scale noisy features output by the encoder and fed into the next-level module of the decoder part of the U-Net network;

[0022] S7: Optical Flow Alignment Processing

[0023] After the U-Net network finishes processing, its output features are sent to the optical flow alignment module. Using the optical flow map calculated in step S2, the features are curled to align the features between adjacent frames.

[0024] S8: Generate and output high-resolution video frames

[0025] The features after optical flow alignment are subjected to a T-iteration denoising process to gradually eliminate noise and restore high-resolution video frame details. The feature f obtained after the denoising process is completed is... denoise The diffuse model's decoder decodes the video into a high-resolution video sequence H.

[0026] Furthermore, in step S2, a coarse high-resolution video frame sequence The calculation formula is as follows:

[0027]

[0028] In this context, "↑" indicates interpolation.

[0029] Furthermore, in step S2, the formula for calculating the optical flow map is as follows:

[0030]

[0031] Furthermore, in step S3, the specific processing procedure is as follows:

[0032] The features of each frame are mapped into the latent space through a pre-trained encoder ε to generate a latent feature representation z:

[0033]

[0034] Furthermore, in step S4, the initial input x T The formula for generating it is as follows:

[0035]

[0036] in, These are learnable parameters that can be optimized during training.

[0037] Furthermore, in step S5, the spatiotemporal information fusion processing procedure is as follows:

[0038] S51: The input features are first expanded according to the frame channels, and each frame is processed in a loop;

[0039] S52: For each frame's feature f i The features f are obtained by concatenating the features of the adjacent frames along the channel dimension. c(i) ;

[0040] S53: Through channel compression, the spliced ​​features are compressed back to their original channel size, thereby obtaining a new feature f that fuses spatiotemporal information. new(i) Thus, the feature map f is obtained. new .

[0041] Furthermore, in step S52, feature f c(i) The formula for generating it is as follows:

[0042] f c(i) =concate(f i-1 ,f i ,f i+1 );

[0043] Among them, fi-1 f i f i+1 These represent the features of the target frame and the features of its preceding and following frames, respectively. "concate" indicates connection by channel dimension.

[0044] Furthermore, in step S53, feature f new(i) The formula for generating it is as follows:

[0045] f new(i) =Conv(f c(i) );

[0046] Here, Conv represents the convolutional layer processing operation.

[0047] Furthermore, in step S6, the feature map f query And output f out The formula for generating it is as follows:

[0048] f query =Grid_Sample(f new (coord);

[0049] f out =INF(f query );

[0050] Where Coord represents the generated two-dimensional coordinate grid of the target size, and Grid_Sample represents the search for f at each coordinate in the two-dimensional coordinate grid. new The feature value at the corresponding position in the diagram, INF represents the implicit neural upsampling module formed by the implicit neural function layer parameterized by the multilayer perceptron, which processes features through the multilayer perceptron.

[0051] Furthermore, in step S8, the formula for generating the high-resolution video sequence H is as follows:

[0052] H = D(f) denoise );

[0053] Where D represents the decoder.

[0054] The present invention has the following advantages over the prior art:

[0055] 1. Adaptable to upsampling at any scale

[0056] This invention uses an implicit neural network to replace the traditional convolutional upsampling module, making the system more scale-adaptive. The implicit neural network can achieve upsampling at any scale without relying on convolution at a specific scale through continuous spatial representation. This continuous feature representation method ensures that high-resolution results can be generated smoothly under different resolution requirements.

[0057] The introduction of implicit neural networks enhances the adaptability of the upsampling module, enabling this method to be applied to video super-resolution tasks of any scale without redesigning or adjusting the model structure. This not only increases the model's flexibility but also reduces its dependence on training data at a specific scale. In practical applications, this advantage can meet the needs of video processing tasks with different resolution requirements, making it particularly suitable for application scenarios with varying display requirements on different devices.

[0058] 2. Improved spatiotemporal consistency

[0059] This invention introduces an implicit neural upsampling module based on spatiotemporal information fusion, which utilizes information from adjacent frames while generating the target frame. By leveraging the spatiotemporal fusion mechanism, the motion information of adjacent frames is deeply fused with the target frame. This spatiotemporal fusion mechanism effectively extracts dynamic features between frames, ensuring the continuity and consistency of the generated high-resolution video between frames.

[0060] By deeply fusing spatiotemporal information, this invention reduces frame jumps and blurring during video generation, achieving smooth video transitions. This method is particularly suitable for video processing with fast motion or complex scenes, significantly improving video viewing quality and visual continuity, ensuring no noticeable visual breaks during playback of super-resolution videos. This improvement enhances the user viewing experience in practical applications, and has significant application value, especially in streaming media playback and virtual reality scenarios.

[0061] 3. Efficient diffusion model generation process

[0062] This invention employs a diffusion model for generating high-resolution frames. The diffusion model uses a multi-step reverse generation process to gradually denoise and restore image details at each step. This method is more stable and reliable than the single-step generation method. The multi-step iterative reverse generation process of the diffusion model can effectively improve the accuracy of detail restoration and reduce the occurrence of artifacts.

[0063] The progressive generation process of the diffusion model allows noise in low-resolution input frames to be gradually removed, resulting in high-resolution frames with high detail and low artifacts. This method is more stable in terms of generation results and has significant advantages when processing low-resolution videos containing noise or compression artifacts. Compared with traditional methods, the diffusion model scheme of this invention can significantly improve the overall quality of videos in practical video applications, making the generated videos more refined in detail and suitable for the needs of high-quality video playback and data storage. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the arbitrary-scale video super-resolution method based on a diffusion model in Embodiment 2 of the present invention.

[0065] Figure 2 This is a schematic diagram of the overall architecture of the diffusion model in Embodiment 3 of the present invention;

[0066] Figure 3 These are integer multiples of the results obtained by applying the method of the present invention in Embodiment 3 of the present invention, including 2x super-resolution magnification, 4x super-resolution magnification, and 8x super-resolution magnification.

[0067] Figure 4 These are non-integer multiples of the super-resolution magnification results obtained by applying the method of the present invention in Embodiment 3 of the present invention, including 2.5x super-resolution magnification results, 3.2x super-resolution magnification results, and 4.1x super-resolution magnification results. Detailed Implementation

[0068] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0069] Example 1

[0070] This embodiment provides a technical solution: an arbitrary-scale video super-resolution method based on a diffusion model, the details of which are as follows:

[0071] The objective of arbitrary-scale video super-resolution

[0072] Video super-resolution aims to generate corresponding high-resolution video sequences from input low-resolution video sequences. Many existing video super-resolution methods exist, such as Transformer-based methods, convolution-based methods, and diffusion model-based methods. However, most of these methods are limited to fixed-magnification magnification, which greatly restricts their application in real-world scenarios. In this invention, we propose a diffusion model for arbitrary-scale video super-resolution. Specifically, we use an implicit neural upsampling module to process the features of the input video sequence, generating features of different resolutions at different levels in the U-Net decoder, concatenating them with the corresponding features in the encoder, and feeding them into the next level. Spatiotemporal fusion sampling fuses the information of the current frame with the information of its neighboring frames, focusing on information outside the current frame during sampling. This effectively solves the problem of inconsistency between consecutive frames in video super-resolution, achieving arbitrary-scale multiplier super-resolution and generating high-quality videos of arbitrary resolution. This embodiment achieves the above objectives through the following steps:

[0073] Implicit Neural Upsampling: To enable the U-Net network to handle features of arbitrary resolution and facilitate smooth connection between the encoder and decoder layer outputs, we designed an implicit neural upsampling module. During upsampling at each level of the U-Net decoder, we generate a grid of features corresponding to the encoder layer output using a coordinate-based method. Both the grid and the features are scaled to a two-dimensional coordinate range of -1 to 1. Each point in the grid finds its corresponding position in the feature map based on its coordinates. At that position, the feature value is calculated using either nearest neighbor or interpolation based on the corresponding feature value in the feature map and returned. After all grid points have been queried, the grid is scaled back to its original size and fed into the implicit neural function layer parameterized by the multilayer perceptron. The result is then concatenated with the encoder output and fed into the next layer of the decoder.

[0074] Spatiotemporal information fusion of adjacent frames: In the grid sampling process mentioned above, since the query range of each point is only within the target features, the useful information contained in its adjacent frames is ignored, which easily leads to inconsistencies. To solve this problem, we adopted a spatiotemporal fusion of adjacent frames. When the features are sent to the stealth neural upsampling module, we process the features by expanding them according to the frame dimension, extracting adjacent frames for each frame and connecting them according to the channel dimension. Then, we process the channel dimension to finally obtain a spatiotemporal information fusion feature with the same channel size as the target frame. After spatiotemporal fusion processing, this feature map is used for the above sampling process.

[0075] Example 2

[0076] In this embodiment, the video super-resolution method in Embodiment 1 is further described. This method achieves arbitrary-scale video super-resolution by introducing an implicit neural network to replace the upsampling module in the traditional U-Net network and adding a spatiotemporal information fusion mechanism to this module. The method includes the following main steps: inputting a low-resolution video frame sequence, gradually recovering high-resolution frames through a diffusion model, and simultaneously adding spatiotemporal fusion of the target frame and adjacent frames in the implicit neural upsampling module to improve the continuity and consistency of the generated video frames.

[0077] The diffusion model is a method for progressively generating data, used in this invention to gradually recover high-resolution video from low-resolution video. The diffusion model restores an initial low-resolution frame to a target high-resolution frame through a multi-step reverse process. In each step, the model generates a higher-resolution intermediate frame based on the degree of noise reduction, ultimately achieving the target sharpness and resolution.

[0078] In traditional U-Net networks, the upsampling module is typically implemented through convolutional operations, which cannot effectively handle resolution changes at arbitrary scales. This invention employs an implicit neural network as the upsampling module, enabling the generative network to handle resolution enhancements at any scale. Through continuous feature representation, the implicit neural network eliminates the dependence on convolutional operations at specific scales for upsampling, exhibiting strong scale adaptability and flexibility, and generating more detailed, high-resolution frames.

[0079] To improve the continuity and consistency of generated video frames, this invention introduces a spatiotemporal fusion method based on the target frame and adjacent frames into the implicit neural upsampling module. This spatiotemporal fusion mechanism deeply fuses the spatiotemporal features of the target frame and adjacent frames, capturing motion information and correlations between frames.

[0080] Specifically, when the features are fed into the stealth neural upsampling module, we process the features by expanding them according to the frame dimension, extracting adjacent frames before and after each frame and connecting them according to the channel dimension, and then processing the channel dimension to finally obtain spatiotemporal information fusion features with the same channel size as the target frame; the high-resolution frames generated in this way are not only clear, but also maintain a high degree of continuity in dynamic videos.

[0081] like Figure 1 As shown, the specific technical solution implementation process of this invention includes the following main steps, in sequence: low-resolution video frame input, diffusion model processing, spatiotemporal information fusion, implicit neural upsampling module processing, generation of high-resolution video frames, and output. The following is a detailed description of each step:

[0082] Step 1: First, obtain the input low-resolution video sequence L = {L1, L2, ..., L...} n The video sequence L contains each low-resolution frame, and information from these frames will be used for feature extraction and upscaling. The preset magnification factor s or target resolution can flexibly adapt to user needs to generate super-resolution video output that meets specific display requirements.

[0083] Step 2: First, interpolate the low-resolution video sequence L to the target resolution size, thus obtaining a rough high-resolution video frame sequence. The coarse high-resolution frame sequence obtained in this step is not the final output, but rather provides a preliminary high-resolution reference for subsequent processing. Next, a pre-trained optical flow network is used to calculate the optical flow information between adjacent frames, generating optical flow maps O. These optical flow maps effectively capture motion information between frames, providing important dynamic guidance for the subsequent diffusion process. The interpolation processing and the calculation formula for optical flow map O are as follows:

[0084]

[0085] Step 3: Encode the coarse high-resolution video frames. Specifically, the features of each frame are mapped into the latent space using a pre-trained encoder ε, generating a latent feature representation z. This feature representation in the latent space compresses the information in the original video frames and provides a more efficient feature description for subsequent generation processes. This encoding significantly reduces redundant information in the original video data while retaining the key information needed to generate high-resolution video. The formula for generating the latent feature representation z is as follows:

[0086]

[0087] Step 4: Based on the encoded latent feature representation z, generate a noise of the same size, and initialize an initial input x based on z and the noise. T The latent feature representation z is then fed into a guided conditional network (similar to the encoder structure of U-Net, which processes and downsamples the input features step by step to prevent connection failures due to size mismatches in the decoder part of the U-Net network) to generate a conditional input. This conditional input serves as auxiliary feature information to guide the denoising process, helping the network gradually restore video details during denoising. Then, the conditional input and the initial input x are combined... T They are fed into the U-Net network together. Initial input x T The formula for generating it is as follows:

[0088]

[0089] in, These are learnable parameters that can be optimized during training.

[0090] Step 5: In the encoder part of the U-Net network, the input features are processed according to the original structure to extract multi-scale noisy features. In the decoder part, the features are processed by residual blocks and Transformer blocks and then sent to the implicit neural upsampling module. In the implicit neural upsampling module, the input features are first expanded according to the frame channels, and each frame is processed cyclically. For each frame feature f i The features of the frame and its adjacent frames are obtained and concatenated together according to the channel dimension to obtain the feature f. c(i) Then, through channel compression, the spliced ​​features are compressed back to their original channel size, thereby obtaining a new feature f that fuses spatiotemporal information. new(i) This processing method can effectively fuse inter-frame information during the decoding stage, improving the coherence between frames. Feature f c(i) With feature f new(i)The formula for generating it is as follows:

[0091] f c(i) =concate(f i-1 ,f i ,f i+1 );

[0092] f new(i) =Conv(f c(i) );

[0093] In the first formula above, the brackets represent the target frame features and the features of its preceding and following frames; concate means connecting them along the channel dimension; and Conv means convolutional layer processing operation.

[0094] Step 6: Based on the multi-scale noisy features generated by the encoder part of the U-Net network, this invention further introduces the generation of a coordinate grid (coord). A coordinate grid of corresponding size is generated according to the size of the multi-scale noisy features in the encoder part of the U-Net network. The processed features (feature f) are then... new(i) Both the coordinate grid size and the coordinate grid size are scaled to the range of -1 to 1. A feature map f is then found for the position of each point in the coordinate grid. new The corresponding position is selected and the feature value is queried. Then, the image is scaled back to its original size, and the resulting feature map f is displayed. query The output f is obtained by feeding the data into a hidden neural function layer parameterized by a multilayer perceptron. out Then, it is connected to the encoder's output and sent to the next-level module of the decoder. The specific processing procedure is as follows:

[0095] f query =Grid_Sample(f new (coord);

[0096] f out =INF(f query );

[0097] Where Coord represents the generated two-dimensional coordinate grid of the target size, and Grid_Sample represents the search for f at each coordinate in the two-dimensional coordinate grid. new The feature value at the corresponding position in the diagram, INF represents the implicit neural upsampling module formed by the implicit neural function layer parameterized by the multilayer perceptron, which processes features through the multilayer perceptron.

[0098] Step 7: After all processing by the U-Net network is complete, the noisy feature f outThe data will be fed into the optical flow alignment module (SpyNet). This module uses the optical flow map calculated in step 2 to curl the features and align the features between adjacent frames, thereby further improving the spatiotemporal consistency between frames. The optical flow alignment operation ensures the continuity of features between frames, effectively reducing inconsistencies such as jitter and blurring in video frames, resulting in a smoother and more natural high-resolution video.

[0099] Step 8: The features after optical flow alignment are subjected to a T-iteration denoising process to gradually eliminate noise, thereby restoring high-resolution video frame details. In each iteration, the denoising process further eliminates noise and restores details. After the final denoising process is completed, the resulting feature f... denoise It will be decoded by the decoder into a high-resolution video sequence H:

[0100] H = D(f) denoise );

[0101] Where D represents the decoder.

[0102] Example 3

[0103] In recent years, some works have also explored how to achieve super-resolution at arbitrary scales. While these methods have achieved the desired results for arbitrary-scale super-resolution, they often struggle to achieve high-fidelity detail at high magnifications. This is because these regression-based methods tend to average the possible super-resolution predictions due to regression loss. Since their inception, diffusion models have demonstrated compelling performance in generating high-fidelity, high-quality images, achieving considerable success in video super-resolution. Nevertheless, due to the fixed upsampling and downsampling of the U-Net network, diffusion-based methods remain limited to a fixed magnification. Retraining the model or employing complex cascaded structures is required for tasks with different magnifications, resulting in additional training costs.

[0104] To address this issue, this embodiment proposes an innovative implicit representation super-resolution diffusion model for achieving arbitrary-scale video super-resolution with high-quality details. We overcome the fixed-scale limitation by introducing implicit neural representation functions (i.e., the implicit neural function layer in Embodiment 1) into the diffusion model, while retaining the advantages of the diffusion model in generating higher-quality details. Specifically, as follows... Figure 2As shown, we first calculate the optical flow between adjacent frames, encoding the video frames into a continuous latent feature space. To achieve arbitrary resolution representation of the image, we employ an implicit neural representation function in the decoder structure of the U-Net network and parameterize it using a coordinate-based multilayer perceptron (MLP). Before performing the upsampling step, we perform a spatiotemporal information fusion operation on the input features. Overall, the model alternates between the diffusion model and the implicit neural representation function. To retain more information from low-resolution video frames, we fuse the original features obtained from the low-resolution video frames after encoding with the features encoded at each stage of the encoder part of the U-Net network during the diffusion process. Simultaneously, to allow the diffusion model to obtain outputs of different resolutions, we introduce a magnification factor as the network input. This magnification factor is encoded as a feature vector and used as the weight for fusion of the original features and the noise processed by the U-Net network. After each denoising process, the features are convoluted using the initially obtained optical flow to obtain new features, which serve as the input for the next denoising step. Finally, the decoder decodes the features to obtain the high-resolution video frame. Our experimental results are shown below. Figure 3 , Figure 4 As shown.

[0105] In summary, the arbitrary-scale video super-resolution method based on the diffusion model described above effectively addresses the shortcomings of existing video super-resolution methods in terms of arbitrary scales through spatiotemporal feature fusion and arbitrary-scale implicit neural upsampling. It not only overcomes the limitations of existing methods that restrict resolution to integer multiples and single multiples, enabling the generation of high-resolution video sequences of arbitrary sizes, but also ensures consistency between video sequences, enriching the application scenarios of video super-resolution and expanding the technological boundaries of this field.

[0106] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A video super-resolution method of arbitrary scale based on a diffusion model, characterized in that, Includes the following steps: S1: Low-resolution video frame input Obtain the input low-resolution video sequence L and the preset magnification factor s or target resolution; S2: Interpolation and Optical Flow Information Calculation Interpolation is performed on the low-resolution video sequence L to bring it up to the target resolution, resulting in a coarse high-resolution video frame sequence. An optical flow network of pretrained optical flow is used to calculate the optical flow information between adjacent frames and generate an optical flow map O. S3: Latent Feature Representation Generation The coarse high-resolution video frames are encoded to generate latent feature representations z; S4: Initial Input Generation Based on the encoded latent feature representation z, generate a noise of the same size, and initialize an initial input x based on the latent feature representation z and the noise. T The latent feature representation z is fed into a guided conditional network for processing to generate conditional inputs; then the conditional inputs and the initial input x are combined. T They are fed together into the U-Net network of the diffusion model; S5: Spatiotemporal Information Fusion In the encoder part of the U-Net network, the input features in step S4 are processed according to the original structure to extract noisy features at multiple scales. In the decoder part of the U-Net network, the noisy features at multiple scales are processed by the residual block and the Transformer block and then sent to the implicit neural upsampling module, where spatiotemporal information fusion processing is performed. S6: Implicit Neural Upsampling Processing Based on the multi-scale noisy feature sizes of the encoder part of the U-Net network, a coordinate grid (coord) of corresponding size is generated. The size of both the spatiotemporally fused features and the coordinate grid (coord) is scaled to the range of -1 to 1. For each point in the coordinate grid, the spatiotemporally fused feature map (f) is located. new The corresponding position is selected and the feature value is queried. Then, the image is scaled back to its original size, and the resulting feature map f is displayed. query The output f is obtained by feeding the data into a hidden neural function layer parameterized by a multilayer perceptron. out Then, it is connected to the multi-scale noisy features output by the encoder and fed into the next-level module of the decoder part of the U-Net network; S7: Optical Flow Alignment Processing After the U-Net network finishes processing, its output features are sent to the optical flow alignment module. Using the optical flow map calculated in step S2, the features are curled to align the features between adjacent frames. S8: Generate and output high-resolution video frames The features after optical flow alignment are subjected to a T-iteration denoising process to gradually eliminate noise and restore high-resolution video frame details. The feature f obtained after the denoising process is completed is... denoise The diffuse model's decoder decodes the video into a high-resolution video sequence H.

2. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S2, a coarse high-resolution video frame sequence The calculation formula is as follows: In this context, "↑" indicates interpolation.

3. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S2, the formula for calculating the optical flow map is as follows:

4. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S3, the specific processing procedure is as follows: The features of each frame are mapped into the latent space through a pre-trained encoder ε to generate a latent feature representation z:

5. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S4, the initial input x T The formula for generating it is as follows: in, These are learnable parameters that can be optimized during training.

6. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S5, the spatiotemporal information fusion processing procedure is as follows: S51: The input features are first expanded according to the frame channels, and each frame is processed in a loop; S52: For each frame's feature f i The features f are obtained by concatenating the features of the adjacent frames along the channel dimension. c(i) ; S53: Through channel compression, the spliced ​​features are compressed back to their original channel size, thereby obtaining a new feature f that fuses spatiotemporal information. new(i) Thus, the feature map f is obtained. new .

7. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 6, characterized in that, In step S52, feature f c(i) The formula for generating it is as follows: f c(i) =concate(f i-1 ,f i ,f i+1 ); Among them, f i-1 f i f i+1 These represent the features of the target frame and the features of its preceding and following frames, respectively. "concate" indicates connection by channel dimension.

8. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 7, characterized in that, In step S53, feature f new(i) The formula for generating it is as follows: f new(i) =Conv(f c(i) ); Here, Conv represents the convolutional layer processing operation.

9. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S6, the feature map f query and output f out The formula for generating it is as follows: f query =Grid_Sample(f new ,coord); in out =INF(f query ); Where coord represents the generated two-dimensional coordinate grid of the target size, and Grid_Sample represents the search for f at each coordinate in the two-dimensional coordinate grid. new The feature value at the corresponding position in the diagram, INF represents the implicit neural upsampling module formed by the implicit neural function layer parameterized by the multilayer perceptron, which processes features through the multilayer perceptron.

10. The arbitrary-scale video super-resolution method based on a diffusion model according to claim 1, characterized in that, In step S8, the formula for generating the high-resolution video sequence H is as follows: H=D(f denoise ); Where D represents the decoder.

Images