A universal restoration method for multi-degraded videos

Through the video super-resolution state space model VSRMamba, by feature expansion and splicing of multiple degraded video frames in time and space dimensions, the problem of limited receptive field of convolutional neural networks is solved, and higher quality high-resolution videos are generated.

CN120410857BActive Publication Date: 2025-08-29SUN YAT SEN UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202510873219.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-06-27
Publication Date
2025-08-29
Estimated Expiration
2045-06-27

AI Technical Summary

Technical Problem

In the existing video super-resolution methods, the receptive field size of the convolutional neural network is limited, and the global information of the video frame cannot be fully utilized, resulting in the low quality of the generated high-resolution video frames.

Method used

The video super-resolution state space model VSRMamba is adopted, including shallow feature extraction module, cyclic feature propagation module and high-resolution reconstruction module. By forward and reversely expanding and splicing multiple shallow features in the time and space dimensions, deep features are generated and fused.

Benefits of technology

It solves the problem of neighbor pixel forgetting, enhances the use of adjacent pixel information between video frames and within video frames, and generates high-resolution videos with more natural and clear texture details, improves video quality, and reduces computing resource requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120410857B_ABST
    Figure CN120410857B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of computer software technology and discloses a universal method for restoring multiple degraded videos. The method comprises: obtaining multiple degraded videos; inputting the multiple degraded videos into VSRMamba; extracting shallow features of the current video frame and multiple consecutive video frames preceding the current video frame using a shallow feature extraction module; inputting the multiple shallow features into a cyclic feature propagation module, aligning the multiple shallow features using the cyclic feature propagation module, performing forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the temporal and spatial dimensions to obtain multiple one-dimensional sequences, and generating deep features of the current video frame based on the multiple one-dimensional sequences; and fusing the shallow features and deep features of the current video frame using a high-resolution reconstruction module to generate a high-resolution video frame corresponding to the current video frame. This method solves the problem of forgotten neighbor pixels, and the generated high-resolution video has more natural and clear texture details.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer software, and in particular relates to a universal repair method for multiple degraded videos. Background Art

[0002] Video super-resolution (VSR) is a method that uses software or hardware technology to increase the resolution of video frames. It generates high-resolution video frames from a series of low-resolution video frames, thereby improving the overall video quality.

[0003] Common video super-resolution methods in the prior art are typically based on convolutional neural networks, using a sliding window approach to generate high-resolution video frames. Specifically, the convolutional neural network used in these methods typically consists of a multi-frame alignment module, a feature extraction module, and a feature fusion module. Using a sliding window approach, these three modules are sequentially processed on low-resolution video frames to generate high-resolution video frames. However, due to the limited receptive field of convolutional neural networks, they cannot fully utilize the global information of the video frames, resulting in low-quality high-resolution video frames. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of forgotten neighbor pixels and enhance the use of neighbor pixel information between and within video frames, so that the generated high-resolution video has more natural and clear texture details, thereby improving the quality of the generated high-resolution video.

[0005] In a first aspect, an embodiment of the present invention provides a universal method for restoring multiple degraded videos, the method comprising:

[0006] Acquire a multi-degraded video to be restored, where the resolution of the multi-degraded video is less than a preset resolution;

[0007] Input the multi-degraded video into a pre-trained video super-resolution state space model VSRMamba, wherein the VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module;

[0008] For any current video frame in the multiple degraded videos, extracting shallow features of the current video frame and a plurality of consecutive video frames before the current video frame by the shallow feature extraction module;

[0009] Inputting the extracted multiple shallow features into the cyclic feature propagation module, aligning the multiple shallow features through the cyclic feature propagation module, performing forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generating the deep features of the current video frame based on the multiple one-dimensional sequences;

[0010] The shallow features and deep features of the current video frame are fused through the high-resolution reconstruction module, and a high-resolution video frame corresponding to the current video frame is generated based on the fused features.

[0011] Optionally, the cyclic feature propagation module includes multiple residual state space modules, each of which is composed of a first layer of normalization, a spatiotemporal state space module, a first residual connection, a second layer of normalization, a convolution layer, a channel attention, and a second residual connection; the spatiotemporal state space module is composed of a first linear layer, a channel-by-channel convolution, a first activation function, a three-dimensional selective scanning module, a third layer of normalization, a second linear layer, a second activation function, a Hadamard product, and a third linear layer; the three-dimensional selective scanning module is composed of a three-dimensional selective expansion sub-module, a state space model, and a sequence summation sub-module.

[0012] Optionally, the cyclic feature propagation module performs forward expansion and concatenation operations, and reverse expansion and concatenation operations on the aligned multiple shallow features in the time dimension and the space dimension, respectively, to obtain multiple one-dimensional sequences, including:

[0013] Inputting the aligned multiple shallow features into a residual state space module, and normalizing the aligned multiple shallow features through a first layer normalization of the residual state space module to obtain multiple normalized shallow features;

[0014] Inputting the normalized multiple shallow features into the spatiotemporal state space module, and sequentially processing the normalized multiple shallow features through a first linear layer, a channel-by-channel convolution, and a first activation function of the spatiotemporal state space module to obtain processed shallow features;

[0015] The processed shallow features are input into a three-dimensional selective scanning module, and the processed shallow features are subjected to forward expansion and splicing operations, as well as reverse expansion and splicing operations in the time dimension and the space dimension respectively through the three-dimensional selective expansion submodule to obtain multiple one-dimensional sequences.

[0016] Optionally, the processed shallow features are input into a three-dimensional selective scanning module, and the three-dimensional selective scanning module performs forward expansion and splicing operations, and reverse expansion and splicing operations on the processed shallow features in the time dimension and the space dimension, respectively, to obtain multiple one-dimensional sequences, including:

[0017] In the first spatial dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain a first one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a second one-dimensional sequence;

[0018] In the second spatial dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain a third one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a fourth one-dimensional sequence;

[0019] In the time dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain the fifth one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain the sixth one-dimensional sequence.

[0020] Optionally, generating the deep features of the current video frame based on the multiple one-dimensional sequences includes:

[0021] The multiple one-dimensional sequences are processed by the state space model and sequence summation submodule of the three-dimensional selective scanning module to obtain a single one-dimensional sequence, and restored to a three-dimensional feature through a folding operation; the restored three-dimensional feature is normalized by the third layer normalization of the spatiotemporal state space module to obtain a normalized three-dimensional feature;

[0022] The normalized multiple shallow features input into the spatiotemporal state space module are processed by a second linear layer and a second activation function, and are processed with the normalized three-dimensional features by a Hadamard product and a third linear layer to obtain processed three-dimensional features;

[0023] Inputting the processed three-dimensional features into the residual state space module, and sequentially processing the processed three-dimensional features through the first residual connection, the second normalization layer, the convolution layer, the channel attention, and the second residual connection of the residual state space module to obtain a re-processed three-dimensional feature;

[0024] The reprocessed three-dimensional features are repeatedly processed by a plurality of the residual state space modules to obtain the deep features of the current video frame.

[0025] Optionally, the VSRMamba training process includes:

[0026] Acquire a high-resolution video dataset, and obtain a low-resolution video dataset corresponding to the high-resolution video dataset by downsampling;

[0027] Inputting the low-resolution video dataset into the VSRMamba to be trained, wherein the VSRMamb to be trained includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module;

[0028] For any low-resolution video frame of a target low-resolution video in the low-resolution video dataset, extracting shallow features of the low-resolution video frame and a plurality of consecutive low-resolution video frames preceding the low-resolution video frame by the shallow feature extraction module, wherein the target low-resolution video is any low-resolution video in the low-resolution video dataset;

[0029] Inputting the extracted multiple shallow features into the cyclic feature propagation module, aligning the multiple shallow features through the cyclic feature propagation module, performing forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generating deep features of the low-resolution video frame based on the obtained multiple one-dimensional sequences;

[0030] The high-resolution reconstruction module fuses the shallow features and deep features of the low-resolution video frame, and generates a high-resolution video frame corresponding to the low-resolution video frame based on the fused features, thereby obtaining a target high-resolution video corresponding to the target low-resolution video;

[0031] Based on the target high-resolution video and the high-resolution video corresponding to the target low-resolution video before downsampling, the loss function value of the VSRMamba to be trained is calculated, and the network weights in each module are updated through optimizer training to reduce the loss function value. When the calculated loss function value is less than a preset value, the trained VSRMamba is determined to be obtained.

[0032] In a second aspect, an embodiment of the present invention provides a universal restoration device for multiple degraded videos, the device comprising:

[0033] a multi-degraded video acquisition module, configured to acquire a multi-degraded video to be restored, wherein the resolution of the multi-degraded video is less than a preset resolution;

[0034] A multi-degraded video input module is used to input the multi-degraded video into a pre-trained video super-resolution state space model VSRMamba, wherein the VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module;

[0035] a shallow feature extraction module, configured to extract, for any current video frame in the multi-degraded video, shallow features of the current video frame and a plurality of consecutive video frames preceding the current video frame through the shallow feature extraction module;

[0036] a deep feature generation module, configured to input the extracted multiple shallow features into the cyclic feature propagation module, align the multiple shallow features through the cyclic feature propagation module, perform forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generate deep features of the current video frame based on the multiple one-dimensional sequences;

[0037] The high-resolution video frame generation module is used to fuse the shallow features and deep features of the current video frame through the high-resolution reconstruction module, and generate a high-resolution video frame corresponding to the current video frame based on the fused features.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0039] at least one processor;

[0040] a memory for storing the at least one processor-executable instruction;

[0041] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method described in the first aspect.

[0043] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0044] The technical solution provided by the embodiment of the present invention is to input the multiple degraded videos into a pre-trained video super-resolution state space model VSRMamba when repairing multiple degraded videos. VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module. After the shallow feature extraction module extracts the shallow features of any video frame and multiple consecutive video frames before the video frame, the cyclic feature propagation module aligns the multiple shallow features and performs forward expansion and splicing operations, as well as reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generates deep features of the current video frame based on the multiple one-dimensional sequences. Finally, the high-resolution reconstruction module fuses the shallow features and deep features of the current video frame, and generates a high-resolution video frame corresponding to the current video frame based on the fused features.

[0045] From the above description, it can be seen that by performing forward expansion and splicing operations, as well as reverse expansion and splicing operations on the aligned shallow features in the time dimension and the spatial dimension, the problem of forgotten neighbor pixels can be solved, and the use of neighboring pixel information between and within video frames can be enhanced. The generated deep features integrate the features within the video frame and the features between the video frames, making the information of the deep features more comprehensive. The generated high-resolution video has more natural and clear texture details, thereby improving the quality of the generated high-resolution video. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the structure of a cyclic feature propagation module provided in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the structure of a residual state space module provided in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a spatiotemporal state space module provided in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of operations performed by a three-dimensional selective scanning module provided in an embodiment of the present invention;

[0050] Figure 5 A flowchart of a general method for repairing multiple degraded videos provided by an embodiment of the present invention;

[0051] Figure 6 A schematic structural diagram of a universal restoration device for multiple degraded videos provided by an embodiment of the present invention;

[0052] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be described in detail below through examples.

[0054] In recent years, deep neural networks such as convolutional neural networks and Transformers have significantly driven technological progress in video super-resolution research. Currently, mainstream video super-resolution methods based on deep neural networks fall into two categories: those based on temporal sliding windows and those based on recurrent frameworks.

[0055] For the method based on time sliding window, the convolutional neural network used in this method usually includes a multi-frame alignment module, a feature extraction module and a feature fusion module; using the sliding window method, the low-resolution video frames are processed in sequence according to the sequence of these three modules to generate high-resolution video frames.

[0056] For the method based on the recurrent framework, this method performs long-term propagation of inter-frame information of multiple video frames based on the structure of the recurrent neural network.

[0057] For the two methods mentioned above, due to the limited receptive field size of the convolutional neural network, the global information of the video frame cannot be fully utilized, resulting in low quality of the generated high-resolution video frames. To achieve better results, the neural network architecture in the existing technology all adopts the Transformer. Although the Transformer improves the visual effect to a certain extent, the processing efficiency of longer and larger videos is low, and the storage requirements are still high. Existing methods all require a trade-off between performance and computational efficiency, which limits their application on resource-constrained devices (such as TVs, smartphones, personal computers, etc.). In their research, the inventors found that the Mamba model based on the state-space model shows potential in long sequence modeling, but directly using it to replace the Transformer and apply it to video super-resolution will face some problems. For example, neighboring pixels are forgotten, multi-frame information utilization is low, and channel redundancy is a problem.

[0058] To address the above-mentioned issues, the present invention aims to provide a universal method for restoring multiple degraded videos. Multiple degraded videos can be understood as low-resolution videos, i.e., the resolution of multiple degraded videos is less than a preset resolution. The preset resolution can be set according to actual conditions and is not specifically limited in the embodiments of the present invention.

[0059] Specifically, multiple degraded videos can be repaired by the video super-resolution state space model VSRMamba. VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module. The cyclic feature propagation module includes a plurality of residual state space modules, each residual state space module includes a spatiotemporal state space module, and the spatiotemporal state space module includes a three-dimensional selective scanning module. The present invention improves the state space model Mamba's processing capability for video data through a three-dimensional selective scanning module, solves the problem of forgetting neighboring pixels and retains the advantages of Mamba. Compared with the Transformer-based method in the prior art, the present invention significantly reduces the demand for computing resources while improving the effect of video super-resolution.

[0060] In order to describe the solution clearly, the training process and testing process of VSRMamba used in the embodiment of the present invention will be described in detail below.

[0061] The execution subject of the solution of the present invention may be a computer, which may be an enterprise server or a personal computer. The execution subject is capable of processing low-resolution videos and generating high-resolution videos, thereby repairing multiple degraded videos.

[0062] The training and testing process of VSRMamba is mainly divided into the following three stages.

[0063] The first stage is training data preparation. Specifically, a high-resolution video dataset is collected and downsampled to obtain corresponding low-resolution data. The downsampling algorithm can be determined based on the actual situation. For example, the downsampling algorithm can be bilinear or bicubic. This stage allows the construction of training data.

[0064] The second phase is neural network training. Specifically, the training data from the first phase is fed into VSRMamba, which is trained using the Adam optimizer, with parameters such as batch size and learning rate. VSRMamba employs a recurrent framework consisting of a shallow feature extraction module, a recurrent feature propagation module, and a high-resolution reconstruction module.

[0065] The shallow feature extraction module performs shallow feature extraction on each low-resolution video frame in the input low-resolution video through a convolutional neural network.

[0066] Assume that the height and width of the input low-resolution video are H and W respectively, there are T frames in total, and three-channel color encoding is used. Then the t-th video frame of the input can be calculated as .

[0067] The shallow feature extraction module can be composed of a 3x3 convolution kernel with an output channel of C, denoted as The input of the shallow feature extraction module is each video frame of the input video, and the output is the shallow feature of each video frame. , the whole process can be expressed by the following formula:

[0068]

[0069] The cyclic feature propagation module adopts the bidirectional, second-order grid propagation framework proposed by BasicVSR++. This framework is used to extract deep features. The cyclic feature propagation module is composed of block alignment and N residual state space modules stacked together to capture long-range dependencies in the spatiotemporal dimension, such as Figure 1 As shown. The number N can be set according to actual conditions. For example, N can be 24. The embodiment of the present invention does not specifically limit the size of N. Specifically, for any low-resolution video in the input low-resolution video, the shallow features of the low-resolution video frame and the multiple consecutive video frames before the low-resolution video frame can be block-aligned. For example, the shallow features of the low-resolution video frame can be aligned with the shallow features of the two video frames before the low-resolution video frame.

[0070] For example, l The output features of layer propagation are , taking the shallow feature extraction module as the 0th layer, then , block alignment is denoted as PA, concatenation operation is denoted as Concat, and residual state space module is denoted as RSSB. If a total of L layers are propagated, the process can be expressed by the following formula:

[0071]

[0072] The residual state space module will be described in detail below. Figure 2 As shown in Figure 1, the module consists of layer normalization (for the sake of clarity, it can be called the first layer normalization), spatiotemporal state space module, residual connection (for the sake of clarity, it can be called the first residual connection), layer normalization (for the sake of clarity, it can be called the second layer normalization), convolutional layer, channel attention, residual connection (for the sake of clarity, it can be called the second residual connection). The input of the residual state space module is , the normalized input of the second layer is , the layer normalization is LN, the spatiotemporal state space module is TSSM, and the two residual connection coefficients are and , the convolution layer is Conv, and the channel attention is CA. The whole process can be expressed by the following formula:

[0073]

[0074] The spatiotemporal state space module will be described in detail below. Figure 3 As shown in the figure, the spatiotemporal state space module consists of a linear layer (for the sake of clarity in the scheme description, it can be called the first linear layer), channel-by-channel convolution, an activation function (for the sake of clarity in the scheme description, it can be called the first activation function), a three-dimensional selective scanning module, layer normalization (for the sake of clarity in the scheme description, it can be called the third layer normalization), a linear layer (for the sake of clarity in the scheme description, it can be called the second linear layer), an activation function (for the sake of clarity in the scheme description, it can be called the second activation function), a Hadamard product, and a linear layer (for the sake of clarity in the scheme description, it can be called the third linear layer). Let the linear layer be Linear, the channel-by-channel convolution be DWConv, the activation function be SiLU, the three-dimensional selective scanning module be SS3D, and the Hadamard product be . Let the input of SS3D be , the whole process can be expressed by the following formula:

[0075]

[0076] The following will describe the three-dimensional selective scanning module in detail. Figure 4 As shown in the figure, the 3D selective scanning module expands and splices the feature blocks of multiple video frames (the feature blocks of two video frames are used as an example in the figure) in the forward and reverse directions of three dimensions (2 dimensions in space + 1 dimension in time) to obtain 6 one-dimensional sequences, which are then input into the state space model for processing. This ensures that pixels that are similar in any dimension are at least similar in a certain expanded sequence, thus solving the problem of forgetting adjacent pixels. The sequence is then input into the state space model, and the output sequence is finally summed. are the unit vectors of the width, height, and time direction of the input video frame respectively. according to The operation of sequential forward expansion is , the reverse expansion operation is The state space model is called SSM. This module can be expressed by the following formula:

[0077]

[0078] exist Figure 4 In the first spatial dimension, the first shallow feature block is forward expanded, and the result is 1234. The second shallow feature block is forward expanded, and the result is 5678. The result of splicing the two is 12345678. Correspondingly, the result of reverse expansion and splicing is 87654321.

[0079] In the second spatial dimension, the first shallow feature block is forward expanded, and the result is 1324. The second shallow feature block is reverse expanded, and the result is 5768. The two are spliced ​​together, and the result is 13245768. Correspondingly, the result of reverse expansion and splicing is 86754231.

[0080] In the temporal dimension, the results of forward expansion are 1526 and 3748, respectively. The result of concatenating the two is 15263748. Correspondingly, the result of reverse expansion and concatenation is 84736251. Since in three-dimensional space, 1 and 5 are adjacent pixels, 2 and 6 are adjacent pixels, 3 and 7 are adjacent pixels, and 5 and 8 are adjacent pixels, forward and reverse expansion of shallow feature blocks in the temporal dimension can solve the problem of forgotten neighboring pixels. Furthermore, by performing forward and reverse expansion in two spatial dimensions and one temporal dimension, respectively, the use of neighboring pixel information between and within video frames can be enhanced, resulting in more natural and clear texture details in the subsequent high-resolution video output.

[0081] Six one-dimensional sequences are fed into the state-space model and processed by the state-space model to obtain processed one-dimensional sequences. The six processed one-dimensional sequences are then summed to obtain a single one-dimensional sequence, which is then restored to three-dimensional features through a folding operation and used as the output of the three-dimensional selective scanning module.

[0082] By looping and executing the residual state space module, the shallow features of the input video frame are Continuously extracted and propagated until the deep features of the video frame are obtained .

[0083] The high-resolution reconstruction module is used to reconstruct high-resolution video from shallow and deep features. With shallow features Add them together and reconstruct the high-resolution output through pixel shuffling. The pixel shuffling operation is called PS, the magnification is s, and the final output high-resolution video is frame t. for:

[0084] VSRMamba uses the L1 loss function for training:

[0085]

[0086] VSRMamba uses the Adam optimizer to train and update the network weights in each module, thereby reducing the loss function value. If the loss function value of VSRMamba is less than the preset threshold, it indicates that the loss function value of VSRMamba is sufficiently small, and the trained VSRMamba is determined. Otherwise, it is necessary to continue adjusting the model parameters of VSRMamba and continue training VSRMamba until the loss function value is less than the preset threshold.

[0087] The third stage is the test data inference stage. The test data is fed into VSRMamba trained in the second stage, and the magnification factor is selected. For example, if the magnification factor is 2x, if the input video resolution is 720p, the output video resolution will be 1440p; if the input video resolution is 1080p, the output video resolution will be 2160p.

[0088] The technical solution provided by the embodiment of the present invention improves Mamba's video data processing capabilities while retaining the advantages of the Mamba model through the three-dimensional selective scanning module included in the video super-resolution state-space model VSRMamba. The three-dimensional selective scanning module solves the problem of forgotten neighboring pixels and enhances the use of neighboring pixel information between and within video frames, so that the output high-resolution video has more natural and clear texture details, thereby improving the video quality of the generated high-resolution video. In addition, compared to the quadratic time complexity O(n²) of the Transformer, VSRMamba has a linear time complexity O(n), thus significantly reducing computational overhead and being suitable for deployment on resource-limited devices.

[0089] After introducing the training and testing processes of VSRMamba, a general restoration method for multi-degraded videos provided by an embodiment of the present invention will be described in detail below.

[0090] like Figure 5 As shown, a universal method for repairing multiple degraded videos provided by an embodiment of the present invention may include the following steps:

[0091] S510: Obtain multiple degraded videos to be restored.

[0092] The resolution of the multi-degraded video is smaller than the preset resolution.

[0093] Specifically, a multi-degraded video can be understood as a low-resolution video, that is, the resolution of the multi-degraded video is less than a preset resolution. The preset resolution can be set according to actual conditions and is not specifically limited in the embodiment of the present invention. To repair the multi-degraded video, the following steps S520 to S550 need to be performed.

[0094] S520 , inputting multiple degraded videos into a pre-trained video super-resolution state space model VSRMamba.

[0095] Among them, VSRMamba includes: shallow feature extraction module, cyclic feature propagation module and high-resolution reconstruction module.

[0096] As an implementation method of the embodiment of the present invention, Figure 2 As shown in the figure, the cyclic feature propagation module includes multiple residual state space modules, each of which is composed of a first layer normalization, a spatiotemporal state space module, a first residual connection, a second layer normalization, a convolutional layer, a channel attention, and a second residual connection.

[0097] like Figure 3 As shown in the figure, the spatiotemporal state space module consists of a first linear layer, channel-by-channel convolution, a first activation function, a 3D selective scanning module, a third normalization layer, a second linear layer, a second activation function, a Hadamard product, and a third linear layer. The 3D selective scanning module consists of a 3D selective expansion submodule, a state space model, and a sequence summation submodule (not shown in the figure).

[0098] S530 : For any current video frame in the multi-degraded video, extract shallow features of the current video frame and a plurality of consecutive video frames before the current video frame by using a shallow feature extraction module.

[0099] For any current video frame in the multi-degraded video, the shallow features of the current video frame and the shallow features of multiple consecutive video frames before the current video frame can be extracted by the shallow feature extraction module. For example, the shallow features of the two video frames before the current video frame can be extracted.

[0100] Assume that the current video frame is , where H and W are the height and width of the current video frame respectively, 3 is the number of channels of the current video frame, and the shallow feature extraction module can be composed of a 3x3 convolution kernel with an output channel of C, which is recorded as , the shallow features of the current video frame output from the shallow feature extraction module are .

[0101] S540, input the extracted multiple shallow features into the cyclic feature propagation module, align the multiple shallow features through the cyclic feature propagation module, perform forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and space dimension, respectively, to obtain multiple one-dimensional sequences, and generate deep features of the current video frame based on the multiple one-dimensional sequences.

[0102] Specifically, after obtaining the shallow features of the current video frame and the shallow features of multiple consecutive video frames before the current video frame, the multiple shallow features can be aligned. In order to solve the problem of forgotten neighbor pixels and enhance the use of neighboring pixel information between and within video frames, the aligned multiple shallow features can be forward expanded and spliced ​​in three dimensions: time dimension (one dimension) and spatial dimension (two dimensions), as well as reverse expanded and spliced ​​to obtain multiple one-dimensional sequences, and the deep features of the current video frame are generated based on the multiple one-dimensional sequences. In this way, the generated deep features fuse the features within the video frame and the features between video frames, thereby solving the problem of forgotten neighbor pixels and making the information of the deep features more comprehensive, which helps to make the subsequent high-resolution video output have more natural and clear texture details.

[0103] As an implementation of an embodiment of the present invention, a cyclic feature propagation module is used to perform forward expansion and concatenation operations, as well as reverse expansion and concatenation operations, on multiple aligned shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences. The method may include the following steps, namely steps a1 to a3:

[0104] In step a1, the aligned multiple shallow features are input into the residual state space module, and the aligned multiple shallow features are normalized by the first layer normalization of the residual state space module to obtain multiple normalized shallow features.

[0105] In step a2, the normalized multiple shallow features are input into the spatiotemporal state space module, and the normalized multiple shallow features are processed in sequence through the first linear layer, channel-by-channel convolution and first activation function of the spatiotemporal state space module to obtain the processed shallow features.

[0106] It should be noted that those skilled in the art should be able to understand layer normalization, linear layer, channel-by-channel convolution and the first activation function, which will not be described here in detail.

[0107] In step a3, the processed shallow features are input into the three-dimensional selective scanning module. Through the three-dimensional selective expansion submodule, the processed shallow features are subjected to forward expansion and splicing operations, as well as reverse expansion and splicing operations in the time dimension and space dimension, respectively, to obtain multiple one-dimensional sequences.

[0108] As an implementation of an embodiment of the present invention, step a3, inputting the processed shallow features into a three-dimensional selective scanning module, and performing forward expansion and splicing operations, as well as reverse expansion and splicing operations, on the processed shallow features in the time dimension and the spatial dimension, respectively, through the three-dimensional selective scanning module to obtain multiple one-dimensional sequences, can include the following steps, namely steps a31 to a33:

[0109] In step a31, the processed shallow features are forward expanded in the first spatial dimension, and the forward expanded sequences are spliced ​​to obtain a first one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a second one-dimensional sequence.

[0110] Step a32: In the second spatial dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain a third one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a fourth one-dimensional sequence.

[0111] Step a33: In the time dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain a fifth one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a sixth one-dimensional sequence.

[0112] Specifically, by performing forward expansion and splicing operations, as well as reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension (one dimension) and the spatial dimension (two dimensions), multiple one-dimensional sequences are obtained, which can solve the problem of forgetting neighbor pixels and enhance the use of neighbor pixel information between and within video frames. Figure 4 In the illustrated embodiment, the specific implementation methods of forward expansion and splicing, as well as reverse expansion and splicing of the aligned shallow features in the spatial dimension and the temporal dimension have been described in detail with examples, which will not be repeated here.

[0113] As an implementation of the embodiment of the present invention, generating deep features of the current video frame based on multiple one-dimensional sequences may include the following steps, namely steps b1 to b4:

[0114] In step b1, multiple one-dimensional sequences are processed through the state space model and sequence summation submodule of the three-dimensional selective scanning module to obtain a single one-dimensional sequence, and then restored to a three-dimensional feature through a folding operation; the restored three-dimensional features are normalized through the third layer normalization of the spatiotemporal state space module to obtain a normalized three-dimensional feature.

[0115] In step b2, the multiple shallow features normalized in step a1 are processed through the second linear layer and the second activation function of the spatiotemporal state space module, and the processing results and the normalized three-dimensional features are processed through the Hadamard product and the third linear layer to obtain the processed three-dimensional features.

[0116] In step b3, the processed three-dimensional features are input into the residual state space module, and the processed three-dimensional features are processed in sequence through the first residual connection, second layer normalization, convolution layer, channel attention, and second residual connection of the residual state space module to obtain the re-processed three-dimensional features.

[0117] Step b4: repeatedly processing the reprocessed three-dimensional features through multiple residual state space modules to obtain deep features of the current video frame.

[0118] Specifically, those skilled in the art should be able to understand the specific operations of the state space model, the sequence summation submodule, the folding operation, the third layer normalization, the second linear layer, the second activation function, the Hadamard product and the third linear layer, and can understand the specific operations of the first residual connection, the second layer normalization, the convolution layer, the channel attention, and the second residual connection, which will not be elaborated here.

[0119] The sequence summation submodule may perform weighted summation on multiple one-dimensional sequences by directly adding the multiple one-dimensional sequences or setting weighted coefficients for the multiple one-dimensional sequences according to actual conditions and then adding them up. Both are reasonable.

[0120] S450: Fusing shallow features and deep features of the current video frame through a high-resolution reconstruction module, and generating a high-resolution video frame corresponding to the current video frame based on the fused features.

[0121] Specifically, after the feature propagation is completed, the deep features of the current video frame are obtained , the high-resolution reconstruction module transforms deep features With shallow features Add, and then reconstruct the high-resolution video frame corresponding to the current video frame through pixel shuffling, and output the high-resolution video frame corresponding to the current video frame. The pixel shuffling operation is denoted as PS, the magnification is s, and the final output high-resolution video is for:

[0122]

[0123] The technical solution provided by the embodiment of the present invention is to input the multiple degraded videos into a pre-trained video super-resolution state space model VSRMamba when repairing multiple degraded videos. VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module. After the shallow feature extraction module extracts the shallow features of any video frame and multiple consecutive video frames before the video frame, the cyclic feature propagation module aligns the multiple shallow features and performs forward expansion and splicing operations, as well as reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generates deep features of the current video frame based on the multiple one-dimensional sequences. Finally, the high-resolution reconstruction module fuses the shallow features and deep features of the current video frame, and generates a high-resolution video frame corresponding to the current video frame based on the fused features.

[0124] From the above description, it can be seen that by performing forward expansion and splicing operations, as well as reverse expansion and splicing operations on the aligned shallow features in the time dimension and the spatial dimension, the problem of forgotten neighbor pixels can be solved, and the use of neighboring pixel information between and within video frames can be enhanced. The generated deep features integrate the features within the video frame and the features between the video frames, making the information of the deep features more comprehensive. The generated high-resolution video has more natural and clear texture details, thereby improving the quality of the generated high-resolution video.

[0125] exist Figure 5 Based on the above embodiment, as an implementation of the embodiment of the present invention, the training process of VSRMamba may include the following steps, namely step d1 to step d4:

[0126] In step d1, a high-resolution video dataset is obtained, and a low-resolution video dataset corresponding to the high-resolution video dataset is obtained by downsampling.

[0127] In step d2, the low-resolution video dataset is input into the VSRMamba to be trained. The VSRMamba to be trained includes: a shallow feature extraction module, a cyclic feature propagation module, and a high-resolution reconstruction module.

[0128] Step d3: for any low-resolution video frame of the target low-resolution video in the low-resolution video dataset, shallow features of the low-resolution video frame and multiple consecutive low-resolution video frames before the low-resolution video frame are extracted by a shallow feature extraction module.

[0129] The target low-resolution video is any low-resolution video in the low-resolution video dataset.

[0130] In step d4, the extracted multiple shallow features are input into a cyclic feature propagation module, and the multiple shallow features are aligned through the cyclic feature propagation module. In the time dimension and the space dimension, the aligned multiple shallow features are respectively subjected to forward expansion and splicing operations, as well as reverse expansion and splicing operations to obtain multiple one-dimensional sequences, and deep features of low-resolution video frames are generated based on the obtained multiple one-dimensional sequences.

[0131] In step d5, the shallow features and deep features of the low-resolution video frame are fused through the high-resolution reconstruction module, and a high-resolution video frame corresponding to the low-resolution video frame is generated based on the fused features to obtain a target high-resolution video corresponding to the target low-resolution video.

[0132] Step d6, based on the target high-resolution video and the high-resolution video corresponding to the target low-resolution video before downsampling, calculate the loss function value of the VSRMamba to be trained, and update the network weights in each module through optimizer training to reduce the loss function value. When the calculated loss function value is less than the preset value, it is determined that the trained VSRMamba is obtained.

[0133] It should be noted that, in the above embodiment, the training process of VSRMamba has been described in detail in the introduction to the overall technical solution, and will not be repeated here.

[0134] In a second aspect, an embodiment of the present invention provides a universal restoration device 60 for multiple degraded videos, such as Figure 6 As shown, the device includes:

[0135] The multi-degraded video acquisition module 610 is configured to acquire a multi-degraded video to be restored, wherein the resolution of the multi-degraded video is smaller than a preset resolution;

[0136] A multi-degraded video input module 620 is configured to input the multi-degraded video into a pre-trained video super-resolution state space model VSRMamba, wherein the VSRMamba includes a shallow feature extraction module, a cyclic feature propagation module, and a high-resolution reconstruction module;

[0137] A shallow feature extraction module 630 is configured to extract shallow features of any current video frame in the multi-degraded video and a plurality of consecutive video frames before the current video frame through the shallow feature extraction module;

[0138] A deep feature generation module 640 is configured to input the extracted multiple shallow features into the cyclic feature propagation module, align the multiple shallow features through the cyclic feature propagation module, perform forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generate deep features of the current video frame based on the multiple one-dimensional sequences;

[0139] The high-resolution video frame generation module 650 is configured to fuse the shallow features and deep features of the current video frame through the high-resolution reconstruction module, and generate a high-resolution video frame corresponding to the current video frame based on the fused features.

[0140] In a third aspect, an embodiment of the present invention provides an electronic device 700, such as Figure 7 Shown, including:

[0141] at least one processor 701;

[0142] a memory 702 for storing the at least one processor-executable instruction;

[0143] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0144] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method described in the first aspect.

[0145] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0146] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A universal restoration method for multiple degraded videos, characterized in that: The method comprises: Acquire a multi-degraded video to be restored, where the resolution of the multi-degraded video is less than a preset resolution; Input the multi-degraded video into a pre-trained video super-resolution state space model VSRMamba, wherein the VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module; For any current video frame in the multiple degraded videos, extracting shallow features of the current video frame and a plurality of consecutive video frames before the current video frame by the shallow feature extraction module; Inputting the extracted multiple shallow features into the cyclic feature propagation module, aligning the multiple shallow features through the cyclic feature propagation module, performing forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generating the deep features of the current video frame based on the multiple one-dimensional sequences; Fusing the shallow features and deep features of the current video frame through the high-resolution reconstruction module, and generating a high-resolution video frame corresponding to the current video frame based on the fused features; The cyclic feature propagation module includes multiple residual state space modules, each of which is composed of a first layer of normalization, a spatiotemporal state space module, a first residual connection, a second layer of normalization, a convolution layer, a channel attention, and a second residual connection; the spatiotemporal state space module is composed of a first linear layer, a channel-by-channel convolution, a first activation function, a three-dimensional selective scanning module, a third layer of normalization, a second linear layer, a second activation function, a Hadamard product, and a third linear layer; the three-dimensional selective scanning module is composed of a three-dimensional selective expansion submodule, a state space model, and a sequence summation submodule; Through the cyclic feature propagation module, forward expansion and splicing operations, as well as reverse expansion and splicing operations, are performed on the aligned multiple shallow features in the time dimension and the space dimension, respectively, to obtain multiple one-dimensional sequences, including: Inputting the aligned multiple shallow features into a residual state space module, and normalizing the aligned multiple shallow features through a first layer normalization of the residual state space module to obtain multiple normalized shallow features; Inputting the normalized multiple shallow features into the spatiotemporal state space module, and sequentially processing the normalized multiple shallow features through a first linear layer, a channel-by-channel convolution, and a first activation function of the spatiotemporal state space module to obtain processed shallow features; The processed shallow features are input into a three-dimensional selective scanning module, and the processed shallow features are subjected to forward expansion and splicing operations, as well as reverse expansion and splicing operations in the time dimension and the space dimension respectively through the three-dimensional selective expansion submodule to obtain multiple one-dimensional sequences.

2. The method according to claim 1, characterized in that The processed shallow features are input into a three-dimensional selective scanning module. The three-dimensional selective scanning module performs forward expansion and splicing operations, as well as reverse expansion and splicing operations on the processed shallow features in the time dimension and the space dimension, respectively, to obtain multiple one-dimensional sequences, including: In the first spatial dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain a first one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a second one-dimensional sequence; In the second spatial dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain a third one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain a fourth one-dimensional sequence; In the time dimension, the processed shallow features are forward expanded, and the forward expanded sequences are spliced ​​to obtain the fifth one-dimensional sequence; and the processed shallow features are reverse expanded, and the reverse expanded sequences are spliced ​​to obtain the sixth one-dimensional sequence.

3. The method according to claim 1, characterized in that Generating the deep features of the current video frame based on the multiple one-dimensional sequences includes: The multiple one-dimensional sequences are processed by the state space model and sequence summation submodule of the three-dimensional selective scanning module to obtain a single one-dimensional sequence, and restored to a three-dimensional feature through a folding operation; the restored three-dimensional feature is normalized by the third layer normalization of the spatiotemporal state space module to obtain a normalized three-dimensional feature; The normalized multiple shallow features input into the spatiotemporal state space module are processed by a second linear layer and a second activation function, and are processed with the normalized three-dimensional features by a Hadamard product and a third linear layer to obtain processed three-dimensional features; Inputting the processed three-dimensional features into the residual state space module, and sequentially processing the processed three-dimensional features through the first residual connection, the second normalization layer, the convolution layer, the channel attention, and the second residual connection of the residual state space module to obtain a re-processed three-dimensional feature; The reprocessed three-dimensional features are repeatedly processed by a plurality of the residual state space modules to obtain the deep features of the current video frame.

4. The method according to any one of claims 1 to 3, characterized in that The VSRMamba training process includes: Acquire a high-resolution video dataset, and obtain a low-resolution video dataset corresponding to the high-resolution video dataset by downsampling; Inputting the low-resolution video dataset into the VSRMamba to be trained, wherein the VSRMamb to be trained includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module; For any low-resolution video frame of a target low-resolution video in the low-resolution video dataset, extracting shallow features of the low-resolution video frame and a plurality of consecutive low-resolution video frames preceding the low-resolution video frame by the shallow feature extraction module, wherein the target low-resolution video is any low-resolution video in the low-resolution video dataset; Inputting the extracted multiple shallow features into the cyclic feature propagation module, aligning the multiple shallow features through the cyclic feature propagation module, performing forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generating deep features of the low-resolution video frame based on the obtained multiple one-dimensional sequences; The high-resolution reconstruction module fuses the shallow features and deep features of the low-resolution video frame, and generates a high-resolution video frame corresponding to the low-resolution video frame based on the fused features, thereby obtaining a target high-resolution video corresponding to the target low-resolution video; Based on the target high-resolution video and the high-resolution video corresponding to the target low-resolution video before downsampling, the loss function value of the VSRMamba to be trained is calculated, and the network weights in each module are updated through optimizer training to reduce the loss function value. When the calculated loss function value is less than a preset value, the trained VSRMamba is determined to be obtained.

5. A universal restoration device for multiple degraded videos, characterized in that: The device comprises: a multi-degraded video acquisition module, configured to acquire a multi-degraded video to be restored, wherein the resolution of the multi-degraded video is less than a preset resolution; A multi-degraded video input module is used to input the multi-degraded video into a pre-trained video super-resolution state space model VSRMamba, wherein the VSRMamba includes: a shallow feature extraction module, a cyclic feature propagation module and a high-resolution reconstruction module; a shallow feature extraction module, configured to extract, for any current video frame in the multi-degraded video, shallow features of the current video frame and a plurality of consecutive video frames preceding the current video frame through the shallow feature extraction module; a deep feature generation module, configured to input the extracted multiple shallow features into the cyclic feature propagation module, align the multiple shallow features through the cyclic feature propagation module, perform forward expansion and splicing operations, and reverse expansion and splicing operations on the aligned multiple shallow features in the time dimension and the spatial dimension, respectively, to obtain multiple one-dimensional sequences, and generate deep features of the current video frame based on the multiple one-dimensional sequences; a high-resolution video frame generation module, configured to fuse the shallow features and deep features of the current video frame through the high-resolution reconstruction module, and generate a high-resolution video frame corresponding to the current video frame based on the fused features; The cyclic feature propagation module includes multiple residual state space modules, each of which is composed of a first layer of normalization, a spatiotemporal state space module, a first residual connection, a second layer of normalization, a convolution layer, a channel attention, and a second residual connection; the spatiotemporal state space module is composed of a first linear layer, a channel-by-channel convolution, a first activation function, a three-dimensional selective scanning module, a third layer of normalization, a second linear layer, a second activation function, a Hadamard product, and a third linear layer; the three-dimensional selective scanning module is composed of a three-dimensional selective expansion submodule, a state space model, and a sequence summation submodule; Through the cyclic feature propagation module, forward expansion and splicing operations, as well as reverse expansion and splicing operations, are performed on the aligned multiple shallow features in the time dimension and the space dimension, respectively, to obtain multiple one-dimensional sequences, including: Inputting the aligned multiple shallow features into a residual state space module, and normalizing the aligned multiple shallow features through a first layer normalization of the residual state space module to obtain multiple normalized shallow features; Inputting the normalized multiple shallow features into the spatiotemporal state space module, and sequentially processing the normalized multiple shallow features through a first linear layer, a channel-by-channel convolution, and a first activation function of the spatiotemporal state space module to obtain processed shallow features; The processed shallow features are input into a three-dimensional selective scanning module, and the processed shallow features are subjected to forward expansion and splicing operations, as well as reverse expansion and splicing operations in the time dimension and the space dimension respectively through the three-dimensional selective expansion submodule to obtain multiple one-dimensional sequences.

6. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 4 when the computer program is executed by a processor.

Citation Information

Patent Citations

  • Bidirectional residual fusion video super-resolution reconstruction method

    CN113658042A

  • Binocular image super-resolution reconstruction method and device and storage medium

    CN118154432A