Real-time online video super-resolution method assisted by compressed domain information

Abstract

The application provides a compression domain information assisted real-time online video super-resolution method, which fully utilizes compression domain information to assist inter-frame feature alignment and fusion, avoids traditional methods depending on large amount of calculation of optical flow estimation, and fundamentally reduces calculation complexity and processing delay; the application utilizes motion vectors and introduces offset prediction, realizes high-precision frame alignment, improves video quality, recovers more texture details, and greatly reduces operation amount; compared with prior art depending on high-end graphics cards, the application can realize low-delay real-time processing on edge devices, and fully meets application requirements of online video real-time transmission and display.

Description

Technical Field

[0001] This invention belongs to the field of image and video processing technology, and particularly relates to a real-time online video super-resolution method assisted by compression domain information. Background Technology

[0002] Video is a crucial carrier of information. In recent years, with technological advancements, new media, live-streaming e-commerce, online education, telemedicine, and other fields have experienced rapid growth, leading to an ever-expanding industry scale. As the user base of various video-related industries continues to grow, bandwidth resources have become a bottleneck restricting their rapid development. This makes high-bitrate video streams prone to latency or stuttering, thus impacting user experience.

[0003] To address these issues, a common solution is to first transmit low-resolution, low-definition video at a lower bitrate, and then perform super-resolution processing on the device to recover the high-resolution, high-definition image. This way, even under poor network conditions or limited bandwidth, a clearer video experience can be provided. The goal of video super-resolution is to reconstruct high-resolution frames by restoring the details missing in the low-resolution frames.

[0004] With the development of deep learning, significant progress has been made in edge video super-resolution, but many challenges remain in practical applications. Most existing algorithms are studied under ideal conditions, neglecting distortions caused by video compression coding (such as blockiness and blurring), resulting in poor performance in real-world applications. In recent years, researchers have begun to focus on the super-resolution problem of compressed videos, combining compression distortion removal with video super-resolution tasks, achieving preliminary results. For example, in the paper "COMISR: Compression-Informed Video Super-Resolution," Li et al. proposed the compressed sensing super-resolution algorithm COMISR, which consists of a bidirectional recurrent module, a detail-aware flow estimation module, and a Laplacian enhancement module, achieving good super-resolution results. In the paper "Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolution," Qiu et al. proposed FTVSR, which performs self-attention computation in the joint spatial-temporal-frequency domain, achieving even better super-resolution results.

[0005] It is worth noting that compressed domain information, such as frame type, quantization parameters, motion vectors, and residual maps, can be extracted during video decoding. This information provides additional temporal and spatial prior information, which can further optimize the performance and efficiency of super-resolution methods. In the paper "Compressed Domain Deep Video Super-Resolution," Chen et al. first proposed using compressed coding prior information to assist in the super-resolution of compressed videos, proposing the CDVSR framework. Specifically, they modulated the extracted features by using residual maps, prediction signals, and partition maps, spatially enhancing local feature differences and achieving content-adaptive feature weighting. In addition, they used motion vectors to align features of adjacent frames and compensated for motion vector errors by generating local attention offsets. This method not only improves video quality but also optimizes computational efficiency, laying the foundation for subsequent research. Building on this, Zhang et al. proposed the CIAF framework in the paper "A CodecInformation Assisted Framework for Efficient Compressed Video Super-resolution." This framework performs inter-frame alignment using motion vectors and uses residual information to dynamically adjust the computational load, thereby significantly improving the super-resolution effect while maintaining low latency. However, there are still limitations to deploying these methods on edge devices, as existing methods are difficult to meet real-time requirements due to the limited computing power of edge devices.

[0006] Existing compressed video super-resolution methods have the following drawbacks: (1) High computational complexity and processing latency. For example, the COMISR method uses a bidirectional recursive network to utilize future frame information, making inherent latency unavoidable; FTVSR performs inter-frame alignment through complex optical flow estimation, resulting in slow computation speed; although CDVSR and CIAF improve frame alignment speed by using motion vectors in the compressed domain, the video reconstruction module still has a large computational load. The above methods can achieve a maximum processing speed of 40 FPS on high-end graphics cards (such as GeForce RTX 3090), but it is difficult to implement real-time applications on actual edge devices. (2) The quality of video super-resolution still needs to be improved. CDVSR and CIAF use motion vectors in the compressed domain for frame alignment, but since motion vectors are block-based, using motion vectors for inter-frame alignment may introduce alignment errors, affecting the final super-resolution quality. In addition, when performing inter-frame feature fusion, existing methods directly convolve the previous frame features and the current frame features by channel dimension, ignoring the quality differences of frame features at different locations in space, thus affecting the super-resolution quality. Summary of the Invention

[0007] To address the problems of high processing latency caused by high computational complexity and poor quality of super-resolution results due to imperfect inter-frame feature alignment and fusion design in existing compressed video super-resolution methods, this invention provides a real-time online video super-resolution method assisted by compression domain information. This method fully utilizes compression domain information to improve alignment and fusion effects, thereby improving the reconstruction effect of super-resolution and significantly reducing computational complexity and processing latency.

[0008] A real-time online video super-resolution method assisted by compression domain information decodes a compressed video stream containing compression domain information to obtain low-resolution video frames and the corresponding compression domain information of each video frame. Then, each video frame is used as the current frame to perform super-resolution operation to obtain the super-resolution video frame corresponding to each low-resolution video frame. The compression domain information includes frame type, motion vector, and residual map. The frame type includes P-frame and I-frame. P-frame has motion vector and residual map, while I-frame does not.

[0009] The super-resolution operation is as follows: extract the frame features of the current frame and determine the frame type of the current frame. If it is an I-frame, perform feature reconstruction and upsampling on the extracted frame features to obtain the super-resolution video frame corresponding to the current frame. If it is a P-frame, take the motion vector corresponding to the current frame as the reference, align the reconstructed features obtained when performing feature reconstruction on the previous frame with the frame features extracted in the current frame, then use the residual map corresponding to the current frame to fuse the aligned reconstructed features of the previous frame and the frame features extracted in the current frame, and finally perform feature reconstruction and upsampling on the fused frame features to obtain the super-resolution video frame corresponding to the current frame.

[0010] Furthermore, if the current frame is a P-frame, the method for aligning the reconstructed features obtained during feature reconstruction of the previous frame with the frame features extracted in the current frame, using the motion vector corresponding to the current frame as a reference, is as follows:

[0011] Use the motion vector corresponding to the current frame as the basic offset Δp base ;

[0012] The reconstructed features F obtained during feature reconstruction of the previous frame. t-1 Based on the basic offset Δp base The distortion is performed to obtain the reconstructed features F of the previous frame after preliminary alignment. t-1 ;

[0013] Reconstruct the features F from the previous frame t-1 The frame features F extracted from the current frame t Concatenate along the channel dimension to form a joint feature [F] t-1 ;F t ];

[0014] Joint features [F]t-1 ;F t Input the pre-trained offset prediction module to obtain F t-1 With F t residual offset Δp res ;

[0015] The base offset Δp base With residual offset Δp res Adding each element together, we obtain the final learnable offset Δp = Δp base +Δp res ;

[0016] Deformable convolution pairs with a learnable offset Δp are used to reconstruct features F. t-1 Perform deformable convolution operations to obtain the final aligned reconstructed features F from the previous frame. t-1 .

[0017] Furthermore, the offset prediction module consists of multiple convolutional layers, and each convolutional layer is nonlinearly transformed using the ReLU activation function.

[0018] Furthermore, the method of fusing the aligned reconstructed features of the previous frame and the frame features extracted from the current frame using the residual map corresponding to the current frame is as follows:

[0019] Reconstruct the aligned features F from the previous frame. t-1 The frame features F extracted from the current frame t and the residual map R corresponding to the current frame t By concatenating along the channel dimension, we obtain the joint feature U = [F]. t-1 ;F t ;R t ];

[0020] The joint feature U is input into the pre-trained gating parameter learning module to obtain the single-channel gating original image G;

[0021] Reconstructing features F from the previous frame using the single-channel gated original image G t-1 The frame features F extracted from the current frame t We perform weighted fusion to obtain the fused frame features F. fuse =G⊙F t +(1-G)⊙F t-1 , where ⊙ represents element-wise multiplication.

[0022] Furthermore, the gating parameter learning module consists of multiple 1*1 convolutional layers and one Sigmoid activation function. The convolutional layers use the ReLU nonlinear activation function for nonlinear transformation, and the use of the Sigmoid activation function ensures that G∈[0,1].

[0023] Furthermore, the upsampling method is as follows:

[0024] The features obtained after feature reconstruction are transformed into RGB image space using one convolutional layer and one pixel shuffle layer. The image resolution of the feature map transformed into RGB image space is then upscaled to the required level, and finally, a super-resolution video frame is generated.

[0025] Furthermore, a pre-trained feature extraction module is used to extract frame features of the current frame. The feature extraction module consists of m convolutional blocks, where 1≤m≤5. Each convolutional block includes two 3×3 convolutional layers, and the non-linear activation function ReLU is used to perform non-linear transformation between each convolutional layer.

[0026] Furthermore, a pre-trained feature reconstruction module is used for feature reconstruction, wherein the feature reconstruction module consists of multiple base blocks, and the base blocks are convolutional blocks or RSTB in SwinIR.

[0027] Beneficial effects:

[0028] This invention provides a real-time online video super-resolution method assisted by compressed domain information. By fully utilizing compressed domain information to assist in inter-frame feature alignment and fusion, it avoids the traditional reliance on computationally intensive optical flow estimation methods, fundamentally reducing computational complexity and processing latency. This invention utilizes motion vectors and introduces offset prediction, achieving not only high-precision frame alignment, improved video quality, and restoration of more texture details, but also a significant reduction in computational load. Compared to existing technologies that rely on high-end graphics cards, this invention can achieve low-latency real-time processing on edge devices, fully meeting the application requirements of real-time online video transmission and display. Attached Figure Description

[0029] Figure 1 A flowchart of the real-time online video super-resolution method assisted by compression domain information provided by the present invention;

[0030] Figure 2 A structural diagram of the motion vector-assisted frame feature alignment module provided by this invention;

[0031] Figure 3 The structural diagram of the gated fusion module with residual map assistance provided by the present invention;

[0032] Figure 4 The principle block diagram of the real-time online video super-resolution method assisted by compression domain information provided by the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the video super-resolution method designed in this invention is used on the edge side. The video frames received on the edge side only consider compression distortion and do not consider distortions such as noise and flicker. Since it is applied to online scenarios, the video frame types only include I-frames and P-frames, and B-frames are not present.

[0035] Based on this, the present invention provides a real-time online video super-resolution method assisted by compression domain information, such as... Figure 1 As shown, the compressed video stream containing compression domain information is decoded to obtain low-resolution video frames and the compression domain information corresponding to each video frame. Then, each video frame is used as the current frame to perform super-resolution operation to obtain the super-resolution video frame corresponding to each low-resolution video frame. The compression domain information includes frame type, motion vector, and residual map. The frame type includes P-frame and I-frame. P-frame has motion vector and residual map, while I-frame does not have motion vector and residual map.

[0036] The super-resolution operation is as follows: extract the frame features of the current frame and determine the frame type of the current frame. If it is an I-frame, perform feature reconstruction and upsampling on the extracted frame features to obtain the super-resolution video frame corresponding to the current frame. If it is a P-frame, take the motion vector corresponding to the current frame as the reference, align the reconstructed features obtained when performing feature reconstruction on the previous frame with the frame features extracted in the current frame, then use the residual map corresponding to the current frame to fuse the aligned reconstructed features of the previous frame and the frame features extracted in the current frame, and finally perform feature reconstruction and upsampling on the fused frame features to obtain the super-resolution video frame corresponding to the current frame.

[0037] The following section provides a detailed explanation of the real-time online video super-resolution method assisted by compression domain information, using 4x super-resolution and H.264 encoding as examples. Figure 4 As shown.

[0038] (1) Obtain the compressed video stream.

[0039] The compressed video stream is obtained by downsampling and compressing the original high-resolution, high-quality video frames. This invention uses the REDS dataset to generate training and test sets. The REDS dataset consists of 240 training video sequences, 30 validation video sequences, and 30 test video sequences. Each video sequence consists of 100 consecutive video frames with a resolution of 720*1280. This invention combines the original training and validation sets as the training set and selects four video sequences ("000", "011", "015", and "020") from the original training set as the test set, named REDS4. First, MATLAB is used to downsample 100 consecutive video frames from the 270 video sequences available in REDS by a factor of 4, resulting in consecutive video frames with a resolution of 180*320. Then, the H.264 encoder of FFmpeg software is used to compress and encode the downsampled video frames to obtain the compressed video stream.

[0040] (2) Decode the compressed video stream to obtain low-resolution video frames and compressed domain information (frame type, motion vector, residual map, etc.).

[0041] Considering the lack of a suitable dataset, this invention designs and generates a video super-resolution dataset containing compression domain information. The compressed video stream obtained in step (1) is decoded using FFmpeg software to obtain low-resolution compressed video frames. Simultaneously, compression domain information, including frame type, motion vectors, and residual maps, is extracted. The compressed video frames and compression domain information are then matched with the 720*1280 video frames from step (1) to form the dataset. In summary, the dataset contains high-resolution clear video frames, low-resolution compressed video frames, frame types, motion vectors (present in P-frames, absent in I-frames), and residual maps (present in P-frames, absent in I-frames).

[0042] (3) Extract video frame features.

[0043] Low-resolution compressed video frames are input into the feature extraction module for video frame feature extraction. The feature extraction module consists of m (1≤m≤5) convolutional blocks. Each convolutional block includes two 3×3 convolutional layers, and a ReLU activation function is used for non-linear transformation between the convolutional layers.

[0044] (4) Determine the frame type of the current frame.

[0045] Determine the current frame type. If it is an I-frame, proceed directly to step (7) for feature reconstruction; if it is a P-frame, proceed to steps (5) and (6) for frame feature alignment and fusion.

[0046] (5) Input the reconstructed features obtained when reconstructing the previous frame into the motion vector-assisted frame feature alignment module to obtain the aligned reconstructed features of the previous frame.

[0047] To fully utilize inter-frame redundancy information and mitigate feature misalignment caused by cross-frame geometric changes, this embodiment uses the reconstructed feature F obtained during feature reconstruction of the previous frame after reconstruction. t-1 The frame features F extracted from the current frame t Together they are input into the motion vector-assisted frame feature alignment module, such as Figure 2 As shown, the specific process is as follows.

[0048] (5a) Motion vector M obtained by the video decoder t←t-1 This describes the displacement from the previous frame to the current frame, providing prior information for subsequent alignment. The module first assigns the motion vector M corresponding to the current frame... t←t-1 Extended to the feature space, as the basic offset Δp base .

[0049] (5b) To reduce the difficulty of subsequent learning, F t-1 Press Δp base The distortion is performed to obtain the reconstructed features F of the previous frame after preliminary alignment. t-1 Since the motion vector is block-based, there is some error involved. This operation roughly compensates for the overall displacement, but it cannot handle local motion.

[0050] (5c) F t-1 and F t Concatenate along the channel dimension to form a joint feature [F] t-1 ;F t ].

[0051] (5d) The joint features are processed by the offset prediction module to obtain the residual offset Δp. res The residual offset Δp res It describes the difference between the actual pixel displacement and the motion vector, and can adaptively capture complex local motions. The offset prediction module consists of n convolutional layers, and the convolutional layers are non-linearly transformed using the ReLU activation function.

[0052] (5e) Set the base offset Δp base With residual offset Δp res Adding each element together, we obtain the final learnable offset Δp = Δp base +Δp res .

[0053] (5f) In deformable convolution, Δp is used as the sampling shift for F t-1 Perform deformable convolution to generate aligned reconstructed features F from the previous frame. t-1Compared to the fixed sampling grid of traditional convolution, deformable convolution allows each convolution kernel to dynamically adjust its sampling position according to Δp, thereby accurately aligning the edges and textures of objects with nonlinear deformation.

[0054] In other words, this invention inputs the reconstructed features of the previous frame into a motion vector-assisted frame feature alignment module. The frame feature alignment module uses deformable convolution for frame feature alignment, uses motion vectors as the base values ​​for offsets, and uses an offset prediction module to learn the residual between the true offset and the motion vector; the final output is the reconstructed features F of the previous frame. t-1 The frame features F extracted from the current frame t Alignment within the same spatial coordinate system significantly enhances the complementarity of cross-frame information.

[0055] (6) Reconstruct the features F of the aligned previous frame t-1 The frame features F extracted from the current frame t The input is fed into the residual map-assisted gated fusion module to obtain the fused features.

[0056] This embodiment addresses the quality differences of frame features at different locations in the frame space between consecutive frames. Based on the characteristic that residual maps can reflect local differences between consecutive frames after alignment, a residual map-assisted gated fusion module is designed to adaptively integrate the reconstructed features F of the previous frame after alignment. t-1 The frame features F extracted from the current frame t Its overall structure is as follows Figure 3 As shown, the specific process is as follows:

[0057] (6a) will With residual plot By splicing along the channel dimension, a joint feature is formed. Among them, R t It can be obtained during the video decoding process.

[0058] (6b) The joint feature U is processed by the gating parameter learning module to output the single-channel gating original image. The gating parameter learning module consists of n 1*1 convolutional layers and one sigmoid activation function. The ReLU function is used in the middle of the convolutional layers for nonlinear transformation. The use of the sigmoid activation function ensures that G∈[0,1].

[0059] (6c) Reconstruct feature F from the previous frame using G t-1 The frame features F extracted from the current frame t Perform weighted fusion to obtain the fused feature F. fuse =G⊙F t +(1-G)⊙F t-1 , where ⊙ represents element-wise multiplication. The larger G is, the more likely it is to retain current features; conversely, the smaller G is, the more it emphasizes historical features.

[0060] The difference between two frames in the feature domain is explicitly measured by the residual map. Then, a gating network generates weights G pixel by pixel, which can dynamically determine whether to "preserve history" or "rely on the present" for each spatial location. In areas with large residuals (usually corresponding to misalignment, occlusion, or violent movement), the gating weight G is increased and 1-G is decreased, thereby suppressing historical information that is misaligned. In static or aligned areas, G is smaller, which can make full use of historical features with high signal-to-noise ratio and improve the quality of detail restoration.

[0061] (7) Input the frame features / fused frame features into the feature reconstruction module to obtain the reconstructed frame features.

[0062] If the current frame is an I-frame, its frame features are input to the feature reconstruction module; if the current frame is a P-frame, the fused frame features are input to the feature reconstruction module. The feature reconstruction module consists of q basic blocks, which can be convolutional blocks or RSTB blocks from SwinIR, etc. After processing by the feature reconstruction module, the reconstructed frame features are obtained.

[0063] (8) Input the reconstructed frame features into the upsampling module to obtain the super-resolution video frame.

[0064] The upsampling module consists of one convolutional layer and one pixel shuffle layer. The reconstructed frame features are input into the upsampling module. First, a 3*3 convolutional layer is used to transform the feature map dimensions to 48×H×W. Then, the pixel shuffle layer is used to transform the reconstructed feature map to RGB space and upscale the image resolution to the required level, that is, to transform the dimensions to 3×4H×4W, thus obtaining the super-resolution video frame.

[0065] The aforementioned real-time online video super-resolution method assisted by compression domain information requires training. Charbonnier loss is used as the loss function, Adam is chosen as the training iterator, and the batch size is set to 8, for a total of 6 × 10⁻⁶ iterations. 5 Second-rate.

[0066] In summary, to address the issues of errors in block-based motion vector alignment and the neglect of spatial information quality differences in direct splicing of inter-frame features in existing methods, this invention uses compressed domain information such as frame type, motion vectors, and residual maps to assist in video super-resolution. This can achieve high-quality super-resolution results on edge devices. The core lies in the design of the motion vector-assisted frame feature alignment module and the residual map-assisted gating parameter learning module.

[0067] The motion vector-assisted frame feature alignment module consists of a warping operation, a channel stitching operation, an offset prediction module, and a deformable convolution. After using motion vectors to perform preliminary alignment of the reconstructed features of the previous frame, the preliminarily aligned reconstructed features of the previous frame and the frame features extracted from the current frame are input into the offset prediction module to learn the residual between the true offset and the motion vector. Based on the offset, deformable convolution is used to obtain the aligned frame features.

[0068] The residual map-assisted gated fusion module consists of channel stitching operations, a gate parameter learning module, and a weighted fusion operation. After stitching the aligned reconstructed features from the previous frame, the frame features extracted from the current frame, and the residual map through channels, the gate parameter learning module learns the gate parameter G. Based on the gate parameter G, the aligned reconstructed features from the previous frame and the frame features extracted from the current frame are fused to obtain the fused frame features.

[0069] In other words, the offset prediction module introduced in this invention can automatically learn and correct the residual between the motion vector and the actual offset, thereby using deformable convolution for feature alignment and effectively mitigating alignment errors. At the same time, the gated fusion module uses the residual map to adaptively adjust the weights G of each branch at different spatial locations, thereby obtaining high-quality fusion features and ensuring that the reconstructed video frames have better performance in texture and detail restoration.

[0070] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A real-time online video super-resolution method assisted by compressed domain information, characterized in that, The compressed video stream containing compression domain information is decoded to obtain low-resolution video frames and the corresponding compression domain information of each video frame. Then, each video frame is used as the current frame to perform super-resolution operation to obtain the super-resolution video frame corresponding to each low-resolution video frame. The compression domain information includes frame type, motion vector, and residual map. The frame type includes P-frame and I-frame. P-frame has motion vector and residual map, while I-frame does not have motion vector and residual map. The super-resolution operation is as follows: extract the frame features of the current frame and determine the frame type of the current frame. If it is an I-frame, perform feature reconstruction and upsampling on the extracted frame features to obtain the super-resolution video frame corresponding to the current frame. If it is a P-frame, take the motion vector corresponding to the current frame as the reference, align the reconstructed features obtained when performing feature reconstruction on the previous frame with the frame features extracted in the current frame, then use the residual map corresponding to the current frame to fuse the aligned reconstructed features of the previous frame and the frame features extracted in the current frame, and finally perform feature reconstruction and upsampling on the fused frame features to obtain the super-resolution video frame corresponding to the current frame. If the current frame is a P-frame, the method for aligning the reconstructed features obtained from feature reconstruction in the previous frame with the frame features extracted in the current frame, using the motion vector corresponding to the current frame as a reference, is as follows: Use the motion vector corresponding to the current frame as the base offset. ; The reconstructed features obtained during feature reconstruction of the previous frame Based on the base offset The distortion is performed to obtain the reconstructed features of the previous frame after preliminary alignment. ; Reconstruct features from the previous frame Frame features extracted from the current frame By splicing along the channel dimension, a joint feature is formed. ; Joint features Input the pre-trained offset prediction module to obtain and residual offset between ; base offset With residual offset Adding each element together yields the final learnable offset. ; Using sampled displacement as a learnable offset Deformable convolution for reconstructing features Perform deformable convolution operations to obtain the final aligned reconstructed features from the previous frame. .

2. The real-time online video super-resolution method assisted by compression domain information as described in claim 1, characterized in that, The offset prediction module consists of multiple convolutional layers, and each convolutional layer is nonlinearly transformed using the ReLU activation function.

3. The real-time online video super-resolution method assisted by compression domain information as described in claim 1, characterized in that, The method for fusing the aligned reconstructed features of the previous frame and the frame features extracted from the current frame using the residual map corresponding to the current frame is as follows: Reconstruct features from the aligned previous frame Frame features extracted from the current frame and the residual map corresponding to the current frame By concatenating along the channel dimension, joint features are obtained. ; Joint features Input the pre-trained gating parameter learning module to obtain the original single-channel gating image. ; Using single-channel gating original image Reconstructing features from the previous frame Frame features extracted from the current frame Perform weighted fusion to obtain the fused frame features. ,in, Represents element-wise multiplication.

4. The real-time online video super-resolution method assisted by compression domain information as described in claim 3, characterized in that, The gating parameter learning module consists of multiple 1x1 convolutional layers and one sigmoid activation function. The convolutional layers use the ReLU activation function for non-linear transformation, and the use of the sigmoid activation function ensures... .

5. The real-time online video super-resolution method assisted by compression domain information as described in claim 1, characterized in that, The upsampling method is as follows: The features obtained after feature reconstruction are transformed into RGB image space using one convolutional layer and one pixel shuffle layer. The image resolution of the feature map transformed into RGB image space is then upscaled to the required level, and finally, a super-resolution video frame is generated.

6. The real-time online video super-resolution method assisted by compression domain information as described in claim 1, characterized in that, A pre-trained feature extraction module is used to extract frame features of the current frame. The feature extraction module consists of... Composed of several convolutional blocks, Each convolutional block consists of two Convolutional layers are used, and nonlinear transformations are performed between each convolutional layer using the ReLU activation function.

7. The real-time online video super-resolution method assisted by compression domain information as described in claim 1, characterized in that, Feature reconstruction is performed using a pre-trained feature reconstruction module, which consists of multiple base blocks, and the base blocks are convolutional blocks or RSTB in SwinIR.

Images