A Multi-Scale Spatial Optimization Video Super-Resolution Method
This video super-resolution method, which employs adaptive multi-scale feature extraction and spatial alignment optimization, addresses the inconvenience and inefficiency of multi-scale feature extraction in existing technologies. It achieves efficient restoration of video scene details while reducing computational load, making it suitable for video super-resolution tasks with high real-time requirements.
Patent Information
- Application Number
- CN202510149404.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-11
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2045-02-11
AI Technical Summary
Existing video super-resolution techniques are neither simple nor efficient in multi-scale feature extraction, resulting in poor restoration of video scene details.
An adaptive multi-scale feature extraction and spatial alignment optimization method is adopted. The feature maps are aligned and fused frame by frame through a recurrent neural network. The video super-resolution algorithm is optimized by combining shallow feature maps and multi-scale feature maps.
It improves the performance of video scene detail recovery, reduces noise interference, and has a lower computational cost than traditional methods, making it suitable for video super-resolution tasks with high real-time requirements.
Smart Images
Figure CN119831848B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of super-resolution technology, and specifically relates to a multi-scale spatial optimization video super-resolution method. Background Technology
[0002] The pursuit of high-definition video is an important goal in life, especially in key areas. For example, in medical image analysis, clear video helps improve diagnostic accuracy; in traffic management, high-definition video enables more accurate assessment of traffic flow and density. Video super-resolution technology, which converts low-definition video into high-definition video, has made significant progress in recent years driven by deep learning. Through neural network-based methods, it has achieved superior video reconstruction results compared to traditional techniques. Despite significant progress in image super-resolution, the development of video super-resolution still faces challenges, mainly because current methods for multi-scale feature extraction are neither simple nor efficient, resulting in the need for further optimization and improvement in capturing scene details within videos.
[0003] Therefore, this study introduces a multi-scale spatially optimized video super-resolution technique. The core of this technique lies in accurately extracting details at various scales from videos in a simple yet efficient manner, thereby enhancing the performance of video super-resolution technology in restoring scene details. Summary of the Invention
[0004] To overcome the problems in the prior art, this invention discloses a multi-scale spatial optimization video super-resolution method, which introduces adaptive multi-scale feature extraction and spatial alignment optimization, and designs an improved video super-resolution algorithm to achieve effective and efficient spatiotemporal information extraction, thereby further improving the quality of high-resolution videos.
[0005] The technical solution adopted by this invention to solve its technical problem is as follows:
[0006] On one hand, this invention proposes a multi-scale spatial optimization video super-resolution method, including:
[0007] S1: Obtain the video sequence data to be processed;
[0008] S2: Extract shallow feature maps for each video frame in the video sequence data;
[0009] S3: For the shallow feature maps of each video frame, further extract the multi-scale feature maps of each video frame through adaptive multi-scale feature extraction.
[0010] S4: The multi-scale feature maps of each video frame are used as input to the recurrent neural network in sequence, and spatial optimization alignment and fusion are performed frame by frame to obtain a one-way super-resolution result.
[0011] S5: Input the multi-scale feature maps of each video frame into the recurrent neural network in reverse, and perform spatially optimized alignment and fusion frame by frame to obtain the super-resolution result in the reverse direction;
[0012] S6: The super-resolution results in one direction and the super-resolution results in the opposite direction are fused together, and then super-resolution reconstruction is performed to obtain a high-resolution video, thus completing the super-resolution of the video sequence data.
[0013] Furthermore, S3 includes:
[0014] S3.1: Divide the shallow feature map into C feature blocks X of different channels according to the channel dimension. i , 0≤i≤C-1;
[0015] S3.2: For feature blocks X with C different channels i Feature block X0 is not processed, while the other feature blocks are downsampled by different multiples.
[0016] S3.3: Extract depth feature blocks from feature block X0 and the remaining downsampled feature blocks using depthwise convolution.
[0017] S3.4: Depth Feature Block Without processing, the remaining depth feature blocks are upsampled to depth feature blocks at different multiples. The size of the upsampled feature block and the feature block New feature map obtained by splicing
[0018] S3.5: Transfer the new feature map After channel fusion, the feature map is multiplied with the shallow feature map to obtain a multi-scale feature map.
[0019] Furthermore, S4 specifically includes:
[0020] S4.1: Obtain the multi-scale feature map of the first frame as the feature map to be aligned. The feature map to be aligned The next frame is used as the current frame;
[0021] S4.2: For the feature maps to be aligned An optical flow alignment method is used to align it with the multi-scale feature map of the current frame. Alignment is performed to obtain coarsely aligned feature maps.
[0022] S4.3: Construct a coarsely aligned feature map based on the optical flow information during optical flow alignment. Each pixel Feature map to be aligned Each pixel The correspondence;
[0023] S4.4: Preparing the feature map for alignment Each pixel Create a window of the same size with the center point as the center, and use all pixels within the window as its center pixel. The neighboring pixels;
[0024] S4.5: Calculate the coarsely aligned feature map medium pixel and corresponding pixel The similarity of all neighboring pixels is used to select the neighboring pixel with the highest similarity as pixel I(x, y) in the spatially optimized aligned feature map. This process is repeated for each coarsely aligned feature map. By analyzing all pixels in the dataset, we can obtain the single-frame super-resolution result for the current frame.
[0025] S4.6 uses the single-frame super-resolution result of the current frame as the feature map to be aligned. The updated value is used to sequentially execute S4.2-S4.5 on all video frames to obtain the unidirectional super-resolution result.
[0026] Furthermore, the spatially optimized alignment fusion described in S4 and S5 employs the same method.
[0027] Furthermore, the super-resolution results in one direction and the super-resolution results in the opposite direction are fused into a single image stitched along the channel dimension.
[0028] On the other hand, the present invention also proposes a multi-scale spatial optimization video super-resolution system for implementing the above-mentioned method.
[0029] The beneficial effects of this invention are as follows:
[0030] This invention designs a multi-scale spatially optimized video super-resolution method, primarily addressing the common shortcomings of current super-resolution models in multi-scale feature extraction and their weakness in restoring partial texture details. It proposes adaptive multi-scale feature extraction and spatial optimization alignment methods. These two methods further optimize video detail features without introducing noise like bilinear methods used in traditional super-resolution models, making them specifically designed to handle multi-scale texture details. Furthermore, this invention has a significantly lower computational cost than the U-Net method used in current mainstream multi-scale extraction models, making it more suitable for video super-resolution with higher real-time requirements. Attached Figure Description
[0031] Figure 1This is a flowchart of the multi-scale spatial optimization video super-resolution method used in the implementation of this invention.
[0032] Figure 2 This is a structural block diagram of adaptive multi-scale feature extraction during the implementation of this invention.
[0033] Figure 3 This is a network structure diagram for spatial optimization alignment during the implementation of this invention. Detailed Implementation
[0034] The method of the present invention will be further described below with reference to the accompanying drawings.
[0035] A flowchart of a multi-scale spatial optimization video super-resolution method of the present invention is shown below. Figure 1 As shown, it includes the following steps:
[0036] Step 1: Acquire low-resolution video sequence data;
[0037] Step 2: Extract shallow features from the video sequence using a network.
[0038] Step 3: Extract multi-scale features from the video sequence using an adaptive multi-scale feature extraction method.
[0039] Step 4: Introduce multi-scale information into the recurrent neural network and perform spatial optimization alignment and fusion with the previous state of the recurrent neural network, i.e., the feature map of the single-frame super-resolution result.
[0040] Step 5: Repeat steps 2-4 to traverse each frame in the video sequence and obtain the super-resolution result in one direction;
[0041] Step 6: Reverse process each frame in the video sequence to obtain the super-resolution result in the reverse direction;
[0042] Step 7: Fuse the unidirectional super-resolution result output in Step 5 with the reverse-direction super-resolution result output in Step 6, and then perform super-resolution reconstruction to complete the output of high-resolution frame video.
[0043] In this embodiment, the low-resolution video sequence data obtained in step 1 is obtained by downsampling high-resolution video for testing the super-resolution method. In practical applications, the low-resolution video to be optimized can be used directly.
[0044] In step 2, shallow feature extraction of the video sequence is performed. Specifically, for the obtained low-resolution video sequence data, shallow features of the video sequence are extracted through the residual blocks of the residual network, including shallow feature maps of video frames.
[0045] In step 3, multi-scale information of the video sequence is extracted using an adaptive multi-scale feature extraction method, such as... Figure 2 As shown, the specific steps include the following:
[0046] Step 3.1: First, denote the shallow feature map as X. T , feature map X T It is divided into four parts according to the channel dimension: X0, X1, X2, and X3.
[0047] [X0, X1, X2, X3] = Chunk(X T )
[0048] Step 3.2: No processing is performed on feature map X0. The other three feature maps X1, X2, and X3 are downsampled by 2x, 4x, and 8x respectively. In this embodiment, the downsampling is performed through a max pooling layer. All features are then extracted using depthwise convolution. In this embodiment, a 3×3 convolution is used, and the extracted deep feature maps are denoted as...
[0049]
[0050]
[0051] Where Conv 3×3 This represents a 3×3 convolution. Indicates 2 i The downsampling is multiplied by a factor of 1. The downsampling method is nearest neighbor interpolation, which is done to prevent other interpolation information from introducing noise that affects the accuracy of multi-scale feature extraction.
[0052] Step 3.3: Incorrect The features are processed using nearest neighbor interpolation. Upsampled feature map The size is used to obtain four feature maps with the same length and width. These feature maps are concatenated to form a new feature map. New feature map obtained by splicing With shallow feature map X T Same size:
[0053]
[0054]
[0055] in Indicates 2 i The upsampling is multiplied by a factor of 1, and Concat represents the concatenation of feature maps.
[0056] Step 3.4: Transfer the new feature map Channel fusion is performed using an l×l convolution and activation function to obtain a feature map containing multi-scale feature information. Feature maps containing multi-scale feature information As an attention matrix, let the original feature map X T Adaptive adjustment of feature information, focusing on information at different scales, yields adaptive multi-scale feature information.
[0057]
[0058] Where Conv 1×1 Let represent an l×l convolution, and Tanh represent the activation function. This step obtains multi-scale information through an adaptive method of multiplication. The purpose of this is to avoid information loss that occurred in previous processes, and to prevent this information loss from interfering with the final result.
[0059] This step extracts multi-scale information adaptively, with its biggest advantage being computational cost. The adaptive approach has a significant computational advantage over the typical multi-scale information extraction model, U-Net.
[0060] Step 4 introduces the obtained multi-scale features into the recurrent neural network, and performs spatial optimization alignment and fusion with the previous states of the recurrent neural network, such as... Figure 3 As shown, the specific steps include the following:
[0061] Step 4.1: Obtain the processing result of the recurrent neural network on the previous video frame, and denote it as the feature map to be aligned. Multiscale features used with the current video frame Perform alignment and fusion; if there is no previous video frame, obtain the multi-scale features of the current frame and record them as follows.
[0062] Step 4.2: For the feature maps to be aligned First, feature maps are extracted using conventional optical flow alignment. The optical flow information is used to analyze the feature map. Multi-scale features of the current frame Alignment is performed to obtain coarsely aligned feature maps.
[0063]
[0064] Warp stands for optical flow alignment.
[0065] Obtain feature map The objectives are twofold: first, to determine the positional information of each pixel in the aligned target feature map I obtained through alignment optimization; and second, to coarsely align the feature map. Information for each pixel Q serves as the query vector in the attention mechanism:
[0066]
[0067] Where q is a unit vector.
[0068] Step 4.3: Based on the position information of each pixel in feature map I and the optical flow information used in the previous conventional optical flow alignment, backtrack from each pixel I(x, y) in the target feature map to obtain the feature map to be aligned. The position of each pixel dx and dy are obtained from optical flow information. (Regarding the feature map...) Each pixel in the data is represented by a pixel. Create an h×w window M centered at the center, and collect information about all pixels in this window. Each pixel in window M is denoted as M(i,j) and is called a pixel. The neighboring pixels, where i∈{0,1…,h}, j∈{0,1…,w}:
[0069]
[0070] Information from all pixels in window M will serve as the key vector K and value vector V in the attention mechanism:
[0071] K = Mk
[0072] V = Mv
[0073] Where k and v are unit vectors.
[0074] Step 4.4: Through the self-attention mechanism, each pixel can be obtained. and corresponding pixel The similarity matrix of all neighboring pixels, preferably cosine similarity, is used to select the neighboring pixel pair with the highest similarity. Alignment is performed, and for each pixel, the optimized alignment result I(x, y) is obtained, as shown in formula 4-8:
[0075] I(x, y) = Max(A(Q, K))V
[0076] Where A(Q, K) represents the similarity matrix between query vector Q and key vector K.
[0077] The improvement in step 4 lies in optimizing the resampling process. In existing common methods, bilinear interpolation introduces new noise, while nearest neighbor interpolation may not select the most suitable pixel for alignment. Step 4 of this method is an improvement based on the nearest neighbor interpolation method. Through the above method, each pixel can select the most ideal result during the alignment process.
[0078] Step 5: Repeat step 4 to traverse each frame in the video sequence and obtain the super-resolution result in one direction;
[0079] Step 6: Following the method in Step 5, traverse the video sequence in reverse to obtain the super-resolution result in the opposite direction; specifically, this includes:
[0080] Obtain the super-resolution result of the next frame output from the previous round of reverse traversal, and denote it as the feature map to be aligned. Multiscale features for the current frame Perform alignment and fusion; if there is no previous reverse traversal round (i.e., the current round is the first round, and the current frame is the last frame of the video), then obtain the multi-scale features of the current frame and denote them as follows.
[0081] Follow the steps 4.2-4.4 to extract the feature map. Multi-scale feature map of the current frame Alignment and fusion are performed to obtain the reverse super-resolution result of a single frame. The video sequence is traversed from back to front, and all the reverse super-resolution results of a single frame are arranged in the original video frame order to obtain the reverse super-resolution result. Step 7: The single-direction super-resolution result output in step 5 and the reverse super-resolution result output in step 6 are fused, and then super-resolution reconstruction is performed to complete the output of high-resolution frame video.
[0082] The fusion process specifically involves directly stitching together the unidirectional and reverse-directional super-resolution results of the same video frame along the channel dimension to obtain a bidirectional super-resolution result.
[0083] The super-resolution results of bidirectional fusion are still at the feature level, so it is necessary to reconstruct the bidirectional super-resolution results frame by frame into images. The single-frame image reconstruction method is a well-known method in the field, and a high-resolution video sequence is obtained. The above method is called super-resolution reconstruction.
[0084] This embodiment also provides a multi-scale spatially optimized video super-resolution system for implementing the above embodiments. The terms "module," "unit," etc., used below refer to combinations of software and / or hardware that perform the intended function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.
[0085] The data input module is used to acquire the video sequence data to be processed;
[0086] The shallow feature extraction module is used to extract shallow feature maps for each video frame;
[0087] The multi-scale feature extraction module is used to further extract multi-scale feature maps for each video frame;
[0088] The recurrent neural network module is used to perform spatial optimization alignment and fusion of multi-scale feature maps of each video frame frame by frame to obtain unidirectional super-resolution results and reverse-directional super-resolution results.
[0089] The image reconstruction module is used to fuse the super-resolution results in one direction and the super-resolution results in the opposite direction, and then perform super-resolution reconstruction to obtain a high-resolution video.
[0090] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0091] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.
[0092] The above description is merely a specific embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A multi-scale spatial optimization video super-resolution method, characterized in that, include: S1: Obtain the video sequence data to be processed; S2: Extract shallow feature maps for each video frame in the video sequence data; S3: For the shallow feature maps of each video frame, further extract multi-scale feature maps of each video frame using an adaptive multi-scale feature extraction method, specifically including: S3.1: Divide the shallow feature map into C feature blocks X of different channels according to the channel dimension. i ,0≤i≤C-1; S3.2: For feature blocks X with C different channels i Feature block X0 is not processed, while the other feature blocks are downsampled by different multiples. S3.3: Extract depth feature blocks from feature block X0 and the remaining downsampled feature blocks using depthwise convolution. S3.4: Depth Feature Block Without processing, the remaining depth feature blocks are upsampled to depth feature blocks at different multiples. The size of the upsampled feature block and the feature block New feature map obtained by splicing S3.5: Transfer the new feature map After channel fusion, the feature map is multiplied with the shallow feature map to obtain a multi-scale feature map; S4: The multi-scale feature maps of each video frame are sequentially used as input to the recurrent neural network. Spatial optimization alignment and fusion are performed frame by frame to obtain a unidirectional super-resolution result, specifically including: S4.1: Obtain the multi-scale feature map of the first frame as the feature map to be aligned. The feature map to be aligned The next frame is used as the current frame; S4.2: For the feature maps to be aligned An optical flow alignment method is used to align it with the multi-scale feature map of the current frame. Alignment is performed to obtain coarsely aligned feature maps. S4.3: Construct a coarsely aligned feature map based on the optical flow information during optical flow alignment. Each pixel Feature map to be aligned Each pixel The correspondence; S4.4: Preparing the feature map for alignment Each pixel Create a window of the same size with the center point as the center, and use all pixels within the window as its center pixel. The neighboring pixels; S4.5: Calculate the coarsely aligned feature map medium pixel and corresponding pixel The similarity of all neighboring pixels is used to select the neighboring pixel with the highest similarity as pixel I(x,y) in the spatially optimized aligned feature map. This process is repeated for each coarsely aligned feature map. By analyzing all pixels in the dataset, we can obtain the single-frame super-resolution result for the current frame. S4.6: Use the single-frame super-resolution result of the current frame as the feature map to be aligned. The updated value is used to sequentially execute S4.2-S4.5 on all video frames to obtain the unidirectional super-resolution result; S5: Input the multi-scale feature maps of each video frame into the recurrent neural network in reverse, and perform spatially optimized alignment and fusion frame by frame to obtain the super-resolution result in the reverse direction; S6: The super-resolution results in one direction and the super-resolution results in the opposite direction are fused together, and then super-resolution reconstruction is performed to obtain a high-resolution video, thus completing the super-resolution of the video sequence data.
2. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that, The shallow feature map extraction uses a residual block network.
3. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that, In S3.2, feature block X0 is not processed, while the remaining feature blocks X... i Perform 2 separately i Downsampling by a factor of 1.
4. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that, Both the downsampling described in S3.2 and the upsampling described in S3.4 use the nearest neighbor interpolation method.
5. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that, The channel fusion described in S3.5 specifically involves using a 1×1 convolution and activation function to fuse the channels.
6. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that, The spatial optimization alignment fusion described in S4 and S5 uses the same method.
7. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that, Step S6 involves fusing the super-resolution results in one direction and the super-resolution results in the opposite direction into a stitched result along the channel dimension.
8. A multi-scale spatial optimization video super-resolution system, used to implement the method of claim 1, characterized in that, The system includes: The data input module is used to acquire the video sequence data to be processed; The shallow feature extraction module is used to extract shallow feature maps for each video frame; The multi-scale feature extraction module is used to further extract multi-scale feature maps for each video frame; The recurrent neural network module is used to perform spatial optimization alignment and fusion of multi-scale feature maps of each video frame frame by frame to obtain unidirectional super-resolution results and reverse-directional super-resolution results. The image reconstruction module is used to fuse the super-resolution results in one direction and the super-resolution results in the opposite direction, and then perform super-resolution reconstruction to obtain a high-resolution video.