Multi-scale space optimization video super-resolution method

Abstract

The invention discloses a multi-scale space optimization video super-resolution method, and belongs to the technical field of image super-resolution. The method comprises the following steps: acquiring low-resolution video sequence data; extracting a shallow feature map of the video sequence through a network; extracting a multi-scale feature map of the video sequence through a self-adaptive multi-scale feature extraction mode; introducing the multi-scale feature map into a recurrent neural network, performing alignment fusion of space optimization, namely traversing all video frames according to a feature map of a single-frame super-division result, and obtaining a single-direction super-division result; reversely traversing all the video frames to obtain a super-resolution result in a reverse direction; and integrating bidirectional super-division results, outputting a final super-division result, and reconstructing an image to obtain high-resolution video sequence data. According to the method, the multi-scale details are simply and efficiently extracted from the video, so that the performance of the video super-resolution technology in the aspect of scene detail recovery is enhanced, and the high-resolution video quality is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of super-resolution, and in particular relates to a multi-scale space optimization video super-resolution method. Background Art

[0002] The pursuit of high-definition video is an important goal in life, especially in some key areas. For example, in medical image analysis, clear video helps improve the accuracy of diagnosis; in traffic management, high-definition video can more accurately assess traffic flow and density. Video super-resolution technology, that is, the technology of converting low-definition video into high-definition video, has made significant progress in recent years driven by deep learning. Through neural network-based methods, it has achieved better video reconstruction effects than traditional technologies. Although there have been significant progress in the field of image super-resolution, the development of video super-resolution still faces challenges, mainly because the current technology methods for multi-scale feature extraction are neither simple nor efficient, resulting in the need for further optimization and improvement of scene details in the video.

[0003] Therefore, this study introduces a multi-scale spatial optimization video super-resolution technology. The core of this technology is to accurately extract details of various scales from the video in a simple and efficient way, so as to enhance the performance of video super-resolution technology in restoring scene details. Summary of the invention

[0004] To overcome the problems in the prior art, the present invention discloses a multi-scale spatial optimization video super-resolution method, introduces adaptive multi-scale feature extraction and spatial alignment optimization, and designs an improved video super-resolution algorithm to achieve effective and efficient spatiotemporal information extraction, thereby further improving the quality of high-resolution video.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] On the one hand, the present invention proposes a multi-scale spatial optimization video super-resolution method, comprising:

[0007] S1: Obtain the video sequence data to be processed;

[0008] S2: extracting shallow feature maps of each video frame in the video sequence data;

[0009] S3: for the shallow feature map of each video frame, further extracting the multi-scale feature map of each video frame by adaptive multi-scale feature extraction;

[0010] S4: The multi-scale feature maps of each video frame are used as the input of the recurrent neural network in turn, and spatially optimized alignment and fusion are performed frame by frame to obtain a unidirectional super-resolution result;

[0011] S5: Reversely input the multi-scale feature map of each video frame into the recurrent neural network, perform spatially optimized alignment and fusion frame by frame, and obtain a super-resolution result in the reverse direction;

[0012] S6: The super-resolution result of one direction is combined with the super-resolution result of the opposite direction, and then a high-resolution video is obtained through super-resolution reconstruction to complete the super-resolution of the video sequence data.

[0013] Furthermore, the S3 includes:

[0014] S3.1: Split the shallow feature map into feature blocks X of C different channels according to the channel dimension i , 0≤i≤C-1;

[0015] S3.2: For feature blocks X of C different channels i , where the feature block X 0 No processing is performed, and the remaining feature blocks are downsampled by different multiples;

[0016] S3.3: Feature block X 0 The remaining feature blocks after downsampling are respectively extracted through deep convolution to extract deep feature blocks

[0017] S3.4: Deep feature block Without processing, the remaining depth feature blocks are upsampled to the depth feature blocks at different multiples. The size of the upsampled feature blocks and feature blocks Splicing to get a new feature map

[0018] S3.5: The new feature map After channel fusion, the multi-scale feature map is obtained by multiplying the shallow feature map.

[0019] Furthermore, the S4 specifically includes:

[0020] S4.1: Get the multi-scale feature map of the first frame as the feature map to be aligned The feature map prepared for alignment The next frame is taken as the current frame;

[0021] S4.2: For feature maps ready for alignment The optical flow alignment method is used to align it with the multi-scale feature map of the current frame Align to obtain the feature map after rough alignment

[0022] S4.3: Construct the feature map after rough alignment based on the optical flow information during optical flow alignment Each pixel in Feature maps to be aligned Each pixel in The corresponding relationship;

[0023] S4.4: Prepare feature maps for alignment Each pixel As the center, create a window of the same size, and all the pixels in the window are taken as its central pixel Neighborhood pixels of

[0024] S4.5: Compute the feature map after rough alignment Medium pixel Corresponding pixel The similarity of all neighboring pixels is calculated, and the neighboring pixel with the highest similarity is selected as the pixel I (x, y) in the feature map after spatial optimization alignment. The feature map after rough alignment is traversed. All pixels in the image are super-resolved to obtain the single-frame super-resolution result of the current frame;

[0025] S4.6 Use the single frame super-resolution result of the current frame as the feature map to be aligned Update value, repeat S4.2-S4.5 for all video frames in sequence to obtain the unidirectional super-resolution result.

[0026] Furthermore, the spatially optimized alignment fusion described in S4 and S5 adopts the same method.

[0027] Furthermore, the super-resolution result in one direction and the super-resolution result in the opposite direction are fused into splicing according to the channel dimension.

[0028] On the other hand, the present invention also proposes a multi-scale space optimized video super-resolution system for implementing the above method.

[0029] The beneficial effects of the present invention are:

[0030] The present invention designs a multi-scale spatial optimization video super-resolution method, which mainly aims at the fact that the current super-resolution model generally lacks multi-scale feature extraction and is weak in restoring some texture details. It proposes a method of adaptive multi-scale feature extraction and spatial optimization alignment. These two methods further optimize the video detail features without introducing noise like the traditional super-resolution model using bilinear methods, and can be used specifically to deal with multi-scale detail textures. At the same time, the computational complexity of the present invention is far less than the U-Net method used in the current mainstream multi-scale extraction model, and is more suitable for video super-resolution with higher real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1The present invention is a flowchart of a multi-scale spatial optimization video super-resolution method used in the implementation process of the present invention.

[0032] Figure 2 It is a structural block diagram of adaptive multi-scale feature extraction during the implementation of the present invention.

[0033] Figure 3 This is a network structure diagram of spatially optimized alignment during the implementation of the present invention. DETAILED DESCRIPTION

[0034] The method of the present invention will be further described below in conjunction with the accompanying drawings.

[0035] The flowchart of the multi-scale spatial optimization video super-resolution method of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0036] Step 1: Obtain low-resolution video sequence data;

[0037] Step 2: Extract shallow features of video sequences through the network.

[0038] Step 3: Extract multi-scale features of the video sequence through adaptive multi-scale feature extraction.

[0039] Step 4: Introduce multi-scale information into the recurrent neural network and perform spatially optimized alignment and fusion with the previous state of the recurrent neural network, that is, the feature map of the single-frame super-resolution result.

[0040] Step 5: Repeat steps 2-4, traverse each frame in the video sequence, and obtain the super-resolution result in one direction;

[0041] Step 6: Reversely process each frame in the video sequence to obtain the super-resolution result in the reverse direction;

[0042] Step 7: The unidirectional super-resolution result output from step 5 and the reverse direction super-resolution result output from step 6 are merged, and then the high-resolution frame video output is completed through super-resolution reconstruction.

[0043] Among them, for the acquisition of low-resolution video sequence data in step 1, the present embodiment adopts high-resolution video downsampling to obtain it for testing the super-resolution method, and the low-resolution video to be optimized can be directly used in actual application.

[0044] In the step 2, shallow features of the video sequence are extracted. Specifically, for the obtained low-resolution video sequence data, shallow features of the video sequence are extracted by residual blocks (ResidualBlock) of the residual network, including shallow feature maps of video frames.

[0045] In step 3, multi-scale information of the video sequence is extracted by adaptive multi-scale feature extraction, such as Figure 2 As shown, it specifically includes the following sub-steps:

[0046] Step 3.1: First, record the shallow feature map as X T , the feature map X T Split into X according to the channel dimension 0 , X 1 , X 2 , X 3 Four parts:

[0047] [X 0 , X 1 , X 2 , X 3 ]=Chunk(X T )

[0048] Step 3.2: For feature map X 0 Without processing, the other three feature maps X 1 , X 2 , X 3 The downsampling is performed by 2 times, 4 times and 8 times respectively. The downsampling in this embodiment is performed by the maximum pooling layer, and all features are extracted by deep convolution. In this embodiment, 3×3 convolution is used, and the extracted deep feature map is recorded as

[0049]

[0050]

[0051] Among them, Conv 3×3 represents 3×3 convolution, Representation 2 i The downsampling method is the nearest neighbor interpolation method, which is used to prevent other interpolation information from introducing noise and affecting the accuracy of multi-scale feature extraction.

[0052] Step 3.3: No Processing, using the nearest neighbor interpolation method to obtain the features Upsampling to feature map The size of the four feature maps with the same length and width is obtained Concatenate these feature maps to form a new feature map The new feature map obtained by splicing With the shallow feature map X T The same dimensions as:

[0053]

[0054]

[0055] in Representation 2 i times upsampling, and Concat represents the concatenation of feature maps.

[0056] Step 3.4: The new feature map Through an l×l convolution and activation function, channels are fused to obtain a feature map containing multi-scale feature information. The feature map containing multi-scale feature information As an attention matrix, let the original feature map X T Adaptively adjust feature information, pay attention to information at different scales, and obtain adaptive multi-scale feature information

[0057]

[0058] Among them, Conv 1×1 represents l×l convolution, and Tanh represents the activation function. This step obtains multi-scale information through multiplication in an adaptive way. The purpose of doing this is that information loss occurs in the previous process, and an adaptive way can avoid this information loss from interfering with the final result.

[0059] This step extracts multi-scale information in an adaptive way, and its biggest advantage lies in the amount of computation. The adaptive method has a significant advantage in terms of computational complexity compared to the general multi-scale information extraction model U-Net.

[0060] The step 4 introduces the obtained multi-scale features into the recurrent neural network and performs spatially optimized alignment fusion with the previous state of the recurrent neural network, such as Figure 3 As shown, it specifically includes the following sub-steps:

[0061] Step 4.1: Get the processing result of the recurrent neural network on the previous video frame and record it as the feature map to be aligned Multi-scale features for the current frame video frame Perform alignment fusion; if there is no previous video frame, obtain the multi-scale features of the current frame and record it as

[0062] Step 4.2: For feature maps ready for alignment First, extract the feature map through conventional optical flow alignment The optical flow information is used to transform the feature map Multi-scale features with the current frame Align to obtain the feature map after rough alignment

[0063]

[0064] Warp represents optical flow alignment.

[0065] Get feature map There are two purposes: one is to determine the position information of each pixel in the aligned target feature map I obtained by alignment optimization, and the other is to Information about each pixel As the query vector Q in the attention mechanism:

[0066]

[0067] Where q is a unit vector.

[0068] Step 4.3: Based on the position information of each pixel in the feature map I and the optical flow information used in the previous conventional optical flow alignment, the feature map to be aligned is obtained by backtracking each pixel I(x, y) in the target feature map The position of each pixel in Among them, dx and dy are obtained through optical flow information. Each pixel in Create a h×w window M as the center and collect information about all pixels in this window. Each pixel in the window M is denoted as M(i, j), called a pixel point Neighborhood pixels, where i∈{0, 1…, h}, j∈{0, 1…, w}:

[0069]

[0070] The information of all pixels in the window M will be used as the key vector K and value vector V in the attention mechanism:

[0071] K=Mk

[0072] V=Mv

[0073] Where k and v are unit vectors.

[0074] Step 4.4: Through the self-attention mechanism, each pixel can be obtained Corresponding pixel The similarity matrix of all neighboring pixels is preferably cosine similarity, and the neighboring pixel pair with the highest similarity is selected. Perform alignment and obtain the optimized alignment result I(x, y) for each pixel. The formula is shown in 4-8:

[0075] I(x,y)=Max(A(Q,K))V

[0076] Where A(Q, K) represents the similarity matrix between the query vector Q and the key vector K.

[0077] The improvement of step 4 is that the resampling process is optimized. In the existing commonly used methods, bilinear interpolation will introduce new noise, and the nearest neighbor interpolation may not select the most suitable pixel point for alignment. Step 4 of this method is improved based on the nearest neighbor interpolation method. Through the above method, each pixel point can select the most ideal result during the alignment process.

[0078] Step 5: Repeat step 4, traverse each frame in the video sequence, and obtain the super-resolution result in one direction;

[0079] Step 6: According to the method of step 5, reversely traverse the video sequence to obtain the super-resolution result in the reverse direction; specifically, it includes:

[0080] Get the next frame super-resolution result output in the previous round of reverse traversal, and record it as the feature map to be aligned Multi-scale features for and current frame Align and fuse; if there is no previous reverse traversal round (that is, the current round is the first round and the current frame is the last frame of the video), the multi-scale features of the current frame are obtained and recorded as

[0081] Follow steps 4.2-4.4 to convert the feature map Multi-scale feature map with the current frame Perform alignment fusion to obtain the reverse super-resolution result of a single frame, traverse the video sequence from back to front, arrange all the reverse super-resolution results of a single frame in the order of the original video frame, and obtain the reverse super-resolution result. Step 7: Fuse the single-direction super-resolution result outputted in step 5 with the reverse super-resolution result outputted in step 6, and then complete the output of the high-resolution frame video through super-resolution reconstruction.

[0082] The fusion specifically includes: directly splicing the super-resolution results of the unidirectional and reverse directions of the same video frame according to the channel dimension to obtain the bidirectional super-resolution result.

[0083] The super-resolution result of bidirectional fusion is still at the feature level, so it is necessary to reconstruct the bidirectional super-resolution result into images frame by frame. The image reconstruction method of a single frame is a well-known method in the field to obtain a high-resolution video sequence. The above method is called super-resolution reconstruction.

[0084] In the present embodiment, a multi-scale spatial optimization video super-resolution system is also provided, which is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible.

[0085] A data input module, used for acquiring video sequence data to be processed;

[0086] A shallow feature extraction module is used to extract shallow feature maps of each video frame;

[0087] A multi-scale feature extraction module, used to further extract a multi-scale feature map of each video frame;

[0088] The recurrent neural network module is used to perform spatially optimized alignment and fusion of the multi-scale feature maps of each video frame frame by frame to obtain super-resolution results in one direction and in the opposite direction;

[0089] The image reconstruction module is used to fuse the super-resolution results in one direction with the super-resolution results in the opposite direction, and then obtain a high-resolution video through super-resolution reconstruction.

[0090] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0091] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0092] The above description is only a specific embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the present application. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A multi-scale spatial optimization video super-resolution method, characterized in that: include: S1: Obtain the video sequence data to be processed; S2: extracting shallow feature maps from each video frame in the video sequence data; S3: for the shallow feature map of each video frame, further extracting the multi-scale feature map of each video frame by adaptive multi-scale feature extraction; S4: The multi-scale feature maps of each video frame are used as the input of the recurrent neural network in turn, and spatially optimized alignment and fusion are performed frame by frame to obtain a unidirectional super-resolution result; S5: Reversely input the multi-scale feature map of each video frame into the recurrent neural network, perform spatially optimized alignment and fusion frame by frame, and obtain a super-resolution result in the reverse direction; S6: The super-resolution result of one direction is combined with the super-resolution result of the opposite direction, and then a high-resolution video is obtained through super-resolution reconstruction to complete the super-resolution of the video sequence data.

2. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that: The extraction of shallow feature maps adopts a residual block network.

3. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that: The S3 includes: S3.1: Split the shallow feature map into feature blocks X of C different channels according to the channel dimension i ,0≤i≤C-1; S3.2: For feature blocks X of C different channels i , where feature block X0 is not processed, and the remaining feature blocks are downsampled by different multiples; S3.3: Extract deep feature blocks from feature block X0 and the remaining feature blocks after downsampling through deep convolution S3.4: Deep feature block Without processing, the remaining depth feature blocks are upsampled to the depth feature blocks at different multiples. The size of the upsampled feature blocks and feature blocks Splicing to get a new feature map S3.5: The new feature map After channel fusion, the multi-scale feature map is obtained by multiplying the shallow feature map.

4. The multi-scale spatial optimization video super-resolution method according to claim 3, characterized in that: In S3.2, feature block X0 is not processed, and the remaining feature blocks X i Conduct 2 i times downsampling.

5. The multi-scale spatial optimization video super-resolution method according to claim 3, characterized in that: The downsampling described in S3.2 and the upsampling described in S3.4 both use the nearest neighbor interpolation method.

6. The multi-scale spatial optimization video super-resolution method according to claim 3, characterized in that: The channel fusion described in S3.5 specifically uses a 1×1 convolution and activation function to fuse the channels.

7. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that: The S4 specifically includes: S4.1: Get the multi-scale feature map of the first frame as the feature map to be aligned The feature map prepared for alignment The next frame is taken as the current frame; S4.2: For feature maps ready for alignment The optical flow alignment method is used to align it with the multi-scale feature map of the current frame Align to obtain the feature map after rough alignment S4.3: Construct the feature map after rough alignment based on the optical flow information during optical flow alignment Each pixel in Feature maps to be aligned Each pixel in The corresponding relationship; S4.4: Prepare feature maps for alignment Each pixel As the center, create a window of the same size, and all the pixels in the window are taken as its central pixel Neighborhood pixels of S4.5: Compute the feature map after rough alignment Medium pixel Corresponding pixel The similarity of all neighboring pixels is calculated, and the neighboring pixel with the highest similarity is selected as the pixel I(x, y) in the feature map after spatial optimization alignment. The feature map after rough alignment is traversed. All pixels in the image are super-resolved to obtain the single-frame super-resolution result of the current frame; S4.6: Use the single frame super-resolution result of the current frame as the feature map to be aligned Update value, repeat S4.2-S4.5 for all video frames in sequence to obtain the unidirectional super-resolution result.

8. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that: The spatially optimized alignment fusion described in S4 and S5 was performed using the same method.

9. The multi-scale spatial optimization video super-resolution method according to claim 1, characterized in that: In step S6, the super-resolution result in one direction and the super-resolution result in the opposite direction are merged into a channel-based splicing.

10. A multi-scale spatial optimization video super-resolution system, used to implement the method of claim 1, characterized in that: The system comprises: A data input module, used for acquiring video sequence data to be processed; A shallow feature extraction module is used to extract shallow feature maps of each video frame; A multi-scale feature extraction module, used to further extract a multi-scale feature map of each video frame; The recurrent neural network module is used to perform spatially optimized alignment and fusion of the multi-scale feature maps of each video frame frame by frame to obtain super-resolution results in one direction and in the opposite direction; The image reconstruction module is used to fuse the super-resolution results in one direction with the super-resolution results in the opposite direction, and then obtain a high-resolution video through super-resolution reconstruction.

Images