Video super-resolution reconstruction method and device based on surround attention mechanism and medium

Abstract

A video super-resolution reconstruction method and device based on a surround attention mechanism and a medium are provided.A video super-resolution reconstructor is constructed,firstly, shallow features of each frame of image in a video are extracted by using a feature extraction network;secondly, the extracted shallow features are input into a surround attention alignment module in a sequential and reverse order to align the features of each frame and the features of adjacent frames;then, time complementary information fusion is realized through a feature fusion network;finally, the fused features of each frame are converted into image residuals with higher resolution through an up-sampling network, the original image after bilinear interpolation is superimposed, and a high-resolution video with more details is reconstructed.The surround attention mechanism designed in the application can simplify the process of video super-resolution reconstruction, effectively improve the performance of the reconstructor, and make it applicable to reconstruct video input with large motion.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to video super-resolution reconstruction technology, specifically a video super-resolution reconstruction method, device and medium based on a surround attention mechanism. Background Technology

[0002] The goal of video super-resolution reconstruction is to reconstruct a video segment into a higher-resolution video with richer details and greater visual clarity. Currently, mainstream advanced video super-resolution reconstructors mainly employ two paradigms: one based on recurrent neural networks (RNNs), which utilizes features generated during the reconstruction of the previous frame, iteratively executing this process to ensure each frame's reconstruction leverages information from all previously processed frames; the other based on sliding windows, which divides an image window centered on the image to be reconstructed and then uses information from all images within that window for reconstruction. RNN-based reconstruction methods often perform better because they utilize more temporal complementary information from multiple frames; while sliding window-based methods, although capable of reconstructing each frame in parallel, have slightly inferior reconstruction results due to insufficient utilization of information from other frames. Furthermore, to achieve better reconstruction results, current advanced methods based on both paradigms are designed as two-stage processes: first, a motion estimation network is introduced to predict the motion information of pixels between frames; second, the predicted motion information is used to assist in the fusion process of features from different frames. The disadvantages of this two-stage process are: first, additional training data is needed to train the motion estimation network separately to ensure the accuracy of the estimated motion information; second, the motion estimation network and the video super-resolution reconstruction network need to be deployed and executed simultaneously and sequentially during the testing process, which makes the reconstruction process complex, and the reconstruction effect is directly limited by the accuracy of the motion estimation network.

[0003] In recent years, attention-based video super-resolution reconstruction methods have emerged. These methods employ a sliding window paradigm, using an attention mechanism to fuse information from different frames within a window, achieving single-stage video super-resolution reconstruction with promising results. However, these methods have drawbacks: firstly, due to the limitations of the sliding window paradigm, they cannot utilize the temporal complementary information of more frames, resulting in suboptimal reconstruction performance; secondly, they do not specifically modify the attention mechanism for video super-resolution reconstruction, leading to issues such as imprecise reconstruction results and unsuitability for reconstructing videos with high motion. Summary of the Invention

[0004] The problem this invention aims to solve is that mainstream video super-resolution reconstructors cannot be trained end-to-end, and the reconstruction process is complex and the reconstruction effect is limited by the accuracy of the motion estimation network. Newly emerging video super-resolution reconstructors based on attention mechanisms have the problem that the reconstruction effect is limited by the sliding window paradigm, and the attention mechanism has not been modified for the video super-resolution reconstruction problem, resulting in imprecise reconstruction effect and unsuitability for reconstructing videos with large amounts of motion.

[0005] The technical solution of this invention is as follows: a video super-resolution reconstruction method based on a surround attention mechanism. A video super-resolution reconstructor is constructed for video super-resolution reconstruction. The video super-resolution reconstructor first uses a feature extraction network to extract shallow features of each frame of the video. Then, the shallow features are input into the surround attention alignment module in both sequential and reverse order to cyclically align the features of each frame and its neighboring frames. Then, a feature fusion network is used to fuse the shallow features of each frame and the alignment features of the neighboring frames as the fused features of each frame. Finally, an upsampling network is used to transform the fused features of each frame into image residuals with higher resolution. The original image frames after bilinear interpolation are superimposed to reconstruct a high-resolution video and complete the video super-resolution reconstruction task.

[0006] The surrounding attention alignment module includes two methods: sequential loop and reverse loop. In the sequential loop, for the current frame i, the shallow features s of frame i and frame (i-1) are used. i s i-1 and the fusion features of the (i-1)th frame As input, output the alignment feature of the (i-1)th frame after alignment with the i-th frame. The wraparound attention alignment module performs as follows:

[0007] a) Feature transformation: shallow features s of frame i and frame i-1 i s i-1 and the fusion features of the (i-1)th frame Feature transformation is performed using layer normalization and convolutional layers, converting them into query Q. i Key K i-1 Sum V i-1 This is used for subsequent wraparound window partitioning. When i = 1, zero values ​​are used to replace non-existent features s. i-1 and

[0008] b) Wrap-around window partitioning: First, for query Q... i Divide the data evenly to obtain N query windows of size p×p. Let Q be the j-th query window. i，j Secondly, for each query window Q i，j In key K i-1 Sum Vi-1 The above are divided into key windows K, each with a size of (1+δ)×p×(1+δ)×p. i-1，j Sum Window V i-1，j The centers of the three windows are at the same position and δ > 0, therefore K i-1，j and V i-1，j Surround Q i，j Subsequently, the query window features of the i-th frame are aligned with the wraparound key and value window features of the (i-1)-th frame, thereby fusing the temporal complementary information of neighboring frames.

[0009] c) Soft and Hard Attention Alignment: First, N groups of query windows, key windows, and value windows are input into a soft attention module. For each group of windows, pixel-level attention weights are calculated based on the features of the query and key windows. Then, the calculated attention weights are used to weight and sum the features of the value window. The weighted features incorporate information related to frame i in frame (i-1). Simultaneously, the calculated attention weights and value windows from each group are input into a hard attention module. The hard attention module selects the feature with the highest attention weight from the value window, which retains high-frequency information related to frame i in frame (i-1). Finally, the features output from the soft and hard attention modules are concatenated and feature-mapped using convolutional layers and a multilayer perceptron to obtain the aligned features of frame i-1 after alignment with frame i.

[0010] Similarly, during the reverse loop, for the current frame i, the shallow features s of frame i and frame (i+1) are used. i s i+1 and the fusion features of the (i+1)th frame As input, output the alignment feature of the (i+1)th frame after alignment with the i-th frame.

[0011] Furthermore, the implementation of the video super-resolution reconstructor includes a network configuration phase, a training phase, and a testing phase:

[0012] 1) Network Configuration Phase: Construct the network framework for a video super-resolution reconstructor based on a surround attention mechanism, including the following configurations:

[0013] 1.1) Feature Extraction Network: A feature extraction network based on convolutional layers and residual blocks is used to extract features from each frame of the video, obtaining shallow features that encode the spatial information of each frame. The shallow features of the i-th frame are denoted as s. i ;

[0014] 1.2) Surround Attention Alignment Module: An alignment module based on a surround attention mechanism is used to cyclically align video frame features in both sequential and reverse order.

[0015] 1.3) Feature Fusion Network: Based on the sequential and reverse looping methods of the surrounding attention alignment module, in the sequential looping method, when executing the i-th frame of the video, the input is the shallow features s of the i-th frame. i And the alignment features of the (i-1)th frame after alignment with the i-th frame. The feature fusion network concatenates two sets of feature inputs, then performs feature fusion based on convolutional layers and residual blocks, utilizing... The temporal complementary information in the matrix is ​​used to recover the details of the i-th frame, and the fused feature of the i-th frame is denoted as . Similarly, in the reverse loop approach, the feature fusion network will combine the shallow features s of the i-th frame. i and the alignment features of the (i+1)th frame Perform feature fusion to obtain the fused features of the i-th frame.

[0016] 1.4) Upsampling Network: For the i-th frame, the upsampling network uses... and As input, the two fused features are concatenated and then output as a high-resolution residual of the i-th frame image through a convolutional layer and pixel shuffling operation. Next, the original image of the i-th frame is directly magnified through a bilinear interpolation algorithm. Finally, the image residual of the i-th frame and the magnified original image are superimposed to obtain a high-resolution version of the i-th frame image. After obtaining the high-resolution versions of all frames, a high-resolution video is generated, completing the video super-resolution reconstruction task.

[0017] 2) Training phase: Given the high-resolution video ground truth, the video super-resolution reconstruction network configured in step 1) is trained using low-resolution versions of the video with bicubic downsampling and Gaussian blur downsampling, respectively, so that the generated reconstruction network can be used to reconstruct different types of videos. During training, the Charbonnier loss function is used to calculate the loss between the high-resolution video reconstructed by the network and the real high-resolution video. The Adam optimizer and backpropagation algorithm are used to update the network parameters, and the cosine annealing strategy is used to update the learning rate. The above process is repeated until the maximum number of iterations of the network is reached.

[0018] 3) Testing phase: Input the video after bicubic downsampling or Gaussian blur downsampling into the corresponding trained video super-resolution reconstructor to obtain a high-resolution version of the reconstructed video, and measure the performance of the video super-resolution reconstructor.

[0019] Based on the above method, the present invention also provides an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the above-described video super-resolution reconstruction method based on the surround attention mechanism, thereby obtaining the video super-resolution reconstructor, which is used to complete the video super-resolution reconstruction task on the input video.

[0020] The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein when the at least one instruction or at least one program is executed, the above-described video super-resolution reconstruction method based on the surround attention mechanism is implemented to obtain the video super-resolution reconstructor.

[0021] This invention proposes a novel video super-resolution reconstructor. Based on a recurrent neural network, it only requires a designed wraparound attention alignment module, combined with sequential and reverse recurrent methods, to fuse temporal complementary information from different frames, thus helping to recover high-resolution videos with more details. The designed alignment module simplifies the fusion process of temporal complementary information, thereby realizing an end-to-end, single-stage video super-resolution reconstructor. This improves the reconstructor's performance and enhances its applicability to videos with high motion.

[0022] The present invention has the following advantages compared with the prior art.

[0023] The single-stage video super-resolution reconstructor proposed in this invention has a straightforward process, requiring no additional training data or training process, and can be directly trained end-to-end with fast training convergence speed.

[0024] The video super-resolution reconstructor proposed in this invention combines the advantages of recurrent neural networks and attention mechanisms. The recurrent neural network paradigm it employs enables the utilization of information from multiple frames, while the attention-based alignment module effectively fuses temporal complementary information from multiple frames, thereby significantly improving the reconstruction effect.

[0025] The surround attention alignment module proposed in this invention can quickly and effectively align the pixels in the current frame query window with the complementary information of related pixels in the surrounding window of the neighboring frames, thereby improving the reconstruction effect with low computational complexity. Furthermore, the design of the surround attention mechanism makes the reconstructor suitable for reconstructing video inputs with large amounts of motion.

[0026] This invention offers advantages in video super-resolution reconstruction tasks, including a direct reconstruction process, accurate reconstruction results, and applicability to reconstructing videos with significant motion. Compared to existing methods, this invention achieves better performance on multiple mainstream datasets. Attached Figure Description

[0027] Figure 1 This is the system framework diagram used in this invention.

[0028] Figure 2 This is a schematic diagram of the structure of the surround attention alignment module of the present invention.

[0029] Figure 3 This is the result of the present invention compared with previous work on multiple mainstream datasets.

[0030] Figure 4 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0031] This invention proposes a single-stage video super-resolution reconstruction method based on a surround attention mechanism, such as... Figure 4 As shown, a feature extraction network is first used to extract shallow features from each frame of the video. Then, the shallow features are cyclically input into the surrounding attention alignment module in both sequential and reverse order to align the features of each frame with the features of its neighboring frames. A feature fusion network is then used to achieve temporal complementary information fusion. Finally, an upsampling network is used to transform the fused features of each frame into image residuals with higher resolution. These residuals are then superimposed on the original image after bilinear interpolation to reconstruct a high-resolution video, thus completing the video super-resolution reconstruction task.

[0032] This invention constructs a video super-resolution reconstructor based on the above process, combining the advantages of recurrent neural networks and attention mechanisms to effectively improve the performance of the video super-resolution reconstructor. The implementation of the video super-resolution reconstructor of this invention includes a network configuration phase, a training phase, and a testing phase, as follows: Figure 1 As shown, the specific explanation is as follows.

[0033] 1) Network Configuration Phase: Construct the network framework for a video super-resolution reconstructor based on a surround attention mechanism, including the following configurations:

[0034] 1.1) Feature Extraction: A feature extraction network based on convolutional layers and residual blocks is used to extract features from each frame of the video, obtaining shallow features that encode the spatial information of each frame. Using s i The shallow features of the i-th frame are represented by the following calculation process:

[0035] s i =Res 5 (LReLU(Conv(x i )))

[0036] in Let H represent the i-th RGB image to be reconstructed. During training, because some memory resources are used to store the parameter update gradients, all video frames are cropped to reduce memory usage. During training, the sizes of H and W are cropped to 64. During testing, since parameter gradients do not need to be stored, more memory resources are available, and H and W retain the original video size. Conv indicates that a convolutional layer is used to map the RGB image to a C-dimensional feature space. In this example, C is set to 64. LReLU represents the LeakyReLU activation function, and Res... 5 This represents five stacked residual blocks suitable for video super-resolution reconstruction. Each residual block contains two convolutional layers, an intermediate LeakyReLU activation function, and residual connections. The negative slope parameter in all LeakyReLU activation functions is set to 0.1. The receptive field size of all convolutional layers is 3x3. Except for the first convolutional layer, which maps the RGB image to a C-dimensional feature space, the other convolutional layers maintain the same feature dimension. No downsampling is performed on the feature maps in any of the convolutional layers. The output shallow features of the i-th frame are...

[0037] 1.2) Wrap-around Attention Alignment: An alignment module based on a wrap-around attention mechanism is used to cyclically align video frame features in both sequential and reverse order. The alignment module executed sequentially is explained below. When processing the i-th frame, it uses the shallow features s extracted from the i-th and (i-1)-th frames... i s i-1 and the fusion features of the (i-1)th frame As input, output the features of the (i-1)th frame after alignment with the i-th frame.

[0038] First, feature transformation is performed, using layer normalization and convolutional layers to transform the shallow features s. i s i-1 and fusion features Convert them to query Q respectively i Key K i-1 Sum V i-1 .

[0039] Next, we divide the window into wraparound sections. First, we perform a query Q... i Divide the data evenly to obtain N query windows of size p×p. Let Q be the j-th query window. i，j Secondly, for each query window Q i，j In key K i-1 Sum V i-1 The above are divided into key windows K, each with a size of (1+δ)×p×(1+δ)×p. i-1，j Sum Window V i-1，j The centers of the three windows are at the same position and δ > 0, therefore K i-1，jand V i-1，j Surround Q i，i Subsequently, the query window features of the i-th frame are aligned with the wraparound key and value window features of the (i-1)-th frame, thereby fusing the temporal complementary information of neighboring frames.

[0040] Finally, a soft and hard attention mechanism is used to align the features. First, N groups of windows are grouped and input into a soft attention module. For each group of windows, pixel-level attention weights are calculated based on the features of the query window and the key window. Then, the calculated attention weights are used to perform a weighted sum of the features of the value window. Next, the calculated attention weights and value windows in each group are input into a hard attention module, which selects the features with the maximum attention weight from the value window. Then, the features output by the soft and hard attention modules are concatenated and feature-mapped through convolutional layers and a multilayer perceptron to obtain the features of the (i-1)th frame aligned with the i-th frame.

[0041] The above process is implemented as follows: Figure 2 As shown, the specific calculation process is as follows.

[0042] 1.2.1) Feature Transformation:

[0043] Q i =Conv(LayerNorm(s) i ))

[0044] K i-1 =Conv(LayerNorm(s) i-1 ))

[0045]

[0046] Among them, s i s i-1 and These represent the shallow features of frame i, the shallow features of frame i-1, and the fused features of frame i-1, respectively. When i = 1, s i-1 and The default padding value is zero; LayerNorm indicates layer normalization operation, and Conv indicates a convolutional layer with a receptive field size of 3*3; This represents the transformed query, key, and value.

[0047] 1.2.2) Wrap-around window division:

[0048] First, query Divide the data into N query windows of size p×p. During training or testing, the feature map Q... i After trimming or padding, to ensure that H and W are divisible by p, therefore... In this example, p is set to 8, and the feature map Q is cropped during training. i H and W are fixed at 64; feature map Q is filled during the test. i To ensure that H and W are divisible by p. The j-th query window Q i，j The calculation formula is:

[0049]

[0050] Q i，j ={Q i [*,u,v]|u j ≤u<u j +p, v j ≤v<v j +p}

[0051] Among them, (u j v j () represents the coordinates of the top-left corner of the j-th query window in the feature map. The top-left corner of the feature map is set as the origin. The index j of the query window starts from 1. (u, v) indicates that it is located in Q. i，j Coordinates within.

[0052] Secondly, for any query window Q i，j From key K i-1 Sum V i-1 The query window, key window, and value window are each divided into (1+δ)×p×(1+δ)×p windows and surrounded by them. The centers of the query window, key window, and value window coincide. δ is the window magnification factor, which is set to 1 in this embodiment to make the key and value windows larger and surround the query window. The calculation formulas for the j-th key window and value window are as follows:

[0053] o j =u j -0.5×δ×p,t j =v j -0.5×δ×p

[0054] K i-1，j ={K i-1 [*,o,t]|o j ≤o<o j +(1+δ)×p,t j ≤t<t j +(1+δ)×p}

[0055] V i-1，j ={V i-1 [*,o,t]|o j ≤o<o j +(1+δ)×p,t j ≤t<t j+(1+δ)×p}

[0056] Among them, (o j , t j () represents the coordinates of the top-left corner of the j-th key or value window in the feature map. For query windows located at the edge of the feature map, such as the top-left query window, there are no features to its left and above. In this case, attempting to create a wraparound key or value window around it would encounter the problem of missing features, i.e., an out-of-bounds problem. Therefore, the coordinates of the top-left corner of the feature map K are... i-1 and V i-1 The edges are filled with default values ​​to avoid out-of-bounds issues. By dividing the query, key, and value, N groups of feature windows are obtained, each containing one query window and two wraparound key-value windows.

[0057] 1.2.3) Soft and hard attention alignment:

[0058]

[0059] Soft i，j =A i，j V i-1，j

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] The N feature windows of the (i-1)th frame and the i-th frame are grouped and aligned. Specifically, the j-th query window Q i，j It contains p×p C-dimensional feature vectors, each vector corresponding to a pixel in the i-th frame; the j-th key window K i-1，j Sum Window V i-1，j Each of them contains (1+δ)×p×(1+δ)×p C-dimensional feature vectors, and each vector corresponds to a pixel in the (i-1)th frame; Softmax represents the Soffmax function; Let A represent the pixel-level attention weight matrix, with p × p rows and (1 + δ) × p × (1 + δ) × p columns. It measures the correlation between p × p pixels in the i-th frame and (1 + δ) × p × (1 + δ) × p pixels in the (i-1)-th frame within the j-th window. i，j，m，n Representing matrix A i，jThe element in the m-th row and n-th column of the attention weight matrix A measures the correlation between the m-th pixel in frame i and the n-th pixel in frame (i-1). i，j Value window V i-1，j Soft weighting is performed to obtain the soft-weighted feature of the j-th window group. i，j Meanwhile, in the j-th window, the attention weight matrix A... i，j Apply the argmax function line by line to find the most relevant pixel in the (i-1)th frame for any m-th pixel in the i-th frame, and label the most relevant pixel as idx. i，j，m The maximum attention weight for the m-th pixel. The most relevant pixel in the sum window Hard weighting is performed to obtain the hard-weighted feature of the m-th pixel in the j-th query window. i，j，m Aggregate the hard-weighted features of all pixels in the j-th query window to obtain the hard-weighted features of the j-th group of windows. i，j After aggregating the soft-weighted and hard-weighted features of all windows respectively, a convolutional layer is used to map the soft and hard features, and a residual connection is used to superimpose the shallow features of the i-th frame to obtain the intermediate alignment features. Finally, an MLP (Multilayer Perceptron) and residual connections are used to map the intermediate alignment features to obtain the features of the (i-1)th frame aligned with the i-th frame.

[0066] 1.3) Feature Fusion: In this embodiment, video frame features are fused sequentially and in reverse order to utilize temporal complementary information in different frames. The feature fusion process is described using sequential cyclic execution. When the i-th frame of the video is reached, the shallow features s from the i-th frame in 1.1) are used. i And 1.2) the features of the (i-1)th frame after being aligned with the i-th frame. As input, the two feature sets are concatenated and then fused using convolutional layers and residual blocks. The temporal complementary information in the matrix is ​​used to recover the details of the i-th frame, and the fused features of the i-th frame are denoted as follows: The calculation process is as follows:

[0067]

[0068] Res 30 This represents 30 sequentially stacked residual blocks, and the fused feature of the i-th frame is denoted as...

[0069] The specific implementations of the surrounding attention alignment module and the feature fusion network under the reverse loop execution method differ from the sequential loop execution method only in that when processing the i-th frame, the features of the (i+1)-th frame are used to replace the features of the (i-1)-th frame; the other processes are the same. During the reverse loop, for the current i-th frame, the shallow features s of the i-th and (i+1)-th frames are used... i s i+1 and the fusion features of the (i+1)th frame As input, output the alignment feature of the (i+1)th frame after alignment with the i-th frame. The feature fusion network combines the shallow features s of the i-th frame. i and the alignment features of the (i+1)th frame Perform feature fusion to obtain the fused features of the i-th frame.

[0070] 1.4) Upsampling: For the i-th frame, the features output by the sequentially cyclically executed wrap-around attention alignment process and feature fusion process. It incorporates temporal complementary information from the (i-1)th frame; similarly, the features output by the reverse-order cyclical alignment and feature fusion processes are... It incorporates temporal complementary information from the (i+1)th frame. When processing the i-th frame, the upsampling process... and As input, two features are concatenated and then processed through a convolutional layer and pixel shuffling to output a high-resolution residual of the i-th frame image. Next, the original RGB image of the i-th frame is directly upscaled using bilinear interpolation. Finally, the image residual output from this process is superimposed with the upscaled original image to obtain a high-resolution version of the i-th frame RGB image. This process can be formulated as follows:

[0071]

[0072]

[0073] In this diagram, Conv1 represents a convolutional layer used to fuse features; PixelShuffle represents a module consisting of convolutional layers, pixel shuffling operations, and LeakyReLU activation functions, used to amplify feature resolution; Conv2 represents a convolutional layer that maps the amplified features while maintaining the number of feature channels; and Conv3 represents a convolutional layer that maps the features to 3D space, producing a high-resolution image residual for the i-th frame. Secondly, Bilinear represents the bilinear interpolation operation, which is used to interpolate the original RGB image x of the i-th frame. i The resolution of the first frame is directly magnified and then added to the residual of the high-resolution image to produce a high-resolution version of the i-th frame. After upsampling to obtain high-resolution versions of all frames, a high-resolution video is generated, completing the video super-resolution reconstruction task.

[0074] 2) Training Phase: Given the ground truth high-resolution video, the reconstructor is trained using different low-resolution versions of the video, employing both bicubic downsampling and Gaussian blur downsampling. These different reconstructor versions are suitable for reconstructing different types of videos. The Charbonnier loss function is used to calculate the reconstructed high-resolution video. Compared to real high-resolution video y={y1,y2,...,y F The loss is calculated as follows: F represents the number of video frames.

[0075]

[0076]

[0077] Among them, y i This represents the high-resolution ground truth value of the i-th frame image. For the reconstruction result of the i-th frame image, the parameter ∈ is set to 0.000001. This represents the reconstruction loss of the i-th frame. This represents the reconstruction loss of the entire video.

[0078] Because the reconstructor needs to save the parameter update gradients during training, the available computational memory is limited. Therefore, input videos with too many frames and ground truth videos are cropped to fix the number of frames in the training video samples at 15. Simultaneously, to enable the recurrent neural network to effectively capture temporal complementary information, input videos with too few frames and ground truth videos are padded with reflection. For example, a 7-frame video with image encoding 1, 2, ..., 7 is padded to 14 frames, resulting in image encodings of 1, 2, ..., 7, 7, 6, ..., 1. Videos that have undergone temporal cropping or padding are then augmented sequentially through random spatial cropping, vertical flipping, horizontal flipping, and 90° rotation. The Adam optimizer and backpropagation algorithm are used to iterate the network parameters, with 300,000 iterations. The initial learning rate is set to 0.0002, and cosine annealing is used to update the learning rate. This process is repeated until the maximum number of iterations is reached.

[0079] 3) Testing Phase: Testing is conducted on datasets that have undergone bicubic downsampling or Gaussian blur downsampling. For video samples with fewer frames, reflection padding is applied before inputting them into the reconstructor for high-resolution video reconstruction. The reconstructed portion without padding is then extracted and compared with the ground truth to measure the reconstructor's performance. For other video samples with sufficiently long frame counts, the testing process remains the same except that reflection padding is not required. Given a reconstructed RGB image X and its ground truth Y, the reconstruction effect is evaluated using PSNR and SSIM metrics. PSNR measures the difference in pixel size between the two images, while SSIM measures the structural difference between them. Their calculation formulas are as follows:

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Where, μ x Let μ be the mean of X. Y Let σ be the mean of Y. X Let σ be the variance of X. Y Let c1 be the variance of Y, and c1 = (0.01 × 255). 2 c2 = (0.03 × 255) 2 The performance metric of the reconstructor across the entire video is the average of the metrics across all frames; higher values ​​for both metrics indicate better reconstructor performance. All experimental metrics for all examples are as follows: Figure 3 As shown, for the REDS, Vimeo-90K-T, and Vid4 datasets after bicubic downsampling (BIdegradation), and the Vimeo-90K-T dataset after Gaussian blur downsampling (BDdegradation), this invention achieves the best results on all metrics compared to other state-of-the-art video super-resolution reconstruction methods.

[0088] The video super-resolution reconstructor of the present invention is program-based and implemented by a processor. Based on the above method, the present invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program segment. The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the video super-resolution reconstruction method based on the surround attention mechanism, thereby obtaining the video super-resolution reconstructor, which is used to complete the video super-resolution reconstruction task on the input video.

[0089] The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein when the at least one instruction or at least one program is executed, the above-described video super-resolution reconstruction method based on the surround attention mechanism is implemented to obtain the video super-resolution reconstructor.

Claims

1. A video super-resolution reconstruction method based on a surround attention mechanism, characterized by: A video super-resolution reconstructor was constructed and trained end-to-end for video super-resolution reconstruction. The video super-resolution reconstructor first uses a feature extraction network to extract shallow features of each frame of the video. Then, the shallow features are input into a surround attention alignment module in both sequential and reverse order to align the features of each frame and its neighboring frames in a loop. Then, a feature fusion network is used to fuse the shallow features of each frame and the alignment features of the neighboring frames as the fused features of each frame. Finally, an upsampling network is used to transform the fused features of each frame into image residuals with higher resolution. The residuals are then superimposed on the original image frames after bilinear interpolation to reconstruct a high-resolution video and complete the video super-resolution reconstruction task. The surrounding attention alignment module includes two methods: sequential loop and reverse loop. In the sequential loop, for the current frame i, the shallow features s of frame i and frame (i-1) are used. i ,s i-1 and the fusion features of the (i-1)th frame As input, output the alignment feature of the (i-1)th frame after alignment with the i-th frame. The wraparound attention alignment module performs as follows: a) Feature transformation: shallow features s of frame i and frame i-1 i ,s i-1 and the fusion features of the (i-1)th frame Feature transformation is performed using layer normalization and convolutional layers, converting them into query Q. i Key K i-1 Sum V i-1 This is used for subsequent wraparound window partitioning. When i = 1, zero values ​​are used to replace non-existent features s. i-1 and b) Wrap-around window partitioning: First, for query Q... i Divide the data evenly to obtain N query windows of size p×p. Let Q be the j-th query window. i,j Secondly, for each query window Q i,j In key K i-1 Sum V i-1 The above are divided into key windows K, each with a size of (1+δ)×p×(1+δ)×p. i-1,j Sum value window V i-1,j The centers of the three windows are in the same position and δ>0, therefore K i-1,j and V i-1,j Surround Q i,j Subsequently, the query window features of the i-th frame are aligned with the wraparound key and value window features of the (i-1)-th frame, thereby fusing the temporal complementary information of neighboring frames. c) Soft and Hard Attention Alignment: First, N groups of query windows, key windows, and value windows are input into a soft attention module. For each group of windows, pixel-level attention weights are calculated based on the features of the query and key windows. Then, the calculated attention weights are used to weight and sum the features of the value window. The weighted features incorporate information related to frame i in frame (i-1). Simultaneously, the calculated attention weights and value windows from each group are input into a hard attention module. The hard attention module selects the feature with the highest attention weight from the value window, which retains high-frequency information related to frame i in frame (i-1). Finally, the features output from the soft and hard attention modules are concatenated and feature-mapped using convolutional layers and a multilayer perceptron to obtain the aligned features of frame i-1 after alignment with frame i. Similarly, during the reverse loop, for the current frame i, the shallow features s of frame i and frame (i+1) are used. i ,s i+1 and the fusion features of the (i+1)th frame As input, output the alignment feature of the (i+1)th frame after alignment with the i-th frame.

2. The video super-resolution reconstruction method based on the surround attention mechanism according to claim 1, characterized in that: The implementation of the video super-resolution reconstructor includes a network configuration phase, a training phase, and a testing phase: 1) Network Configuration Phase: Construct the network framework for a video super-resolution reconstructor based on a surround attention mechanism, including the following configurations: 1.1) Feature Extraction Network: A feature extraction network based on convolutional layers and residual blocks is used to extract features from each frame of the video, obtaining shallow features that encode the spatial information of each frame. The shallow features of the i-th frame are denoted as s. i ; 1.2) Surround Attention Alignment Module: An alignment module based on a surround attention mechanism is used to cyclically align video frame features in both sequential and reverse order. 1.3) Feature Fusion Network: Based on the sequential and reverse looping methods of the surrounding attention alignment module, in the sequential looping method, when executing the i-th frame of the video, the input is the shallow features s of the i-th frame. i And the alignment features of the (i-1)th frame after alignment with the i-th frame. The feature fusion network concatenates two sets of feature inputs, then performs feature fusion based on convolutional layers and residual blocks, utilizing... The temporal complementary information in the matrix is ​​used to recover the details of the i-th frame, and the fused feature of the i-th frame is denoted as . Similarly, in the reverse loop approach, the feature fusion network will combine the shallow features s of the i-th frame. i and the alignment features of the (i+1)th frame Perform feature fusion to obtain the fused features of the i-th frame. 1.4) Upsampling Network: For the i-th frame, the upsampling network uses... and As input, the two fused features are concatenated and then output as a high-resolution residual of the i-th frame image through a convolutional layer and pixel shuffling operation. Next, the original image of the i-th frame is directly magnified through a bilinear interpolation algorithm. Finally, the image residual of the i-th frame and the magnified original image are superimposed to obtain a high-resolution version of the i-th frame image. After obtaining the high-resolution versions of all frames, a high-resolution video is generated, completing the video super-resolution reconstruction task. 2) Training phase: Given the high-resolution video ground truth, the video super-resolution reconstruction network configured in step 1) is trained using low-resolution versions of the video with bicubic downsampling and Gaussian blur downsampling, respectively, so that the generated reconstruction network can be used to reconstruct different types of videos. During training, the Charbonnier loss function is used to calculate the loss between the high-resolution video reconstructed by the network and the real high-resolution video. The Adam optimizer and backpropagation algorithm are used to update the network parameters, and the cosine annealing strategy is used to update the learning rate. The above process is repeated until the maximum number of iterations of the network is reached. 3) Testing phase: Input the video after bicubic downsampling or Gaussian blur downsampling into the corresponding trained video super-resolution reconstructor to obtain a high-resolution version of the reconstructed video, and measure the performance of the video super-resolution reconstructor.

3. The video super-resolution reconstruction method based on the surround attention mechanism according to claim 2, characterized in that: Step 1.1) specifically involves: Let s i This indicates that the feature extraction network extracts shallow features for the i-th frame, and its calculation formula is as follows: s i =Res 5 (LReLU(Conv(x i ))) in Let represent the i-th frame RGB image to be reconstructed, Conv denotes mapping the RGB image to a C-dimensional feature space using a convolutional layer, LReLU denotes the LeakyReLU activation function, and Res 5 This represents five sequentially stacked residual blocks. Each residual block contains two convolutional layers, a LeakyReLU activation function in between, and residual connections. The receptive field size of all convolutional layers in the network is 3x3. Except for the first convolutional layer, which maps the RGB image to a C-dimensional feature space, the other convolutional layers maintain the same feature dimension. No downsampling is performed on the feature maps in any of the convolutional layers. The output shallow feature of the i-th frame is...

4. The video super-resolution reconstruction method based on the surround attention mechanism according to claim 2, characterized in that: In the sequential loop execution mode of the surrounding attention alignment module, the specific implementation when processing the i-th frame image is as follows: 1.2.1) Feature Transformation: Q i =Conv(LayerNorm(s i )) K i-1 =Conv(LayerNorm(s i-1 )) Among them, s i s i-1 and These represent the shallow features of frame i, the shallow features of frame i-1, and the fused features of frame i-1, respectively. When i = 1, s i-1 and The default padding value is zero; LayerNorm indicates layer normalization operation, and Conv indicates a convolutional layer with a receptive field size of 3*3; This represents the transformed query, key, and value; 1.2.2) Wrap-around window division: Given the query obtained by the transformation Divide it into N query windows of size p×p, where the j-th query window Q i,j It is represented as: Q i,j ={Q i [*,u,v]|u j ≤u<u j +p,v j ≤v<v j +p} Wherein, the top left corner of the feature map is the origin (0,0), j=1,…,N, (u j ,v j () represents the coordinates of the top-left corner of the j-th query window; Secondly, for any query window Q i,j The key K obtained from the conversion i-1 Sum V i-1 The code divides the data into larger windows surrounding it, with the size of the surrounding window being (1+δ)×p×(1+δ)×p, where δ is the window magnification factor. The corresponding j-th key window and value window are represented as follows: o j =u j -0.5×δ×p,t j =v j -0.5×δ×p K i-1,j ={K i-1 [*,o,t]|o j ≤o<o j +(1+δ)×p,t j ≤t<t j +(1+δ)×p} V i-1,j ={V i-1 [*,o,t]|o j ≤o<o j +(1+δ)×p,t j ≤t<t j +(1+δ)×p} Among them, (o j ,t j ) represents the coordinates of the top-left corner of the j-th key window or value window; simultaneously, it is directed to the feature map K. i-1 and V i-1 The area around the query window is filled with default values ​​to avoid out-of-bounds issues when dividing the query window located at the edge of the feature map into key windows or value windows. By dividing the query, key and value, N groups of feature windows are obtained, each containing one query window and two wraparound key-value windows. 1.2.3) Soft and hard attention alignment: Given N feature windows, use soft and hard attention mechanisms respectively to align the features of frame i-1 with the features of frame i: Soft i,j =A i,j V i-1,j The j-th query window Q i,j It contains p×p C-dimensional feature vectors, each vector corresponding to a pixel in the i-th frame; the j-th key window K i-1,j Sum value window V i-1,j Each of them contains (1+δ)×p×(1+δ)×p C-dimensional feature vectors, and each vector corresponds to a pixel in the (i-1)th frame; Softmax represents the Softmax function; Let A represent the pixel-level attention weight matrix, with p × p rows and (1 + δ) × p × (1 + δ) × p columns. It measures the correlation between p × p pixels in the i-th frame and (1 + δ) × p × (1 + δ) × p pixels in the (i-1)-th frame within the j-th window. i,j,m,n A represents i,j The element in the m-th row and n-th column of the attention weight matrix A; i,j Value window V i-1,j Soft weighting is performed to obtain the soft-weighted feature of the j-th window group. i,j Meanwhile, in the j-th window, the attention weight matrix A is... i,j Apply the argmax function line by line to find the most relevant pixel in the (i-1)th frame for any m-th pixel in the i-th frame, and label the most relevant pixel as idx. i,j,m The maximum attention weight for the m-th pixel The most relevant pixel in the sum window Hard weighting is performed to obtain the hard-weighted feature of the m-th pixel in the j-th query window. i,j,m Aggregate the hard-weighted features of all pixels in the j-th query window to obtain the hard-weighted features of the j-th group of windows. i,j After aggregating the soft-weighted and hard-weighted features of all windows respectively, a convolutional layer is used to map the soft and hard features, and a residual connection is used to superimpose the shallow features of the i-th frame to obtain the intermediate alignment features. Finally, an MLP (Multilayer Perceptron) and residual connections are used to map the intermediate alignment features to obtain the features of the (i-1)th frame aligned with the i-th frame. The reverse loop execution method of the surrounding attention alignment module can be obtained in the same way.

5. The video super-resolution reconstruction method based on the surround attention mechanism according to claim 2, characterized in that: In step 1.3), the specific implementation of the feature fusion network processing the i-th frame image under sequential loop execution is as follows: Where s i These are the shallow features of the i-th frame. The features of the (i-1)th frame are aligned with the features of the i-th frame. These two features are mapped by a convolutional layer and the Leaky ReLU activation function, and then... (The sentence is incomplete and requires more context to translate accurately.) 30 The 30 sequentially stacked residual blocks are subjected to deep feature fusion to output the fused features of the i-th frame. Similarly, the specific implementation of the feature fusion network in processing the i-th frame image under reverse loop execution can be obtained.

6. The video super-resolution reconstruction method based on the surround attention mechanism according to claim 2, characterized in that: The specific implementation of the upsampling network in step 1.4) when processing the i-th frame image is as follows: in, and Let Conv1 and Conv2 represent the fused features of the i-th frame in sequential and reverse cyclic methods, respectively. Conv1 is the convolutional layer used to fuse features. PixelShuffle represents a module consisting of a convolutional layer, pixel shuffling operation, and LeakyReLU activation function, used to amplify the feature resolution. Conv2 maps the amplified features while keeping the number of feature channels unchanged. Conv3 maps the features to 3D space, producing a high-resolution image residual of the i-th frame. Bilinear represents the bilinear interpolation operation, which is used to interpolate the original RGB image x of the i-th frame. i After the resolution is directly magnified, the residual between the high-resolution image and the high-resolution image is... Add them together to produce a high-resolution version of the i-th frame of the RGB image. After obtaining high-resolution RGB images of all frames through an upsampling network, a high-resolution video is generated, completing the video super-resolution reconstruction task.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the video super-resolution reconstruction method based on the surround attention mechanism as described in any one of claims 1-6, thereby obtaining the video super-resolution reconstructor, which is used to complete the video super-resolution reconstruction task on the input video.

8. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the video super-resolution reconstruction method based on the surround attention mechanism as described in any one of claims 1-6, thereby obtaining the video super-resolution reconstructor.

Images