A video super-resolution method based on gated cross-attention

Abstract

The application provides a video super-resolution method based on a gated mutual attention, comprising the following steps: S1, acquiring a low-resolution video sequence and performing shallow sequence feature extraction; S2, performing joint motion estimation and feature fusion according to the shallow sequence features by using a multi-scale coding and decoding model based on an efficient attention module and a bidirectional parallel distortion module and in combination with optical flow estimation to obtain new sequence features; the efficient attention module comprises a plurality of gated mutual attention modules; S3, adding the new sequence features after Pixelshuffle up-sampling and convolution to the low-resolution video sequence element by element to obtain a high-resolution video sequence. The application provides a video super-resolution method based on a gated mutual attention, and solves the problem that the VSR performance obtained by the existing super-resolution reconstruction technology is not high enough.

Description

Technical Field

[0001] This invention relates to the technical field of video super-resolution, and more specifically, to a video super-resolution method based on gated mutual attention. Background Technology

[0002] Today, there is a need to capture massive amounts of high-quality images and videos to meet the specific requirements of many applications, such as medical imaging, astronomy, surveillance, and remote sensing.

[0003] However, there are two problems in acquiring high-quality video: First, due to limitations in hardware storage capacity and the need to accommodate large amounts of data, the length and number of videos captured in real time are significantly restricted. Second, the limited accuracy of data collected by various sensors in existing video capture equipment results in relatively low frame quality, which may not be suitable for the corresponding application fields. Both of these problems can be solved from a hardware perspective, namely by increasing the storage capacity of the capture equipment and the accuracy of the sensor data, but this would greatly increase costs. Therefore, for economic reasons, more and more people are focusing on software solutions, and one of the more popular directions currently is super-resolution reconstruction technology.

[0004] In the field of super-resolution reconstruction, video super-resolution (VSR) reconstruction methods refer to the task of upscaling low-resolution (LR) video to high-resolution (HR). It has been successfully applied to several computer vision applications, such as video surveillance and high-definition television. It can provide more accurate super-resolution (SR) results than single-image super-resolution (SISR). Compared to image super-resolution algorithms, VSR tasks are more complex due to the introduction of temporal information into video data.

[0005] VSR can be viewed as a spatiotemporal sequence prediction problem. However, existing super-resolution reconstruction techniques often neglect to utilize the locality of data and the correlation between adjacent frames. They cannot align features between video frames and jointly capture past and future information, resulting in insufficient VSR performance. Summary of the Invention

[0006] To overcome the technical shortcomings of existing super-resolution reconstruction techniques in achieving insufficient VSR performance, this invention provides a video super-resolution method based on gated mutual attention.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0008] A video super-resolution method based on gated mutual attention includes the following steps:

[0009] S1: Obtain low-resolution video sequences and perform shallow sequence feature extraction;

[0010] S2: Using a multi-scale encoding and decoding model based on an efficient attention module (EAB) and a bidirectional parallel twist (BPW) module, and combining optical flow estimation, new sequence features are obtained by performing joint motion estimation and feature fusion based on shallow sequence features;

[0011] The High Efficiency Attention Module (EAB) includes multiple Gated Mutual Attention (GMA) modules;

[0012] S3: The new sequence features are upsampled by Pixelshuffle and convolutioned, and then added element-wise to the low-resolution video sequence to obtain a high-resolution video sequence.

[0013] In the above scheme, the gated mutual attention (GMA) module in the multi-scale encoding and decoding model has the ability to predict multiple frames and model long-range temporal dependencies. It is combined with optical flow estimation to use local and spatiotemporal correlation information at different scales to perform joint motion estimation, feature alignment and feature fusion on shallow sequence features. The bidirectional parallel warp (BPW) module is used to further fuse information from adjacent video frames through parallel feature warping. This can improve the performance of VSR by performing feature propagation and alignment. Moreover, the multi-scale encoding and decoding model calculates attention scores at the channel level, so that the computational complexity is linearly related to the spatial resolution of the input image, which can effectively save computational resources.

[0014] Preferably, the multi-scale encoding / decoding model includes an encoder and a decoder; wherein,

[0015] The encoder end includes an M-layer encoder. The feature output of the bidirectional parallel twist (BPW) module of the m-th layer encoder and the feature input of the efficient attention module (EAB) of the m+1-th layer encoder are downsampled by a 4×4 convolution with a stride of 2.

[0016] The decoder includes an M-layer decoder.

[0017] The feature output of the bidirectional parallel twist (BPW) module of the m-th layer decoder and the feature input of the efficient attention module (EAB) of the (m+1)-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

[0018] The feature output of the bidirectional parallel twist (BPW) module of the M-th layer encoder and the feature input of the efficient attention module (EAB) of the 1-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

[0019] Preferably, the low-resolution sequence information is embedded into the encoder and decoder by the optical flow estimation module, and the output of the combined efficient attention module (EAB) provides twisting information to the bidirectional parallel twist (BPW) module.

[0020] Preferably, the efficient attention module (EAB) of the last layer decoder at the decoder end is not combined with the bidirectional parallel warp (BPW) module.

[0021] Preferably, N=3, and the features output by the bidirectional parallel twist (BPW) modules of the encoders in layers 1 and 2 at the encoder end are transmitted to the bidirectional parallel twist (BPW) modules in layers 2 and 1 at the decoder end via skip connections, respectively.

[0022] Preferably, the gated mutual attention (GMA) module includes a multi-head depth transposed attention (MDTA) module and a multi-head mutual attention (MMA) module; wherein,

[0023] The Multi-Head Deep Transposed Attention (MDTA) module first encodes local information through three-branch 1×1 convolutions and 3×3 depthwise separable convolutions, generating query tensor features, key tensor features, and value tensor features. Then, the dimensions are transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate the transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and 1×1 convolution to obtain the output features of the Multi-Head Deep Transposed Attention (MDTA) module.

[0024] In the Multi-Head Mutual Attention (MMA) module, shallow sequence features are divided into sets of 2 frames at each layer and cross-interact with intermediate features output from other layers.

[0025] Preferably, the gated mutual attention (GMA) module further includes a deep convolutional gated feedforward network (DGFN) module; the features flowing into the DGFN module are first normalized by layers, and then flow into the parallel two branches of 1×1 convolution and 3×3 deep convolution, and the number of feature channels is doubled; the features of one branch are first processed by the ReLU activation function, and then element-wise multiplied with the features of the other branch to suppress irrelevant information, and then processed by 1×1 convolution and then channel-wise added with the original features flowing into the DGFN module.

[0026] Preferred shallow sequence features The process of encoding and decoding after one layer is represented as:

[0027]

[0028] in, This indicates the operation of the first gated mutual attention (GMA) module; This indicates the operation of the second gated mutual attention (GMA) module; Represents the operation of the nth gated mutual attention (GMA) module; Φ(·) represents the optical flow estimation operation; f BPW (·) indicates the operation of the Bidirectional Parallel Twist (BPW) module.

[0029] Preferably, the optical flow estimation operation is as follows:

[0030] For the feature X of frame t t Given the estimated optical flow O t-1，t and O t+1，t The feature X of frame t-1 is distorted respectively. t-1 and the feature X of frame t+1 t+1 get:

[0031]

[0032] Where W(·) represents the image warping function operation; X′ t-1 X′ represents the initial warping feature of frame t-1; t+1 This represents the initial warping feature of the (t+1)th frame;

[0033] Next, the offset residual o of the (t-1)th frame is predicted using a convolutional layer C. t-1，t The offset residual o in frame t+1 t+1，t The motion mask m of frame t-1 t-1，t The motion mask m of frame t+1 t+1，t The relationship is expressed as follows:

[0034] o t-1，t o t+1，t m t-1，t m t+1，t =C(Concat(O) t-1，t O t+1，t , X′ t-1 , X′ t+1 ))

[0035] Where Concat(·) represents a join operation;

[0036] Then, twist X again. t-1 and X t+1 get:

[0037]

[0038] Where D(·) represents the deformable convolution operation; X′ t-1Represents the warping feature of the (t-1)th frame; X′ t+1 This represents the warping feature of the (t+1)th frame;

[0039] Finally, X t , and Connected together, features are fused and reduced in dimensionality using a multilayer perceptron, dynamically outputting new features for the t-th frame.

[0040] Preferably, the high-resolution video sequence is obtained using the following formula:

[0041]

[0042] Where UP(·) represents the Pixelshuffle upsampling operation; This indicates the operation of the sixth Efficient Attention Module (EAB); This represents a 3×3 convolution; This represents the new sequence features after pixel shuffle upsampling and convolution; This represents the low-resolution video image of the i-th frame; This represents the i-th low-resolution video image after Pixelshuffle upsampling; This indicates a channel-level addition operation; This represents the high-resolution video image of the i-th frame.

[0043] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0044] This invention provides a video super-resolution method based on gated mutual attention. It utilizes a gated mutual attention (GMA) module within a multi-scale encoding / decoding model, capable of multi-frame prediction and long-range temporal dependency modeling, and combines optical flow estimation to leverage local and spatiotemporal correlation information at different scales for joint motion estimation, feature alignment, and feature fusion of shallow sequence features. Furthermore, a bidirectional parallel warp (BPW) module is employed to further fuse information from adjacent video frames through parallel feature warping, improving VSR performance by performing feature propagation and alignment. Moreover, the multi-scale encoding / decoding model calculates attention scores at the channel level, making the computational complexity linearly related to the spatial resolution of the input image, effectively saving computational resources. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the implementation steps of the technical solution of the present invention;

[0046] Figure 2 This is a schematic diagram of the overall framework of the present invention;

[0047] Figure 3 This is a schematic diagram of the framework of the efficient attention module in this invention;

[0048] Figure 4 This is a schematic diagram of the framework of the gated mutual attention module in this invention;

[0049] Figure 5 This is a schematic diagram of the framework of the multi-head depth transpose attention module in this invention;

[0050] Figure 6 This is a schematic diagram of the framework of the bidirectional parallel twisting module in this invention. Detailed Implementation

[0051] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0052] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0053] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0054] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] Example 1

[0056] like Figure 1-3 As shown, a video super-resolution method based on gated mutual attention includes the following steps:

[0057] S1: Obtain low-resolution video sequences and perform shallow sequence feature extraction;

[0058] S2: Using a multi-scale encoding and decoding model based on an efficient attention module (EAB) and a bidirectional parallel twist (BPW) module, and combining optical flow estimation, new sequence features are obtained by performing joint motion estimation and feature fusion based on shallow sequence features;

[0059] The High Efficiency Attention Module (EAB) includes multiple Gated Mutual Attention (GMA) modules;

[0060] S3: The new sequence features are upsampled by Pixelshuffle and convolutioned, and then added element-wise to the low-resolution video sequence to obtain a high-resolution video sequence.

[0061] In the specific implementation process, the gated mutual attention (GMA) module in the multi-scale encoding and decoding model, which has the ability to predict multiple frames and model long-range temporal dependencies, is used in conjunction with optical flow estimation to utilize local and spatiotemporal correlation information at different scales to perform joint motion estimation, feature alignment, and feature fusion on shallow sequence features. Furthermore, the bidirectional parallel warp (BPW) module is used to further fuse information from adjacent video frames through parallel feature warping, which can improve the performance of VSR by performing feature propagation and alignment. Moreover, the multi-scale encoding and decoding model calculates attention scores at the channel level, making the computational complexity linearly related to the spatial resolution of the input image, which can effectively save computational resources.

[0062] Example 2

[0063] A video super-resolution method based on gated mutual attention includes the following steps:

[0064] S1: Obtain low-resolution video sequences and perform shallow sequence feature extraction;

[0065] Extracting low-resolution video sequences X i : {X0, ..., X N Shallow sequence features of} It is expressed as follows:

[0066]

[0067] Among them, f S This indicates a shallow feature extraction module, which includes two convolutional layers, each followed by a ReLU activation function layer;

[0068] S2: Using a multi-scale encoding and decoding model based on an efficient attention module (EAB) and a bidirectional parallel twist (BPW) module, and combining optical flow estimation, new sequence features are obtained by performing joint motion estimation and feature fusion based on shallow sequence features;

[0069] The High Efficiency Attention Module (EAB) includes multiple Gated Mutual Attention (GMA) modules;

[0070] More specifically, the multi-scale encoding / decoding model includes an encoder end and a decoder end; wherein,

[0071] The encoder end includes an M-layer encoder. The feature output of the bidirectional parallel twist (BPW) module of the m-th layer encoder and the feature input of the efficient attention module (EAB) of the m+1-th layer encoder are downsampled by a 4×4 convolution with a stride of 2.

[0072] The decoder includes an M-layer decoder.

[0073] The feature output of the bidirectional parallel twist (BPW) module of the m-th layer decoder and the feature input of the efficient attention module (EAB) of the (m+1)-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

[0074] The feature output of the bidirectional parallel twist (BPW) module of the M-th layer encoder and the feature input of the efficient attention module (EAB) of the 1-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

[0075] More specifically, low-resolution sequence information is embedded into the encoder and decoder through the optical flow estimation module, and the output of the joint efficient attention module (EAB) provides twisting information to the bidirectional parallel twist (BPW) module.

[0076] More specifically, the efficient attention module (EAB) of the last layer of the decoder is not combined with the bidirectional parallel warp (BPW) module.

[0077] In practice, the efficient attention module (EAB) of the last layer of the decoder is not combined with the bidirectional parallel warp (BPW) module, thus preserving the detailed information between sequences.

[0078] More specifically, such as Figure 4-5 As shown, the gated mutual attention (GMA) module includes a multi-head depth transposed attention (MDTA) module and a multi-head mutual attention (MMA) module; wherein,

[0079] The Multi-Head Deep Transposed Attention (MDTA) module first encodes local information through three-branch 1×1 convolutions and 3×3 depthwise separable convolutions, generating query tensor features, key tensor features, and value tensor features. Then, the dimensions are transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate the transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and 1×1 convolution to obtain the output features of the Multi-Head Deep Transposed Attention (MDTA) module.

[0080] In the Multi-Head Mutual Attention (MMA) module, shallow sequence features are divided into sets of 2 frames at each layer and cross-interact with intermediate features output from other layers.

[0081] In practice, the MMA module does not have local induction bias and can assist the optical flow estimation module in joint motion estimation and feature fusion.

[0082] More specifically, the gated mutual attention (GMA) module further includes a deep convolutional gated feedforward network (DGFN) module. The DGFN module includes layer normalization, 1×1 convolution, ReLU activation function, element-wise multiplication, and channel addition operations. The features flowing into the DGFN module are first normalized, and then flow into the parallel two branches of 1×1 convolution and 3×3 deep convolution, with the number of feature channels doubled. The features of one branch are first activated by ReLU, and then element-wise multiplied with the features of the other branch to suppress irrelevant information. After passing through a 1×1 convolution, they are then added to the original features flowing into the DGFN module at the channel level.

[0083] More specifically, shallow sequence features The process of encoding and decoding after one layer is represented as:

[0084]

[0085] in, This indicates the operation of the first gated mutual attention (GMA) module; This indicates the operation of the second gated mutual attention (GMA) module; Represents the operation of the nth gated mutual attention (GMA) module; Φ(·) represents the optical flow estimation operation; f BPW (·) indicates the operation of the Bidirectional Parallel Twist (BPW) module.

[0086] In practical implementation, due to the division of the spatial window, the MMA module has difficulty handling large motions. Therefore, the BPW module is used to calculate the optical flow of the output features of the GMA module and its neighboring frame features, and then flips them to the previous and next frames. Figure 6 As shown, the sequence features flowing into the BPW module first flow into the forward twist branch and the reverse twist branch to tilt the features to the previous frame and the next frame. Then, they are concatenated with the original features and feature fusion and dimensionality reduction are performed using a multilayer perceptron (MLP) before flowing into the next layer GMA module.

[0087] More specifically, the optical flow estimation operation is as follows:

[0088] For the feature X of frame t t , t≤T, given the estimated optical flow O t-1，t and O t+1，t The feature X of frame t-1 is distorted respectively. t-1 and the feature X of frame t+1 t+1 get:

[0089]

[0090] Where W(·) represents the image warping function operation; X′ t-1 X′ represents the initial warping feature of frame t-1; t+1 This represents the initial warping feature of the (t+1)th frame;

[0091] Next, the offset residual o of the (t-1)th frame is predicted using a convolutional layer C. t-1，t The offset residual o in frame t+1 t+1，t The motion mask m of frame t-1 t-1，t The motion mask m of frame t+1 t+1，t The relationship is expressed as follows:

[0092] o t-1，t o t+1，t m t-1，t m t+1，t =C(Concat(O) t-1，t O t+1，t , X′ t-1 , X′ t+1 ))

[0093] Where Concat(·) represents a join operation;

[0094] Then, twist X again. t-1 and X t+1 get:

[0095]

[0096] Where D(·) represents the deformable convolution operation; X′ t-1 Represents the warping feature of the (t-1)th frame; X′ t+1 This represents the warping feature of the (t+1)th frame;

[0097] Finally, X t , and Connected together, features are fused and reduced in dimensionality using a multilayer perceptron, dynamically outputting new features for the t-th frame.

[0098] S3: The new sequence features are upsampled by Pixelshuffle and convolutioned, and then added element-wise to the low-resolution video sequence to obtain a high-resolution video sequence.

[0099] Example 3

[0100] A video super-resolution method based on gated mutual attention includes the following steps:

[0101] S1: Obtain low-resolution video sequences and perform shallow sequence feature extraction;

[0102] Extracting low-resolution video sequences X i : {x0, ..., X N Shallow sequence features of} It is expressed as follows:

[0103]

[0104] Among them, f S This indicates a shallow feature extraction module, which includes two convolutional layers, each followed by a ReLU activation function layer;

[0105] S2: Using a multi-scale encoding and decoding model based on an efficient attention module (EAB) and a bidirectional parallel twist (BPW) module, and combining optical flow estimation, new sequence features are obtained by performing joint motion estimation and feature fusion based on shallow sequence features;

[0106] The High Efficiency Attention Module (EAB) includes multiple Gated Mutual Attention (GMA) modules;

[0107] More specifically, the multi-scale encoding / decoding model includes an encoder end and a decoder end; wherein,

[0108] The encoder end includes an M-layer encoder. The feature output of the bidirectional parallel twist (BPW) module of the m-th layer encoder and the feature input of the efficient attention module (EAB) of the m+1-th layer encoder are downsampled by a 4×4 convolution with a stride of 2.

[0109] The decoder includes an M-layer decoder.

[0110] The feature output of the bidirectional parallel twist (BPW) module of the m-th layer decoder and the feature input of the efficient attention module (EAB) of the (m+1)-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

[0111] The feature output of the bidirectional parallel twist (BPW) module of the M-th layer encoder and the feature input of the efficient attention module (EAB) of the 1-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

[0112] More specifically, low-resolution sequence information is embedded into the encoder and decoder through the optical flow estimation module, and the output of the joint efficient attention module (EAB) provides twisting information to the bidirectional parallel twist (BPW) module.

[0113] More specifically, the efficient attention module (EAB) of the last layer of the decoder is not combined with the bidirectional parallel warp (BPW) module.

[0114] More specifically, N=3, the features output by the bidirectional parallel twist (BPW) modules of the encoders at layers 1 and 2 are transmitted to the bidirectional parallel twist (BPW) modules at layers 2 and 1 of the decoder via skip connections, respectively.

[0115] More specifically, the gated mutual attention (GMA) module includes a multi-head depth transposed attention (MDTA) module and a multi-head mutual attention (MMA) module; wherein,

[0116] The Multi-Head Deep Transposed Attention (MDTA) module first encodes local information through three-branch 1×1 convolutions and 3×3 depthwise separable convolutions, generating query tensor features, key tensor features, and value tensor features. Then, the dimensions are transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate the transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and 1×1 convolution to obtain the output features of the Multi-Head Deep Transposed Attention (MDTA) module.

[0117] Given the input features F of the GMA module, a layer normalization operation is first performed, followed by channel-level spatial context encoding using 1×1 and 3×3 depthwise convolutions in the Multi-Head Deep Transposed Attention (MDTA) module. Next, a dot product multiplication reconstruction is performed on the query and key projection to generate a transposed attention map.

[0118] The process of the MDTA module is shown in the following formula:

[0119]

[0120] in, Indicates dimensional reshaping; LN(·) represents a 3×3 depth-separable convolution operation; LN(·) represents a layer normalization operation. F represents the matrix multiplication operation; MDTA represents the output feature of the MDTA module; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represent the features of the query tensor; Represent the key tensor features; Represents the features of the value tensor;

[0121] In the Multi-Head Mutual Attention (MMA) module, shallow sequence features are divided into sets of 2 frames at each layer and cross-interact with intermediate features output from other layers.

[0122] In practice, the MDTA module models the global context across channels, making its complexity linear with the input image. The non-linear gating mechanism removes the non-linear activation function, enabling more efficient learning of feature representations and further reducing information redundancy.

[0123] The MMA module does not have local induction bias and can assist the optical flow estimation module in joint motion estimation and feature fusion.

[0124] More specifically, the gated mutual attention (GMA) module further includes a deep convolutional gated feedforward network (DGFN) module. The DGFN module includes layer normalization, 1×1 convolution, ReLU activation function, element-wise multiplication, and channel addition operations. The features flowing into the DGFN module are first normalized, and then flow into the parallel two branches of 1×1 convolution and 3×3 deep convolution, with the number of feature channels doubled. The features of one branch are first activated by ReLU, and then element-wise multiplied with the features of the other branch to suppress irrelevant information. After passing through a 1×1 convolution, they are then added to the original features flowing into the DGFN module at the channel level.

[0125] More specifically, shallow sequence features The process of encoding and decoding after one layer is represented as:

[0126]

[0127] in, This indicates the operation of the first gated mutual attention (GMA) module; This indicates the operation of the second gated mutual attention (GMA) module; Represents the operation of the nth gated mutual attention (GMA) module; Φ(·) represents the optical flow estimation operation; f BPW (·) indicates the operation of the Bidirectional Parallel Twist (BPW) module.

[0128] More specifically, the optical flow estimation operation is as follows:

[0129] For the feature X of frame t t , t≤T, given the estimated optical flow O t-1，t and O t+1，t The feature X of frame t-1 is distorted respectively. t-1 and the feature X of frame t+1 t+1 get:

[0130]

[0131] Where W(·) represents the image warping function operation; X′ t-1 X′ represents the initial warping feature of frame t-1; t+1 This represents the initial warping feature of the (t+1)th frame;

[0132] Next, the offset residual o of the (t-1)th frame is predicted using a convolutional layer C. t-1，t The offset residual o in frame t+1 t+1，t The motion mask m of frame t-1 t-1，t The motion mask m of frame t+1 t+1，t The relationship is expressed as follows:

[0133] o t-1，t o t+1，t m t-1，t m t+1，t =C(Concat(O) t-1，t O t+1，t , X′ t-1 , X′ t+1 ))

[0134] Where Concat(·) represents a join operation;

[0135] Then, twist X again. t-1 and X t+1 get:

[0136]

[0137] Where D(·) represents the deformable convolution operation; X′ t-1 Represents the warping feature of the (t-1)th frame; X′ t+1 This represents the warping feature of the (t+1)th frame;

[0138] Finally, X t , and Connected together, features are fused and reduced in dimensionality using a multilayer perceptron, dynamically outputting new features for the t-th frame.

[0139] S3: The new sequence features are upsampled by Pixelshuffle and convolutioned, and then added element-wise to the low-resolution video sequence to obtain a high-resolution video sequence.

[0140] More specifically, the high-resolution video sequence is obtained using the following formula:

[0141]

[0142] Where UP(·) represents the Pixelshuffle upsampling operation; This indicates the operation of the sixth Efficient Attention Module (EAB); This represents a 3×3 convolution; This represents the new sequence features after pixel shuffle upsampling and convolution; This represents the low-resolution video image of the i-th frame; This represents the i-th low-resolution video image after Pixelshuffle upsampling; This indicates a channel-level addition operation; This represents the high-resolution video image of the i-th frame.

[0143] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A video super-resolution method based on gated mutual attention, characterized in that, Includes the following steps: S1: Obtain low-resolution video sequences and perform shallow sequence feature extraction; S2: Using a multi-scale encoding / decoding model based on an efficient attention module and a bidirectional parallel twisting module, and in conjunction with optical flow estimation, new sequence features are obtained by performing joint motion estimation and feature fusion based on shallow sequence features. The high-efficiency attention module includes multiple gated mutual attention modules; S3: The new sequence features are upsampled by pixel shuffle and convolutioned, and then added element-wise to the low-resolution video sequence to obtain a high-resolution video sequence; The gated mutual attention module includes a multi-head depth transpose attention module and a multi-head mutual attention module; wherein... The multi-head deep transposed attention module first encodes local information through three-branch 1×1 convolutions and 3×3 depthwise separable convolutions, generating query tensor features, key tensor features, and value tensor features. Then, the dimensions are transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate the transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and 1×1 convolution to obtain the output features of the multi-head deep transposed attention module. In the multi-head mutual attention module, shallow sequence features are divided into sets of 2 frames in each layer and interact with intermediate features output from other layers. The gated mutual attention module also includes a deep convolutional gated feedforward network module. The features flowing into the deep convolutional gated feedforward network module are first normalized by layers, and then flow into the parallel two branches of 1×1 convolution and 3×3 deep convolution, and the number of feature channels is doubled. The features of one branch are first activated by the ReLU function, and then element-wise multiplied with the features of the other branch to suppress irrelevant information. Then, after passing through a 1×1 convolution, they are added to the original features flowing into the deep convolutional gated feedforward network module at the channel level.

2. The video super-resolution method based on gated mutual attention according to claim 1, characterized in that, The multi-scale encoding / decoding model includes an encoder and a decoder; wherein... The encoder end includes an M-layer encoder. The feature output of the bidirectional parallel twisting module of the m-th layer encoder and the feature input of the efficient attention module of the (m+1)-th layer encoder are downsampled by a 4×4 convolution with a stride of 2. The decoder includes an M-layer decoder. The feature output of the bidirectional parallel twisting module of the m-th layer decoder and the feature input of the efficient attention module of the (m+1)-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2. The feature output of the bidirectional parallel twisting module of the M-th layer encoder and the feature input of the efficient attention module of the 1-th layer decoder are upsampled by a 2×2 transposed convolution with a stride of 2.

3. The video super-resolution method based on gated mutual attention according to claim 2, characterized in that, Low-resolution sequence information is embedded into the encoder and decoder through the optical flow estimation module, and the output of the combined efficient attention module provides distortion information to the bidirectional parallel distortion module.

4. The video super-resolution method based on gated mutual attention according to claim 3, characterized in that, The efficient attention module of the last layer of the decoder is not combined with the bidirectional parallel twist module.

5. A video super-resolution method based on gated mutual attention according to claim 4, characterized in that, When N=3, the features output by the bidirectional parallel twisting modules of the encoders in layers 1 and 2 at the encoder end are transmitted to the bidirectional parallel twisting modules in layers 2 and 1 at the decoder end via skip connections, respectively.

6. The video super-resolution method based on gated mutual attention according to claim 1, characterized in that, Shallow sequence features The process of encoding and decoding through one layer can be represented as: in, This indicates the operation of the first gated mutual attention module; This indicates the operation of the second gated mutual attention module; Represents the nth gated mutual attention module operation; Φ represents the optical flow estimation operation; f BPW (·) indicates a bidirectional parallel twist module operation.

7. A video super-resolution method based on gated mutual attention according to claim 6, characterized in that, The optical flow estimation operation is as follows: For the t-th frame of features X t , given the estimated optical flow O t-1,t and O t+1,t , warp the t-1-th frame of features X t-1 and the t+1-th frame of features X t+1 respectively to get: where W(·) denotes an image warping function operation; X ′ t-1 denotes the initial warped feature of the t-1th frame; X ′ t+1 denotes the initial warped feature of the t+1th frame; Next, the offset residual o of the (t-1)th frame is predicted using a convolutional layer C. t-1,t The offset residual o in frame t+1 t+1,t The motion mask m of frame t-1 t-1,t The motion mask m of frame t+1 t+1,t The relationship is expressed as follows: o t-1,t ,o t+1,t ,m t-1,t ,m t+1,t =C(Concat(O t-1,t ,O t+1,t ,X′ t-1 ,X′ t+1 )) Where Concat(·) represents a join operation; Then, twist X again. t-1 and X t+1 get: Where D(·) represents deformable convolution operation; This represents the warping feature of the (t-1)th frame; This represents the warping feature of the (t+1)th frame; Finally, X t , and Connected together, features are fused and reduced in dimensionality using a multilayer perceptron, dynamically outputting new features for the t-th frame.

8. A video super-resolution method based on gated mutual attention according to claim 5, characterized in that, The high-resolution video sequence is obtained using the following formula: Where UP(·) represents the Pixelshuffle upsampling operation; This indicates the operation of the sixth efficient attention module; This represents a 3×3 convolution; This represents the new sequence features after pixel shuffle upsampling and convolution; This represents the low-resolution video image of the i-th frame; This represents the i-th low-resolution video image after Pixelshuffle upsampling; This indicates a channel-level addition operation; This represents the high-resolution video image of the i-th frame.

Images