Video super-resolution method and system based on strip-like and global spatiotemporal attenuated attention

Abstract

The present invention discloses a video super-resolution method and system based on strip-shaped and global spatiotemporal attenuation attention, which belongs to the field of video super-resolution technology. The present invention includes the following steps: 1. extracting the spatial features of the original video frame image after enhancement and up-sampling; 2. calculating the spatial strip-shaped attention and the temporal strip-shaped attention on the up-sampled spatial features; 3. processing the spatial strip-shaped attention and the temporal strip-shaped attention using depthwise separable convolution and residual structure respectively, respectively obtaining the final attention output, fusing the two attention outputs to obtain fused features; 4. calculating the global spatiotemporal attenuation attention of the spatial feature map, fusing the spatial features, the fused features and the global spatiotemporal attenuation attention to obtain the final high-resolution reconstruction result. The video super-resolution method proposed by the present invention can improve PSNR / SSIM and restore sharp detail information on commonly used data sets.

Description

Technical Field

[0001] The present invention belongs to the technical field of video super-resolution, and in particular designs a video super-resolution method and system based on strip-like and global spatiotemporal attenuation attention. Background Art

[0002] Video, a common multimedia format, consists of a series of consecutive images. Super-resolution aims to reconstruct a low-resolution input image or video into a high-resolution output. This task is a classic and challenging one in computer science, with numerous practical applications, such as satellite imagery, remote sensing, digital high-definition imaging, and microscopic imaging.

[0003] Super-resolution input can be broadly categorized into two types: images and videos. Image super-resolution aims to reconstruct a single low-resolution image into a high-resolution result; video super-resolution, on the other hand, takes multiple consecutive frames as input and leverages the relationships between them to restore the high-resolution image. Traditional video super-resolution methods have evolved through methods based on interpolation and motion estimation, but these methods struggle to effectively estimate complex motion in videos. With the widespread application of deep learning in various computing fields, the field of video super-resolution has also seen extensive research on deep learning-based methods. Currently, deep learning-based video super-resolution methods primarily include three architectures: convolutional neural network (CNN)-based methods, which use a sliding window approach to extract effective detail features between adjacent video frames for computation; recurrent neural network (RNN)-based methods, which utilize hidden states to focus on historical information in previous video frames and future information in future video frames for computation; and Transformer-based methods, which utilize a self-attention mechanism to extract both long-range and short-range features for computation.

[0004] However, existing methods suffer from high computational complexity and high memory usage, limiting the model's deployment on devices. Furthermore, for video super-resolution, the model also needs to process information in the temporal dimension, increasing computational complexity and limiting model training on long frames. Furthermore, existing methods pay insufficient attention to prior details of different spatial and temporal locations in the video, lacking a reasonable approach to modeling key, effective information within the video frame. These shortcomings result in poor visual quality from existing video super-resolution methods. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention discloses a video super-resolution method and system based on strip-shaped and global spatiotemporal attenuated attention, and proposes to perform strip-shaped attention calculation to reduce the computational complexity of the neural network from the square level to the linear level, making the model easy to deploy; using the attenuated attention matrix, it better processes the prior detail information of different spatial positions and different temporal positions in the video, helping the network to better consider important information when reconstructing high-resolution videos, so that the super-resolution video has lower computational complexity and better visual effects.

[0006] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0007] In a first aspect, the present invention proposes a video super-resolution method based on strip-wise and global spatiotemporal attenuated attention, comprising the following steps:

[0008] Step (1). Obtain low-resolution video and perform image enhancement on each video frame;

[0009] Step (2). Extract the original spatial features of each video frame after image enhancement and interpolate the original spatial features to obtain the upsampled spatial features of each video frame;

[0010] Step (3). For the upsampled spatial features obtained in step (2), calculate the spatial strip attention of each video frame and the temporal strip attention between frames;

[0011] Step (4). The spatial strip attention and temporal strip attention obtained in step (3) are processed using depthwise separable convolution and residual structure respectively, and then the two results are fused to obtain the fused features of each video frame;

[0012] Step (5). For the upsampled spatial features obtained in step (2), obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuation attention;

[0013] Step (6). The original spatial features obtained in step (2), the fused features obtained in step (4), and the global spatiotemporal attenuated attention obtained in step (5) are fused and reconstructed to obtain a high-resolution reconstruction result for each video frame; all video frames are traversed to obtain a high-resolution video after super-resolution.

[0014] In steps (1) and (2), the network first performs image enhancement on the low-resolution video sequence by rotating and stacking, and then extracts the spatial features of each frame of the video sequence. The extracted shallow features retain the original information of the video frame more completely, and serve as a guide in the subsequent video frame processing process to help the network reconstruct the high-resolution video frame. These shallow features are then subjected to strip attention calculations in the spatial and temporal dimensions and global spatiotemporal decay attention calculations to perform feature reconstruction.

[0015] In the steps (3)-(4), the spatial strip attention and temporal strip attention of the low-resolution video sequence are calculated respectively. When calculating the spatial strip attention, the video sequence is divided into a horizontal feature tensor and a vertical feature tensor in the channel dimension; the same is true when calculating the temporal strip attention. The advantage of using strip attention is that it uses spatial and temporal strip attention to extract spatial and temporal information respectively, which can more efficiently capture long-distance dependencies in the video than the global self-attention mechanism; the strip attention mechanism reduces memory consumption, and its spatial and temporal memory complexity are: O((H 2 +W 2 )T) and O((H+W)T 2 ), which is significantly lower than the original Transformer’s time and space complexity O((HWT) 2 ).

[0016] In step (5), the Manhattan decay matrix of the time, height, and width dimensions of each low-resolution video sequence frame is calculated, and then the spatiotemporal attention is calculated to obtain the global spatiotemporal decay attention. The advantage of using the decay attention matrix is ​​that it can better process the prior detail information of different spatial and temporal positions in the video, helping the network to better consider important information when reconstructing the high-resolution video.

[0017] In the step (6), the spatial features obtained in steps (1)-(2), the spatial and temporal strip fusion features obtained in steps (3)-(4), and the global spatiotemporal attenuation attention obtained in step (5) are fused, and the features are reconstructed and upsampled to obtain a super-resolution high-resolution result.

[0018] Furthermore, in the step (1), the image enhancement includes horizontal 90° flipping, vertical 90° flipping, and mirror symmetry.

[0019] Furthermore, in the step (2), the channel dimension of each video frame of the low-resolution video after image enhancement in step (1) is expanded from 3 to 64 while the resolution remains unchanged, thereby obtaining the original spatial features of each video frame.

[0020] Furthermore, the step (3) includes:

[0021] Step (3.1). Perform layer normalization and feature segmentation on the upsampled spatial features to obtain two types of feature tensors, namely spatial horizontal feature tensors and spatial vertical feature tensors;

[0022] Step (3.2). The spatial horizontal feature tensor and spatial vertical feature tensor obtained in step (3.1) are mapped to query, key and value respectively;

[0023] Step (3.3). For the query, key, and value obtained in step (3.2), respectively calculate the spatial horizontal attention output and the spatial vertical attention output; the spatial horizontal attention output and the spatial vertical attention output are used as the spatial strip attention result;

[0024] Step (3.4). For the upsampled spatial features of each video frame, consider the feature expressions of the n frames before and after it to obtain the joint upsampled spatial features X t ′ represents the upsampled spatial features of the t-th video frame;

[0025] Step (3.5). Combine the joint upsampled spatial features obtained in step (3.4) Perform layer normalization and feature segmentation to obtain two types of feature tensors, namely time horizontal feature tensor and time longitudinal feature tensor;

[0026] Step (3.6). Split the temporal transverse feature tensor and temporal longitudinal feature tensor obtained in step (3.5) into a number of transverse strips with non-overlapping temporal dimensions and the same spatial position, and a number of longitudinal strips with non-overlapping temporal dimensions and the same spatial position, respectively;

[0027] Step (3.7). For the horizontal strips and vertical strips obtained in step (3.6), map them to query, key, and value respectively;

[0028] Step (3.8). For the queries, keys and values ​​of the horizontal strips and vertical strips obtained in step (3.7), calculate their temporal horizontal attention output and temporal vertical attention output respectively; the temporal horizontal attention output and temporal vertical attention output are used as the temporal strip attention results.

[0029] Furthermore, the step (4) includes:

[0030] Step (4.1). Concatenate the spatial horizontal attention output and the spatial vertical attention output in the spatial strip attention and perform a depth-wise separable convolution operation to obtain the spatial convolution result; concatenate the temporal horizontal attention output and the temporal vertical attention output in the temporal strip attention and perform a depth-wise separable convolution operation to obtain the temporal convolution result;

[0031] Step (4.2). Apply residual structure and normalization to the spatial convolution result to obtain the final spatial strip attention; apply residual structure and normalization to the temporal convolution result to obtain the final temporal strip attention;

[0032] Step (4.3). Add the final spatial strip attention and the final spatial strip attention to obtain the fused feature.

[0033] Furthermore, the step (5) includes:

[0034] Step (5.1). For the upsampled spatial features of each video frame, calculate the Manhattan attenuation matrix of its time dimension, the Manhattan attenuation matrix of its height dimension, and the Manhattan attenuation matrix of its width dimension respectively;

[0035] Step (5.2). Add the three-dimensional Manhattan attenuation matrices element by element to obtain a three-dimensional Manhattan attenuation matrix;

[0036] Step (5.3). Map the upsampled spatial features of each video frame into query, key and value, and perform attention calculation in combination with the three-dimensional Manhattan attenuation matrix to obtain the global spatiotemporal attenuated attention result.

[0037] Furthermore, the step (6) includes:

[0038] Step (6.1). Add the fusion features of each video frame and the global spatiotemporal attenuated attention to obtain the spliced ​​features;

[0039] Step (6.2). Upsample the spliced ​​features, and perform residual connection between the upsampled results and the original spatial features of each video frame to obtain the residual features;

[0040] Step (6.3). Perform 3×3 convolution and sub-pixel convolution on the residual features to obtain the final high-resolution output;

[0041] Step (6.4). Traverse all video frames to obtain a high-resolution result after the entire video resolution is increased by 4 times.

[0042] In a second aspect, the present invention proposes a video super-resolution system based on strip-shaped and global spatiotemporal attenuated attention, which is used to implement the above-mentioned video super-resolution method based on strip-shaped and global spatiotemporal attenuated attention.

[0043] Beneficial effects of the present invention:

[0044] This paper designs a complete network structure for video super-resolution based on strip-based and global spatiotemporal attenuated attention. The structure comprises a data preprocessing module, a spatial feature extraction module, a strip-based attention calculation module, an attenuated attention calculation module, and a super-resolution reconstruction module, forming a multi-layered network architecture. The present invention utilizes the strip-based attention mechanism to significantly reduce spatiotemporal complexity while retaining a large amount of detailed information. Furthermore, the attenuated attention mechanism effectively accounts for the importance of distance dependence in the video reconstruction process, effectively reconstructing the video's detailed information and achieving a better visual effect after super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 4 is a structural block diagram of a video super-resolution method based on strip-shaped and global spatiotemporal attenuated attention adopted in an embodiment of the present invention;

[0046] Figure 2 is an overall flow chart of an embodiment of the present invention;

[0047] Figure 3 : is a structural block diagram of a strip attention calculation module in an embodiment of the present invention;

[0048] Figure 4 This is a structural block diagram of the attenuated attention calculation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The method of the present invention will be further described below with reference to the accompanying drawings.

[0050] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0051] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0052] The structural block diagram of the video super-resolution method based on strip and global spatiotemporal attenuation attention of the present invention is as follows: Figure 1 As shown in Figure 2, five modules are designed in this method: data preprocessing module, spatial feature extraction module, strip attention calculation module, attenuated attention calculation module and super-resolution reconstruction module. The flow chart of the video super-resolution method implemented by each module is shown in Figure 2. Figure 2 shown.

[0053] The data preprocessing module is used to process the input original video data stream, obtain low-resolution video and perform image enhancement on each video frame; specifically, the method in the following step (1) is executed.

[0054] Step (1). Obtain a low-resolution video sequence and perform mirror symmetry, horizontal 90° flip, and vertical 90° flip on each frame to achieve image enhancement. The low-resolution video sequence after image enhancement is recorded as I t LR , where t represents the t-th frame image; it is then input into the spatial feature extraction module frame by frame.

[0055] The spatial feature extraction module is used to extract the spatial features of each frame image in the low-resolution video and interpolate the original spatial features to obtain the upsampled spatial features of each video frame; specifically, the method in the following step (2) is executed.

[0056] Step (2). Take the t(t∈[1,T])th frame of the low-resolution video obtained in step (1) The convolutional layer is used to change the channel dimension of the input frame from 3 to 64, and then the original spatial feature extraction is performed.

[0057] The spatial feature extraction process is expressed as:

[0058]

[0059] Among them, X t Represents the t-th frame image The original spatial features of , R(.) represents the stacked residual block structure used to extract features, RELU(.) represents the activation function, and conv(.) represents the convolutional layer. The spatial resolution is kept unchanged during the spatial feature calculation process.

[0060] Interpolate the spatial features of each frame of the low-resolution video with an upsampling rate of 0.5:

[0061] X t ′=Bilinear(X t ,0.5)

[0062] Among them, X t ′ represents the t-th frame picture X t The upsampled spatial features, the spatial resolution is the t-th frame image X t 0.5 times, Bilinear(.,.) represents the bilinear interpolation method. At this time, the feature expression after upsampling of each frame is: in t∈{1,2,...,T} and X t ′ has a channel dimension of Xt 4 times the channel dimension, that is

[0063] A core design point of the present invention is to use the strip attention calculation module to calculate the strip attention of each frame in the low-resolution video, such as Figure 3 As shown, the strip attention calculation module is used to calculate the spatial strip attention of each video frame and the temporal strip attention between frames for the upsampled spatial features; the spatial strip attention and the temporal strip attention are processed using depthwise separable convolution and residual structure respectively, and then the results of the two are fused to obtain the fused features of each video frame; specifically, the methods in the following steps (3)-(4) are executed.

[0064] Step (3). For the spatial features obtained in step (2), calculate the spatial strip attention of each video frame and the temporal strip attention between frames, specifically:

[0065] Step (3.1). The upsampled spatial feature X obtained in step (2) t 'Perform layer normalization and segmentation according to channel dimension features to obtain two types of feature tensors, namely spatial horizontal feature tensors and spatial longitudinal feature tensor

[0066]

[0067] in, and Represents the horizontal and vertical feature tensors respectively, and the number of channels of each tensor is

[0068] Step (3.2). For the horizontal feature tensor and vertical feature tensor obtained in step (3.1), map them into queries: Q (h) , Q (v) ; Key: K (h) , K (v) Sum V (h) 、V (v) , for the lateral feature tensor:

[0069]

[0070] Among them, W Q 、W K and W V is the linear transformation matrix; the same is true for the longitudinal feature tensor:

[0071]

[0072] Step (3.3). For the query, key, and value of the horizontal feature tensor and vertical feature tensor obtained in step (3.2), calculate their spatial horizontal attention outputs respectively and spatial longitudinal attention output

[0073]

[0074]

[0075] Among them, H′, W′ and C′ represent the t-th frame image X t The upsampled spatial features X t ’’s length, width, and number of channels.

[0076] The above spatial horizontal attention output and spatial longitudinal attention output As a result of spatial strip attention.

[0077] Step (3.4). For each frame of the image, the upsampled spatial feature X t ′, consider the feature expression of each n frames before and after, and record X at this time t 'for

[0078]

[0079] Wherein, n is a positive integer, and its value range is 0≤n≤min(t-1,Tt).

[0080] Step (3.5). The features obtained in step (3.4) Perform layer normalization and segmentation by channel dimension features to obtain two feature tensors, namely the time horizontal feature tensor and the time longitudinal feature tensor

[0081]

[0082] in, and Represents the horizontal and vertical feature tensors respectively, and the number of channels of each tensor is

[0083] Step (3.7). The time lateral feature tensor obtained in step (3.6) and the time longitudinal feature tensor Divide into H′ horizontal strips with non-overlapping time dimensions and the same spatial position and W′ longitudinal strips that do not overlap in time dimensions and have the same spatial position

[0084]

[0085]

[0086] Among them, a∈{1,2,...,H′}, b∈{1,2,...,W′}.

[0087] Step (3.8). For the horizontal strips obtained in step (3.7) and longitudinal stripes Map them into queries respectively: key: Sum For horizontal strips:

[0088]

[0089] Among them, W Q 、W K and W V is the linear transformation matrix; the same is true for the vertical strips:

[0090]

[0091] Step (3.9). For the query, key and value of the horizontal strip and vertical strip obtained in step (3.8), calculate their temporal horizontal attention output respectively and temporal longitudinal attention output

[0092]

[0093]

[0094] Among them, H′, W′ and C′ represent the t-th frame image X t The upsampled spatial features X t ’’s length, width, and number of channels.

[0095] The above temporal horizontal attention output and temporal longitudinal attention output As a temporal strip of attention results.

[0096] Step (4). The spatial strip attention and temporal strip attention obtained in step (3) are processed using depthwise separable convolution and residual structure respectively, and then the two results are fused to obtain spatiotemporal strip fusion attention, specifically:

[0097] Step (4.1). Output the spatial horizontal attention obtained in step (3.4) and spatial longitudinal attention output Perform splicing and depth-separable convolution operations to obtain spatial convolution results

[0098]

[0099] in, Denotes concatenation, and DepthwiseConv(.) denotes depthwise separable convolution.

[0100] Step (4.2). The spatial convolution result obtained in step (4.1) Using the residual structure and normalization processing, we get the final spatial strip attention O space :

[0101]

[0102] Among them, RELU(.) represents nonlinear activation, Conv 3×3 (.) represents a 3×3 convolution, and GN(.) represents a group normalization operation.

[0103] Step (4.3). The temporal horizontal attention output obtained in step (3.9) and temporal longitudinal attention output Repeat steps (4.1)-(4.2) to get the final temporal strip attention O time ;

[0104] Step (4.4). The spatial strip attention O obtained in step (4.2) space And the temporal strip attention O obtained in step (4.3) time Add them together to get the spatiotemporal strip fusion attention O fusion :

[0105] O fusion =O space +O time

[0106] In the video super-resolution task, it is also necessary to consider the details of the time and space dimensions, and the importance of the details of the two dimensions must be calculated separately. Figure 4 As shown, the attenuated attention calculation module is used to upsample the spatial features, obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to calculate the global spatiotemporal attenuated attention; specifically, the method in the following step (5) is executed.

[0107] Step (5). Upsample the spatial feature X obtained in step (2) t ′, and obtain the three-dimensional Manhattan attenuation matrix of the spatial features of each frame respectively, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuation attention, specifically:

[0108] Step (5.1). For the upsampled feature X obtained in step (2) t ′, calculate the Manhattan decay matrix D of its time dimension T,man ; In the time dimension T, for any two time steps t i and t j , the calculation formula of Manhattan distance is:

[0109] manDis tan ce(t i ,t j )=|t i -t j |

[0110] The Manhattan decay matrix of the time dimension is obtained as:

[0111] D T,man [i,j]=α·manDis tan ce(t i ,t j )

[0112] Among them, α is the attenuation factor, which is used to control the influence of Manhattan distance on attenuation.

[0113] Step (5.2). For the upsampled feature X obtained in step (3.1) t ′, calculate the Manhattan attenuation matrix D of its height dimension H′,man ; In the height dimension H′, for any two height positions h j and h k , the calculation formula of Manhattan distance is:

[0114] manDis tan ce(h j ,h k )=|h j -h k |

[0115] The resulting height-dimensional Manhattan attenuation matrix is:

[0116] D H′,man [j,k]=β·manDis tan ce(h j ,h k )

[0117] Among them, β is the attenuation factor, which is used to control the influence of Manhattan distance on attenuation.

[0118] Step (5.3). For the upsampled feature X obtained in step (3.1) t ′, calculate the Manhattan attenuation matrix D of its width dimension W′,man; In the width dimension W′, for any two height positions w k and w i , the calculation formula of Manhattan distance is:

[0119] manDis tan ce(w k ,w i )=|w k -w i |

[0120] The resulting Manhattan attenuation matrix of the width dimension is:

[0121] D W′,man [k,i]=γ·manDis tan ce(w k ,w i )

[0122] Among them, γ is the attenuation factor, which is used to control the influence of Manhattan distance on attenuation.

[0123] Step (5.4). Add the three-dimensional Manhattan attenuation matrices obtained in steps (5.1), (5.2), and (5.3) element by element to obtain the final three-dimensional Manhattan attenuation matrix D man :

[0124] D man [i,j,k]=D T,man [i,j]+D H′,man [j,k]+D W′,man [k,i]

[0125] Among them, i, j and k correspond to the indexes in the time, spatial height and width dimensions respectively.

[0126] Step (5.5). For the upsampled feature X obtained in step (2) t ′, and maps it into query Q, key K and value V:

[0127] Q=XW Q ′,K=XW K ′,V=XW V '

[0128] Among them, W Q ′、W K ′ and W V ′ is the weight matrix of the linear transformation.

[0129] Step (5.6). Perform attention calculation on the query Q, key K and value V of the upsampled features obtained in step (5.5) to obtain the global spatiotemporal attenuated attention O decay :

[0130] Odecay =Soft max((Q·K T )⊙D man )·V

[0131] Where ⊙ represents element-wise multiplication.

[0132] After calculating the spatiotemporal strip fusion attention and spatiotemporal fusion attenuation attention for each frame of the low-resolution video, the super-resolution reconstruction module is used to fuse and reconstruct the upsampled spatial features, fusion features and global spatiotemporal attenuation attention to obtain a high-resolution reconstruction result for each video frame; specifically, the method in the following step (6) is executed.

[0133] Step (6). Fuse the original spatial features obtained in step (2), the spatiotemporal strip fusion attention obtained in step (4), and the global spatiotemporal attenuation attention obtained in step (5) to obtain a high-resolution reconstruction result for each video frame, specifically:

[0134] Step (6.1). For the spatiotemporal strip fusion attention O obtained in step (4) fusion And the global spatiotemporal attenuation attention O obtained in step (5) decay Add and get the added feature O out :

[0135] O out =O fusion +O decay

[0136] Step (6.2). For the added feature O obtained in step (6.1) out , perform 2 times bilinear interpolation upsampling, and change its dimension from H′×W′×C (where, ) becomes To facilitate O out With X t Perform residual connection to obtain residual feature O res :

[0137] O res =X t +Bilinear(O out , 2)

[0138] Among them, Bilinear(·,·) represents the bilinear interpolation method,

[0139] Step (6.3) for the residual feature O res , use 3×3 convolution to adjust the channel so that its dimension changes from It becomes H×W×C to facilitate subsequent high-resolution upsampling operations; then PixelShuffle is used to perform sub-pixel convolution, making its dimension become Finally, 3×3 convolution is used to adjust the channels to obtain the final high-resolution output X HR , the dimension of the output feature is 2H×2W×3:

[0140] X HR =Conv 3×3 (PixelShuffle(Conv 3×3 (O res ), 4))

[0141] Among them, Conv 3×3 (·) represents a 3×3 convolution operation, PixelShuffle(·,4) represents a sub-pixel convolution operation, and 4 represents a 4x reduction in the channel size, expanding the channel dimension to the spatial resolution. This achieves a 4x increase in resolution. In this embodiment, 4x can also be replaced by other multiples of perfect square numbers, such as 9x.

[0142] Step (6.4). Traverse all video frames to obtain a high-resolution result after the entire video resolution is increased by 4 times.

[0143] The combination of the image preprocessing module, spatial feature extraction module, strip attention calculation module, attenuated attention calculation module, and super-resolution reconstruction module described above can construct a complete video super-resolution system based on strip and global spatiotemporal attenuated attention. Each module may or may not be physically separated, may be located in one place, or may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. Those skilled in the art can understand and implement it without any creative effort.

[0144] The video super-resolution system based on strip-shaped and global spatiotemporal attenuated attention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, as a logical device, software is implemented by the processor of any device with data processing capabilities reading the corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0145] The above description is merely a specific embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the concept of this application. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A video super-resolution method based on strip-wise and global spatiotemporal attenuated attention, characterized in that: The following steps are involved: Step (1). Obtain low-resolution video and perform image enhancement on each video frame; Step (2). Extract the original spatial features of each video frame after image enhancement and interpolate the original spatial features to obtain the upsampled spatial features of each video frame; Step (3). For the upsampled spatial features obtained in step (2), calculate the spatial strip attention of each video frame and the temporal strip attention between frames; Step (4). The spatial strip attention and temporal strip attention obtained in step (3) are processed using depthwise separable convolution and residual structure respectively, and then the two results are fused to obtain the fused features of each video frame; Step (5). For the upsampled spatial features obtained in step (2), obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuation attention; Step (6). The original spatial features obtained in step (2), the fused features obtained in step (4), and the global spatiotemporal attenuated attention obtained in step (5) are fused and reconstructed to obtain a high-resolution reconstruction result for each video frame; all video frames are traversed to obtain a high-resolution video after super-resolution.

2. The video super-resolution method based on strip-wise and global spatiotemporal attenuated attention according to claim 1, characterized in that: In the step (1), the image enhancement includes horizontal 90° flipping, vertical 90° flipping, and mirror symmetry.

3. The video super-resolution method based on strip-like and global spatiotemporal attenuated attention according to claim 1, characterized in that: In the step (2), the channel dimension of each video frame of the low-resolution video after image enhancement in step (1) is expanded from 3 to 64 while the resolution remains unchanged, thereby obtaining the original spatial features of each video frame.

4. The video super-resolution method based on strip-wise and global spatiotemporal attenuated attention according to claim 1, characterized in that: The step (3) comprises: Step (3.1). Perform layer normalization and feature segmentation on the upsampled spatial features to obtain two types of feature tensors, namely spatial horizontal feature tensors and spatial vertical feature tensors; Step (3.2). The spatial horizontal feature tensor and spatial vertical feature tensor obtained in step (3.1) are mapped to query, key and value respectively; Step (3.3). For the query, key, and value obtained in step (3.2), respectively calculate the spatial horizontal attention output and the spatial vertical attention output; the spatial horizontal attention output and the spatial vertical attention output are used as the spatial strip attention result; Step (3.4). For the upsampled spatial features of each video frame, consider the feature expressions of the n frames before and after it to obtain the joint upsampled spatial features X t ′ represents the upsampled spatial features of the t-th video frame; Step (3.5). The joint up-sampled spatial features obtained in step (3.4) Perform layer normalization and feature segmentation to obtain two types of feature tensors, namely time horizontal feature tensor and time longitudinal feature tensor; Step (3.6). Split the temporal horizontal feature tensor and the temporal vertical feature tensor obtained in step (3.5) into a number of horizontal strips with non-overlapping temporal dimensions and the same spatial position, and a number of vertical strips with non-overlapping temporal dimensions and the same spatial position, respectively; Step (3.7). For the horizontal strips and vertical strips obtained in step (3.6), map them to query, key, and value respectively; Step (3.8). For the queries, keys and values ​​of the horizontal strips and vertical strips obtained in step (3.7), calculate their temporal horizontal attention output and temporal vertical attention output respectively; the temporal horizontal attention output and temporal vertical attention output are used as the temporal strip attention results.

5. The video super-resolution method based on strip-like and global spatiotemporal attenuated attention according to claim 1, characterized in that: The step (4) comprises: Step (4.1). Concatenate the spatial horizontal attention output and the spatial vertical attention output in the spatial strip attention and perform a depth-wise separable convolution operation to obtain the spatial convolution result; concatenate the temporal horizontal attention output and the temporal vertical attention output in the temporal strip attention and perform a depth-wise separable convolution operation to obtain the temporal convolution result; Step (4.2). Apply residual structure and normalization to the spatial convolution result to obtain the final spatial strip attention; apply residual structure and normalization to the temporal convolution result to obtain the final temporal strip attention; Step (4.3). Add the final spatial strip attention and the final spatial strip attention to obtain the fused feature.

6. The video super-resolution method based on strip-wise and global spatiotemporal attenuated attention according to claim 1, characterized in that: The step (5) comprises: Step (5.1). For the upsampled spatial features of each video frame, calculate the Manhattan attenuation matrix of its time dimension, the Manhattan attenuation matrix of its height dimension, and the Manhattan attenuation matrix of its width dimension respectively; Step (5.2). Add the three-dimensional Manhattan attenuation matrices element by element to obtain a three-dimensional Manhattan attenuation matrix; Step (5.3). Map the upsampled spatial features of each video frame into query, key and value, and perform attention calculation in combination with the three-dimensional Manhattan attenuation matrix to obtain the global spatiotemporal attenuated attention result.

7. The video super-resolution method based on strip-like and global spatiotemporal attenuated attention according to claim 1, characterized in that: The step (6) comprises: Step (6.1). Add the fusion features of each video frame and the global spatiotemporal attenuated attention to obtain the spliced ​​features; Step (6.2). Upsample the spliced ​​features, and perform residual connection between the upsampled results and the original spatial features of each video frame to obtain the residual features; Step (6.3). Perform 3×3 convolution and sub-pixel convolution on the residual features to obtain the final high-resolution output; Step (6.4). Traverse all video frames to obtain a high-resolution result after the entire video resolution is increased by 4 times.

8. A video super-resolution system based on strip-like and global spatiotemporal attenuation attention, used to implement the video super-resolution method according to claim 1, characterized in that: The system comprises: An image preprocessing module is used to obtain low-resolution videos and perform image enhancement on each video frame; A spatial feature extraction module is used to extract the original spatial features of each video frame after image enhancement and interpolate the original spatial features to obtain the upsampled spatial features of each video frame; The strip attention calculation module is used to calculate the spatial strip attention of each video frame and the temporal strip attention between frames for the upsampled spatial features. The spatial strip attention and temporal strip attention are processed using depthwise separable convolution and residual structure respectively, and then the results of the two are fused to obtain the fused features of each video frame. The attenuated attention calculation module is used to obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuated attention; The super-resolution reconstruction module is used to fuse and reconstruct the original spatial features, fused features and global spatiotemporal attenuated attention to obtain a high-resolution reconstruction result for each video frame; it traverses all video frames to obtain a high-resolution video after super-resolution.

Images