Video super-resolution method and system based on strip-shaped and global time-space attenuation attention

Abstract

The invention discloses a video super-resolution method and system based on strip-shaped and global time-space attenuation attention, and belongs to the technical field of video super-resolution. The method comprises the following steps of: 1, extracting spatial features of an original video frame image after enhancement and performing up-sampling; 2, calculating spatial strip attention and time strip attention on the upsampling spatial features; 3, processing the spatial strip-shaped attention and the time strip-shaped attention by using depth separable convolution and a residual structure to obtain final attention outputs, and performing fusion processing on the two attention outputs to obtain fusion features; and 4, calculating the global space-time attenuation attention of the spatial feature map, and fusing the spatial features, the fusion features and the global space-time attenuation attention to obtain a final high-resolution reconstruction result. Based on the video super-resolution method provided by the invention, the PSNR / SSIM can be improved on a common data set, and sharp detail information can be recovered.

Description

Technical Field

[0001] The present invention belongs to the technical field of video super-resolution, and in particular designs a video super-resolution method and system based on strip and global spatiotemporal attenuation attention. Background Art

[0002] Video, as a common form of multimedia, is a sequence of continuous images. The super-resolution task aims to reconstruct the input low-resolution image or video into a high-resolution output. This task is a classic and challenging task in the computer field, and also has a large number of practical scenarios, such as satellite image remote sensing, digital high-definition, microscopic imaging, etc.

[0003] The input form of super-resolution can be roughly divided into two types: image and video. Image super-resolution aims to reconstruct a single low-resolution image into a high-resolution result; video super-resolution restores a high-resolution image by inputting multiple frames of continuous images at a time and using the relationship between frames. Traditional video super-resolution methods have evolved from methods based on interpolation and motion estimation, but these methods are difficult to effectively estimate complex motion processes in videos. With the widespread application of deep learning in various fields of computing, the field of video super-resolution has also carried out extensive research on deep learning-based methods. At present, deep learning-based video super-resolution methods mainly include three structures: convolutional neural network (CNN)-based methods, which use sliding windows to extract effective detail features between adjacent video frames for calculation; recurrent neural network (RNN)-based methods, which use hidden states to pay attention to historical information in previous video frames and future information in future video frames for calculation; Transformer-based methods, which use self-attention mechanisms to extract long-distance and short-distance features for calculation.

[0004] However, existing methods have high computational complexity and high memory usage. This limits the deployment of the model on the device. Moreover, for video super-resolution, the model also needs to process information in the time dimension, which increases computational complexity and limits the training of the model on long frames. Secondly, existing methods do not pay enough attention to the prior detailed information of different spatial positions and different time positions in the video, and lack a reasonable modeling method for key effective information in the video frame. The above defects lead to poor visual effects of the results obtained by the video super-resolution method of the existing method. Summary of the invention

[0005] In order to solve the above technical problems, the present invention discloses a video super-resolution method and system based on strip and global spatiotemporal attenuated attention, and proposes to perform strip attention calculation to reduce the computational complexity of the neural network from the square level to the linear level, so that the model is easy to deploy; using the attenuated attention matrix, the prior detail information of different spatial positions and different time positions in the video is better processed, helping the network to better consider important information when reconstructing high-resolution video, so that the video after super-resolution has lower computational complexity and better visual effect.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention proposes a video super-resolution method based on stripe and global spatiotemporal attenuation attention, comprising the following steps:

[0008] Step (1). Obtain a low-resolution video and perform image enhancement on each video frame;

[0009] Step (2). Extracting the original spatial features of each video frame after image enhancement and interpolating the original spatial features to obtain the upsampled spatial features of each video frame;

[0010] Step (3). For the upsampled spatial features obtained in step (2), the spatial strip attention of each video frame and the temporal strip attention between frames are calculated respectively;

[0011] Step (4). The spatial strip attention and temporal strip attention obtained in step (3) are processed using depthwise separable convolution and residual structure respectively, and then the results of the two are fused to obtain the fusion features of each video frame;

[0012] Step (5). For the upsampled spatial features obtained in step (2), obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuation attention;

[0013] Step (6). The original spatial features obtained in step (2), the fused features obtained in step (4) and the global spatiotemporal attenuated attention obtained in step (5) are fused and reconstructed to obtain a high-resolution reconstruction result for each video frame; all video frames are traversed to obtain a high-resolution video after super-resolution.

[0014] In the steps (1)-(2), the network first performs operations such as rotation and stacking on the low-resolution video sequence to enhance the image, and then extracts the spatial features of each frame of the video sequence. The extracted shallow features retain the original information of the video frame more completely, and serve as a guide in the subsequent video frame processing process to help the network reconstruct the high-resolution video frame. These shallow features are subjected to strip attention calculations in the spatial and temporal dimensions and global spatiotemporal attenuation attention calculations to perform feature reconstruction.

[0015] In the steps (3)-(4), the spatial strip attention and temporal strip attention of the low-resolution video sequence are calculated respectively, wherein the video sequence is divided into a horizontal feature tensor and a vertical feature tensor in the channel dimension when calculating the spatial strip attention; the same is true when calculating the temporal strip attention. The advantage of using strip attention is that the spatial and temporal strip attentions are used to extract spatial and temporal information respectively, which can more efficiently capture long-distance dependencies in the video than the global self-attention mechanism; the strip attention mechanism reduces memory consumption, and its spatial and temporal memory complexity are respectively: O((H 2 +W 2 )T) and O((H+W)T 2 ), which is significantly lower than the original Transformer’s time and space complexity O((HWT) 2 ).

[0016] In step (5), the Manhattan decay matrix of the three dimensions of time, height and width of each low-resolution video sequence frame is calculated, and then the spatiotemporal attention calculation is performed to obtain the global spatiotemporal decay attention. The advantage of using the decay attention matrix is ​​that it can better process the prior detail information of different spatial positions and different temporal positions in the video, helping the network to better consider important information when reconstructing high-resolution video.

[0017] In the step (6), the spatial features obtained in steps (1)-(2), the spatial and temporal strip fusion features obtained in steps (3)-(4), and the global spatiotemporal attenuation attention obtained in step (5) are fused, and the features are reconstructed and upsampled to obtain a super-resolution high-resolution result.

[0018] Furthermore, in the step (1), image enhancement includes horizontal 90° flipping, vertical 90° flipping, and mirror symmetry.

[0019] Furthermore, in the step (2), the channel dimension of each video frame of the low-resolution video after the image enhancement in step (1) is expanded from 3 to 64 while the resolution remains unchanged, so as to obtain the original spatial features of each video frame.

[0020] Furthermore, the step (3) comprises:

[0021] Step (3.1). Perform layer normalization and feature segmentation on the upsampled spatial features to obtain two types of feature tensors, namely, spatial horizontal feature tensors and spatial vertical feature tensors;

[0022] Step (3.2). The spatial horizontal feature tensor and the spatial vertical feature tensor obtained in step (3.1) are mapped into query, key and value respectively;

[0023] Step (3.3). For the query, key and value obtained in step (3.2), respectively calculate the spatial horizontal attention output and the spatial vertical attention output; the spatial horizontal attention output and the spatial vertical attention output are used as the spatial strip attention result;

[0024] Step (3.4). For the upsampled spatial features of each video frame, consider the feature expressions of the n frames before and after it to obtain the joint upsampled spatial features X t ′ represents the upsampled spatial features of the tth video frame;

[0025] Step (3.5). The joint upsampled spatial features obtained in step (3.4) Perform layer normalization and feature segmentation to obtain two types of feature tensors, namely, time-horizontal feature tensor and time-vertical feature tensor;

[0026] Step (3.6). Divide the temporal transverse feature tensor and temporal longitudinal feature tensor obtained in step (3.5) into a number of transverse strips with non-overlapping temporal dimensions and the same spatial position and a number of longitudinal strips with non-overlapping temporal dimensions and the same spatial position;

[0027] Step (3.7). For the horizontal strips and vertical strips obtained in step (3.6), map them into query, key and value respectively;

[0028] Step (3.8). For the queries, keys and values ​​of the horizontal strips and vertical strips obtained in step (3.7), calculate their temporal horizontal attention output and temporal vertical attention output respectively; the temporal horizontal attention output and temporal vertical attention output are used as the temporal strip attention results.

[0029] Furthermore, the step (4) comprises:

[0030] Step (4.1). Concatenate the spatial horizontal attention output and the spatial vertical attention output in the spatial strip attention and perform a depth-separable convolution operation to obtain a spatial convolution result; concatenate the temporal horizontal attention output and the temporal vertical attention output in the temporal strip attention and perform a depth-separable convolution operation to obtain a temporal convolution result;

[0031] Step (4.2). The spatial convolution result is processed using a residual structure and normalization to obtain the final spatial strip attention; the temporal convolution result is processed using a residual structure and normalization to obtain the final temporal strip attention;

[0032] Step (4.3). Add the final spatial strip attention and the final spatial strip attention to obtain the fused feature.

[0033] Furthermore, the step (5) comprises:

[0034] Step (5.1). For the upsampled spatial features of each video frame, calculate the Manhattan attenuation matrix of its time dimension, the Manhattan attenuation matrix of its height dimension, and the Manhattan attenuation matrix of its width dimension respectively;

[0035] Step (5.2). Add the three-dimensional Manhattan attenuation matrices element by element to obtain a three-dimensional Manhattan attenuation matrix;

[0036] Step (5.3). Map the upsampled spatial features of each video frame into query, key and value, and perform attention calculation in combination with the three-dimensional Manhattan attenuation matrix to obtain the global spatiotemporal attenuated attention result.

[0037] Furthermore, the step (6) comprises:

[0038] Step (6.1). Add the fusion features of each video frame and the global spatiotemporal attenuation attention to obtain the spliced ​​features;

[0039] Step (6.2). Upsample the spliced ​​features, and perform residual connection between the upsampled results and the original spatial features of each video frame to obtain the residual features;

[0040] Step (6.3). Perform 3×3 convolution and sub-pixel convolution operations on the residual features to obtain the final high-resolution output;

[0041] Step (6.4). Traverse all video frames to obtain a high-resolution result after the entire video resolution is increased by 4 times.

[0042] In a second aspect, the present invention proposes a video super-resolution system based on strip-like and global spatiotemporal attenuated attention, which is used to implement the above-mentioned video super-resolution method based on strip-like and global spatiotemporal attenuated attention.

[0043] Beneficial effects of the present invention:

[0044] The present invention designs a complete network structure of video super-resolution based on strip-shaped and global spatiotemporal attenuation attention, including a data preprocessing module, a spatial feature extraction module, a strip-shaped attention calculation module, an attenuation attention calculation module and a super-resolution reconstruction module, which belongs to a multi-level network architecture. The present invention utilizes the strip-shaped attention mechanism to greatly reduce the spatiotemporal complexity while retaining a large amount of detail information. At the same time, the attenuation attention mechanism takes into account the importance of distance dependence in the video reconstruction process, so that the detailed information of the video can be effectively reconstructed, so that the super-resolution video has a better visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 4 is a structural block diagram of a video super-resolution method based on strip-shaped and global spatiotemporal attenuation attention used in an embodiment of the present invention;

[0046] Figure 2 is an overall flow chart of an embodiment of the present invention;

[0047] Figure 3 is a structural block diagram of a strip attention calculation module in an embodiment of the present invention;

[0048] Figure 4 4 is a structural block diagram of the attenuated attention calculation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The method of the present invention will be further described below in conjunction with the accompanying drawings.

[0050] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0051] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0052] The structural block diagram of the video super-resolution method based on strip and global spatiotemporal attenuation attention of the present invention is as follows: Figure 1 As shown in Figure 1, five modules are designed in this method: data preprocessing module, spatial feature extraction module, strip attention calculation module, attenuation attention calculation module and super-resolution reconstruction module. The flow chart of the video super-resolution method implemented by each module is shown in Figure 1. Figure 2 shown.

[0053] The data preprocessing module is used to process the input raw video data stream, obtain low-resolution video and perform image enhancement on each video frame; specifically, the method in the following step (1) is executed.

[0054] Step (1) Obtain a low-resolution video sequence and perform mirror symmetry, horizontal 90° flip, and vertical 90° flip on each frame of the image to achieve image enhancement. The low-resolution video sequence after image enhancement is recorded as I t LR , where t represents the tth frame of the picture; then it is input into the spatial feature extraction module frame by frame.

[0055] The spatial feature extraction module is used to extract the spatial features of each frame image in the low-resolution video, and interpolate the original spatial features to obtain the upsampled spatial features of each video frame; specifically, the method in the following step (2) is executed.

[0056] Step (2). Take the t(t∈[1,T])th frame of the low-resolution video obtained in step (1) The convolutional layer is used to change the channel dimension of the input frame from 3 to 64, and then the original spatial feature extraction is performed.

[0057] The spatial feature extraction process is expressed as:

[0058]

[0059] Among them, X t Indicates the t-th frame image The original spatial features of , R(.) represents the stacked residual block structure used to extract features, RELU(.) represents the activation function, and conv(.) represents the convolutional layer. The spatial resolution is kept unchanged during the spatial feature calculation process.

[0060] Interpolate the spatial features of each frame of the low-resolution video with an upsampling rate of 0.5:

[0061] X t ′=Bilinear(X t ,0.5)

[0062] Among them, X t ′ represents the t-th frame image X t The upsampled spatial features, the spatial resolution is the t-th frame image X t 0.5 times, Bilinear(.,.) represents the bilinear interpolation method. At this time, the feature expression after upsampling of each frame is: in t∈{1,2,...,T} and X t The channel dimension of ′ is Xt 4 times the channel dimension, that is

[0063] A core design point of the present invention is to use the strip attention calculation module to calculate the strip attention of each frame in the low-resolution video, such as Figure 3 As shown, the strip attention calculation module is used to upsample the spatial features, and respectively calculate the spatial strip attention of each video frame and the temporal strip attention between frames; the spatial strip attention and the temporal strip attention are processed using a deep separable convolution and a residual structure, and then the results of the two are fused to obtain a fused feature of each video frame; specifically, the methods in the following steps (3)-(4) are executed.

[0064] Step (3). For the spatial features obtained in step (2), the spatial strip attention of each video frame and the temporal strip attention between frames are calculated respectively, specifically:

[0065] Step (3.1). The upsampled spatial feature X obtained in step (2) t ′Perform layer normalization and feature segmentation according to channel dimension to obtain two types of feature tensors, namely spatial lateral feature tensors and the spatial longitudinal feature tensor

[0066]

[0067] in, and Represent the horizontal and vertical feature tensors respectively, and the number of channels of each tensor is

[0068] Step (3.2). For the horizontal feature tensor and vertical feature tensor obtained in step (3.1), map them into queries: Q (h) , Q (v) ; Key: K (h) , K (v) Sum value V (h) 、V (v) , for the lateral feature tensor:

[0069]

[0070] Among them, W Q , W K and W V is the linear transformation matrix; the same is true for the longitudinal feature tensor:

[0071]

[0072] Step (3.3). For the query, key, and value of the horizontal feature tensor and vertical feature tensor obtained in step (3.2), calculate their spatial horizontal attention outputs respectively and spatial longitudinal attention output

[0073]

[0074]

[0075] Among them, H′, W′ and C′ represent the t-th frame image X t The upsampled spatial features X t ’’s length, width and number of channels.

[0076] The above spatial horizontal attention output and spatial longitudinal attention output As a result of spatial strip attention.

[0077] Step (3.4). For each frame of the image, the upsampled spatial feature X t ′, consider the feature expression of each n frames before and after, record X at this time t 'for

[0078]

[0079] Wherein, n is a positive integer, and its value range is 0≤n≤min(t-1,Tt).

[0080] Step (3.5). The features obtained in step (3.4) Perform layer normalization and feature segmentation by channel dimension to obtain two feature tensors, namely the temporal horizontal feature tensor and the time longitudinal feature tensor

[0081]

[0082] in, and Represent the horizontal and vertical feature tensors respectively, and the number of channels of each tensor is

[0083] Step (3.7). The time-horizontal feature tensor obtained in step (3.6) and the time longitudinal feature tensor Divide into H′ horizontal strips with non-overlapping time dimensions and the same spatial position and W′ longitudinal strips that do not overlap in time dimensions and have the same spatial position

[0084]

[0085]

[0086] Among them, a∈{1,2,...,H′}, b∈{1,2,...,W′}.

[0087] Step (3.8). For the horizontal strips obtained in step (3.7) and longitudinal stripes Map them into queries respectively: key: Sum For horizontal strips:

[0088]

[0089] Among them, W Q , W K and W V is the linear transformation matrix; the same is true for the vertical strips:

[0090]

[0091] Step (3.9). For the query, key and value of the horizontal strip and vertical strip obtained in step (3.8), calculate their temporal horizontal attention output respectively and temporal longitudinal attention output

[0092]

[0093]

[0094] Among them, H′, W′ and C′ represent the t-th frame image X t The upsampled spatial features X t ’’s length, width and number of channels.

[0095] The above temporal horizontal attention output and temporal longitudinal attention output As a temporal strip of attention results.

[0096] Step (4). The spatial strip attention and temporal strip attention obtained in step (3) are processed using depthwise separable convolution and residual structure respectively, and then the two results are fused to obtain spatiotemporal strip fusion attention, specifically:

[0097] Step (4.1). Output the spatial horizontal attention obtained in step (3.4) and spatial longitudinal attention output Perform concatenation and depth-separable convolution operations to obtain spatial convolution results

[0098]

[0099] in, represents concatenation, and DepthwiseConv(.) represents depthwise separable convolution.

[0100] Step (4.2). The spatial convolution result obtained in step (4.1) Using the residual structure and normalization processing, we get the final spatial strip attention O space :

[0101]

[0102] Among them, RELU(.) represents nonlinear activation, Conv 3×3 (.) represents a 3×3 convolution, and GN(.) represents a group normalization operation.

[0103] Step (4.3). The temporal horizontal attention output obtained in step (3.9) and temporal longitudinal attention output Repeat steps (4.1)-(4.2) to obtain the final temporal strip attention O time ;

[0104] Step (4.4). The spatial strip attention O obtained in step (4.2) space And the temporal strip attention O obtained in step (4.3) time Add them together to get the spatiotemporal strip fusion attention O fusion :

[0105] O fusion =O space +O time

[0106] In the video super-resolution task, it is also necessary to consider the detail information in the time and space dimensions, and the importance of the detail information in the two dimensions must be calculated separately. Figure 4 As shown, the attenuated attention calculation module is used to upsample the spatial features, obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to calculate the global spatiotemporal attenuated attention; specifically, the method in the following step (5) is executed.

[0107] Step (5). Upsample the spatial feature X obtained in step (2) t ′, and obtain the three-dimensional Manhattan attenuation matrix of the spatial features of each frame respectively, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuation attention, specifically:

[0108] Step (5.1). For the upsampled feature X obtained in step (2) t ′, calculate the Manhattan attenuation matrix D of its time dimension T,man ; In the time dimension T, for any two time steps t i and t j , the calculation formula of Manhattan distance is:

[0109] manDis tan ce(t i ,t j )=|t i -t j |

[0110] The Manhattan attenuation matrix of the time dimension is obtained as:

[0111] D T,man [i,j]=α·manDis t ance(t i ,t j )

[0112] Among them, α is the attenuation factor, which is used to control the impact of Manhattan distance on attenuation.

[0113] Step (5.2). For the upsampled feature X obtained in step (3.1) t ′, calculate the Manhattan attenuation matrix D of its height dimension H′,man ; In the height dimension H′, for any two height positions h j and h k , the calculation formula of Manhattan distance is:

[0114] manDis tan ce(h j ,h k )=|h j -h k |

[0115] The resulting Manhattan attenuation matrix of the height dimension is:

[0116] D H′,man [j,k]=β·manDis tan ce(h j ,h k )

[0117] Among them, β is the decay factor, which is used to control the effect of Manhattan distance on decay.

[0118] Step (5.3). For the upsampled feature X obtained in step (3.1) t ′, calculate the Manhattan attenuation matrix D of its width dimension W′,man; In the width dimension W′, for any two height positions w k and w i , the calculation formula of Manhattan distance is:

[0119] manDis tan ce(w k ,w i )=|w k -w i |

[0120] The resulting Manhattan attenuation matrix of the width dimension is:

[0121] D W′,man [k,i]=γ·manDis tan ce(w k ,w i )

[0122] Among them, γ is the attenuation factor, which is used to control the impact of Manhattan distance on attenuation.

[0123] Step (5.4). Add the three-dimensional Manhattan attenuation matrices obtained in steps (5.1), (5.2), and (5.3) element by element to obtain the final three-dimensional Manhattan attenuation matrix D man :

[0124] D man [i,j,k]=D T,man [i,j]+D H′,man [j,k]+D W′,man [k,i]

[0125] Among them, i, j and k correspond to the indexes in the time, spatial height and width dimensions respectively.

[0126] Step (5.5). For the upsampled feature X obtained in step (2) t ′, and maps it into query Q, key K and value V:

[0127] Q=XW Q ′,K=XW K ′,V=XW V '

[0128] Among them, W Q ′、W K ′ and W V ′ is the weight matrix of the linear transformation.

[0129] Step (5.6). Perform attention calculation on the query Q, key K and value V of the upsampled features obtained in step (5.5) to obtain the global spatiotemporal attenuated attention O decay :

[0130] Odecay =Soft max((Q·K T )⊙D man )·V

[0131] Where ⊙ represents element-by-element multiplication.

[0132] After calculating the spatiotemporal strip fusion attention and spatiotemporal fusion attenuation attention for each frame of the low-resolution video, the super-resolution reconstruction module is used to fuse and reconstruct the upsampled spatial features, fusion features and global spatiotemporal attenuation attention to obtain a high-resolution reconstruction result for each video frame; specifically, the method in the following step (6) is executed.

[0133] Step (6). The original spatial features obtained in step (2), the spatiotemporal strip fusion attention obtained in step (4), and the global spatiotemporal attenuation attention obtained in step (5) are fused to obtain a high-resolution reconstruction result for each video frame, specifically:

[0134] Step (6.1). For the spatiotemporal strip fusion attention O obtained in step (4) fusion And the global spatiotemporal attenuation attention O obtained in step (5) decay Add them together to get the added feature O out :

[0135] O out =O fusion +O decay

[0136] Step (6.2). For the added feature O obtained in step (6.1) out , perform 2 times bilinear interpolation upsampling, and change its dimension from H′×W′×C (where, ) becomes For convenience out With X t Perform residual connection to obtain residual feature O res :

[0137] O res =X t +Bilinear(O out , 2)

[0138] Among them, Bilinear(·,·) represents the bilinear interpolation method,

[0139] Step (6.3) for the residual feature O res , use 3×3 convolution to adjust the channel, so that its dimension changes from It becomes H×W×C to facilitate subsequent high-resolution upsampling operations; then PixelShuffle is used to perform sub-pixel convolution to make its dimension become Finally, 3×3 convolution is used to adjust the channels to obtain the final high-resolution output X HR , the dimension of the output feature is 2H×2W×3:

[0140] X HR =Conv 3×3 (PixelShuffle(Conv 3×3 (O res ), 4))

[0141] Among them, Conv 3×3 (·) represents a 3×3 convolution operation, PixelShuffle(·,4) represents a sub-pixel convolution operation, and 4 represents a 4-fold reduction of the channel, which expands the dimension of the channel to the spatial resolution. Thus, the resolution is increased by 4 times. In this embodiment, 4 times can also be replaced by other multiples of perfect square numbers, such as 9 times.

[0142] Step (6.4). Traverse all video frames to obtain a high-resolution result after the entire video resolution is increased by 4 times.

[0143] The combination of the image preprocessing module, spatial feature extraction module, strip attention calculation module, attenuated attention calculation module, and super-resolution reconstruction module described above can construct a complete video super-resolution system based on strip and global spatiotemporal attenuated attention. Each module may or may not be physically separated, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art can understand and implement it without creative work.

[0144] The video super-resolution system based on strip and global spatiotemporal attenuation attention can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory and run by the processor of any device with data processing capability.

[0145] The above description is only a specific embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the present application. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A video super-resolution method based on strip-wise and global spatiotemporal attenuation attention, characterized in that: The following steps are involved: Step (1). Obtain a low-resolution video and perform image enhancement on each video frame; Step (2). Extracting the original spatial features of each video frame after image enhancement and interpolating the original spatial features to obtain the upsampled spatial features of each video frame; Step (3). For the upsampled spatial features obtained in step (2), the spatial strip attention of each video frame and the temporal strip attention between frames are calculated respectively; Step (4). The spatial strip attention and temporal strip attention obtained in step (3) are processed using depthwise separable convolution and residual structure respectively, and then the results of the two are fused to obtain the fusion features of each video frame; Step (5). For the upsampled spatial features obtained in step (2), obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuation attention; Step (6). The original spatial features obtained in step (2), the fused features obtained in step (4) and the global spatiotemporal attenuated attention obtained in step (5) are fused and reconstructed to obtain a high-resolution reconstruction result for each video frame; all video frames are traversed to obtain a high-resolution video after super-resolution.

2. The video super-resolution method based on strip-like and global spatiotemporal attenuation attention according to claim 1, characterized in that: In the step (1), image enhancement includes horizontal 90° flipping, vertical 90° flipping, and mirror symmetry.

3. The video super-resolution method based on strip-like and global spatiotemporal attenuation attention according to claim 1, characterized in that: In the step (2), the channel dimension of each video frame of the low-resolution video after the image enhancement in step (1) is expanded from 3 to 64 while the resolution remains unchanged, thereby obtaining the original spatial features of each video frame.

4. The video super-resolution method based on strip-like and global spatiotemporal attenuation attention according to claim 1, characterized in that: The step (3) comprises: Step (3.1). Perform layer normalization and feature segmentation on the upsampled spatial features to obtain two types of feature tensors, namely, spatial horizontal feature tensors and spatial vertical feature tensors; Step (3.2). The spatial horizontal feature tensor and the spatial vertical feature tensor obtained in step (3.1) are mapped into query, key and value respectively; Step (3.3). For the query, key and value obtained in step (3.2), respectively calculate the spatial horizontal attention output and the spatial vertical attention output; the spatial horizontal attention output and the spatial vertical attention output are used as the spatial strip attention result; Step (3.4). For the upsampled spatial features of each video frame, consider the feature expressions of the n frames before and after it to obtain the joint upsampled spatial features X t ′ represents the upsampled spatial features of the tth video frame; Step (3.5). The joint upsampled spatial features obtained in step (3.4) Perform layer normalization and feature segmentation to obtain two types of feature tensors, namely, time-horizontal feature tensor and time-vertical feature tensor; Step (3.6). Divide the temporal transverse feature tensor and temporal longitudinal feature tensor obtained in step (3.5) into a number of transverse strips with non-overlapping temporal dimensions and the same spatial position and a number of longitudinal strips with non-overlapping temporal dimensions and the same spatial position; Step (3.7). For the horizontal strips and vertical strips obtained in step (3.6), map them into query, key and value respectively; Step (3.8). For the queries, keys and values ​​of the horizontal strips and vertical strips obtained in step (3.7), calculate their temporal horizontal attention output and temporal vertical attention output respectively; the temporal horizontal attention output and temporal vertical attention output are used as the temporal strip attention results.

5. The video super-resolution method based on strip-like and global spatiotemporal attenuation attention according to claim 1, characterized in that: The step (4) comprises: Step (4.1). Concatenate the spatial horizontal attention output and the spatial vertical attention output in the spatial strip attention and perform a depth-separable convolution operation to obtain a spatial convolution result; concatenate the temporal horizontal attention output and the temporal vertical attention output in the temporal strip attention and perform a depth-separable convolution operation to obtain a temporal convolution result; Step (4.2). The spatial convolution result is processed using a residual structure and normalization to obtain the final spatial strip attention; the temporal convolution result is processed using a residual structure and normalization to obtain the final temporal strip attention; Step (4.3). Add the final spatial strip attention and the final spatial strip attention to obtain the fused feature.

6. The video super-resolution method based on strip-like and global spatiotemporal attenuation attention according to claim 1, characterized in that: The step (5) comprises: Step (5.1). For the upsampled spatial features of each video frame, calculate the Manhattan attenuation matrix of its time dimension, the Manhattan attenuation matrix of its height dimension, and the Manhattan attenuation matrix of its width dimension respectively; Step (5.2). Add the three-dimensional Manhattan attenuation matrices element by element to obtain a three-dimensional Manhattan attenuation matrix; Step (5.3). Map the upsampled spatial features of each video frame into query, key and value, and perform attention calculation in combination with the three-dimensional Manhattan attenuation matrix to obtain the global spatiotemporal attenuated attention result.

7. The video super-resolution method based on strip-like and global spatiotemporal attenuation attention according to claim 1, characterized in that: The step (6) comprises: Step (6.1). Add the fusion features of each video frame and the global spatiotemporal attenuation attention to obtain the spliced ​​features; Step (6.2). Upsample the spliced ​​features, and perform residual connection between the upsampled results and the original spatial features of each video frame to obtain the residual features; Step (6.3). Perform 3×3 convolution and sub-pixel convolution operations on the residual features to obtain the final high-resolution output; Step (6.4). Traverse all video frames to obtain a high-resolution result after the entire video resolution is increased by 4 times.

8. A video super-resolution system based on strip-like and global spatiotemporal attenuation attention, used to implement the video super-resolution method of claim 1, characterized in that: The system comprises: An image preprocessing module, which is used to obtain low-resolution video and perform image enhancement on each video frame; A spatial feature extraction module is used to extract the original spatial features of each video frame after image enhancement and interpolate the original spatial features to obtain upsampled spatial features of each video frame; The strip attention calculation module is used to calculate the spatial strip attention of each video frame and the temporal strip attention between frames for the upsampled spatial features; the spatial strip attention and the temporal strip attention are processed using a deep separable convolution and a residual structure respectively, and then the results of the two are fused to obtain the fused features of each video frame; The attenuated attention calculation module is used to obtain the three-dimensional Manhattan attenuation matrix of the upsampled spatial features of each video frame for the upsampled spatial features, and then perform spatiotemporal attention calculation to obtain the global spatiotemporal attenuated attention; The super-resolution reconstruction module is used to fuse and reconstruct the original spatial features, fused features and global spatiotemporal attenuation attention to obtain a high-resolution reconstruction result for each video frame; it traverses all video frames to obtain a high-resolution video after super-resolution.

Images