Spatiotemporal mixed video super-resolution method based on deformable attention

Abstract

The application discloses a kind of spatio-temporal mixed video super-resolution methods based on deformable attention, belong to video super-resolution technical field.The application mainly includes the following steps:1. after being pretreated, local feature extraction is carried out to low-resolution video data stream;2. then further global feature extraction is carried out, after the block discarding attention module and deformable attention module in series, the video frame feature vector of gathering global information is obtained;3. local feature extraction is carried out again and shallow feature information is combined by jump connection;4. the feature information of each spatial block vector in time dimension is extracted using time trajectory, and feature alignment is completed;5. the forward and backward super-resolution results obtained are fused, and the final high-resolution frame result is output through reconstruction module.The video super-resolution reconstruction method proposed based on the application can obtain high-quality high-resolution video, and can restore clearer image edges and reduce ghosting compared to previous methods.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video super-resolution, and in particular to a spatio-temporal hybrid super-resolution method based on deformable attention. BACKGROUND

[0002] High-resolution videos have always been the goal of daily life, such as in the medical imaging field, high-definition video streams are beneficial for more accurate analysis and diagnosis; in the traffic monitoring field, high-definition road monitoring videos are beneficial for more accurate analysis and detection of traffic flow and traffic density. Video super-resolution is a technology that converts low-resolution videos into high-resolution videos. In recent years, with the development of deep learning, neural network-based super-resolution technology can achieve higher video reconstruction quality compared to traditional methods. The current mainstream video super-resolution methods based on neural networks can be roughly divided into three categories: convolutional neural network (CNN) based video super-resolution method, recurrent neural network (RNN) based super-resolution method, and Transformer based video super-resolution method. The CNN-based method uses convolution kernels to extract local features to complete feature alignment and feature fusion; the RNN-based method uses hidden states to store and propagate past or future time information; the Transformer-based method uses self-attention mechanism to parallelize feature extraction and feature alignment.

[0003] However, current video super-resolution methods either only use spatial or temporal correlation in the video, or cannot effectively combine spatio-temporal information to complete the video reconstruction task. The existing CNN-based video super-resolution method uses a windowed approach to extract information within a local space and a local time range, lacking the ability to model information over a long distance range; the existing RNN-based method has error accumulation, noise amplification, and gradient vanishing / explosion problems due to sharing a single network, making it difficult to achieve effective super-resolution over a long distance range; the existing Transformer-based method requires a lot of computational resources to extract spatio-temporal information, making it difficult to achieve efficient computational efficiency. SUMMARY

[0004] The application discloses a spatio-temporal hybrid video super-resolution method based on deformable attention, which aims to use attention mechanisms to achieve effective and efficient spatio-temporal information extraction on a multi-level network structure, thereby better completing the detail reconstruction of low-resolution videos, and providing an effective enhancement means for practical applications.

[0005] The technical solution adopted by the application to solve its technical problems is as follows:

[0006] A spatio-temporal hybrid video super-resolution method based on deformable attention, comprising the following steps:

[0007] Step (1). Obtain a low-resolution video data stream and perform image enhancement;

[0008] Step (2). Obtain the shallow feature map of each frame of the enhanced image in the video;

[0009] Step (3). Local feature extraction and down-sampling are performed on the shallow feature map obtained in step (2) to obtain a down-sampled local feature map;

[0010] Step (4). Global feature extraction is performed on the down-sampled local feature map obtained in step (3), and a feature map that aggregates global information is obtained through a series of block dropout attention modules and deformable attention modules;

[0011] Step (5). Local feature extraction and up-sampling are performed on the feature maps output by steps (2)-(4) to obtain an up-sampled local feature map;

[0012] Step (6). Similarity calculation and fusion are performed on each sub-block of the local feature map obtained in step (5) in the time trajectory to obtain an aligned feature map;

[0013] Step (7). Repeat steps (2) to (6) to traverse each frame of the enhanced image in the video to obtain a single-direction super-resolution result;

[0014] Step (8). Repeat steps (2) to (6) to process each frame of the enhanced image in the video in the reverse direction to obtain a reverse-direction super-resolution result;

[0015] Step (9). The single-direction super-resolution result output by step (7) and the reverse-direction super-resolution result output by step (8) are fused, and then a super-resolution reconstruction module is used to complete the output of the high-resolution frame image.

[0016] Further, the step (4) comprises:

[0017] Step (4.1). The down-sampled local feature map output by step (3) is sequentially subjected to L3 block dropout attention modules and L3 deformable mutual attention modules in stage three;

[0018] Step (4.2). The feature map output by step (4.1) is subjected to a down-sampling operation;

[0019] Step (4.3). The feature map output by step (4.2) is sequentially subjected to L4 block dropout attention modules and L4 deformable mutual attention modules in stage four;

[0020] Step (4.4). The feature map output by step (4.3) is subjected to an up-sampling operation;

[0021] Step (4.5). After adding the downsampled local feature map output in step (3) and the upsampled feature map output in step (4.4), the feature map is passed through L5 block discard attention modules and L5 deformable mutual attention modules in stage five in sequence.

[0022] Step (4.6). Upsample the feature map output in step (4.5) to obtain a feature map that aggregates global information.

[0023] Furthermore, in the block discarding attention modules of stages three, four, and five, only important blocks are retained, which can extract information from important regions in the image, thereby reducing the interference of non-critical or interfering information and obtaining feature maps that aggregate important information.

[0024] The calculation process includes:

[0025] a. Each block of the attention module includes M TDB sub-blocks, and each TDB sub-block is set with a drop rate α. j ;

[0026] b. For each TDB sub-block, its input feature map is denoted as... The i-th frame feature map of the j-th sub-block input of the attention module is discarded in the l-th block of stage s; the i-th frame feature map of the first sub-block input of the attention module in the first block of stage 3 is the downsampled local feature map output in step (3), and the output of the previous TDB sub-block is used as the input of the next TDB sub-block;

[0027] In each TDB sub-block, the input feature map is first processed. The feature map is divided into blocks, and the cosine similarity between each block is calculated to obtain the attention matrix. in, Discard the attention matrix of the j-th sub-block of the attention module for the l-th block;

[0028] c. Each TDB sub-block is discarded according to the rate α. j , attention matrix The α with the lowest similarity j *N feature maps are divided into blocks to form a mask matrix For use by the next TDB sub-block;

[0029] d. Based on the attention matrix The values ​​in the attention mechanism are used to obtain the output feature map of the TDB sub-block;

[0030] e. In the block discard attention module, the next TDB sub-block receives the mask matrix from the previous TDB sub-block. This is then combined with the attention matrix computed in the current TDB sub-block to generate a new attention matrix.

[0031] f. Calculate the output feature map based on the new attention matrix.

[0032] g. Repeat step af until all M sub-blocks in the l-th block discard attention module have completed their output. The output of the last sub-block in the l-th block discard attention module is the output of the l-th block discard attention module, denoted as . Let it be used as the input to the (l+1)th block discard attention module, denoted as

[0033] Let l = l + 1, and continue repeating step af until the Lth step is completed. s The output of the attention module is discarded in each block, denoted as .

[0034] Furthermore, in the deformable mutual attention modules of stages three, four, and five, deformable attention is used to predict the offset of the preceding and following frames under optical flow guidance. Predicted values ​​of the preceding and following frames are generated through bilinear interpolation. Then, the current frame is aligned to the preceding and following frames respectively using the mutual attention mechanism. Finally, the local attention mechanism is used for fusion, which can extract information from the preceding and following frames to alleviate the error problems caused by occlusion and image boundaries, and obtain feature maps that gather global information.

[0035] The calculation process includes:

[0036] h. The deformable mutual attention module receives data from the Lth stage in the corresponding phase s. s Output feature map of the block discard attention module As input, the query, key, and value in the attention mechanism are first output through a linear layer.

[0037] i. The query q in the attention mechanism of the l-th deformable mutual attention module. l The offset network Θ(·) predicts the offset of each pixel in the input feature map, generating the offsets of the preceding and following terms:

[0038] j. Using the offset generated by the offset network to modify the input feature map The deformation is performed to generate the offset feature maps of the preceding and following terms. and

[0039] k. Utilize the predicted feature maps of the preceding and following terms and Generates the query, key, and value in the sampled attention mechanism;

[0040] l. will and The query, key, and value (q) in the attention mechanism of the l-th deformable mutual attention module l ,k l ,v l ) Calculate mutual attention and fuse them;

[0041] m. Channels are reduced through simple channel merging and convolutional layers, resulting in bidirectional aligned outputs;

[0042] n. Using the method described above (hm), until step l is completed. s The output of a deformable mutual attention module is denoted as...

[0043] The feature size changes in steps (3), (4), and (5) are as follows: through three downsampling and three upsampling operations, the feature size undergoes a transformation from 64 to 32 to 16 to 8 to 16 to 32 to 64. Simultaneously, the spatial size is reduced three times and finally restored to its original size. The purpose of this step is twofold: firstly, to effectively extract spatial features at different scales by combining multiple receptive fields; and secondly, to adapt to the global computation of the block dropout attention module and the deformable attention module, reducing the image spatial size can improve computational efficiency. Furthermore, combining the local feature extraction of the Swin-Transformer with the global attention computation of both the block dropout attention module and the deformable attention module allows for the simultaneous aggregation of local and spatial information.

[0044] In step (6), each spatial block vector on the time trajectory is processed through bitmap iteration, reducing a significant amount of repetitive work and computational complexity. Each time, only the time offset between the current frame and the previous frame needs to be calculated, and all forward frames are processed simultaneously; the reverse time frames are processed similarly. Furthermore, in step (6), by selecting a specified number of spatial block vectors with the highest confidence for fusion, the high complexity caused by simultaneously mixing and calculating spatiotemporal block vectors is reduced, and there is no over-reliance on the accuracy of selecting the block vector with the highest confidence. This improves the effectiveness and fault tolerance of implicit alignment.

[0045] The beneficial effects of this invention are:

[0046] The application designs a complete network structure of spatio-temporal mixed video super-resolution based on deformable attention, comprising a video preprocessing module, a shallow feature extraction module, a spatial attention module, a time trajectory attention module, and a super-resolution reconstruction module, and belongs to a multi-level network architecture, and combines the characteristics of Swin-Transformer which is good at capturing local feature information and the ability of block attention and deformable mutual attention to extract global feature information, so that sufficient feature information can be effectively extracted to predict sufficient details. In addition, in the extraction of time feature information, the method based on the time trajectory greatly reduces the computational complexity; through the effective combination of time information and spatial information, the high-quality super-resolution effect can be finally completed. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The spatio-temporal mixed video super-resolution method structure block diagram used in the implementation process of the application

[0048] Figure 2 The spatial attention network structure used in the implementation process of the application.

[0049] Figure 3 The overall flowchart used in the implementation process of the application. DETAILED DESCRIPTION

[0050] The method of the application will be further described below in combination with the drawings.

[0051] The structure block diagram of the spatio-temporal mixed super-resolution method based on deformable attention of the application is shown in Figure 1 The whole includes four parts: a video preprocessing module, a shallow feature extraction module, a spatial attention module, a time trajectory attention module, and a super-resolution reconstruction module.

[0052] The video preprocessing module is used for processing the input original video data stream, and the method in step (1) below is executed.

[0053] Step (1). Obtain a low-resolution video data stream And sequentially mirror, horizontally 90° reverse, and vertically 90° flip to realize enhancement, and then input frame by frame to the shallow feature extraction module.

[0054] The shallow feature extraction module is used for preliminarily extracting the features of each frame of the enhanced image in the video, and the method in step (2) below is executed.

[0055] Step (2). For the i-th frame of the enhanced image in the video A convolution layer is used to expand the channel of the input frame from 3 to 64 while keeping the resolution unchanged.

[0056] The shallow feature extraction process is represented as:

[0057]

[0058] wherein f SF (.) represents shallow feature extraction, represents a shallow feature map.

[0059] The spatial attention module is used to extract multi-scale spatial information. Specifically, local spatial information is obtained through an efficient Swin-Transformer module, global spatial information is selectively retained according to the degree of importance, and feature refilling is performed using front and rear frame information to perform the methods in steps (3)-(5) below.

[0060] In one specific implementation of the present application, as Figure 2 shown, the entire spatial attention module is a UNet structure, the resolution output by each layer gradually decreases from left to right, and then gradually increases. The overall structure is a multi-scale information extraction network that combines local and global information, wherein the Swin-Transformer module is used for local feature information extraction at a larger resolution. On the basis of aggregating local information, the present application uses the block dropout attention module and the deformable cross attention module to extract global feature information, and uses the skip connection to further combine the local and global feature information.

[0061] Step (3). The shallow feature map output by step (2) is subjected to local feature extraction, specifically:

[0062] Step (3.1). The shallow feature map output by step (2) After phase one, in this embodiment, the phase one is composed of L1 Swin-Transformer blocks, each Swin-Transformer block is composed of a local self-attention and a sliding window self-attention in series, the similarity calculation is limited in a smaller window, each window shares parameters, the purpose is to aggregate local spatial information of the feature map while reducing the computational complexity.

[0063] The calculation process of the phase one is represented as:

[0064]

[0065] wherein, represents the feature map output by the phase one, represents the function function of the phase one.

[0066] Step (3.2). The feature map output by step (3.1) Down-sampling operation is performed, i.e. spatial size is halved and feature size is doubled, to obtain a down-sampled feature map

[0067] Step (3.3). The down-sampled feature map output by step (3.2) is input into a stage two. In this embodiment, the stage two is composed of L2 Swin-Transformer blocks, L2 = L1.

[0068] The calculation process of the stage two is represented as:

[0069]

[0070] wherein, represents the feature map output by the stage two, represents a function function of the stage two.

[0071] Step (3.4). The feature map output by step (3.3) is input into a stage three. Down-sampling operation is performed, i.e. spatial size is halved and feature size is doubled, to obtain a down-sampled feature map

[0072] Step (4). The down-sampled feature map output by step (3.4) is input into a stage four. Global feature extraction is performed, specifically:

[0073] Step (4.1). The down-sampled feature map output by step (3.4) is input into a stage four. In this embodiment, the stage three is composed of L3 block dropout attention modules (TDB) connected in series with L3 deformable mutual attention modules (DMB).

[0074] The calculation process of the stage three is represented as:

[0075]

[0076] wherein, represents the feature map output by the stage three, represents a function function of the TDB block in the stage three, represents a function function of the DMB block in the stage three.

[0077] In one specific implementation of the present application, the calculation step of the TDB block includes:

[0078] 4.1.a. In each TDB block, M TDB sub-blocks are set in this embodiment, and the dropout rate of each TDB sub-block is set, denoted as:

[0079]

[0080] wherein, αj is the drop rate of the jth TDB sub-block, in particular, the 1st TDB sub-block is not dropped by default.

[0081] 4.1.b. For each TDB sub-block, its input is the feature map denotes the ith frame feature map input to the jth sub-block of the lth TDB block in stage s, where i.e., the ith frame feature map input to the 1st sub-block of the 1st TDB block in stage three is After that, the L3 TDB blocks in stage three are concatenated, and the M sub-blocks in each TDB block are also concatenated, according to the concatenation relationship, the output of the previous sub-block is taken as the input of the next sub-block.

[0082] For each sub-block in the TDB block, first, the input feature map is divided into blocks, and the cosine similarity between each block in the feature map is calculated to obtain the attention matrix:

[0083]

[0084] wherein, is the attention matrix of the jth sub-block of the lth TDB block, is the cosine similarity of the jth sub-block of the lth TDB block, is the query and key in the attention mechanism of the jth sub-block of the lth TDB block, the superscript T represents transposition, and N is the number of input feature map blocks.

[0085] 4.1.c. Each sub-block in the TDB block is dropped according to the drop rate j , the attention matrix with the lowest similarity j *N feature map blocks are made into a mask matrix for use by the next sub-block.

[0086] 4.1.d. The output feature map of the TDB sub-block is obtained by multiplying the attention matrix and the value in the attention mechanism:

[0087]

[0088] wherein, denotes the ith frame feature map output by the jth sub-block of the lth TDB block in stage s, corresponding to the ith frame feature map input to the j+1th sub-block of the lth TDB block in stage s, is the value in the attention mechanism of the jth sub-block of the lth TDB block.

[0089] In one embodiment of the present application, the calculation formula of query, key, value in attention mechanism in formula (3) and (4) is:

[0090]

[0091] wherein, are linear matrices of query, key, value in attention mechanism, respectively.

[0092] 4.1.e. In the TDB block, the next sub-block accepts the mask matrix from the previous sub-block and combines with the attention matrix calculated by the current sub-block, thereby generating a new attention matrix:

[0093]

[0094] 4.1.f. Similar to formula (3), the output feature map is calculated according to the new attention matrix, and the formula is:

[0095]

[0096] 4.1.g. Repeat the process of steps 4.1.a-4.1.f until all M sub-blocks in the lth TDB block complete the output, and the output of the last sub-block in the lth TDB block is the output of the lth TDB block, denoted as which is taken as the input of the (l+1)th TDB block, denoted as

[0097] Let l = l + 1, continue to repeat the process of steps 4.1.a-4.1.f until the output of the L3th TDB block is completed, denoted as

[0098] 4.1.h. The DMB block accepts the output feature map from the L3th TDB block as input, first output query, key, value in attention mechanism through linear layer:

[0099]

[0100]

[0101] wherein, q l , k l , v l are query, key, value in the attention mechanism of the lth DMB block, respectively, is a linear matrix.

[0102] 4.1.i. q lThe offset of each pixel point in the input feature map is predicted by the offset network Θ(·).

[0103] In this embodiment, the DMB block can be used for bidirectional prediction of the front and back frames, so there are two offset networks, which respectively generate the offset of the front and back items:

[0104] The offset network Θ(·) is stacked by convolution layers without changing the spatial size. In order to reduce the computational complexity, grouped offset learning is performed.

[0105] 4.1.j. Using the offset generated by the offset network to deform the input feature map to generate the front and back feature maps after offset and

[0106]

[0107] wherein, is a linear matrix, is a pre-trained forward optical flow, is a pre-trained backward optical flow, and grid represents the position information of the vector block in the input feature map, is the predicted front and back feature maps;

[0108] In this embodiment, in order to stabilize the training, the present application uses the forward and backward optical flows to guide the learning of the offset network.

[0109] 4.1.k. Using the predicted front and back feature maps and to generate the query, key and value in the sampling attention mechanism:

[0110]

[0111]

[0112] wherein, are linear matrices respectively, are the query, key and value in the sampling attention mechanism of the front feature map respectively, are the query, key and value in the sampling attention mechanism of the back feature map respectively.

[0113] 4.1.l. calculating the mutual attention and fusing and and (q l ,k l ,v l ).

[0114] The cross-attention between the front and back frames is:

[0115]

[0116] wherein F l represents the fused front feature map, B l represents the fused back feature map.

[0117] 4.1.m. Reduce the channel by simple channel merging and convolution layer, output the result of bidirectional alignment

[0118]

[0119] wherein, is the feature map after bidirectional alignment, which is the input of the (l+1)th DMB block;

[0120] 4.1.n. Use the method of steps h-m above until the output of the L4th deformable cross-attention module is completed, denoted as

[0121] Step (4.2). The feature map output by step (4.1) is down-sampled, that is, the spatial size is halved and the feature size is doubled, denoted as the down-sampled feature map

[0122] Step (4.3). The feature map output by step (4.2) is subjected to stage four, in this embodiment, the stage four is composed of L4 blocks of attention module in series with L4 deformable cross-attention modules.

[0123] In this embodiment, the execution process of step (4.3) is the same as that of step (4.1), and s=4 is set, and step (4.1) is repeated;

[0124] The calculation process of the stage four is represented as:

[0125]

[0126] wherein, represents the feature map output by the stage four, represents the function function of the TDB block in the stage four, represents the function function of the DMB block in the stage four.

[0127] Step (4.4). The feature map output by step (4.3) is up-sampled, that is, the spatial size is doubled and the feature size is halved, denoted as the up-sampled feature map

[0128] Step (4.5). The down-sampled feature map outputted by step (3.4) and the up-sampled feature map outputted by step (4.4) are added as the input feature map of stage five, which in this embodiment is composed of L5 blocks of the deformable attention module in series with L5 blocks of the variable attention module.

[0129] In this embodiment, step (4.5) is executed in the same way as step (4.1) with s = 5, and step (4.1) is repeated.

[0130] The calculation process of stage five is represented as:

[0131]

[0132] wherein, represents the feature map outputted by stage five, represents the function function of the TDB block in stage five, represents the function function of the DMB block in stage five.

[0133] Step (4.6). The feature map outputted by step (4.5) is up-sampled, i.e. the spatial size is doubled and the feature size is halved, and is denoted as the up-sampled feature map

[0134] Step (5). Local feature extraction and up-sampling are performed on the feature maps outputted by steps (2)-(4) to obtain the up-sampled local feature map.

[0135] Step (5.1). The down-sampled feature map outputted by step (3.2) and the up-sampled feature map outputted by step (4.6) are added as the input feature map of stage six, which in this embodiment is composed of L6 Swin-Transformer blocks.

[0136] The calculation process of stage six is represented as:

[0137]

[0138] wherein, represents the feature map outputted by stage six, represents the function function of stage six.

[0139] Step (5.2). The feature map outputted by step (5.1) is up-sampled, i.e. the spatial size is doubled and the feature size is halved, and is denoted as the up-sampled feature map

[0140] Step (5.3). The shallow feature map output in step (2) The upsampled feature map output from step (5.2) The summation serves as the input feature map for stage seven. In this embodiment, stage seven consists of L7 Swing-Transformer blocks.

[0141] The calculation process for stage seven is expressed as follows:

[0142]

[0143] in, The feature map representing the output of stage seven. This represents the function for stage seven.

[0144] The temporal trajectory attention module is used to extract features of the same spatial block vector in different time dimensions and perform trajectory-based feature alignment, executing the method in step (6) below.

[0145] Step (6). The feature map output in step (5.3) The temporal trajectory attention module extracts N*β feature information of each block in each frame of the image that is most similar in the temporal trajectory, and uses this as the alignment result.

[0146] The entire time-trajectory attention module can be denoted as:

[0147]

[0148] in, f represents the feature map output by the temporal trajectory attention module. traj (.) represents the function of the time trajectory attention module.

[0149] In one specific embodiment of the present invention, the calculation steps of the time trajectory attention module include:

[0150] 6.a. Input feature map The frame is divided into blocks, and the block result of the current frame is used as the query Q in the attention mechanism. i ;

[0151] Feature maps of all frames before the current frame The data is divided into blocks, and the block results of all frames preceding the current frame are used as the key K in the attention mechanism. i ;

[0152] Feature maps of all frames before the current frame The data is divided into blocks, and the block results of all frames preceding the current frame are used as the value V in the attention mechanism. i .

[0153] The Q mentioned i K i V i Represented as:

[0154]

[0155] Among them, Q i Represents the feature map of the current frame The query in the attention mechanism corresponding to the block results, K i Represents the feature maps of all frames preceding the current frame. The keys in the attention mechanism corresponding to the block results. For Q i The nth block in K represents i The nth block of the t-th frame, V i Represents the feature maps of all frames preceding the current frame. The key-value pairs in the attention mechanism corresponding to the block-based results. V represents i The nth block of the t-th frame in the feature map, where N represents the number of blocks in the feature map.

[0156] Use a bitmap to record the coordinate position of each feature map block on the time trajectory:

[0157]

[0158] in, This represents the x-coordinate of the nth block in frame t. and ordinate W is the width of the feature map, H is the height of the feature map, and τ n This represents the set of coordinates of the nth block corresponding to all frames preceding the current frame. This represents the set of coordinates of blocks from all frames preceding the current frame.

[0159] Therefore, formula (13) can be reformulated as a trajectory:

[0160]

[0161] in, express Feature map corresponding to the location The blocks, express Feature map blocks corresponding to coordinate positions in the set. express Feature map corresponding to the location The blocks.

[0162] 6.b. For each feature map block and trajectory K i Cosine similarity of each frame corresponding block in the set is calculated to obtain cosine similarity

[0163]

[0164] wherein D is the feature dimension of the feature block, is the square of L2 norm, is and trajectory K i Cosine similarity of each frame corresponding block in the set.

[0165] 6.c. For each block in the feature map, K i The N*β blocks with the highest similarity in the set:

[0166]

[0167] wherein, is the block set with the highest similarity, and β is the proportion coefficient.

[0168] 6.d. K i The N*(1-β) block set with the lowest similarity in the set is made into a mask, and the trajectory attention matrix is output again through the softmax function

[0169]

[0170] wherein mask_fill(.) is a mask processing function, is the trajectory attention matrix, is the cosine similarity after adding the mask.

[0171] 6.e. For each block in the feature map, time feature information can be extracted from the trajectory according to simple matrix multiplication:

[0172]

[0173] wherein, represents the set of represents the feature map corresponding to the position block.

[0174] 6.f. The feature map output in step 6.e is corrected through L7 residual blocks, and the process is recorded as:

[0175]

[0176] wherein, represents a function function of the residual block, represents a feature map of the residual block output. In the embodiment, the residual block is a conventional structure in the art, for example, the residual block structure in the RestNet residual network can be adopted.

[0177] Step (7). For each frame of the enhanced image in the video Steps (2)-(6) are performed, and a single-direction super-resolution result is output

[0178] Step (8). The enhanced image is processed in the reverse direction, i.e., the first frame of the enhanced image is taken as the T-th frame of the enhanced image after reversing The reverse-direction super-resolution result is obtained by the method of steps (2)-(6)

[0179] The super-resolution reconstruction module is used to aggregate the spatial feature information and the temporal feature information extracted by the foregoing network, and the channel feature information is expanded to the spatial dimension through pixel-level feature arrangement, so that an efficient up-sampling operation is completed. The method in the following step (9) is executed.

[0180] Step (9). The single-direction super-resolution result output by step (7) and the reverse-direction super-resolution result output by step (8) are fused, i.e., channel merging, and then the super-resolution reconstruction module is used to complete the output of the high-resolution frame.

[0181] In a specific implementation of the application, the step (9) includes:

[0182] Step (9.1). The output results of steps (7) and (8) are spliced, denoted as:

[0183]

[0184] wherein, Conv(.) represents convolution, [. ;.] represents splicing operation, X 10 represents the super-resolution feature map after channel merging.

[0185] Step (9.2). The sub-pixel convolution layer is used to realize the up-sampling of each frame of the super-resolution feature map in X 10 , denoted as:

[0186]

[0187] ​​wherein f pixelshuffle (.) denotes a sub-pixel convolution layer, and X 11 denotes the up-sampling feature map output by the sub-pixel convolution layer.

[0188] In this embodiment, the sub-pixel convolution layer is specifically: taking X 10 as input, reducing the feature map of each frame in X by r 2 times in the feature channel, tiling the patches of the feature map to the spatial dimension, and then restoring the channel to 3 through a convolution layer to obtain the up-sampling feature map X 11 output by the sub-pixel convolution layer.

[0189] Step (9.3). Adding the up-sampling feature map X 11 output by step (9.2) and the result after bicubic interpolation of the low-resolution video data stream X obtained in step (1), thereby outputting the final super-resolution result Y, denoted as:

[0190] Y = X 11 + f upsampling (X)

[0191] wherein f upsampling (.) denotes a bicubic interpolation operation.

[0192] The final super-resolution result Y is the high-resolution video output by the low-resolution video through the spatial-temporal mixed video super-resolution method based on deformable attention of the present application.

[0193] The above description is merely specific embodiments of the present application and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the concept of the present application. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A spatio-temporal hybrid video super-resolution method based on deformable attention, characterized in that, The method comprises the following steps: Step (1). Obtain low-resolution video data stream and perform image enhancement; Step (2). Obtain shallow features of each frame of the enhanced image in the video to obtain a shallow feature map; Step (3). Perform local feature extraction and down-sampling on the shallow feature map obtained in step (2) to obtain a down-sampled local feature map; Step (4). Perform global feature extraction on the down-sampled local feature map obtained in step (3), and obtain a feature map with aggregated global information through a block dropout attention module and a deformable attention module connected in series; Step (5). Perform local feature extraction and up-sampling on the feature maps output by steps (2)-(4) to obtain an up-sampled local feature map; Step (6). Calculate the similarity of each block of the local feature map obtained in step (5) on the time track and fuse them to obtain an aligned feature map; Step (7). Repeat steps (2) to (6) to traverse each frame of the enhanced image in the video to obtain a single-direction super-resolution result; Step (8). Repeat steps (2) to (6) to process each frame of the enhanced image in the video in reverse to obtain a reverse-direction super-resolution result; Step (9). Fuse the single-direction super-resolution result output by step (7) and the reverse-direction super-resolution result output by step (8), and then pass through a super-resolution reconstruction module to complete the output of a high-resolution frame image.

2. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 1, characterized in that, In step (1), the image enhancement includes mirroring, horizontal 90° inversion, and vertical 90° inversion.

3. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 1, characterized in that, In step (2), the channels of the enhanced image are expanded from 3 to 64 while keeping the resolution unchanged to obtain a shallow feature map.

4. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 1, characterized in that, Step (3) comprises: Step (3.1). The shallow feature map output from step (2) is input into the L1 Swin-Transformer blocks in stage one to extract local features. Local features are extracted through L1 Swin-Transformer blocks in stage one. Step (3.2). Down-sample the feature map output by step (3.1); Step (3.3). Extract local features from the down-sampled feature map output by step (3.2) through L2 Swin-Transformer blocks in stage two; Step (3.4). Down-sample the feature map output by step (3.3) to obtain a down-sampled local feature map.

5. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 1, characterized in that, Step (4) comprises: Step (4.1). The down-sampled local feature map output by step (3) is sequentially passed through L3 block dropout attention modules and L3 deformable attention modules in stage three; Step (4.2). Down-sample the feature map output by step (4.1); Step (4.3). The feature map output by step (4.2) is sequentially passed through L4 block dropout attention modules and L4 deformable attention modules in stage four; Step (4.4). Up-sample the feature map output by step (4.3); Step (4.5). Add the down-sampled local feature map output by step (3) and the up-sampled feature map output by step (4.4), and then sequentially pass through L5 block dropout attention modules and L5 deformable attention modules in stage five; Step (4.6). Up-sample the feature map output by step (4.5) to obtain a feature map with aggregated global information.

6. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 5, characterized in that, The calculation process of the block discard attention module in the third, fourth and fifth stages includes: a. Each block discard attention module includes M TDB sub-blocks, and the discard rate of each TDB sub-block is: wherein a j is the discard rate of the jth TDB sub-block; b. For each TDB sub-block, the input feature map is denoted as represents the i-th frame feature map input by the j-th sub-block of the i-th block in stage s; the i-th frame feature map input by the 1-st sub-block of the 1-st block in stage three is the down-sampled local feature map output by step (3), and the output of the previous TDB sub-block is taken as the input of the following TDB sub-block; In each TDB sub-block, first, the input feature map is divided into sub-blocks The cosine similarity between each sub-block in the feature map is calculated to obtain an attention matrix wherein, is the attention matrix of the jth sub-block of the attention drop module of the lth block. c. Each TDB sub-block according to the discard rate a j Attention matrix The lowest a in similarity j *N feature map patches are made into a mask matrix For the next TDB sub-block; d. According to the attention matrix and the value in the attention mechanism, the output feature map of the TDB sub-block is obtained: wherein, represents the i-th frame feature map output by the j-th sub-block of the l-th block drop attention module in stage s, corresponding to the i-th frame feature map input by the j+1-th sub-block of the l-th block drop attention module in stage s, is a value in the j-th sub-block attention mechanism of the l-th block drop attention module. e.In the block discard attention module, the next TDB sub-block receives the mask matrix from the previous TDB sub-block and combines it with the attention matrix computed by the current TDB sub-block, resulting in a new attention matrix: f. Calculate the output feature map according to the new attention matrix, and the formula is: g.The process of steps a-f is repeated until the output of the last sub-block in the l-th block drop-attention module is outputted, which is the output of the l-th block drop-attention module, denoted as which is taken as the input of the (l+1)-th block drop-attention module, denoted as Let l = l + 1, continue to repeat the process of steps a-f until the output of the Lth block discard attention module is completed, denoted as s ​ 7. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 5, characterized in that, The calculation process of the deformable mutual attention module in the third, fourth and fifth stages includes: h.The deformable cross-attention module accepts the output feature maps from the Lth block discard attention module in the corresponding stage s s The deformable cross-attention module accepts the output feature maps from the Lth block discard attention module in the corresponding stage s As input, first output the query, key, value in the attention mechanism through a linear layer l e [1, L s ], s e [3, 4, 5] wherein q l , k l , v l are respectively the query, key, value in the l-th deformable self-attention module attention mechanism, is a linear matrix; i. the query q in the first deformable cross-attention module attention mechanism l The offset of each pixel point in the input feature map is predicted by the offset network Θ(·) to generate the offset of the front and rear items: j. offsets generated by the offset network to the input feature map deformation, thereby generating the offset feature map and wherein, is a linear matrix, is a pre-trained forward optical flow, is a pre-trained backward optical flow, and geid represents position information of the vector block in the input feature map, is a predicted forward and backward feature map; k.Using predicted preceding and succeeding feature maps and Generating queries, keys, values in the attention mechanism after sampling: wherein, are linear matrices, respectively, are query, key, value in the attention mechanism after sampling for the former feature map generation, respectively, are query, key, value in the attention mechanism after sampling for the latter feature map generation, respectively. l. will be and and the first deformable cross-attention module attention mechanism query, key, value (q l ,k l ,v l ) to calculate cross-attention and fusion; The mutual attention between the front and rear frames is: Wherein, F l represents the previous feature map after fusion, B l represents the latter feature map after fusion; m. Reduce the channels by simple channel merging and convolution layers, and output the bidirectional alignment result; n. The method of steps h-m above is followed until the output of the Lth deformable cross-attention module, denoted as s ​ 8. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 4, characterized in that, The step (5) includes: Step (5.1). Add the down-sampling feature map output by step (3.2) and the feature map output by step (4) which aggregates global information, and extract local features through L6 Swin-Transformer blocks in the sixth stage; Step (5.2). Perform up-sampling operation on the feature map output by step (5.1); Step (5.3). Add the shallow feature map output by step (2) and the up-sampled feature map output by step (5.2), extract local features through L7 Swin-Transformer blocks in the seventh stage, and obtain the up-sampled local feature map.

9. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 1, characterized in that, The step (6) includes: Step (6.1). The up-sampled local feature maps obtained in step (5) are divided into blocks, and the block results of the current frame are taken as the query Q in the attention mechanism i ; The up-sampled local feature map corresponding to all frames before the current frame The up-sampled local feature map corresponding to all frames before the current frame is divided into blocks, and the block result of the current frame is used as a key K in the attention mechanism i ; wherein, The up-sampled local feature map corresponding to all frames before the current frame is divided into blocks, and the block result of the current frame is used as a key K in the attention mechanism the aligned feature maps of all the frames before the current frame performing blocking, taking the blocking result of all the frames before the current frame as the value V in the attention mechanism i ; The coordinate positions of each feature map patch on the time track are recorded by a bitmap to obtain Q i , K i , V i trajectory expression forms: wherein, denotes the horizontal coordinate of the nth patch in the tth frame and the vertical coordinate W is the width of the feature map and H is the height of the feature map; denotes the feature map patch corresponding to the position denotes the feature map patch corresponding to the position denotes the feature map patch corresponding to the position N denotes the number of patches of the feature map;​ Step (6.2). For feature map patching and trajectory K i Cosine similarity is calculated for each frame corresponding to the patch in the set, and the cosine similarity is obtained. Step (6.3). For cosine similarity of patches in feature maps, keep K i top N*β patches with highest similarity in the set Step (6.4). The K i N*(1-β) block sets with the lowest similarity in the set The mask is made, and the trajectory attention matrix is output again through the softmax function Step (6.5). For each sub-block in the feature map, extract time feature information from the trajectory: wherein, denotes a set of denotes feature maps corresponding to the positions a patch of [1, i-1], τ n denotes a set of coordinates of the n-th patch corresponding to all frames before the current frame, denotes a set of coordinates of the patch of all frames before the current frame; Step (6.6). Feature maps output from step (6.5) are input into a L7 residual block to correct the features and obtain aligned feature maps. Step (6.6). Feature maps output from step (6.5) are input into a L7 residual block to correct the features and obtain aligned feature maps.

10. The spatio-temporal hybrid video super-resolution method based on deformable attention according to claim 1, characterized in that, The step (9) includes: Step (9.1). Concatenate the single-direction super-resolution result and the reverse-direction super-resolution result to obtain the channel-merged super-resolution feature map; Step (9.2). Perform up-sampling on each frame of the channel-merged super-resolution feature map through sub-pixel convolution layers to obtain the up-sampled feature map; Step (9.3). Add the up-sampled feature map output by step (9.2) and the result after bicubic interpolation of the low-resolution video data stream obtained by step (1), thereby outputting the final super-resolution result and obtaining the high-resolution frame image.

Images