Video super-resolution imaging reconstruction method based on transformer scale flow estimation

Abstract

The application discloses a video super-resolution imaging reconstruction method based on transformer scale flow estimation, and comprises the following steps: 1) data set preprocessing; 2) establishing a reconstruction model; and 3) model training. The method can obtain more high-frequency detail information of different scales, learn better features, and improve the reconstruction effect of video super-resolution imaging.

Description

Technical Field

[0001] This invention relates to image processing technology, specifically a video super-resolution imaging reconstruction method based on transformer scale flow estimation. Background Technology

[0002] Super-resolution (SR) aims to generate high-resolution (HR) images or videos from low-resolution (LR) corresponding images. As a typical low-level computer vision problem, SR has been extensively studied for decades. In recent years, with the increasing demands of smartphones, apps, smart HDTVs, and surveillance equipment, the need for super-resolution technology has grown significantly. With substantial research and promising results in deep learning, significant progress has been made in single-image super-resolution in recent years. And in recent years, the increasing demand for high-definition video has brought video super-resolution into sharper focus.

[0003] Existing deep learning-based video super-resolution imaging and reconstruction methods generally involve four steps: feature extraction, alignment, fusion, and reconstruction. Feature alignment and fusion primarily involve explicit or implicit motion estimation and compensation across multiple frames. When problems such as occlusion and complex motion exist in the video, the feature alignment and fusion strategies play a crucial role in the quality of video frame reconstruction: how to align multiple frames under given large motion and how to effectively fuse different frames. Video super-resolution (VSR) presents an additional challenge because it involves aggregating information from multiple highly correlated but misaligned frames in a video sequence. When the video contains occlusion, large motion, and severe blur, the challenge lies in the design of the alignment and fusion modules. To obtain high-quality output, accurate alignment and correspondence establishment are essential.

[0004] Better utilization of inter-frame and fused information is crucial. Learning-based optical flow estimation has long been dominated by cost volume and convolutional pipelines, which are inherently limited to local correlations, making it difficult to address the long-standing challenge of large displacements. Recurrence-based methods are the strategy adopted by most advanced algorithms, and they can effectively extract implicit alignment and fused features. However, previous alignment and fusion methods rely on local sliding window methods to align multiple frames under given large motions, failing to effectively utilize global information for alignment. Furthermore, when directly fusing features from the fused past hidden state information, they fail to fully leverage the relationships between features and the high-frequency details of the past hidden state information.

[0005] Therefore, a new technology is needed to solve the problems mentioned above. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a video super-resolution imaging reconstruction method based on transformer scale flow estimation. This method can acquire more high-frequency detail information at different scales, learn better features, and improve the reconstruction effect of video super-resolution imaging.

[0007] The technical solution to achieve the objective of this invention is:

[0008] A video super-resolution imaging reconstruction method based on transformer scale flow estimation, which differs from existing techniques, includes the following steps:

[0009] 1) Dataset preprocessing: The obtained high-definition video frame dataset Vimeo-90K is blurred with a Gaussian kernel of 1.6, and then further downsampled by 4 times to obtain low-resolution video frames, resulting in the training set T;

[0010] 2) Establishing a reconstruction model: The training dataset T is processed using the reconstruction model (Model) to obtain the super-resolution image SR. The reconstruction model (Model) includes a pre-scale hidden state process (PSHSnet), a transformer scale flow estimation process (TSFEnet), a scale interaction fusion process (SIFnet), and a reconstruction process (Rnet), including:

[0011] 2-1) Take multiple frames of samples I from the training set T t ,I t-1 ...I t-n Input the pre-scaled hidden state process PSHSnet to obtain initial hidden state features H0, H0, and H0 at multiple scales. 1 / 2 H0 1 / 4 The pre-scale hidden state process in PSHSNet is as follows: First, input multiple low-resolution video frames I t ,I t-1 ...I t-n Each layer undergoes a 3x3 convolution operation (conv3) for shallow feature extraction. The extracted features are then concatenated and fused using conccat to obtain the fused feature x1. Next, feature x1 is processed by a Unet consisting of channel attention layers at each level. attention The model obtains feature x2; then, feature x2 is passed through a residual block RB consisting of 5 layers of residual units to obtain feature x3; finally, feature x3 is passed through a reconstruction network Rnet without subpixels, and the initial hidden state feature information H0 is downsampled by a factor of 2 to obtain the downsampled feature H0. 1 / 2 For the downsampled feature H0 1 / 2 Perform a 2x downsampling to obtain the downsampling feature H0. 1 / 4The process is shown in formulas (1), (2), (3), (4), (5), and (6):

[0012] x1 = conccat(conv3(I t ),conv3(I t-1 ...conv3(I t-n )) (1),

[0013] x2 = Unet attention (x1) (2),

[0014] x3=RB(x2) (3),

[0015] H0 = Rnet(x3) (4),

[0016] H0 1 / 2 =Downsample(H0) (5),

[0017] H0 1 / 4 =Downsample(H0) 1 / 2 (6);

[0018] 2-2) Take samples I from the training set T t ,I t-1 The initial hidden state features are input into the Transformer Scale Flow Estimation Process (TSFEnet) to obtain fused features X1, X2, and X3 at different scales. The Transformer Scale Flow Estimation Process (TSFEnet) is as follows:

[0019] 2-2-1) Input two low-resolution video frames I t ,I t-1 Features f1 and f2 are obtained by using residual blocks with shared weights.

[0020] 2-2-2) Input features f1 and f2 into two layers of residual blocks for downsampling to obtain the feature f1 after the first layer of downsampling. 1 / 2 f2 1 / 2 The features f1 after the second layer downsampling 1 / 4 f2 1 / 4 ;

[0021] 2-2-3) This will yield features f1 and f2 at different scales, with the first-layer downsampled feature f1 1 / 2 f2 1 / 2 Second layer downsampling f1 1 / 4 f2 1 / 4 Inputting each of the respective transformer flow estimation networks TFEnet yields flow estimates, respectively. 1 / 2,flow 1 / 4 ;

[0022] 2-2-4) For feature f1 1 / 4 Hidden state feature H0 1 / 4 Perform flow 1 / 4 Flow estimation yields the distortion feature f12. 1 / 4 H0 1 / 4warp ;

[0023] 2-2-5) For feature f12 1 / 4 H0 1 / 4warp f2 1 / 4 Channels are spliced ​​and fused to obtain feature X3;

[0024] 2-2-6) Input feature X3 into the residual block and upsample it to obtain feature X32;

[0025] 2-2-7) For feature f1 1 / 2 Hidden state feature H0 1 / 2 f2 1 / 2 Repeat steps 2-2-5 and 2-2-6 to obtain feature x2;

[0026] 2-2-8) Channel number concatenation and fusion of features X32 and x2 are performed to obtain feature X2;

[0027] 2-2-9) Input feature X2 into the residual block and upsample it to obtain feature X21;

[0028] 2-2-10) Repeat operations 2-2-5) and 2-2-6 on feature f1, hidden state feature H0, and f2 to obtain feature x1;

[0029] 2-2-11) Channel number concatenation and fusion of features X23 and x1 are performed to obtain feature X1;

[0030] 2-3) Input the fused features X1, X2, and X3 obtained in step 2-2) into the scale interaction fusion network SIFnet to obtain high-frequency detail feature information f3. The scale interaction fusion network SIFnet is composed of a Unet model consisting of four scales, where the bottom layer of the model consists of a transformer, and the remaining layers consist of residual multi-level blocks. The specific scale interaction fusion process of SIFnet is as follows:

[0031] 2-3-1) Input feature X1 is processed through the first layer of residual multi-level module for downsampling to obtain feature Z1;

[0032] 2-3-2) The feature Z1 is downsampled through a convolutional layer to obtain the feature Z2.

[0033] 2-3-3) After concatenating and fusing features Z2 and X2 by channel number, the feature is input into the second layer of the residual multilevel block of downsampling to obtain feature Z3;

[0034] 2-3-4) Downsample feature Z1 through a convolutional layer to obtain feature Z4;

[0035] 2-3-5) After concatenating and fusing features Z4 and X3 by channel number, the feature is input into the third layer of the downsampled residual multilevel block to obtain feature Z5;

[0036] 2-3-6) Downsample feature Z5 by passing it through a convolutional layer to obtain feature Z6;

[0037] 2-3-7) Input feature Z5 into the cross-transformer layer to obtain feature Z7;

[0038] 2-3-8) Deconvolve feature Z7 to obtain upsampled feature Z8;

[0039] 2-3-9) After concatenating and fusing features Z8 and Z5 by channel number, input them into the first layer of the upsampled residual multi-level block to obtain feature Z9;

[0040] 2-3-10) Deconvolve feature Z9 to obtain upsampled feature Z10;

[0041] 2-3-11) After concatenating and fusing features Z10 and Z5 by channel number, the result is input into the second layer of the upsampled residual multi-level block to obtain feature Z11;

[0042] 2-3-12) Deconvolve feature Z11 to obtain upsampled feature Z12;

[0043] 2-3-13) After concatenating and fusing features Z12 and Z3 by channel number, the result is input into the second layer of the upsampled residual multilevel block to obtain feature Z13;

[0044] 2-3-14) Deconvolve feature Z13 to obtain upsampled feature Z14;

[0045] 2-3-15) After concatenating and fusing features Z14 and Z1 by channel number, the result is input into the second layer of the upsampled residual multi-level block to obtain high-frequency detail feature information f3;

[0046] 2-4) Input the high-frequency detail feature information f3 obtained in step 2-3) into the reconstruction network Rnet. The reconstruction network Rnet consists of 4 residual layers, 2 parallel convolutional layers, 1 dropout layer, and 1 sub-pixel convolutional layer. The output is the super-resolution image SR1 of the first frame and the hidden state information feature H. t1 The specific reconstruction process of Rnet is as follows:

[0047] First, the input feature f3 is processed through convolutional layers, activation function layers σ, dropout layers, and convolutional layers to obtain the hidden information feature map h. t Reconstructing the output feature map t Then for h t o t The final hidden information feature map H is obtained by concatting and fusion of the channels. t , for the hidden information feature map H t Two layers of downsampling are performed to obtain the downsampled feature map H. t1 / 2 H t1 / 4 At the same time, for o t Subpixel convolution is performed to obtain the feature map, and the corresponding low-resolution image of the reconstructed frame is obtained by bicubic upsampling. t The final super-resolution image (SR) is obtained by adding elements one by one.

[0048] 3) Model Training: The network is trained end-to-end. First, the L1 loss function is used to accelerate convergence, and then the L2 loss function is used for fine-tuning, as shown in Equations (7) and (8):

[0049] L1 loss function:

[0050] L2 loss function:

[0051] The transformer flow estimation network TFEnet described in step 2) is as follows: First, assuming the input features are F1 and F2, the cross-transformer is used to perform self-similarity feature enhancement to obtain features F3 and F4, where feature F1 is the query (Q) of the cross-attention, and feature F2 is the key (K) and value (V); finally, features F3 and F4 are used for flow estimation through relevance softmax to obtain the final flow estimate flow, where relevance softmax consists of matrix operations and softmax, as shown in formulas (9), (10), and (11):

[0052]

[0053]

[0054] M = softmax(C) ∈ R H×W×H×W (11),

[0055] Where d k D is a standardization factor to avoid large values ​​after matrix operations.

[0056] The multi-level residual block (MRLB) mentioned in step 2) is as follows: First, the input feature x1 is cascaded through three convolutional layers conv3 with 3x3 kernels, and an activation function layer σ is added after each convolutional layer. The output of each layer is represented as F. a F b F c Channel concatenation and fusion are performed (concat), followed by dimensionality reduction of the number of channels using a 1x1 convolution kernel (conv1) to obtain features. These features are then element-wise added to the input features (x1) of the multi-level residual module (MRLB) to form the residual multi-level module. The resulting feature is represented as F. MRLB The formula is as follows:

[0057] F MRLB =conv1(concat(F a ,F b ,F c ))+x1 (12).

[0058] This technical solution can more fully explore the spatial relationships within video frames and the temporal relationships between frames, and can obtain better high-frequency detail information, thereby recovering better reconstructed details and effectively improving the quality of reconstructed image frames in the video.

[0059] This method employs a transformer scale flow estimation network to utilize global multi-scale contextual information for inter-frame alignment, enabling better processing of video frames with large displacements and occlusions. The use of a pre-scale hidden state network can reduce the impact of imbalance in the one-way recurrent method. The use of a scale interaction fusion network to obtain information at different scales allows for the acquisition of more high-frequency detail information and the learning of better features, thereby improving the imaging and reconstruction effect of video super-resolution.

[0060] This method can acquire more high-frequency detail information at different scales, learn better features, and improve the reconstruction effect of video super-resolution imaging. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the method flow in the embodiment;

[0062] Figure 2 This is a schematic diagram of the overall network in the embodiment;

[0063] Figure 3 This is a schematic diagram of the pre-scaled hidden state network in the embodiment;

[0064] Figure 4 This is a schematic diagram of the transformer scale flow estimation network in the embodiment;

[0065] Figure 5 This is a schematic diagram of the scale-interactive fusion network in the embodiment. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0067] Example:

[0068] Reference Figure 1 , Figure 2 A video super-resolution imaging reconstruction method based on transformer scale flow estimation, which differs from existing techniques, includes the following steps:

[0069] 1) Dataset preprocessing: The obtained high-definition video frame dataset Vimeo-90K is blurred with a Gaussian kernel of 1.6, and then further downsampled by 4 times to obtain low-resolution video frames, resulting in the training set T;

[0070] 2) Establishing a reconstruction model: The training dataset T is processed using the reconstruction model (Model) to obtain the super-resolution image SR. The reconstruction model (Model) includes a pre-scale hidden state process (PSHSnet), a transformer scale flow estimation process (TSFEnet), a scale interaction fusion process (SIFnet), and a reconstruction process (Rnet), including:

[0071] 2-1) Take multiple frames of samples I from the training set T t ,I t-1 ...I t-n Input the pre-scaled hidden state process PSHSnet to obtain initial hidden state features H0, H0, and H0 at multiple scales. 1 / 2 H0 1 / 4 Among them, such as Figure 3 As shown, the pre-scaled hidden state process in PSHSNet is as follows: First, input multiple low-resolution video frames I t ,I t-1 ...I t-n Each layer undergoes a 3x3 convolution operation (conv3) for shallow feature extraction. The extracted features are then concatenated and fused using conccat to obtain the fused feature x1. Next, feature x1 is processed by a Unet consisting of channel attention layers at each level. attention The model obtains feature x2; then, feature x2 is passed through a residual block RB consisting of 5 layers of residual units to obtain feature x3; finally, feature x3 is passed through a reconstruction network Rnet without subpixels, and the initial hidden state feature information H0 is downsampled by a factor of 2 to obtain the downsampled feature H0. 1 / 2 For the downsampled feature H0 1 / 2 Perform a 2x downsampling to obtain the downsampling feature H0. 1 / 4The process is shown in formulas (1), (2), (3), (4), (5), and (6):

[0072] x1 = conccat(conv3(I t ),conv3(I t-1 ...conv3(I t-n )) (1),

[0073] x2 = Unet attention (x1) (2),

[0074] x3=RB(x2) (3),

[0075] H0 = Rnet(x3) (4),

[0076] H0 1 / 2 =Downsample(H0) (5),

[0077] H0 1 / 4 =Downsample(H0) 1 / 2 (6);

[0078] 2-2) Take samples I from the training set T t ,I t-1 The initial hidden state features are input into the transformer scale flow estimation process TSFEnet, resulting in fused features X1, X2, and X3 at different scales, where, for example... Figure 4 As shown, the transformer-scaled flow estimation process in TSFEnet is as follows:

[0079] 2-2-1) Input two low-resolution video frames I t ,I t-1 Features f1 and f2 are obtained by using residual blocks with shared weights.

[0080] 2-2-2) Input features f1 and f2 into two layers of residual blocks for downsampling to obtain the feature f1 after the first layer of downsampling. 1 / 2 f2 1 / 2 The features f1 after the second layer downsampling 1 / 4 f2 1 / 4 ;

[0081] 2-2-3) This will yield features f1 and f2 at different scales, with the first-layer downsampled feature f1 1 / 2 f2 1 / 2 Second layer downsampling f1 1 / 4 f2 1 / 4Inputting each of the respective transformer flow estimation networks TFEnet yields flow estimates, respectively. 1 / 2 ,flow 1 / 4 ;

[0082] 2-2-4) For feature f1 1 / 4 Hidden state feature H0 1 / 4 Perform flow 1 / 4 Flow estimation yields the distortion feature f12. 1 / 4 H0 1 / 4warp ;

[0083] 2-2-5) For feature f12 1 / 4 H0 1 / 4warp f2 1 / 4 Channels are spliced ​​and fused to obtain feature X3;

[0084] 2-2-6) Input feature X3 into the residual block and upsample it to obtain feature X32;

[0085] 2-2-7) For feature f1 1 / 2 Hidden state feature H0 1 / 2 f2 1 / 2 Repeat steps 2-2-5 and 2-2-6 to obtain feature x2;

[0086] 2-2-8) Channel number concatenation and fusion of features X32 and x2 are performed to obtain feature X2;

[0087] 2-2-9) Input feature X2 into the residual block and upsample it to obtain feature X21;

[0088] 2-2-10) Repeat operations 2-2-5) and 2-2-6 on feature f1, hidden state feature H0, and f2 to obtain feature x1;

[0089] 2-2-11) Channel number concatenation and fusion of features X23 and x1 are performed to obtain feature X1;

[0090] 2-3) Input the fused features X1, X2, and X3 obtained in step 2-2) into the scale interaction fusion network SIFnet to obtain high-frequency detail feature information f3. The scale interaction fusion network SIFnet is composed of a Unet model consisting of four scales, where the bottom layer of the model consists of transformers, and the remaining layers consist of residual multi-level blocks, as shown below. Figure 5 As shown, the specific scale-based interactive fusion process of SIFnet is as follows:

[0091] 2-3-1) Input feature X1 is processed through the first layer of residual multi-level module for downsampling to obtain feature Z1;

[0092] 2-3-2) The feature Z1 is downsampled through a convolutional layer to obtain the feature Z2.

[0093] 2-3-3) After concatenating and fusing features Z2 and X2 by channel number, the feature is input into the second layer of the residual multilevel block of downsampling to obtain feature Z3;

[0094] 2-3-4) Downsample feature Z1 through a convolutional layer to obtain feature Z4;

[0095] 2-3-5) After concatenating and fusing features Z4 and X3 by channel number, the feature is input into the third layer of the downsampled residual multilevel block to obtain feature Z5;

[0096] 2-3-6) Downsample feature Z5 by passing it through a convolutional layer to obtain feature Z6;

[0097] 2-3-7) Input feature Z5 into the cross-transformer layer to obtain feature Z7;

[0098] 2-3-8) Deconvolve feature Z7 to obtain upsampled feature Z8;

[0099] 2-3-9) After concatenating and fusing features Z8 and Z5 by channel number, input them into the first layer of the upsampled residual multi-level block to obtain feature Z9;

[0100] 2-3-10) Deconvolve feature Z9 to obtain upsampled feature Z10;

[0101] 2-3-11) After concatenating and fusing features Z10 and Z5 by channel number, the result is input into the second layer of the upsampled residual multi-level block to obtain feature Z11;

[0102] 2-3-12) Deconvolve feature Z11 to obtain upsampled feature Z12;

[0103] 2-3-13) After concatenating and fusing features Z12 and Z3 by channel number, the result is input into the second layer of the upsampled residual multilevel block to obtain feature Z13;

[0104] 2-3-14) Deconvolve feature Z13 to obtain upsampled feature Z14;

[0105] 2-3-15) After concatenating and fusing features Z14 and Z1 by channel number, the result is input into the second layer of the upsampled residual multi-level block to obtain high-frequency detail feature information f3;

[0106] 2-4) Input the high-frequency detail feature information f3 obtained in step 2-3) into the reconstruction network Rnet. The reconstruction network Rnet consists of 4 residual layers, 2 parallel convolutional layers, 1 dropout layer, and 1 sub-pixel convolutional layer. The output is the super-resolution image SR1 of the first frame and the hidden state information feature H. t1 The specific reconstruction process of Rnet is as follows:

[0107] First, the input feature f3 is processed through convolutional layers, activation function layers σ, dropout layers, and convolutional layers to obtain the hidden information feature map h. t Reconstructing the output feature map t Then for h t o t The final hidden information feature map H is obtained by concatting and fusion of the channels. t , for the hidden information feature map H t Two layers of downsampling are performed to obtain the downsampled feature map H. t1 / 2 H t1 / 4 At the same time, for o t Subpixel convolution is performed to obtain the feature map, and the corresponding low-resolution image of the reconstructed frame is obtained by bicubic upsampling. t The final super-resolution image (SR) is obtained by adding elements one by one.

[0108] 3) Model Training: The network is trained end-to-end. First, the L1 loss function is used to accelerate convergence, and then the L2 loss function is used for fine-tuning, as shown in Equations (7) and (8):

[0109] L1 loss function:

[0110] L2 loss function:

[0111] The transformer flow estimation network TFEnet described in step 2) is as follows: First, assuming the input features are F1 and F2, the cross-transformer is used to perform self-similarity feature enhancement to obtain features F3 and F4, where feature F1 is the query (Q) of the cross-attention, and feature F2 is the key (K) and value (V); finally, features F3 and F4 are used for flow estimation through relevance softmax to obtain the final flow estimate flow, where relevance softmax consists of matrix operations and softmax, as shown in formulas (9), (10), and (11):

[0112]

[0113]

[0114] M = soft max(C) ∈ R H×W×H×W (11),

[0115] Where d k D is a standardization factor to avoid large values ​​after matrix operations.

[0116] The multi-level residual block (MRLB) mentioned in step 2) is as follows: First, the input feature x1 is cascaded through three convolutional layers conv3 with 3x3 kernels, and an activation function layer σ is added after each convolutional layer. The output of each layer is represented as F. a F b F c Channel concatenation and fusion are performed (concat), followed by dimensionality reduction of the number of channels using a 1x1 convolution kernel (conv1) to obtain features. These features are then element-wise added to the input features (x1) of the multi-level residual module (MRLB) to form the residual multi-level module. The resulting feature is represented as F. MRLB The formula is as follows:

[0117] F MRLB =conv1(concat(F a ,F b ,F c ))+x1 (12).

Claims

1. A video super-resolution imaging reconstruction method based on transformer scale flow estimation, characterized in that, Includes the following steps: 1) Dataset preprocessing: The obtained high-definition video frame dataset Vimeo-90K is blurred with a Gaussian kernel of 1.6, and then further downsampled by 4 times to obtain low-resolution video frames, resulting in the training set T; 2) Establishing a reconstruction model: The training dataset T is processed using the reconstruction model (Model) to obtain the super-resolution image SR. The reconstruction model (Model) includes a pre-scale hidden state process (PSHSnet), a transformer scale flow estimation process (TSFEnet), a scale interaction fusion process (SIFnet), and a reconstruction process (Rnet), including: 2-1) Sample multiple frames from the training set T Input the pre-scaled hidden state process PSHSnet to obtain initial hidden state features H0 at multiple scales. , The pre-scale hidden state process in PSHSNet is as follows: First, input multiple low-resolution video frames. Each layer undergoes a 3x3 convolution operation. Shallow feature extraction is performed, followed by channel concatenation and fusion of the extracted features. , obtain fusion features Secondly, characteristics Through a system composed of channel attention in each layer Model, obtain features Then, features The residual block consists of 5 layers of residual units. , to obtain features Finally, features Reconstruction network without subpixel composition The initial hidden state feature information H0 is then downsampled by a factor of 2 to obtain the downsampled feature information. For downsampling features Perform a 2x downsampling to obtain the downsampling features. ; 2-2) Samples from training set T The initial hidden state features are input into the transformer scale flow estimation process TSFEnet to obtain fused features X1, X2, and X3 at different scales; 2-3) Input the fusion features X1, X2, and X3 obtained in step 2-2) into the scale interaction fusion network SIFnet to obtain high-frequency detail feature information f3. The scale interaction fusion network SIFnet is composed of a Unet model consisting of four scales, where the bottom layer of the model is composed of transformers and the remaining layers are composed of residual multi-level blocks. 2-4) Input the high-frequency detail feature information f3 obtained in step 2-3) into the reconstruction network Rnet. The reconstruction network Rnet consists of 4 residual layers, 2 parallel convolutional layers, 1 dropout layer, and 1 sub-pixel convolutional layer, and outputs the super-resolution image of the first frame. Hidden state information features The specific reconstruction process of Rnet is as follows: First, the input feature f3 is processed through a convolutional layer and an activation function layer. The Dropout layer and convolutional layer obtain hidden information feature maps.

1. Reconstruct the output feature map ; then , Perform channel splicing and fusion Obtain the final hidden information feature map , for hidden information feature map Two layers of downsampling are performed to obtain the downsampled feature map. , At the same time, Perform subpixel convolution The obtained feature map and its corresponding bicubic upsampling are then performed. Low-resolution image of reconstructed frame The final super-resolution image (SR) is obtained by adding elements one by one. 3) Model training: The network is trained end-to-end. First, the L1 loss function is used to speed up the convergence, and then the L2 loss function is used for fine-tuning.

2. The video super-resolution imaging reconstruction method based on transformer scale flow estimation according to claim 1, characterized in that, The transformer flow estimation network TFEnet described in step 2) is as follows: First, assume the input features are... , Input a cross-transformer to perform self-similarity feature enhancement, and obtain the features. , , among which features For cross-attention queries, features For key and value; finally, features , Flow estimation is performed using correlation softmax to obtain the final flow estimate, where correlation softmax consists of matrix operations and softmax.

3. The video super-resolution imaging reconstruction method based on transformer scale flow estimation according to claim 1, characterized in that, The transformer-scale flow estimation process TSFEnet described in step 2-2) is as follows: 2-2-1) Input two low-resolution video frames Features f1 and f2 are obtained by using residual blocks with shared weights. 2-2-2) Input features f1 and f2 into two layers of residual blocks for downsampling to obtain the features after the first layer of downsampling. , Features after second-layer downsampling , ; 2-2-3) This will yield features f1 and f2 at different scales, and the first-layer downsampled features. , Second layer downsampling , Each input to its respective transformer flow estimation network TFEnet yields a flow estimate. , ; 2-2-4) Features Hidden state features conduct Flow estimation yields distortion features. , ; 2-2-5) Features , , Channels are spliced ​​and fused to obtain feature X3; 2-2-6) Input feature X3 into the residual block and upsample it to obtain feature X32; 2-2-7) Features Hidden state features , Repeat steps 2-2-5 and 2-2-6 to obtain feature x2; 2-2-8) Channel number concatenation and fusion of features X32 and x2 are performed to obtain feature X2; 2-2-9) Input feature X2 into the residual block and upsample it to obtain feature X21; 2-2-10) Repeat operations 2-2-5 and 2-2-6 for feature f1, hidden state features H0 and f2 to obtain feature x1; 2-2-11) Perform channel number splicing and fusion on features X23 and x1 to obtain feature X1.

4. The video super-resolution imaging reconstruction method based on transformer scale flow estimation according to claim 1, characterized in that, The scale-interactive fusion process described in steps 2-3) of SIFnet is as follows: 2-3-1) Input feature X1 is processed by the first layer of residual multi-level module for downsampling to obtain feature Z1; 2-3-2) The feature Z1 is downsampled through a convolutional layer to obtain the feature Z2. 2-3-3) After concatenating and fusing features Z2 and X2 by channel number, the result is input into the second layer of the downsampled residual multilevel block to obtain feature Z3; 2-3-4) Downsample feature Z1 through a convolutional layer to obtain feature Z4; 2-3-5) After concatenating and fusing features Z4 and X3 by channel number, the feature is input into the third layer residual multilevel block of downsampling to obtain feature Z5; 2-3-6) Downsample feature Z5 by passing it through a convolutional layer to obtain feature Z6; 2-3-7) Input feature Z5 into the cross-transformer layer to obtain feature Z7; 2-3-8) Deconvolve feature Z7 to obtain upsampled feature Z8; 2-3-9) After concatenating and fusing features Z8 and Z5 by channel number, input them into the first layer of the upsampled residual multi-level block to obtain feature Z9; 2-3-10) Deconvolve feature Z9 to obtain upsampled feature Z10; 2-3-11) After concatenating and fusing features Z10 and Z5 by channel number, the result is input into the second layer of the upsampled residual multi-level block to obtain feature Z11; 2-3-12) Deconvolve feature Z11 to obtain upsampled feature Z12; 2-3-13) After concatenating and fusing features Z12 and Z3 by channel number, the result is input into the second layer of the upsampled residual multilevel block to obtain feature Z13; 2-3-14) Deconvolve feature Z13 to obtain upsampled feature Z14; 2-3-15) After concatenating and fusing features Z14 and Z1 by channel number, the feature is input into the second layer of the upsampled residual multi-level block to obtain high-frequency detail feature information f3.

Images