Video spatiotemporal super-resolution method based on Transformer

Abstract

The present invention discloses a video spatiotemporal super-resolution method based on Transformer. The method comprises the following steps: collecting training samples of a video spatiotemporal super-resolution model; constructing a high frame rate low-resolution video sequence generation model, generating intermediate frames based on the temporal feature fusion of the Transformer attention mechanism, and generating a high frame rate low-resolution video sequence; constructing a high frame rate high-resolution video sequence generation model, and reconstructing a high frame rate high-resolution video sequence based on the spatiotemporal feature fusion of the Transformer attention mechanism; constructing and training a video spatiotemporal super-resolution model based on Transformer, and performing spatiotemporal super-resolution reconstruction on the video sequence according to the trained video spatiotemporal super-resolution model based on Transformer. The present invention combines the Transformer attention mechanism to obtain the spatiotemporal connection in the video sequence, and realizes the spatiotemporal super-resolution enhancement of the video.

Description

Technical Field

[0001] The present invention belongs to the field of video spatiotemporal super-resolution enhancement, and relates to a Transformer-based video spatiotemporal super-resolution method. Background Art

[0002] Video super-resolution enhancement refers to the reconstruction of a high-frame-rate and high-resolution video sequence using multiple low-resolution video sequences in the same dynamic scene. It includes two aspects: incremental super-resolution in time series and video super-resolution in space. The development of deep learning algorithm technology in recent years has greatly improved incremental super-resolution technology in time series and video super-resolution technology in space, such as a video interpolation method, device, electronic device and storage medium disclosed in a patent application with publication number CN111654746B; and a video super-resolution restoration method based on deep learning and adjacent frames disclosed in a patent application with publication number CN112365403A. However, traditional deep learning methods rarely achieve super-resolution in time and space at the same time.

[0003] In the field of video spatiotemporal super-resolution, some researchers have tried methods based on optical flow. However, methods based on optical flow usually model the connection between adjacent video frames with a preset spatiotemporal relationship, which cannot adapt to a variety of motion situations, and the resulting sequence is prone to jittery visual phenomena. For example, a video spatiotemporal super-resolution method and device based on improved deformable convolution correction is disclosed in a patent application with publication number CN113034380A. In fact, video data is related in space and time. Research on video spatiotemporal super-resolution technology that integrates spatiotemporal features can obtain more information in the video, thereby improving the effect of video spatiotemporal super-resolution. Summary of the invention

[0004] In order to solve at least one of the problems existing in the above technical background, the present invention provides a Transformer-based video spatiotemporal super-resolution method.

[0005] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0006] The Transformer-based video spatiotemporal super-resolution method includes the following steps:

[0007] S1, collect model training samples for video spatiotemporal super-resolution;

[0008] S2. Build a high frame rate and low resolution video sequence generation model, generate intermediate frames based on the temporal feature fusion of the Transformer attention mechanism, and generate high frame rate and low resolution video sequences;

[0009] S3. Build a high frame rate and high resolution video sequence generation model, and reconstruct high frame rate and high resolution video sequences based on the spatiotemporal feature fusion of the Transformer attention mechanism;

[0010] S4. Build and train a Transformer-based video spatiotemporal super-resolution model, and perform spatiotemporal super-resolution reconstruction on the video sequence according to the trained Transformer-based video spatiotemporal super-resolution model.

[0011] Further, in step S1, an original dataset of video sequences covering various scenes and actions is selected;

[0012] The video sequence I provided by the original dataset HR is regarded as a high-frame-rate and high-resolution video sequence; each video sequence in the original dataset is represented as T represents the length of the video sequence, that is, the number of frames in a video. Represents the i-th frame image in the video sequence; i=1,…,T;

[0013] The preprocessing of the video sequences in the original dataset is as follows:

[0014] First, we use the original high frame rate and high resolution video sequence I in the original data set HR The high frame rate and low resolution video sequence is obtained by r-fold downsampling through bicubic interpolation, and then a-fold downsampling is performed to obtain a low frame rate and low resolution video sequence to construct a training data set.

[0015] Furthermore, in step S2, in the constructed high frame rate low resolution video sequence generation model, generating intermediate frames based on the temporal feature fusion of the Transformer attention mechanism to generate a high frame rate low resolution video sequence includes the following steps:

[0016] S2.1. For the low frame rate and low resolution video sequence obtained by preprocessing, a low frame rate and low resolution initial feature sequence is calculated through convolution and residual modules, and an intermediate frame image is generated between two adjacent frame images in the low frame rate and low resolution video sequence. For the intermediate frame image to be generated, i.e., the t-th frame image, the feature of the previous frame image is recorded as F t-1 , the feature of the next frame image is recorded as F t+1 ;

[0017] S2.2. Based on the Transformer attention mechanism, the connection information of the adjacent frame image features of the intermediate frame image to be generated is obtained to obtain the feature connection information between the frames; for the features of the two adjacent frames of the intermediate frame image to be generated, the connection information of the adjacent frame image features includes the forward connection information R from the image features of the previous frame of the intermediate frame image to the image features of the next frame of the intermediate frame imaget-1→t+1 , and the reverse connection information R from the image features of the next frame of the intermediate frame image to the image features of the previous frame of the intermediate frame image t+1→t-1 ;

[0018] S2.3. Fusion of the previous frame image features F t-1 and positive contact information R t-1→t+1 , and obtain the forward intermediate frame feature F from front to back t-1→t ; The next frame image feature F fused with the intermediate frame image t+1 and reverse contact information R t+1→t-1 , and obtain the reverse intermediate frame feature F from back to front t+1→t ; The forward intermediate frame feature F t-1→t and reverse intermediate frame feature F t+1→t Fusion is performed by combining the forward intermediate frame feature F t-1→t and reverse intermediate frame feature F t+1→t The concatenation is then performed through convolution to obtain the final intermediate frame feature F middle , as follows:

[0019] F t-1→t =F t-1 +R t-1→t+1 ;

[0020] F t+1→t =F t+1 +R t+1→t-1 ;

[0021] F middle =Conv(Concate(F t-1→t ,F t+1→t ));

[0022] S2.4. The final intermediate frame feature F is generated middle The feature information F of the intermediate frame is obtained by feature learning through the RDN (Residual Dense Network) module. t , and then the feature information of the intermediate frame is converted into image information through convolution calculation to generate an intermediate frame image; the intermediate frame image and the low frame rate and low resolution video sequence obtained by preprocessing are spliced ​​into a high frame rate and low resolution video sequence in chronological order, that is, the intermediate frame image is inserted between the corresponding previous frame image and the next frame image.

[0023] Further, in step S2.2, the forward connection information R from the image features of the previous frame of the intermediate frame image to the image features of the next frame of the intermediate frame image is obtained. t-1→t+1 , and the reverse connection information R from the image features of the next frame of the intermediate frame image to the image features of the previous frame of the intermediate frame image t+1→t-1 , as follows:

[0024] The image feature F of the previous frame t-1 Perform a convolution operation to get the Transformer attention mechanism's 'query': query(q1);

[0025] The image feature F of the next frame t+1 Perform two convolution operations to obtain the Transformer attention mechanism 'key-value pair': key(k1) and value(v1);

[0026] The obtained 'query' q1, 'key-value pair' key(k1) and value(v1) are expanded in blocks respectively; the sizes of q1, k1 and v1 are all f channel ×patch L ×patch L , where f channel Indicates the number of feature channels;

[0027] For each patch of size L ×patch L The feature channel is divided into n blocks, each with a resolution of The original size is patch L ×patch L Each channel feature of can be expressed as n×patch×patch, since there are a total of f channel feature channels, which can eventually be converted into n dimensions of patch×patch×f channel Features;

[0028] The connection information of the features of adjacent frames is calculated through the Transformer attention mechanism. For each layer of Transformer, the specific process is as follows: the relationship between the query (q1) and the key (k1) is obtained through softmax calculation, the value (v1) is calculated in the form of weight distribution, and then the response association Respon of the previous frame image feature on the next frame image feature is obtained through convolution calculation. The specific mathematical expression is as follows:

[0029]

[0030] Respon=Conv(Attention(q1,k1,v1));

[0031] Among them, d k Represents the input feature dimension, i.e., patch×patch×f channel ;

[0032] After the M-layer Transformer calculation, the output of the Transformer is convolved once as the connection information R from the image features of the previous frame to the image features of the next frame. t-1→t+1 , as follows:

[0033] R t-1→t+1 =Conv(Response);

[0034] Use the same method to obtain the connection information R from the image features of the next frame to the image features of the previous frame t+1→t-1 .

[0035] Furthermore, in step S3, in the constructed high frame rate and high resolution video sequence generation model, the high frame rate and high resolution video sequence is reconstructed based on the spatiotemporal feature fusion of the Transformer attention mechanism, including the following steps:

[0036] S3.1, extracting the initial feature sequence of the video sequence: using convolution to extract the initial features of the intermediate frame image t obtained in step S2.4, and splicing the low frame rate and low resolution initial feature sequence generated in step S2.1 in time sequence, and performing convolution operation on the spliced ​​high frame rate and low resolution sequence features to obtain the final high frame rate and low resolution initial feature sequence F;

[0037] S3.2. Use the Transformer-based attention mechanism to learn the total spatiotemporal connection features R of the high frame rate low resolution initial feature sequence F all ;

[0038] S3.3. The total spatiotemporal connection feature R all The reconstruction feature F is obtained through the reconstruction module h , and increase the reconstruction feature F through convolution calculation h The number of channels is calculated and the reconstructed feature F is converted into h The deepened number of channels is converted into the reconstructed feature F h The width of F, i.e. the enlarged reconstruction feature h The feature resolution is magnified r times to obtain the high-resolution features after upsampling;

[0039] S3.4, convert the high-resolution features into image information through convolution operation, and perform bilinear interpolation and amplification on the high frame rate and low resolution video sequence obtained in S2.4; add the two together to obtain the final high frame rate and high resolution video sequence.

[0040] Furthermore, in step S3.2, the total spatiotemporal connection feature R of the high frame rate and low resolution initial feature sequence F is learned using the Transformer attention mechanism all , specifically including the following steps:

[0041] S3.2.1. Divide the high-frame-rate and low-resolution initial feature sequence F into small-scale feature blocks, and use the Transformer-based attention mechanism to extract the spatiotemporal connection features R of the small-scale feature blocks. small ;

[0042] S3.2.2. Divide the high-frame-rate and low-resolution initial feature sequence F into large-scale feature blocks, and use the Transformer-based attention mechanism to extract the temporal connection features R of the large-scale feature blocks. large ;

[0043] S3.2.3. Connecting the spatiotemporal domain features R small and the time domain contact feature R large Add together to get the total spatiotemporal connection feature R all .

[0044] Furthermore, step S3.2.1 is specifically as follows:

[0045] Perform position encoding and convolution calculation on the high frame rate and low resolution initial feature sequence F obtained in step S3.1 to obtain the "query" q (query) and "key-value pairs" k and v (key-value) of the Transformer attention mechanism; divide each feature channel of q, k, v into n s feature blocks; for each feature block, the Transformer attention mechanism is used to calculate the n features of the feature block and the same frame image feature. s -1 feature block, and the relationship between the feature block and other frame image features s The result of Transformer output is convolved to obtain the spatiotemporal connection feature R of the small-scale feature block small .

[0046] Furthermore, step S3.2.2 is specifically as follows:

[0047] Use the high-frame-rate, low-resolution initial feature sequence F obtained in step S3.1 to perform position encoding and convolution calculations to obtain the Transformer attention mechanism's 'query' q (query) and 'key-value pairs' k and v (key-value); divide each feature channel of q, k, v into n L feature blocks; for each feature block, the Transformer attention mechanism is used to calculate the temporal connection between the current feature block and the feature blocks at the same position as the feature block in different frame image features of the video sequence; the result of the Transformer output is convolved to obtain the large-scale feature block temporal connection feature R large .

[0048] Further, step S4 includes the following steps:

[0049] S4.1. Construct a Transformer-based video spatiotemporal super-resolution model, including a high frame rate low-resolution video sequence generation model and a high frame rate high-resolution video sequence generation model. The high frame rate low-resolution video sequence generation model receives a low frame rate low-resolution video sequence and outputs a high frame rate low-resolution video sequence to the high frame rate high-resolution video sequence generation model. The high frame rate high-resolution video sequence generation model outputs a high frame rate high-resolution video sequence to complete the video spatiotemporal super-resolution reconstruction;

[0050] S4.2. Use Charbonnier Loss to construct the loss function L of the Transformer-based video spatiotemporal super-resolution reconstruction model charbonnier , as follows:

[0051]

[0052] in, represents the tth frame of the high frame rate and high resolution video sequence in the original dataset, represents the tth frame of a high-frame-rate and high-resolution video sequence generated by the Transformer-based video spatiotemporal super-resolution model; ε is a constant used to stabilize the training of the Transformer-based video spatiotemporal super-resolution model;

[0053] S4.3. Train the Transformer-based video spatiotemporal super-resolution model according to the original data set and the training data set, evaluate the Transformer-based video spatiotemporal super-resolution model by peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and obtain the trained Transformer-based video spatiotemporal super-resolution model.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1) Transformer-based video spatiotemporal super-resolution reconstruction. Compared with the previous feature extractor RNN of sequence data, Transformer can process the input sequence data in parallel and will not cause the problem of gradient vanishing when the model is too deep; it can also effectively learn the connections between different areas of the image and different frames of the video sequence.

[0056] 2) The present invention combines the Transformer attention mechanism to obtain the spatiotemporal connections in video sequences, and realizes super-resolution enhancement of videos in time and space. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a Transformer-based video spatiotemporal super-resolution method in one embodiment of the present invention.

[0058] Figure 2 It is the structure of a video interpolation processing method based on the time domain feature fusion of the Transformer attention mechanism in one embodiment of the present invention.

[0059] Figure 3 It is the structure of a video super-resolution enhancement method based on the spatiotemporal feature fusion of the Transformer attention mechanism in one embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below through specific implementation modes, but the implementation modes of the present invention are not limited thereto.

[0061] Embodiment 1:

[0062] Transformer-based video spatiotemporal super-resolution methods, such as Figure 1 As shown, the following steps are included:

[0063] S1, collect model training samples for video spatiotemporal super-resolution;

[0064] In this embodiment, the original dataset Vimeo90kTriplet, which covers video sequences of various scenes and actions, is selected;

[0065] The video sequence I provided by the original dataset HR is regarded as a high-frame-rate and high-resolution video sequence; each video sequence in the original dataset is represented as T represents the length of the video sequence, that is, the number of frames in a video. Represents the i-th frame image in the video sequence; i=1,…,T;

[0066] The preprocessing of the video sequences in the original dataset is as follows:

[0067] First, we use the original high frame rate and high resolution video sequence I in the original data set HR The high frame rate and low resolution video sequence is obtained by downsampling by r times through bicubic interpolation, and then downsampling by a times is obtained to obtain a low frame rate and low resolution video sequence, and a training data set is constructed; in this embodiment, r=4, a=2; the training data set is in the form of a sequence consisting of two frames of low resolution images The goal is to generate a sequence of three high-resolution images

[0068] S2. Build a high frame rate and low resolution video sequence generation model, generate intermediate frames based on the temporal feature fusion of the Transformer attention mechanism, and generate high frame rate and low resolution video sequences, including the following steps:

[0069] S2.1. For the low frame rate and low resolution video sequence obtained by preprocessing, a low frame rate and low resolution initial feature sequence is calculated through convolution and residual modules, and an intermediate frame image is generated between two adjacent frame images in the low frame rate and low resolution video sequence. For the intermediate frame image to be generated, i.e., the t-th frame image, the feature of the previous frame image is recorded as F t-1 , the feature of the next frame image is recorded as F t+1 ; In this embodiment, the image size of the low frame rate and low resolution video sequence is 64×64, the number of residual modules is 5, and t is 2, that is, the first frame and the third frame generate the second frame.

[0070] S2.2. Based on the Transformer attention mechanism, the connection information of the adjacent frame image features of the intermediate frame image to be generated is obtained to obtain the feature connection information between the frames; for the features of the two adjacent frames of the intermediate frame image to be generated, the connection information of the adjacent frame image features includes the forward connection information R from the image features of the previous frame of the intermediate frame image to the image features of the next frame of the intermediate frame image 1→3 , and the reverse connection information R from the image features of the next frame of the intermediate frame image to the image features of the previous frame of the intermediate frame image 3→1 ;

[0071] Get the positive connection information R from the image features of the previous frame of the intermediate frame image to the image features of the next frame of the intermediate frame image 1→3 , and the reverse connection information R from the image features of the next frame of the intermediate frame image to the image features of the previous frame of the intermediate frame image 3→1 , as follows:

[0072] Perform a convolution operation on the image feature F1 of the previous frame to obtain the "query" of the Transformer attention mechanism: query(q1);

[0073] Perform two convolution operations on the image feature F3 of the next frame to obtain the Transformer attention mechanism 'key-value pair': key(k1) and value(v1);

[0074] The obtained 'query' q1, 'key-value pair' key(k1) and value(v1) are expanded in blocks respectively; the sizes of q1, k1 and v1 are all f channel ×patch L ×patch L , where f channelrepresents the number of feature channels; in this embodiment, f channel For 64, patch L is 64.

[0075] For each patch of size L ×patch L The feature channel is divided into n blocks, each with a resolution of The original size is patch L ×patch L Each channel feature of can be expressed as n×patch×patch, since there are a total of f channel feature channels, which can eventually be converted into n dimensions of patch×patch×f channel features; in this embodiment, patch is 8 and n is 64.

[0076] The connection information of the features of adjacent frames is calculated through the Transformer attention mechanism. For each layer of Transformer, the specific process is as follows: the relationship between the query (q1) and the key (k1) is obtained through softmax calculation, the value (v1) is calculated in the form of weight distribution, and then the response association Respon of the previous frame image feature on the next frame image feature is obtained through convolution calculation. The specific mathematical expression is as follows:

[0077]

[0078] Respon=Conv(Attention(q1,k1,v1));

[0079] Among them, d k Represents the input feature dimension, i.e., patch×patch×f channel ;

[0080] After the M-layer Transformer calculation, the output of the Transformer is convolved once as the connection information R from the image features of the previous frame to the image features of the next frame. 1→3 , as follows:

[0081] R 1→3 =Conv(Response);

[0082] Use the same method to obtain the connection information R from the image features of the next frame to the image features of the previous frame 3→1 .

[0083] S2.3, in this embodiment, the previous frame image feature F1 and the positive contact information R of the intermediate frame image are fused. 1→3 , and obtain the forward intermediate frame feature F from front to back1→2 ; Fusion of the next frame image feature F3 and the reverse contact information R of the intermediate frame image 3→1 , and obtain the reverse intermediate frame feature F from back to front 2→1 ; The forward intermediate frame feature F 1→2 and reverse intermediate frame feature F 2→1 Fusion is performed by combining the forward intermediate frame feature F 1→2 and reverse intermediate frame feature F 2→1 The concatenation is then performed through convolution to obtain the final intermediate frame feature F2, as follows:

[0084] F 1→2 =F1+R 1→3 ;

[0085] F 2→1 =F3+R 3→1 ;

[0086] F2=Conv(Concate(F 1→2 ,F 2→1 ));

[0087] S2.4. The final intermediate frame feature F2 is learned through the RDN module, and then the feature information of the intermediate frame is converted into image information through convolution calculation to generate the intermediate frame image. Intermediate frame image The low frame rate and low resolution video sequences obtained by preprocessing are spliced ​​into high frame rate and low resolution video sequences in chronological order. Intermediate frame image Insert the corresponding previous frame image and the next frame between.

[0088] S3. Build a high frame rate and high resolution video sequence generation model, based on the spatiotemporal feature fusion of the Transformer attention mechanism, and reconstruct high frame rate and high resolution video sequences, including the following steps:

[0089] S3.1, extracting the initial feature sequence of the video sequence: using convolution to extract the initial features of the intermediate frame image t obtained in step S2.4, and splicing the low frame rate and low resolution initial feature sequence generated in step S2.1 in time sequence, and performing convolution operation on the spliced ​​high frame rate and low resolution sequence features to obtain the final high frame rate and low resolution initial feature sequence F;

[0090] S3.2. Use the Transformer-based attention mechanism to learn the total spatiotemporal connection features R of the high frame rate low resolution initial feature sequence F all , specifically including the following steps:

[0091] S3.2.1. Divide the high-frame-rate and low-resolution initial feature sequence F into small-scale feature blocks, and use the Transformer-based attention mechanism to extract the spatiotemporal connection features R of the small-scale feature blocks. small , as follows:

[0092] Perform position encoding and convolution calculation on the high frame rate and low resolution initial feature sequence F obtained in step S3.1 to obtain the "query" q (query) and "key-value pairs" k and v (key-value) of the Transformer attention mechanism; divide each feature channel of q, k, v into n s feature blocks; for each feature block, the Transformer attention mechanism is used to calculate the n features of the feature block and the same frame image feature. s -1 feature block, and the relationship between the feature block and the features of other frame images s The result of Transformer output is convolved to obtain the spatiotemporal connection feature R of the small-scale feature block small .

[0093] S3.2.2. Divide the high-frame-rate and low-resolution initial feature sequence F into large-scale feature blocks, and use the Transformer-based attention mechanism to extract the temporal connection features R of the large-scale feature blocks. large , as follows:

[0094] Use the high frame rate and low resolution initial feature sequence F obtained in step S3.1 to perform position encoding and convolution calculation to obtain the "query" q (query) and "key-value pairs" k and v (key-value) of the Transformer attention mechanism; divide each feature channel of q, k, v into n L feature blocks; for each feature block, the Transformer attention mechanism is used to calculate the temporal connection between the current feature block and the feature blocks at the same position as the feature block in different frame image features of the video sequence; the result of the Transformer output is convolved to obtain the large-scale feature block temporal connection feature R large .

[0095] S3.2.3. Connecting the spatiotemporal domain features R small and the time domain contact feature R large Add together to get the total spatiotemporal connection feature R all .

[0096] S3.3. The total spatiotemporal connection feature R all The reconstruction feature F is obtained through the reconstruction module h , and increase the reconstruction feature F through convolution calculationh The number of channels is calculated and the reconstructed feature F is converted into h The deepened number of channels is converted into the reconstructed feature F h The width of F, i.e. the enlarged reconstruction feature h The feature resolution is magnified r times to obtain the up-sampled high-resolution features; in this embodiment, the reconstruction module includes 30 residual modules Resblock, and r is 4.

[0097] S3.4, convert the high-resolution features into image information through convolution operation, and perform bilinear interpolation and amplification on the high frame rate and low resolution video sequence obtained in S2.4; add the two together to obtain the final high frame rate and high resolution video sequence

[0098] S4. Build and train a Transformer-based video spatiotemporal super-resolution model, including the following steps:

[0099] S4.1. Construct a Transformer-based video spatiotemporal super-resolution model, including a high frame rate low-resolution video sequence generation model and a high frame rate high-resolution video sequence generation model. The high frame rate low-resolution video sequence generation model receives a low frame rate low-resolution video sequence and outputs a high frame rate low-resolution video sequence to the high frame rate high-resolution video sequence generation model. The high frame rate high-resolution video sequence generation model outputs a high frame rate high-resolution video sequence to complete the video spatiotemporal super-resolution reconstruction;

[0100] S4.2. Use Charbonnier Loss to construct the loss function L of the Transformer-based video spatiotemporal super-resolution reconstruction model charbonnier , as follows:

[0101]

[0102] in, represents the tth frame of the high frame rate and high resolution video sequence in the original dataset, represents the tth frame of a high frame rate and high resolution video sequence generated by the Transformer-based video spatiotemporal super-resolution model; ε is a constant used to stabilize the training of the Transformer-based video spatiotemporal super-resolution model; in this embodiment, ε is 1e-12.

[0103] S4.3. Train the Transformer-based video spatiotemporal super-resolution model according to the original data set and the training data set, evaluate the Transformer-based video spatiotemporal super-resolution model by peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and obtain a trained Transformer-based video spatiotemporal super-resolution model.

[0104] The video sequence is reconstructed with video spatiotemporal super-resolution according to the trained Transformer-based video spatiotemporal super-resolution model.

[0105] This embodiment generates intermediate frames based on Transformer-based time domain feature fusion on the Vimeo90K Triplet test set to generate a high frame rate video sequence.

[0106] This embodiment is mainly compared with the previous interpolation method, mainly to verify the effect of the generated intermediate frame image. Therefore, we use low frame rate and high resolution video sequence input, that is, no spatial downsampling operation is performed on the image. In the specific implementation, our input image size is 3×128×128, and S2.1 uses a 6-layer residual module to extract shallow features, and S2.2 uses an 8-layer Transformer attention mechanism to extract inter-frame information. The experimental comparison results are shown in Table 1. Among them, TOFlow comes from reference 1 (see: Xue T, Chen B, Wu J, et al. Video enhancement with task-oriented flow [J]. International Journal of Computer Vision, 2019, 127 (8): 1106-1125.), DAIN comes from reference 2 (see: Bao W, Lai WS, Ma C, et al. Depth-aware video frame interpolation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 3703-3712.) CAIN comes from reference 3 (see: Choi M, Kim H, Han B, et al. Channel attention is all you need for video frame interpolation [C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34 (07): 10663-10671.), and AdaCoF comes from reference 4 (see: Lee H,Kim T,Chung T,et al.Adacof:Adaptivecollaboration of flows for video frame interpolation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:5316-5325.)

[0107] Table 1

[0108] Method <![CDATA[TOFlow [1] ]]> <![CDATA[DAIN [2] ]]> <![CDATA[CAIN [3] ]]> <![CDATA[AdaCoF [4] ]]> The present invention PSNR 33.73 34.71 34.65 34.27 34.824 SSIM 0.968 0.976 0.973 0.971 0.974

[0109] As can be seen from the table, the method of generating intermediate frames and high frame rate video sequences based on the temporal feature fusion of the Transformer attention mechanism of the present invention achieves the best effect in PSNR and also achieves good results in SSIM performance. This shows that the present invention can generate high-quality intermediate frame images.

[0110] Example 2

[0111] This example generates intermediate frames based on the temporal feature fusion of the Transformer attention mechanism on the Vimeo90K Triplet test set, generating high frame rate low resolution video sequences. The experimental comparison results are shown in Table 2. STAR is from Reference 5 (see: Haris M, Shakhnarovich G, Ukita N. Space-Time-Aware Multi-Resolution Video Enhancement[C] / / 2020IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2020)

[0112] Table 2

[0113] Method <![CDATA[TOFlow [1] ]]> <![CDATA[DAIN [2] ]]> <![CDATA[STAR [5] ]]> The present invention PSNR 36.04 36.69 39.13 39.32 SSIM 0.984 0.986 0.991 0.988

[0114] As can be seen from the table, the method of the present invention for generating intermediate frames based on the fusion of time domain features of Transformer and generating high frame rate and low resolution video sequences has a PSNR that is 0.19 dB higher than that of STAR and a SSIM that is 0.003 lower than that of the previous methods, indicating that the present invention can generate high-quality intermediate frame images and reconstruct high frame rate and low resolution video sequences.

[0115] Example 3

[0116] This embodiment performs Transformer-based video spatiotemporal super-resolution on the Vimeo-Triplet test set of the Vimeo-90K dataset. The input of this embodiment is a low-frame rate and low-resolution image sequence consisting of the first frame and the third frame, and the output is a high-frame rate and high-resolution video sequence consisting of the first frame, the second frame, and the third frame. The experimental comparison results are shown in Table 3, where VSR represents the super-resolution results of the first frame and the third frame, VFI represents the result of the second frame after interpolation and amplification, and AVE represents the average PSNR result of these three frames. Among them, STAR comes from document 5 (see: Haris M Shakhnarovich G, Ukita N. Space-Time-Aware Multi-Resolution Video Enhancement [C] / / 2020 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020).

[0117]

[0118] As can be seen from the table, the video spatiotemporal super-resolution based on Transformer of the present invention has good performance in video super-resolution and video interpolation, among which the PSNR of the second frame after interpolation and amplification is 0.003dB higher than that of STARNet, and the PSNR of the first and third frames after amplification is 0.097dB higher than that of STARNet, and the overall PSNR result is 0.066dB higher than that of STARNet. This shows that the present invention can generate high-quality reconstructed high-frame rate and high-resolution video sequences.

[0119] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. Transformer-based video spatiotemporal super-resolution method, characterized by: The following steps are involved: S1, collect model training samples for video spatiotemporal super-resolution; S2. Build a high frame rate and low resolution video sequence generation model, generate high frame rate and low resolution video sequences based on the intermediate frames generated by the temporal feature fusion of the Transformer attention mechanism, including the following steps: S2.

1. For the low frame rate and low resolution video sequence obtained by preprocessing, a low frame rate and low resolution initial feature sequence is calculated through convolution and residual modules, and an intermediate frame image is generated between two adjacent frame images in the low frame rate and low resolution video sequence. For the intermediate frame image to be generated, i.e., the t-th frame image, the feature of the previous frame image is recorded as F t-1 , the feature of the next frame image is recorded as F t+1 ; S2.

2. Based on the Transformer attention mechanism, the connection information of the adjacent frame image features of the intermediate frame image to be generated is obtained to obtain the feature connection information between the frames; for the features of the two adjacent frames of the intermediate frame image to be generated, the connection information of the adjacent frame image features includes the forward connection information R from the image features of the previous frame of the intermediate frame image to the image features of the next frame of the intermediate frame image t-1→t+1 , and the reverse connection information R from the image features of the next frame of the intermediate frame image to the image features of the previous frame of the intermediate frame image t+1→t-1 ; Get the positive connection information R from the image features of the previous frame of the intermediate frame image to the image features of the next frame of the intermediate frame image t-1→t+1 , and the reverse connection information R from the image features of the next frame of the intermediate frame image to the image features of the previous frame of the intermediate frame image t+1→t-1 , as follows: The image feature F of the previous frame t-1 Perform convolution operation to get the query of Transformer attention mechanism: query(q1); The image feature F of the next frame t+1 Perform two convolution operations to obtain the Transformer attention mechanism key-value pairs: key(k1) and value(v1); The obtained query query(q1), key(k1) and value(v1) are expanded in blocks respectively; the sizes of q1, k1 and v1 are all f channel ×patch L ×patch L , where f channel Indicates the number of feature channels; For each patch of size L ×patch L The feature channel is divided into n blocks, each with a resolution of patch×patch. The original size is patch L ×patch L Each channel feature of can be expressed as n×patch×patch, since there are a total of f channel feature channels, which can eventually be converted into n dimensions of patch×patch×f channel Features; The connection information of the features of adjacent frames is calculated through the Transformer attention mechanism. The specific process for each layer of Transformer is as follows: the relationship between the query query (q1) and the key (k1) is obtained through softmax calculation, the value (v1) is calculated in the form of weight distribution, and then the response association Respon of the previous frame image feature on the next frame image feature is obtained through convolution calculation. The specific mathematical expression is as follows: Respon=Conv(Attention(q1,k1,v1)); Among them, d k Represents the feature dimension of the input, that is, patch×patch×f channel ; After the M-layer Transformer calculation, the output of the Transformer is convolved once as the connection information R from the image features of the previous frame to the image features of the next frame. t-1→t+1 , as follows: R t-1→t+1 =Conv(Respon); Use the same method to obtain the connection information R from the image features of the next frame to the image features of the previous frame t+1→t-1 ; S2.

3. Fusion of the previous frame image features F t-1 and positive contact information R t-1→t+1 , and obtain the forward intermediate frame feature F from front to back t-1→t ; The next frame image feature F fused with the intermediate frame image t+1 and reverse contact information R t+1→t-1 , and obtain the reverse intermediate frame feature F from back to front t+1→t ; The forward intermediate frame feature F t-1→t and reverse intermediate frame feature F t+1→t Fusion is performed by combining the forward intermediate frame feature F t-1→t and reverse intermediate frame feature F t+1→t The concatenation is then performed through convolution to obtain the final intermediate frame feature F middle , as follows: F t-1→t =F t-1 +R t-1→t+1 ; F t+1→t =F t+1 +R t+1→t-1 ; F middle =Conv(Concate(F t-1→t ,F t+1→t )); S2.

4. The final intermediate frame feature F is generated middle The feature information F of the intermediate frame is obtained by feature learning through the RDN module. t , and then convert the feature information of the intermediate frame into image information through convolution calculation to generate an intermediate frame image; the intermediate frame image and the low frame rate and low resolution video sequence obtained by preprocessing are spliced ​​into a high frame rate and low resolution video sequence in chronological order, that is, the intermediate frame image is inserted between the corresponding previous frame image and the next frame image; S3. Build a high frame rate and high resolution video sequence generation model, based on the spatiotemporal feature fusion of the Transformer attention mechanism, and reconstruct high frame rate and high resolution video sequences, including the following steps: S3.1, extracting the initial feature sequence of the video sequence: using convolution to extract the initial features of the intermediate frame image t obtained in step S2.4, and splicing the low frame rate and low resolution initial feature sequence generated in step S2.1 in time sequence, and performing convolution operation on the spliced ​​high frame rate and low resolution sequence features to obtain the final high frame rate and low resolution initial feature sequence F; S3.

2. Use the Transformer-based attention mechanism to learn the total spatiotemporal connection features R of the high frame rate low resolution initial feature sequence F all ; S3.

3. The total spatiotemporal connection feature R all The reconstruction feature F is obtained through the reconstruction module h , and increase the reconstruction feature F through convolution calculation h The number of channels is calculated and the reconstructed feature F is converted into h The deepened number of channels is converted into the reconstructed feature F h The width of F, i.e. the enlarged reconstruction feature h The feature resolution is magnified r times to obtain the high-resolution features after upsampling; S3.4, converting the high-resolution features into image information through convolution operation, and performing bilinear interpolation amplification on the high-frame-rate and low-resolution video sequence obtained in S2.4; adding the two together to obtain the final high-frame-rate and high-resolution video sequence; S4. Construct and train a Transformer-based video spatiotemporal super-resolution model, and perform spatiotemporal super-resolution reconstruction on the video sequence according to the trained Transformer-based video spatiotemporal super-resolution model, including the following steps: S4.

1. Construct a Transformer-based video spatiotemporal super-resolution model, including a high frame rate low-resolution video sequence generation model and a high frame rate high-resolution video sequence generation model; the high frame rate low-resolution video sequence generation model receives a low frame rate low-resolution video sequence and outputs a high frame rate low-resolution video sequence to a high frame rate high-resolution video sequence generation model, and the high frame rate high-resolution video sequence generation model outputs a high frame rate high-resolution video sequence to complete video spatiotemporal super-resolution reconstruction; S4.

2. Use Charbonnier Loss to construct the loss function L of the Transformer-based video spatiotemporal super-resolution reconstruction model charbonnier , as follows: in, represents the tth frame of the high frame rate and high resolution video sequence in the original dataset, represents the tth frame of a high-frame-rate and high-resolution video sequence generated by the Transformer-based video spatiotemporal super-resolution model; ε is a constant used to stabilize the training of the Transformer-based video spatiotemporal super-resolution model; S4.

3. Train the Transformer-based video spatiotemporal super-resolution model according to the original data set and the training data set, evaluate the Transformer-based video spatiotemporal super-resolution model by peak signal-to-noise ratio and structural similarity index, and obtain a trained Transformer-based video spatiotemporal super-resolution model.

2. The Transformer-based video spatiotemporal super-resolution method according to claim 1, characterized in that: In step S1, an original dataset of video sequences covering various scenes and actions is selected; The video sequence I provided by the original dataset HR Treated as a high frame rate and high resolution video sequence; Each video sequence in the original dataset is represented as T represents the length of the video sequence, that is, the number of frames in a video. Represents the i-th frame image in the video sequence; i=1,…,T; The preprocessing of the video sequences in the original dataset is as follows: First, we use the original high frame rate and high resolution video sequence I in the original data set HR The high frame rate and low resolution video sequence is obtained by r-fold downsampling through bicubic interpolation, and then a-fold downsampling is performed to obtain a low frame rate and low resolution video sequence to construct a training data set.

3. The Transformer-based video spatiotemporal super-resolution method according to claim 1, characterized in that: In step S3.2, the total spatiotemporal connection feature R of the high frame rate low resolution initial feature sequence F is learned using the Transformer attention mechanism all , specifically including the following steps: S3.2.

1. Divide the high-frame-rate and low-resolution initial feature sequence F into small-scale feature blocks, and use the Transformer-based attention mechanism to extract the spatiotemporal connection features R of the small-scale feature blocks. small ; S3.2.

2. Divide the high-frame-rate and low-resolution initial feature sequence F into large-scale feature blocks, and use the Transformer-based attention mechanism to extract the temporal connection features R of the large-scale feature blocks. large ; S3.2.

3. Connecting the spatiotemporal domain features R small and the time domain contact feature R large Add together to get the total spatiotemporal connection feature R all .

4. The Transformer-based video spatiotemporal super-resolution method according to claim 1, characterized in that: Step S3.2.1 is as follows: Perform position encoding and convolution calculation on the high frame rate and low resolution initial feature sequence F obtained in step S3.1 to obtain the query q (query) and key-value pairs k and v (key-value) of the Transformer attention mechanism; divide each feature channel of q, k, v into n s feature blocks; for each feature block, the Transformer attention mechanism is used to calculate the n features of the feature block and the same frame image feature. s -1 feature block, and the relationship between the feature block and other frame image features s The result of Transformer output is convolved to obtain the spatiotemporal connection feature R of the small-scale feature block small .

5. The Transformer-based video spatiotemporal super-resolution method according to claim 1, characterized in that: Step S3.2.2 is as follows: Use the high-frame-rate, low-resolution initial feature sequence F obtained in step S3.1 to perform position encoding and convolution calculations to obtain the query q (query) and key-value pairs k and v (key-value) of the Transformer attention mechanism; divide each feature channel of q, k, v into n L feature blocks; for each feature block, the Transformer attention mechanism is used to calculate the temporal connection between the current feature block and the feature blocks at the same position as the feature block in different frame image features of the video sequence; the result of the Transformer output is convolved to obtain the large-scale feature block temporal connection feature R large .

6. A computer-readable medium having data and a computer program stored thereon, characterized in that: The medium stores image data and can be run as a program, which executes the Transformer-based video spatiotemporal super-resolution method described in any one of claims 1 to 5.

Images