A multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment

Abstract

The present invention discloses a multi-scale modulation video super-resolution reconstruction method based on deformable transformer alignment, which relates to the field of computer vision. The method includes: obtaining a shallow feature map sequence of a video; using a multi-scale modulation network based on deformable transformer alignment to process the shallow feature map sequence; the multi-scale modulation network based on deformable transformer alignment includes: a deformable transformer alignment module, a multi-scale modulation module, a Swin Transformer hybrid module and a reconstruction module; the deformable transformer alignment module is used for feature alignment; the multi-scale modulation module is used for multi-scale feature fusion; the Swin Transformer hybrid module captures spatial features of the modulated feature map; the reconstruction module is used to reconstruct a high-resolution image. The present invention proposes a multi-scale modulation network based on deformable transformer alignment, which combines the deformable transformer alignment module with the multi-scale modulation module, while ensuring the reconstruction performance, using relatively few network parameters to improve the computational efficiency of the network, and balance the complexity and performance of the network.

Description

Technical Field

[0001] The present invention relates to the field of video super-resolution in computer vision, and in particular to a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment. Background Art

[0002] With the rapid growth of internet video data, users' demand for higher video quality continues to rise. However, due to bandwidth limitations, videos are often downsampled and compressed, resulting in a degradation in quality. Consequently, video super-resolution (VSR) has emerged as a video enhancement technology. It aims to increase video resolution, thereby enhancing clarity and detail. This improves the quality of low-resolution videos resulting from quality loss during compression and transmission, or from the original low-resolution recording. Traditional video super-resolution methods primarily rely on algorithms based on image processing and statistical inference, which were the mainstream before the advent of deep learning. These include interpolation-based methods such as bilinear interpolation, bicubic interpolation, and nearest neighbor interpolation; sparse representation-based methods such as dictionary learning; and regularization-based methods such as maximum a posteriori estimation and total variation regularization. While these traditional methods have achieved some success in video super-resolution, they also suffer from inherent drawbacks such as insufficient reconstructed detail, poor adaptability to complex motion scenes, and high computational complexity.

[0003] In the existing technology, mature VSR network models usually include an alignment module to compensate for the motion of objects between frames. The alignment module is crucial for CNN-based VSR networks because the local inductive bias of CNN only allows them to effectively utilize spatially densely distributed information. At the same time, the latest progress in CNN-based VSR has shown better performance by adding techniques such as residual learning, dense connections, and deformable convolutions. PSRT proposes a method for aligning image patches. It crops the image into non-overlapping patches with small self-attention window sizes, and then aligns these patches. It locates the motion of objects based on optical flow, and uses nearest neighbor sampling to align video frames to corresponding positions in the reference frame. Finally, these aligned video frame sequences are input into the network of the transformer architecture to obtain higher resolution images.

[0004] However, the limited receptive field of CNN-based VSR networks can only utilize tightly distributed spatial information and cannot handle large, unaligned, and wide-ranging motions. The transformer introduced into the VSR task retains a complex alignment module and can only process low-frequency information, significantly increasing the computational burden and failing to effectively capture multi-scale image features, resulting in less-than-ideal reconstruction results. While the latest existing video super-resolution reconstruction methods achieve relatively ideal reconstruction results, they increase the number of network parameters, increasing network complexity and increasing the computational burden.

[0005] Therefore, when performing video super-resolution reconstruction, how to ensure that the reconstructed image has high resolution while using relatively few network parameters to improve the computational efficiency of video super-resolution reconstruction and achieve a balance between network complexity and performance is a problem that needs to be solved in existing technologies. Summary of the Invention

[0006] Based on this, it is necessary to provide a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment to address the above technical problems.

[0007] The present invention provides a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment, comprising:

[0008] A multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment includes the following steps:

[0009] Obtain a low-resolution video frame sequence from a video source, and extract a shallow feature map sequence from multiple low-resolution video frame sequences;

[0010] Construct a multi-scale modulation network based on deformable transformer alignment, including a deformable transformer alignment module, a multi-scale modulation module, a SwinTransformer hybrid module, and a reconstruction module;

[0011] Input the shallow feature map sequence into the multi-scale modulation network based on deformable transformer alignment;

[0012] Through the deformable transformer alignment module, optical flow is used to pre-align the features of the support frame. The offset generation network is used to generate the offset between the pre-aligned features and the support frame to capture the relevant key areas of the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame, aggregate the key features of the key areas, and obtain the aligned feature map sequence;

[0013] Through the multi-scale modulation module, temporal correlation information is constructed in the linear layer and the input channel of the multi-scale modulation module is evenly divided into multiple heads. Different 3D deep convolutions are used to expand the receptive field of each head. Temporal correlation information is introduced to fuse features of different scales to obtain a modulation feature map.

[0014] The Swin Transformer hybrid module is used to capture the spatial features of the modulation feature map and obtain the detailed texture information of the modulation feature map;

[0015] Through the reconstruction module, the modulation feature map and its detailed texture information are reconstructed using the sub-pixel convolution PixelShuffle upsampling method to obtain a high-resolution image.

[0016] Preferably, the acquisition of the shallow feature map sequence specifically includes:

[0017] Get low-resolution video frames from the video source;

[0018] Assume that any propagation channel j of the multi-scale modulation network based on deformable transformer alignment, then the video frame sequence can be expressed as Where, j = 1, 2, 3, ..., i = 1, 2, 3, ...;

[0019] Extracting shallow feature map sequences from video frame sequences

[0020] in, represents the support frame, i.e., the shallow feature map of the i-th time step in the j-th propagation branch, is the reference frame, Represents the shallow feature maps of the i-1 and i+1 time steps respectively is the shallow feature map of the i+1th time step, Respectively They are adjacent frames.

[0021] Preferably, the network structure of the multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment is a recursive bidirectional network structure based on the improved basic video super-resolution network Basic VSR++.

[0022] Preferably, it is characterized in that the acquisition of the alignment feature map sequence specifically includes:

[0023] The shallow feature sequence Input to the deformable transformer alignment module;

[0024] by is the reference frame, To support the frame, optical flow estimation is used to pre-align the reference frame and the support frame to obtain the pre-aligned features:

[0025]

[0026] in, They are two pre-aligned features of the support frame, W is the distortion operation, and flow i-1→i and Represents the optical flow from the i-1th frame to the i-th frame and from the i+1th frame to the i-th frame respectively;

[0027] The support frame is projected into the high-frequency space through the convolution layer, and the formula Calculate the high-frequency spatial features; where q ～ is the high-frequency spatial feature, W q is a 1×1 convolutional layer, To support frames;

[0028] The offset between the high-frequency spatial features and the pre-aligned features is calculated through the offset generation network. The calculation formula is:

[0029]

[0030]

[0031] Among them, θ offset represents the offset generation network, which consists of a depthwise convolution and a 3×3 convolution, and cat represents the concatenation operation;

[0032] The generated offset is added to the optical flow to obtain the residual fine-tuning of the optical flow to determine the sampling position of the deformation point. The spatial sampling deformation feature is calculated by using the sampling function of bilinear interpolation and the residual fine-tuning of the optical flow to the initial alignment feature. The calculation formula is:

[0033]

[0034] in, They are spatial sampling deformation features, is the sampling function of bilinear interpolation, flow' i-1→i and flow' i+1→i They are respectively the residual fine-tuning of the optical flow of the two reference frames;

[0035] The deformed keys and values are obtained by projection, and the deformed keys and values are queried using the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame and aggregate the key features to obtain the alignment features related to the support frame. The process can be formulated as follows:

[0036]

[0037] Among them, k ～ and v ～ are the keys and values of the spatial sampling deformation features, W k and W v The 1×1 convolutional layers are projections of the keys and values of the spatially sampled deformed features. are the alignment features of the support frames, softmax is the activation function, d is the dimension of the reference frame, B is the relative position bias, and R is q ～ and k ～ The relative displacement between Indicates relative position deviation;

[0038] Merge the support frame with its alignment features to obtain a sequence of aligned feature maps

[0039] Preferably, the offset between the high-frequency spatial feature and the pre-aligned feature reflects the offset required for further feature alignment.

[0040] Preferably, it is characterized in that the acquisition of the modulation characteristic map specifically includes:

[0041] The multi-scale modulation module receives a sequence of aligned feature maps As input, it is reshaped into linear features It builds temporal correlation information in the linear layer and then restores the linear features to a sequence of aligned feature maps;

[0042] Where H represents the image height, W represents the image width, and C is the number of channels;

[0043] The input channel of the multi-scale modulation module is converted into N ] = split(x) is evenly divided into N heads;

[0044] Among them, [x1,x2,…x N ] is the sequence of aligned feature maps of N heads, and split represents the average division operation;

[0045] For each head's aligned feature map sequence, perform 3D depth convolution operations with different receptive fields and aggregate them into feature sequences with different receptive fields. The formula is:

[0046]

[0047] Among them, M(x) represents a feature sequence with different receptive fields, Concat represents an aggregation operation, 3DW represents a 3D depth convolution, and k i ∈[x1,x2,…x N] means that the input features are divided into multiple heads in the channel dimension, i∈[3,5,…k], which indicates the number of receptive fields of each head convolution layer;

[0048] The features in the obtained feature sequences with different receptive fields are shuffled and grouped according to the head to obtain N groups of feature sequences with different receptive fields;

[0049] The feature fusion operation is performed on the features of all groups using deep convolutions of different scales to obtain multi-scale fusion features. The calculation expression is:

[0050]

[0051] in, is the feature of the i-th channel of the j-th group, G represents the sequence of aligned feature maps in the i-th channel of the j-th group. i represents the initial fusion feature, M represents the multi-scale fusion feature;

[0052] The obtained multi-scale fusion features are introduced into the linear layer related information to obtain the modulation feature map.

[0053] Preferably, the linear layer related information is constructed by a 3×3 3D depth convolution operation;

[0054] Preferably, the aligned feature map sequence for each head is subjected to 3D depth convolution operations with different receptive fields and aggregated, specifically including:

[0055] The receptive field of each head is as follows: the first head uses a 3×3 depth convolution kernel, the second head uses a 5×5 depth convolution kernel, and so on. The receptive field of each subsequent head gradually increases by 2 convolution kernels. A larger receptive field can be obtained by adjusting the number of heads;

[0056] Preferably, the method of shuffling and grouping the obtained multi-scale image features according to the heads is to randomly select a channel from each head to construct a group.

[0057] Preferably, obtaining the high-resolution image specifically includes:

[0058] The reconstruction module adopts the upsampling method of sub-pixel convolution PixelShuffle, including:

[0059] Obtain a modulation feature map with more channels through the convolution layer, where the number of channels is the square of the number of channels of the target high-resolution image;

[0060] Using sub-pixel convolution operations, channels are rearranged into pixels of a high-resolution image;

[0061] The resulting high-resolution image is a multiple of the low-resolution input image in size.

[0062] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0063] The present invention proposes a multi-scale modulation video super-resolution reconstruction method based on deformable transformer alignment. Low-resolution video frames are processed through a multi-scale modulation network (Multi-scale (Deformable Transformer Network, MSDT) of deformable transformer alignment. The deformable transformer alignment module dynamically samples useful information to align the reference frame of the input video frame sequence to the support frame. Compared with the CNN-based alignment method and the introduction of transformer in VSR tasks, it can capture more comprehensive and long-term useful information. The multi-scale modulation module can more effectively fuse features of different scales and refine the texture details of the image through multi-scale modulation of the alignment features while using relatively few parameters.

[0064] The video super-resolution reconstruction method provided by the present invention improves the computational efficiency of video super-resolution reconstruction by using fewer network parameters while ensuring the high resolution of the reconstructed image, thereby achieving a good balance between network complexity and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0066] Figure 1 A schematic diagram of the overall process of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0067] Figure 2 A network structure diagram of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0068] Figure 3 A structural diagram of a deformable transformer alignment module of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0069] Figure 4A structural diagram of a multi-scale modulation module of a multi-scale modulation video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0070] Figure 5 A comparison chart of the visualization effects of VSR (X4) on the REDS dataset of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention; DETAILED DESCRIPTION

[0071] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0072] At present, in the field of video super-resolution, although the existing technology has achieved a certain degree of improvement in the level of video super-resolution performance, how to ensure the high resolution of the reconstructed image while using fewer network parameters to improve the computational efficiency of video super-resolution reconstruction and achieve a balance between network complexity and performance is a problem that needs to be solved by the existing technology.

[0073] To solve the above technical problems, the present invention provides an embodiment of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment.

[0074] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0075] Figure 1 The figure is a flow chart of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment in the present invention, which specifically includes the following steps:

[0076] S101: Obtain a low-resolution video frame sequence from a video source, and extract a shallow feature map sequence from multiple low-resolution video frame sequences, including the following steps:

[0077] Get low-resolution video frames from the video source;

[0078] Assume that any propagation channel j of the multi-scale modulation network based on deformable transformer alignment, then the video frame sequence can be expressed as Where, j = 1, 2, 3, ..., i = 1, 2, 3, ...;

[0079] Extracting shallow feature map sequences from video frame sequences

[0080] in, represents the support frame, i.e., the shallow feature map of the i-th time step in the j-th propagation branch, is the reference frame, Represents the shallow feature maps of the i-1 and i+1 time steps respectively is the shallow feature map of the i+1th time step, Respectively They are adjacent frames.

[0081] S102: Construct a multi-scale modulation network based on deformable transformer alignment, including a deformable transformer alignment module, a multi-scale modulation module, a SwinTransformer hybrid module, and a reconstruction module.

[0082] S103: Input the shallow feature map sequence into the multi-scale modulation network based on deformable transformer alignment. Through the deformable transformer alignment module, optical flow is used to pre-align the features of the support frame. The offset generation network is used to generate the offset between the pre-aligned features and the support frame to capture the relevant key areas of the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame, aggregate the key features of the key areas, and obtain the aligned feature map sequence;

[0083] The deformable transformer alignment module is as follows Figure 3 As shown in the figure, only the alignment process from the i-1th frame to the i-th frame is shown. The alignment process from the i+1th frame to the i-th frame is the same as the alignment process from the i-1th frame to the i-th frame, including the following steps:

[0084] The shallow feature sequence Input to the deformable transformer alignment module;

[0085] by is the reference frame, To support the frame, optical flow estimation is used to pre-align the reference frame and the support frame to obtain the pre-aligned features:

[0086]

[0087] in, They are two pre-aligned features of the support frame, W is the distortion operation, and flow i-1→i and Represents the optical flow from the i-1th frame to the i-th frame and from the i+1th frame to the i-th frame respectively;

[0088] The support frame is projected into the high-frequency space through the convolution layer, and the formula Calculate the high-frequency spatial features; where q ～ is the high-frequency spatial feature, W q is a 1×1 convolutional layer, To support frames;

[0089] The offset between the high-frequency spatial features and the pre-aligned features is calculated through the offset generation network. The calculation formula is:

[0090]

[0091] Among them, θ offset represents the offset generation network, which consists of a depthwise convolution and a 3×3 convolution, and cat represents the concatenation operation;

[0092] The generated offset is added to the optical flow to obtain the residual fine-tuning of the optical flow to determine the sampling position of the deformation point. The spatial sampling deformation feature is calculated by using the sampling function of bilinear interpolation and the residual fine-tuning of the optical flow to the initial alignment feature. The calculation formula is:

[0093]

[0094] in, They are spatial sampling deformation features, is the sampling function of bilinear interpolation, flow' i-1→i and flow' i+1→i They are respectively the residual fine-tuning of the optical flow of the two reference frames;

[0095] The deformed keys and values are obtained by projection, and the deformed keys and values are queried using the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame and aggregate the key features to obtain the alignment features related to the support frame. The process can be formulated as follows:

[0096]

[0097] Among them, k ～ and v ～ are the keys and values of the spatial sampling deformation features, W k and W v The 1×1 convolutional layers are projections of the keys and values of the spatially sampled deformed features. are the alignment features of the support frames, softmax is the activation function, d is the dimension of the reference frame, B is the relative position bias, and R is q ～ and k ～ The relative displacement between Indicates relative position deviation;

[0098] Merge the support frame with its alignment features to obtain a sequence of aligned feature maps

[0099] S104: Using a multi-scale modulation module, constructing temporal correlation information in a linear layer and evenly dividing the input channels of the multi-scale modulation module into multiple heads, applying a different 3D deep convolution to each head to expand the receptive field, introducing temporal correlation information to perform feature fusion at different scales, and obtaining a modulation feature map;

[0100] The multi-scale modulation module in the multi-scale modulation network based on deformable transformer alignment is as follows Figure 4 As shown, the following steps are included:

[0101] The multi-scale modulation module receives a sequence of aligned feature maps As input, it is reshaped into linear features It builds temporal correlation information in the linear layer and then restores the linear features to a sequence of aligned feature maps;

[0102] Where H represents the image height, W represents the image width, and C is the number of channels;

[0103] The input channel of the multi-scale modulation module is converted into N ] = split(x) is evenly divided into N heads;

[0104] Among them, [x1,x2,…x N ] is the sequence of aligned feature maps of N heads, and split represents the average division operation;

[0105] For each head's aligned feature map sequence, perform 3D depth convolution operations with different receptive fields and aggregate them into feature sequences with different receptive fields. The formula is:

[0106]

[0107] Among them, M(x) represents a feature sequence with different receptive fields, Concat represents an aggregation operation, 3DW represents a 3D depth convolution, and k i ∈[x1,x2,…x N ] means that the input features are divided into multiple heads in the channel dimension, i∈[3,5,…k], which indicates the number of receptive fields of each head convolution layer;

[0108] The features in the obtained feature sequences with different receptive fields are shuffled and grouped according to the head to obtain N groups of feature sequences with different receptive fields;

[0109] The feature fusion operation is performed on the features of all groups using deep convolutions of different scales to obtain multi-scale fusion features. The calculation expression is:

[0110]

[0111] in, is the feature of the i-th channel of the j-th group, G represents the sequence of aligned feature maps in the i-th channel of the j-th group. i represents the initial fusion feature, M represents the multi-scale fusion feature;

[0112] The obtained multi-scale fusion features are introduced into the linear layer related information to obtain the modulation feature map.

[0113] S105: The modulation feature map is input into the hybrid module, the hybrid module fuses the spatiotemporal information of the modulation feature map, and uses the Swin Transformer-based image restoration module and its residual block to capture the correlation information of the spatial range of the modulation feature map.

[0114] S106: Inputting the modulation feature map into a reconstruction module, which reconstructs the modulation feature map by upsampling the modulation feature map using the temporal and spatial information and correlation information of the modulation feature map to obtain a high-resolution image, specifically including:

[0115] Obtain a modulation feature map with more channels through the convolution layer, where the number of channels is the square of the number of channels of the target high-resolution image;

[0116] Sub-pixel convolution operations are used to rearrange these channels into pixels of a high-resolution image;

[0117]

[0118] Among them, I H,W,C is the pixel value of the output image at position (H, W) with channel C, It is the modulation feature map at position (H, W), after r 2 The corresponding pixel value after channel grouping, represents c divided by r 2 The integer part of c is used to determine the channel to which c belongs, c mod r 2 represents c divided by r 2 The integer part of the quotient is used to determine the specific position of c;

[0119] The resulting high-resolution image is a multiple of the low-resolution input image in size.

[0120] This example uses the basicVSR framework to implement the training task, uses the Charbonnier loss as the training objective, and uses the Adam optimizer for training, where β1 = 0.9, β2 = 0.999, the initial learning rate is set to 2e-4, and a cosine annealing scheme is used to decay the learning rate to a minimum of 1e-7. Spynet is used to predict the optical flow in the video; the total number of iterations is set to 300k, the mini-batch size is 8, and the LR patch size is 64; the number of network channels is set to 64, the number of DTABs is set to 3, the number of attention heads is set to 4, the number of MSMBs is set to 4, and the head N is set to 4.

[0121] Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as evaluation metrics. REDS and Vimeo90K video frame sequences are used as training sets, and REDS4, Vimeo-90K-T, Vid4, UDM10, and Vimeo-900K video frame sequences are used as test sets.

[0122] The training set data is used as the network proposed in this invention to train a multi-scale modulated video super-resolution reconstruction model based on deformable transformer alignment;

[0123] Peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) were used as evaluation metrics;

[0124] The test set data was input into the model to obtain the video super-resolution results, which were compared with the results of the existing state-of-the-art technology. The comparison results are shown in the following table:

[0125] Table 1: Comparison of PSNR and SSIM for X4VSR task on REDS4, Vimeo-90K-T, Vid4 datasets

[0126]

[0127]

[0128] Comparing MSDT with several other methods, most methods achieved the best results in the VSR task. As can be seen from Table 1, MSDT achieved better results and achieved the best performance with the least number of parameters, which proves the beneficial effect of the method of the present invention and achieves a good balance between model complexity and performance.

[0129] like Figure 5 As shown in Figure 3, a visual comparison is provided between the two methods. It is obvious that the video frames reconstructed by MSDT achieve better visual effects.

[0130] In summary, through the verification of this embodiment, it is shown that the multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention improves the computational efficiency of video super-resolution reconstruction while ensuring the high resolution of the reconstructed image by using fewer network parameters, and achieves a good balance between network complexity and performance, thereby providing strong scientific guidance for the practical application of video super-resolution in the field of computer vision.

[0131] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. Any technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment, characterized in that: include: Obtain a low-resolution video frame sequence from a video source, and extract a shallow feature map sequence from multiple low-resolution video frame sequences; Construct a multi-scale modulation network based on deformable transformer alignment, including a deformable transformer alignment module, a multi-scale modulation module, a SwinTransformer hybrid module, and a reconstruction module; Input the shallow feature map sequence into the multi-scale modulation network based on deformable transformer alignment; Through the deformable transformer alignment module, optical flow is used to pre-align the features of the support frame. The offset generation network is used to generate the offset between the pre-aligned features and the support frame to capture the relevant key areas of the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame, aggregate the key features of the key areas, and obtain the aligned feature map sequence; Through the multi-scale modulation module, temporal correlation information is constructed in the linear layer and the input channel of the multi-scale modulation module is evenly divided into multiple heads. Different 3D deep convolutions are used to expand the receptive field of each head. Temporal correlation information is introduced to fuse features of different scales to obtain a modulation feature map. The Swin Transformer hybrid module is used to capture the spatial features of the modulation feature map and obtain the detailed texture information of the modulation feature map; Through the reconstruction module, the modulation feature map and its detailed texture information are reconstructed using the sub-pixel convolution PixelShuffle upsampling method to obtain a high-resolution image.

2. The multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment according to claim 1, characterized in that: The acquisition of the shallow feature map sequence specifically includes: Get low-resolution video frames from the video source; Assume that any propagation channel j of the multi-scale modulation network based on deformable transformer alignment, then the video frame sequence can be expressed as Where, j = 1, 2, 3, ..., i = 1, 2, 3, ...; Extracting shallow feature map sequences from video frame sequences in, represents the support frame, i.e., the shallow feature map of the i-th time step in the j-th propagation branch, and is the reference frame, and Represents the shallow feature maps of the i-1 and i+1 time steps respectively is the shallow feature map of the i+1th time step, and Respectively They are adjacent frames.

3. The multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment according to claim 1, characterized in that: The network structure of the multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment is a recursive bidirectional network structure based on the improved basic video super-resolution network Basic VSR++.

4. The multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment according to claim 1, characterized in that: The acquisition of the alignment feature map sequence specifically includes: The shallow feature sequence Input to the deformable transformer alignment module; by and is the reference frame, To support the frame, optical flow estimation is used to pre-align the reference frame and the support frame to obtain the pre-aligned features: in, and They are two pre-aligned features of the support frame, W is the distortion operation, and flow i-1→i and Represents the optical flow from the i-1th frame to the i-th frame and from the i+1th frame to the i-th frame respectively; The support frame is projected into the high-frequency space through the convolution layer, and the formula Calculate the high-frequency spatial features; where q ～ is the high-frequency spatial feature, W q is a 1×1 convolutional layer, To support frames; The offset between the high-frequency spatial features and the pre-aligned features is calculated through the offset generation network. The calculation formula is: Among them, θ offset represents the offset generation network, which consists of a depthwise convolution and a 3×3 convolution, and cat represents the concatenation operation; The generated offset is added to the optical flow to obtain the residual fine-tuning of the optical flow to determine the sampling position of the deformation point. The spatial sampling deformation feature is calculated by using the sampling function of bilinear interpolation and the residual fine-tuning of the optical flow to the initial alignment feature. The calculation formula is: in, and They are spatial sampling deformation features, is the sampling function of bilinear interpolation, flow' i-1→i and flow' i+1→i They are respectively the residual fine-tuning of the optical flow of the two reference frames; The deformed keys and values are obtained by projection, and the deformed keys and values are queried using the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame and aggregate the key features to obtain the alignment features related to the support frame. The process can be formulated as follows: Among them, k ～ and v ～ are the keys and values of the spatial sampling deformation features, W k and W v The 1×1 convolutional layers are projections of the keys and values of the spatially sampled features. and are the alignment features of the support frames, softmax is the activation function, d is the dimension of the reference frame, B is the relative position bias, and R is q ～ and k ～ The relative displacement between Indicates relative position deviation; Merge the support frame with its alignment features to obtain a sequence of aligned feature maps 5. The multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment according to claim 4, characterized in that: The offset between the high-frequency spatial feature and the pre-aligned feature reflects the offset required for further feature alignment.

6. The multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment according to claim 1, characterized in that: The acquisition of the modulation characteristic graph specifically includes: The multi-scale modulation module receives a sequence of aligned feature maps As input, it is reshaped into linear features It builds temporal correlation information in the linear layer and then restores the linear features to a sequence of aligned feature maps; Where H represents the image height, W represents the image width, and C is the number of channels; The input channel of the multi-scale modulation module is converted into N ] = split(x) is evenly divided into N heads; Among them, [x1,x2,…x N ] is the sequence of aligned feature maps of N heads, and split represents the average division operation; For each head's aligned feature map sequence, perform 3D depth convolution operations with different receptive fields and aggregate them into feature sequences with different receptive fields. The formula is: Among them, M(x) represents a feature sequence with different receptive fields, Concat represents an aggregation operation, 3DW represents a 3D depth convolution, and k i ∈[x1,x2,…x N ] means that the input features are divided into multiple heads in the channel dimension, i∈[3,5,…k], which indicates the number of receptive fields of each head convolution layer; The features in the obtained feature sequences with different receptive fields are shuffled and grouped according to the head to obtain N groups of feature sequences with different receptive fields; The feature fusion operation is performed on the features of all groups using deep convolutions of different scales to obtain multi-scale fusion features. The calculation expression is: in, is the feature of the i-th channel of the j-th group, G represents the sequence of aligned feature maps in the i-th channel of the j-th group. i represents the initial fusion feature, M represents the multi-scale fusion feature; The obtained multi-scale fusion features are introduced into the linear layer related information to obtain the modulation feature map.

7. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment according to claim 6, characterized in that: The linear layer related information is constructed through a 3×3 3D depth convolution operation.

8. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment according to claim 6, wherein: The aligned feature map sequence for each head is subjected to 3D deep convolution operations with different receptive fields and aggregated, specifically including: The receptive field of each head is as follows: the first head adopts a 3×3 depth convolution kernel, the second head adopts a 5×5 depth convolution kernel, and so on. The receptive field of each subsequent head gradually increases by 2 convolution kernels. A larger receptive field can be obtained by adjusting the number of heads.

9. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment according to claim 6, wherein: The method of shuffling and grouping the obtained multi-scale image features according to the heads is to randomly select a channel from each head to construct a group.

10. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment according to claim 1, characterized in that: The obtaining of the high-resolution image specifically includes: The reconstruction module adopts the upsampling method of sub-pixel convolution PixelShuffle, including: Obtain a modulation feature map with more channels through the convolution layer, where the number of channels is the square of the number of channels of the target high-resolution image; Using sub-pixel convolution operations, channels are rearranged into pixels of a high-resolution image; The resulting high-resolution image is a multiple of the low-resolution input image in size.

Images