Multi-scale modulation video super-division reconstruction method based on deformable transformer alignment

Abstract

The invention discloses a multi-scale modulation video super-division reconstruction method based on deformable transformer alignment, and relates to the field of computer vision. The method comprises the following steps: acquiring a shallow feature map sequence of a video; a multi-scale modulation network based on deformable transformer alignment is adopted to process the shallow layer feature map sequence; the multi-scale modulation network based on the deformable transfer form alignment comprises a deformable transfer form alignment module, a multi-scale modulation module, a Swin transfer form mixing module and a reconstruction module, wherein the deformable transfer form alignment module is connected with the multi-scale modulation module; the deformable transform alignment module is used for feature alignment; the multi-scale modulation module is used for multi-scale feature fusion; a Swin Transform hybrid module is used for carrying out spatial feature capture on the modulation feature map; and the reconstruction module is used for reconstructing the high-resolution image. According to the multi-scale modulation network based on the deformable transformer alignment, the deformable transformer alignment module is combined with the multi-scale modulation module, the reconstruction performance is guaranteed, meanwhile, the calculation efficiency of the network is improved through relatively few network parameters, and the complexity and performance of the network are balanced.

Description

Technical Field

[0001] The present invention relates to the field of video super-resolution of computer vision, and in particular to a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment. Background Art

[0002] At present, with the rapid growth of Internet video data, users' demand for video quality continues to increase. However, due to bandwidth limitations, videos are often downsampled and compressed, resulting in quality degradation. Therefore, video super-resolution (VSR) has emerged as a video enhancement technology. It aims to increase the resolution of the video, thereby improving its clarity and details, thereby improving the quality of low-resolution videos caused by quality loss during compression and transmission or low original recording resolution. Traditional video super-resolution methods mainly rely on algorithms based on image processing and statistical inference, which were mainstream before deep learning technology. Such as interpolation-based methods: bilinear interpolation, bicubic interpolation and nearest neighbor interpolation; sparse representation-based methods: dictionary learning; regularization-based methods: maximum a posteriori estimation, total variation regularization. Although traditional methods have achieved certain success in processing video super-resolution, they also have some inherent defects, such as insufficient reconstruction details, poor adaptability to complex motion scenes, and large computational complexity.

[0003] In the prior art, mature VSR network models usually include an alignment module to compensate for the motion of objects between frames. The alignment module is crucial for CNN-based VSR networks because the local inductive bias of CNNs only allows them to effectively utilize spatially densely distributed information. At the same time, the latest progress in CNN-based VSR has shown better performance by adding techniques such as residual learning, dense connections, and deformable convolutions. PSRT proposes a method for aligning image patches by cropping the image into non-overlapping patches of small self-attention window size, then aligning on these patches, locating the motion of objects according to optical flow, and using nearest neighbor sampling to align video frames to corresponding positions in the reference frame. Finally, these aligned video frame sequences are input into the network of the transformer architecture to obtain higher resolution images.

[0004] However, in the CNN-based VSR network, the limited receptive field of CNN can only utilize information that is tightly distributed in space and cannot handle unaligned, wide-ranging large motions. The transformer introduced into the VSR task retains the complex alignment module and can only process low-frequency information, which significantly increases the computational burden and cannot effectively obtain multi-scale image features, resulting in unsatisfactory reconstruction effects. Although the latest existing video super-resolution reconstruction methods have achieved relatively ideal reconstruction effects, they have increased the number of network parameters, increased network complexity, and increased the computational burden.

[0005] Therefore, when performing video super-resolution reconstruction, how to ensure that the reconstructed image has high resolution while using relatively few network parameters to improve the computational efficiency of video super-resolution reconstruction and achieve a balance between network complexity and performance is a problem that needs to be solved in the existing technology. Summary of the invention

[0006] Based on this, it is necessary to provide a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment to address the above technical problems.

[0007] The present invention provides a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment, comprising:

[0008] A multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment includes the following steps:

[0009] Obtain a low-resolution video frame sequence from a video source, and extract a shallow feature map sequence from multiple low-resolution video frame sequences;

[0010] Construct a multi-scale modulation network based on deformable transformer alignment, including a deformable transformer alignment module, a multi-scale modulation module, a SwinTransformer hybrid module, and a reconstruction module;

[0011] Input the shallow feature map sequence into the multi-scale modulation network based on deformable transformer alignment;

[0012] Through the deformable transformer alignment module, optical flow is used to pre-align the features of the support frame, and the offset generation network is used to generate the offset between the pre-aligned features and the support frame to capture the relevant key areas of the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame, aggregate the key features of the key areas, and obtain the alignment feature map sequence;

[0013] Through the multi-scale modulation module, the time correlation information is constructed in the linear layer and the input channel of the multi-scale modulation module is evenly divided into multiple heads. Different 3D deep convolutions are used to expand the receptive field of each head, and the time correlation information is introduced to fuse features of different scales to obtain the modulation feature map.

[0014] The spatial features of the modulation feature map are captured through the Swin Transformer hybrid module to obtain the detailed texture information of the modulation feature map;

[0015] Through the reconstruction module, the modulation feature map and its detailed texture information are reconstructed by using the sub-pixel convolution PixelShuffle upsampling method to obtain a high-resolution image.

[0016] Preferably, the acquisition of the shallow feature map sequence specifically includes:

[0017] Get low-resolution video frames from a video source;

[0018] Assume that any propagation channel j of the multi-scale modulation network based on deformable transformer alignment is, then the video frame sequence can be expressed as Where, j = 1, 2, 3, ..., i = 1, 2, 3, ...;

[0019] Extracting shallow feature map sequences from video frame sequences

[0020] in, represents the support frame, i.e., the shallow feature map of the i-th time step in the j-th propagation branch, is the reference frame, Represents the shallow feature maps of the i-1th and i+1th time steps respectively is the shallow feature map of the i+1th time step, Respectively Adjacent frames.

[0021] Preferably, the network structure of the multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment is a recursive bidirectional network structure based on an improved basic video super-resolution network Basic VSR++.

[0022] Preferably, it is characterized in that the acquisition of the alignment feature map sequence specifically includes:

[0023] The shallow feature sequence Input to the deformable transformer alignment module;

[0024] by is the reference frame, To support the frame, optical flow estimation is used to pre-align the reference frame and the support frame to obtain the pre-alignment feature:

[0025]

[0026] in, They are the two pre-aligned features of the support frame, W is the distortion operation, and flow i-1→i and Respectively represent the optical flow from the i-1th frame to the i-th frame and from the i+1th frame to the i-th frame;

[0027] The support frame is projected into the high-frequency space through the convolution layer, and the formula Calculate the high-frequency spatial features; where q ～ is the high-frequency spatial feature, W q is a 1×1 convolutional layer, To support frames;

[0028] The offset between the high-frequency spatial features and the pre-aligned features is calculated through the offset generation network. The calculation formula is:

[0029]

[0030]

[0031] Among them, θ offset represents the offset generation network, which consists of a depthwise convolution and a 3×3 convolution, and cat represents the concatenation operation;

[0032] The generated offset is added to the optical flow to obtain the residual fine-tuning of the optical flow to determine the sampling position of the deformation point. The spatial sampling deformation feature is calculated by using the sampling function of bilinear interpolation and the residual fine-tuning of the optical flow for the initial alignment feature. The calculation formula is:

[0033]

[0034] in, They are spatial sampling deformation features, is the sampling function of bilinear interpolation, flow' i-1→i and flow' i+1→i They are respectively the residual fine-tuning of the optical flow of the two reference frames;

[0035] The deformed keys and values ​​are obtained by projection, and the deformed keys and values ​​are queried using the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame and aggregate the key features to obtain the alignment features related to the support frame. The process can be formulated as:

[0036]

[0037] Among them, k ～ and v ～ are the keys and values ​​of the spatial sampling deformation features, W k and W v The 1×1 convolutional layers are projections of the keys and values ​​of the spatially sampled and deformed features. are the alignment features of the support frames, softmax is the activation function, d is the dimension of the reference frame, B is the relative position bias, and R is q ～ and k ～ The relative displacement between Indicates relative position deviation;

[0038] Merge the support frame with its alignment features to obtain a sequence of aligned feature maps

[0039] Preferably, the offset between the high-frequency spatial feature and the pre-aligned feature reflects the offset required for further feature alignment.

[0040] Preferably, it is characterized in that the acquisition of the modulation characteristic graph specifically includes:

[0041] The multi-scale modulation module receives a sequence of aligned feature maps As input, it is reshaped into linear features It builds temporal correlation information in the linear layer and then restores the linear features to a sequence of aligned feature maps;

[0042] Among them, H represents the image height, W represents the image width, and C is the number of channels;

[0043] The input channels of the multi-scale modulation module are transformed into N ] = split(x) is evenly divided into N heads;

[0044] Among them, [x1,x2,…x N ] is the sequence of aligned feature maps of N heads, split represents the average division operation;

[0045] For each head’s aligned feature map sequence, 3D deep convolution operations with different receptive fields are performed on it and aggregated into feature sequences with different receptive fields. The formula is:

[0046]

[0047] Among them, M(x) represents a feature sequence with different receptive fields, Concat represents an aggregation operation, 3DW represents a 3D deep convolution, and k i ∈[x1,x2,…x N] means that the input features are divided into multiple heads in the channel dimension, i∈[3,5,…k], which means the number of receptive fields of each head convolution layer;

[0048] The features in the obtained feature sequences with different receptive fields are shuffled and grouped according to the heads to obtain N groups of feature sequences with different receptive fields;

[0049] The deep convolution of different scales is used to perform feature fusion operation on the features of all groups to obtain multi-scale fusion features. The calculation expression is:

[0050]

[0051] in, is the feature of the i-th channel of the j-th group, G represents the sequence of aligned feature maps in the i-th channel of the j-th group. i represents the initial fusion feature, M represents the multi-scale fusion feature;

[0052] The obtained multi-scale fusion features are introduced into the linear layer related information to obtain the modulation feature map.

[0053] Preferably, the linear layer related information is constructed by a 3×3 3D depth convolution operation;

[0054] Preferably, the aligned feature map sequence for each head is subjected to 3D deep convolution operations with different receptive fields and aggregated, specifically including:

[0055] The receptive field of each head is as follows: the first head uses a 3×3 depth convolution kernel, the second head uses a 5×5 depth convolution kernel, and so on. The receptive field of each subsequent head gradually increases by 2 convolution kernels. A larger receptive field can be obtained by adjusting the number of heads;

[0056] Preferably, the method of shuffling and grouping the obtained multi-scale image features according to the heads is to randomly select a channel from each head to construct a group.

[0057] Preferably, the obtaining of the high-resolution image specifically includes:

[0058] The reconstruction module adopts the up-sampling method of sub-pixel convolution PixelShuffle, including:

[0059] A modulation feature map with more channels is obtained through the convolution layer, and the number of channels is the square of the number of channels of the target high-resolution image;

[0060] Sub-pixel convolution operation is used to rearrange the channels into pixels of a high-resolution image;

[0061] The resulting high-resolution image is a multiple of the size of the low-resolution input image.

[0062] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0063] The present invention proposes a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment. Low-resolution video frames are processed by a multi-scale modulation network (Multi-scale (Deformable Transformer Network, MSDT) of deformable transformer alignment. The deformable transformer alignment module dynamically samples useful information to align the reference frame of the input video frame sequence to the support frame. Compared with the alignment method based on CNN and the introduction of transformer in VSR tasks, more comprehensive and long-term useful information can be captured. The multi-scale modulation module can more effectively fuse features of different scales and refine the texture details of the image by multi-scale modulation of the alignment features under the condition of using relatively few parameters.

[0064] The video super-resolution reconstruction method provided by the present invention improves the computational efficiency of video super-resolution reconstruction by using fewer network parameters while ensuring that the reconstructed image has high resolution, thereby achieving a good balance between network complexity and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0066] Figure 1 A schematic diagram of the overall process of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0067] Figure 2 A network structure diagram of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0068] Figure 3 A structural schematic diagram of a deformable transformer alignment module of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0069] Figure 4A structural schematic diagram of a multi-scale modulation module of a multi-scale modulation video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention;

[0070] Figure 5 A comparison diagram of the visualization effect of VSR (X4) on the REDS dataset of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention; DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] At present, in the field of video super-resolution, although the existing technology has improved to a certain extent in terms of video super-resolution performance, how to improve the computational efficiency of video super-resolution reconstruction while ensuring the high resolution of the reconstructed image and achieving a balance between network complexity and performance is a problem that needs to be solved by the existing technology.

[0073] In order to solve the above technical problems, the present invention provides an embodiment of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment.

[0074] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0075] Figure 1 The present invention is a flowchart of a multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment, and the method specifically includes the following steps:

[0076] S101: obtaining a low-resolution video frame sequence from a video source, and extracting a shallow feature map sequence from multiple low-resolution video frame sequences, including the following steps:

[0077] Get low-resolution video frames from a video source;

[0078] Assume that any propagation channel j of the multi-scale modulation network based on deformable transformer alignment is, then the video frame sequence can be expressed as Where, j = 1, 2, 3, ..., i = 1, 2, 3, ...;

[0079] Extracting shallow feature map sequences from video frame sequences

[0080] in, represents the support frame, i.e., the shallow feature map of the i-th time step in the j-th propagation branch, is the reference frame, Represents the shallow feature maps of the i-1th and i+1th time steps respectively is the shallow feature map of the i+1th time step, Respectively Adjacent frames.

[0081] S102: Construct a multi-scale modulation network based on deformable transformer alignment including a deformable transformer alignment module, a multi-scale modulation module, a SwinTransformer hybrid module and a reconstruction module.

[0082] S103: Input the shallow feature map sequence into the multi-scale modulation network based on deformable transformer alignment, use the optical flow to pre-align the features of the support frame through the deformable transformer alignment module, use the offset generation network to generate the offset between the pre-aligned features and the support frame to capture the relevant key areas of the support frame, use the self-attention mechanism to model the relevant key areas of the support frame, aggregate the key features of the key areas, and obtain the aligned feature map sequence;

[0083] The deformable transformer alignment module is as follows Figure 3 As shown, the figure only shows the alignment process from the i-1th frame to the i-th frame, and the alignment process from the i+1th frame to the i-th frame is the same as the alignment process from the i-1th frame to the i-th frame, including the following steps:

[0084] The shallow feature sequence Input to the deformable transformer alignment module;

[0085] by is the reference frame, To support the frame, optical flow estimation is used to pre-align the reference frame and the support frame to obtain the pre-alignment feature:

[0086]

[0087] in, They are the two pre-aligned features of the support frame, W is the distortion operation, and flow i-1→i and Respectively represent the optical flow from the i-1th frame to the i-th frame and from the i+1th frame to the i-th frame;

[0088] The support frame is projected into the high-frequency space through the convolution layer, and the formula Calculate the high-frequency spatial features; where q ～ is the high-frequency spatial feature, W q is a 1×1 convolutional layer, To support frames;

[0089] The offset between the high-frequency spatial features and the pre-aligned features is calculated through the offset generation network. The calculation formula is:

[0090]

[0091] Among them, θ offset represents the offset generation network, which consists of a depthwise convolution and a 3×3 convolution, and cat represents the concatenation operation;

[0092] The generated offset is added to the optical flow to obtain the residual fine-tuning of the optical flow to determine the sampling position of the deformation point. The spatial sampling deformation feature is calculated by using the sampling function of bilinear interpolation and the residual fine-tuning of the optical flow for the initial alignment feature. The calculation formula is:

[0093]

[0094] in, They are spatial sampling deformation features, is the sampling function of bilinear interpolation, flow' i-1→i and flow' i+1→i They are respectively the residual fine-tuning of the optical flow of the two reference frames;

[0095] The deformed keys and values ​​are obtained by projection, and the deformed keys and values ​​are queried using the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame and aggregate the key features to obtain the alignment features related to the support frame. The process can be formulated as:

[0096]

[0097] Among them, k ～ and v ～ are the keys and values ​​of the spatial sampling deformation features, W k and W v The 1×1 convolutional layers are projections of the keys and values ​​of the spatially sampled and deformed features. are the alignment features of the support frames, softmax is the activation function, d is the dimension of the reference frame, B is the relative position bias, and R is q ～ and k ～ The relative displacement between Indicates relative position deviation;

[0098] Merge the support frame with its alignment features to obtain a sequence of aligned feature maps

[0099] S104: constructing time correlation information in a linear layer through a multi-scale modulation module and evenly dividing the input channel of the multi-scale modulation module into multiple heads, using different 3D deep convolutions to expand the receptive field for each head, introducing time correlation information to perform feature fusion of different scales, and obtaining a modulation feature map;

[0100] The multi-scale modulation module in the multi-scale modulation network based on deformable transformer alignment is as follows Figure 4 As shown, the following steps are included:

[0101] The multi-scale modulation module receives a sequence of aligned feature maps As input, it is reshaped into linear features It builds temporal correlation information in the linear layer and then restores the linear features to a sequence of aligned feature maps;

[0102] Among them, H represents the image height, W represents the image width, and C is the number of channels;

[0103] The input channels of the multi-scale modulation module are transformed into N ] = split(x) is evenly divided into N heads;

[0104] Among them, [x1,x2,…x N ] is the sequence of aligned feature maps of N heads, split represents the average division operation;

[0105] For each head’s aligned feature map sequence, 3D deep convolution operations with different receptive fields are performed on it and aggregated into feature sequences with different receptive fields. The formula is:

[0106]

[0107] Among them, M(x) represents a feature sequence with different receptive fields, Concat represents an aggregation operation, 3DW represents a 3D deep convolution, and k i ∈[x1,x2,…x N ] means that the input features are divided into multiple heads in the channel dimension, i∈[3,5,…k], which means the number of receptive fields of each head convolution layer;

[0108] The features in the obtained feature sequences with different receptive fields are shuffled and grouped according to the heads to obtain N groups of feature sequences with different receptive fields;

[0109] The deep convolution of different scales is used to perform feature fusion operation on the features of all groups to obtain multi-scale fusion features. The calculation expression is:

[0110]

[0111] in, is the feature of the i-th channel of the j-th group, G represents the sequence of aligned feature maps in the i-th channel of the j-th group. i represents the initial fusion feature, M represents the multi-scale fusion feature;

[0112] The obtained multi-scale fusion features are introduced into the linear layer related information to obtain the modulation feature map.

[0113] S105: The modulation feature map is input into the hybrid module, the hybrid module fuses the temporal and spatial information of the modulation feature map, and uses the Swin Transformer-based image restoration module and its residual block to capture the correlation information of the spatial range of the modulation feature map.

[0114] S106: inputting the modulation feature map into a reconstruction module, and the reconstruction module reconstructs the modulation feature map by an upsampling method using the temporal and spatial information and correlation information of the modulation feature map to obtain a high-resolution image, specifically including:

[0115] A modulation feature map with more channels is obtained through the convolution layer, and the number of channels is the square of the number of channels of the target high-resolution image;

[0116] Using sub-pixel convolution operations, these channels are rearranged into pixels of a high-resolution image;

[0117]

[0118] Among them, I H,W,C is the pixel value of the output image at position (H, W) with channel C, is the modulation feature map at position (H, W), after r 2 The corresponding pixel value after channel grouping, represents c divided by r 2 The integer part of c is used to determine the channel to which c belongs, c mod r 2 represents c divided by r 2 The integer part of the quotient is used to determine the specific position of c;

[0119] The resulting high-resolution image is a multiple of the size of the low-resolution input image.

[0120] In this embodiment, the basicVSR framework is used to implement the training task, the Charbonnier loss is used as the training target, and the Adam optimizer is used for training, where β1=0.9, β2=0.999, the initial learning rate is set to 2e-4, and the cosine annealing scheme is used to decay the learning rate to a minimum of 1e-7, and Spynet is used to predict the optical flow in the video; the total number of iterations is set to 300k, the mini-batch size is 8, and the LR patch size is 64; the number of network channels is set to 64, the number of DTABs is set to 3, the number of attention heads is set to 4, the number of MSMBs is set to 4, and the head N is set to 4.

[0121] Peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM) are used as evaluation indicators. REDS and Vimeo90K video frame sequences are used as training sets, and REDS4, Vimeo-90K-T, Vid4, UDM10 and Vimeo-900K video frame sequences are used as test sets.

[0122] The training set data is used as the network proposed by the present invention to train, and a multi-scale modulated video super-resolution reconstruction model based on deformable transformer alignment is obtained;

[0123] Peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) were used as evaluation metrics;

[0124] The test set data is input into the model to obtain the video super-resolution results, which are compared with the results of the existing most advanced technology. The comparison results are shown in the following table:

[0125] Table 1: Comparison of PSNR and SSIM for X4VSR task on REDS4, Vimeo-90K-T, Vid4 datasets

[0126]

[0127]

[0128] Comparing MSDT with several other methods, most of them achieved the best results in the VSR task. As can be seen from Table 1, MSDT achieved better results and had the best performance with the least number of parameters, which proved the beneficial effect of the method of the present invention and achieved a good balance between model complexity and performance.

[0129] like Figure 5 As shown, a visual comparison is provided between other VSR methods. Obviously, the video frames reconstructed by MSDT achieve better visual effects.

[0130] In summary, through the verification of this embodiment, it is shown that the multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment provided by the present invention improves the computational efficiency of video super-resolution reconstruction by using fewer network parameters while ensuring the high resolution of the reconstructed image, and achieves a good balance between network complexity and performance, thereby providing powerful scientific guidance for the practical application of video super-resolution in the field of computer vision.

[0131] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment, characterized in that: include: Obtain a low-resolution video frame sequence from a video source, and extract a shallow feature map sequence from multiple low-resolution video frame sequences; Construct a multi-scale modulation network based on deformable transformer alignment, including a deformable transformer alignment module, a multi-scale modulation module, a SwinTransformer hybrid module, and a reconstruction module; Input the shallow feature map sequence into the multi-scale modulation network based on deformable transformer alignment; Through the deformable transformer alignment module, optical flow is used to pre-align the features of the support frame, and the offset generation network is used to generate the offset between the pre-aligned features and the support frame to capture the relevant key areas of the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame, aggregate the key features of the key areas, and obtain the alignment feature map sequence; Through the multi-scale modulation module, the time correlation information is constructed in the linear layer and the input channel of the multi-scale modulation module is evenly divided into multiple heads. Different 3D deep convolutions are used to expand the receptive field of each head, and the time correlation information is introduced to fuse features of different scales to obtain the modulation feature map. The spatial features of the modulation feature map are captured through the Swin Transformer hybrid module to obtain the detailed texture information of the modulation feature map; Through the reconstruction module, the modulation feature map and its detailed texture information are reconstructed by using the sub-pixel convolution PixelShuffle upsampling method to obtain a high-resolution image.

2. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 1, characterized in that: The acquisition of the shallow feature map sequence specifically includes: Get low-resolution video frames from a video source; Assume that any propagation channel j of the multi-scale modulation network based on deformable transformer alignment is, then the video frame sequence can be expressed as Where, j = 1, 2, 3, ..., i = 1, 2, 3, ...; Extracting shallow feature map sequences from video frame sequences in, represents the support frame, i.e., the shallow feature map of the i-th time step in the j-th propagation branch, and is the reference frame, and Represents the shallow feature maps of the i-1th and i+1th time steps respectively is the shallow feature map of the i+1th time step, and Respectively Adjacent frames.

3. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 1, characterized in that: The network structure of the multi-scale modulated video super-resolution reconstruction method based on deformable transformer alignment is a recursive bidirectional network structure based on an improved basic video super-resolution network Basic VSR++.

4. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 1, characterized in that: The acquisition of the alignment feature map sequence specifically includes: The shallow feature sequence Input to the deformable transformer alignment module; by and is the reference frame, To support the frame, optical flow estimation is used to pre-align the reference frame and the support frame to obtain the pre-alignment feature: in, and They are the two pre-aligned features of the support frame, W is the distortion operation, and flow i-1→i and Respectively represent the optical flow from the i-1th frame to the i-th frame and from the i+1th frame to the i-th frame; The support frame is projected into the high-frequency space through the convolution layer, and the formula Calculate the high-frequency spatial features; where q ～ is the high-frequency spatial feature, W q is a 1×1 convolutional layer, To support frames; The offset between the high-frequency spatial features and the pre-aligned features is calculated through the offset generation network. The calculation formula is: Among them, θ offset represents the offset generation network, which consists of a depthwise convolution and a 3×3 convolution, and cat represents the concatenation operation; The generated offset is added to the optical flow to obtain the residual fine-tuning of the optical flow to determine the sampling position of the deformation point. The spatial sampling deformation feature is calculated by using the sampling function of bilinear interpolation and the residual fine-tuning of the optical flow for the initial alignment feature. The calculation formula is: in, and They are spatial sampling deformation features, is the sampling function of bilinear interpolation, flow' i-1→i and flow' i+1→i They are respectively the residual fine-tuning of the optical flow of the two reference frames; The deformed keys and values ​​are obtained by projection, and the deformed keys and values ​​are queried using the support frame. The self-attention mechanism is used to model the relevant key areas of the support frame and aggregate the key features to obtain the alignment features related to the support frame. The process can be formulated as: Among them, k ～ and v ～ are the keys and values ​​of the spatial sampling deformation features, W k and W v The 1×1 convolutional layers are projections of the keys and values ​​of the spatially sampled and deformed features. and are the alignment features of the support frames, softmax is the activation function, d is the dimension of the reference frame, B is the relative position bias, and R is q ～ and k ～ The relative displacement between Indicates relative position deviation; Merge the support frame with its alignment features to obtain a sequence of aligned feature maps 5. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 3, characterized in that: The offset between the high-frequency spatial feature and the pre-aligned feature reflects the offset required for further feature alignment.

6. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment according to claim 1, characterized in that: The acquisition of the modulation characteristic graph specifically includes: The multi-scale modulation module receives a sequence of aligned feature maps As input, it is reshaped into linear features It builds temporal correlation information in the linear layer and then restores the linear features to a sequence of aligned feature maps; Among them, H represents the image height, W represents the image width, and C is the number of channels; The input channels of the multi-scale modulation module are transformed into N ] = split(x) is evenly divided into N heads; Among them, [x1,x2,…x N ] is the sequence of aligned feature maps of N heads, split represents the average division operation; For each head’s aligned feature map sequence, 3D deep convolution operations with different receptive fields are performed on it and aggregated into feature sequences with different receptive fields. The formula is: Among them, M(x) represents a feature sequence with different receptive fields, Concat represents an aggregation operation, 3DW represents a 3D deep convolution, and k i ∈[x1,x2,…x N ] means that the input features are divided into multiple heads in the channel dimension, i∈[3,5,…k], which means the number of receptive fields of each head convolution layer; The features in the obtained feature sequences with different receptive fields are shuffled and grouped according to the heads to obtain N groups of feature sequences with different receptive fields; The deep convolution of different scales is used to perform feature fusion operation on the features of all groups to obtain multi-scale fusion features. The calculation expression is: M=3DW 3×3 ([G1,G2…G N ]) in, is the feature of the i-th channel of the j-th group, G represents the sequence of aligned feature maps in the i-th channel of the j-th group. i represents the initial fusion feature, M represents the multi-scale fusion feature; The obtained multi-scale fusion features are introduced into the linear layer related information to obtain the modulation feature map.

7. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 6, characterized in that: The linear layer related information is constructed through a 3×3 3D depth convolution operation.

8. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 5, characterized in that: The aligned feature map sequence of each head is subjected to 3D deep convolution operations with different receptive fields and aggregated, specifically including: The receptive field of each head is that the first head adopts a 3×3 deep convolution kernel, the second head adopts a 5×5 deep convolution kernel, and so on. The receptive field of each subsequent head gradually increases by 2 convolution kernels, and a larger receptive field can be obtained by adjusting the number of heads.

9. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment as claimed in claim 5, characterized in that: The method of shuffling and grouping the obtained multi-scale image features according to the heads is to randomly select a channel from each head to construct a group.

10. The method for super-resolution reconstruction of multi-scale modulated video based on deformable transformer alignment according to claim 1, characterized in that: The obtaining of the high-resolution image specifically includes: The reconstruction module adopts the up-sampling method of sub-pixel convolution PixelShuffle, including: A modulation feature map with more channels is obtained through the convolution layer, and the number of channels is the square of the number of channels of the target high-resolution image; Sub-pixel convolution operation is used to rearrange the channels into pixels of a high-resolution image; The resulting high-resolution image is a multiple of the low-resolution input image in size.

Images