A deformable transformer-based video super-resolution method and system

Abstract

The application provides a deformable Transformer-based video super-resolution method and system, after collecting low-resolution video dataset samples in the field of video super-resolution, the low-resolution video training set samples are subjected to convolution operation to obtain video sequence feature maps; the video sequence feature maps are subjected to local feature extraction and down-sampling operation; the obtained down-sampled local features are sent into the deformable Transformer to enhance the spatial information of the local features; the local features of each frame of the video are traversed and subjected to up-sampling; the enhanced video features are sent into the deformable convolution network for time alignment; finally, the video features subjected to spatial enhancement and time alignment are sent into a super-resolution reconstruction module, and the output of high-resolution video frames is completed through the super-resolution reconstruction module. The application can obtain high-resolution video with high definition and good texture information, and compared with the prior art, the details of the video can be better restored and the video ghosting can be reduced.

Description

Technical Field

[0001] This invention belongs to the field of video super-resolution video enhancement, and relates to a video super-resolution scheme based on deformable Transformer. Background Technology

[0002] Video super-resolution enhancement technology refers to using technical means to improve the clarity and detail of low-resolution videos to achieve the effect of high-resolution videos. With the development of the internet, video content has become one of the main ways people obtain information. However, due to limitations in shooting equipment and transmission bandwidth, many videos suffer from image quality loss during transmission. Furthermore, some older videos cannot be presented to users in high resolution due to storage media limitations. Therefore, the emergence of video super-resolution enhancement technology solves these problems, allowing users to view video content more clearly.

[0003] Traditional video enhancement techniques mainly include interpolation-based methods and learning-based methods. Interpolation methods generate high-resolution images by making pixel-level predictions on low-resolution images, but this method is prone to blurring and distortion. Learning-based methods, on the other hand, utilize neural network models to train on low-resolution images, thereby generating high-quality high-resolution images. In recent years, the development of convolutional neural network technology has made learning-based video super-resolution enhancement techniques increasingly mainstream. This technique can be trained on large-scale datasets, thus achieving higher accuracy and robustness. However, the performance of existing convolutional neural network-based video super-resolution methods largely depends on the accuracy of motion estimation and compensation. Their performance is also highly dependent on the number of input frames, which is unfavorable for videos lacking long-range temporal information. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, the present invention aims to provide a deformable Transformer-based video super-resolution method. This method utilizes the self-attention structure of the deformable Transformer to fuse similar and clear local features from adjacent frames into the current frame, thereby improving the video reconstruction effect and constructing an effective video super-resolution method.

[0005] To achieve the above objectives, the technical solution proposed by this invention is as follows:

[0006] According to the final draft, further supplementation is needed.

[0007] The present invention proposes the following improvements:

[0008] (1) Design a video super-resolution network based on deformable Transformer. The network consists of a global feature extraction part, a local feature extraction part, a deformable Transformer part, a temporal alignment part, and a reconstruction part.

[0009] (2) Construct corresponding training and test samples in the video dataset, and train the parameters of the convolutional neural network built in step 2 until the network converges;

[0010] (3) Input the extracted global video features into the Swin-Transformer block, reduce the spatial size by downsampling, expand the channel size, and further extract the local features of the video.

[0011] (4) The most relevant n local features are calculated using cosine similarity. These local features are then passed to the deformable Transformer module, where the spatial attention weight matrix is ​​calculated using the self-attention module. Spatially enhanced video feature sequences are obtained through an attention weight fusion formula.

[0012] (5) Time alignment of video feature sequences is performed using deformable convolution.

[0013] (6) Input the video feature sequence into the trained reconstruction module to obtain the super-resolution reconstruction result.

[0014] The proposed Transformer-based video super-resolution method can acquire high-resolution videos with good texture information. Compared with previous methods, it can better restore the details of the video and reduce video ghosting.

[0015] The present invention is simple and convenient to implement, highly practical, and solves the problems of low practicality and inconvenience in actual application of related technologies. It can improve user experience and has significant market value. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the framework of an embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0019] The embodiment proposes a deformable Transformer-based video super-resolution method. It designs a deformable Transformer-based video super-resolution network, which consists of a global feature extraction part, a local feature extraction part, a deformable Transformer part, a temporal alignment part, and a reconstruction part. The Transformer represents...

[0020] The Vimeo90K-T low-resolution video sequence set was selected as the training sample pair for the example. At the same time, the global feature extraction part, local feature extraction part, deformable Transformer part, temporal alignment part and reconstruction part in step (1) were trained until the network converged.

[0021] By inputting video training samples into the trained network model, a super-resolution reconstructed video sequence can be obtained.

[0022] Specifically, the preferred embodiment uses a deformable Transformer video super-resolution network structure as follows: Figure 2 As shown:

[0023] In the global feature extraction section, this example uses 2N convolutional layers to extract global features from the input video frames. The formula is as follows:

[0024]

[0025] in H is the global feature of frame t. global (·) indicates the global feature extraction part. This is the t-th low-resolution video frame. In specific implementations, the preferred value range for N is 1-3.

[0026] In the local feature extraction section, this embodiment uses k Swin-Transformers to extract local features and uncover deeper local information. The expression is as follows:

[0027]

[0028] H local (·) indicates the part that uses the Swing-Transformer for local feature extraction. These are the local features of the t-th low-resolution video frame. The local features are then upsampled to the same dimension as the initial video feature sequence. The cosine similarity formula is then used to calculate the similarity with... Local features of the n most similar adjacent frames In specific implementation, the preferred value range for k is 2-3, and the preferred value range for n is 15-32. The Swin-Transformer is a Transformer-based neural network structure used for computer vision tasks such as image classification, object detection, and semantic segmentation. It combines local features with global contextual information to model and process information at different scales in an image. Specifically, the Swin-Transformer employs a hierarchical Transformer structure, including multiple levels of window attention mechanisms and hierarchical feature fusion methods, effectively capturing local details and global semantic information in images.

[0029] right Local features of adjacent video frames The n most similar vectors are convolved to obtain the 'key-value pair vectors' of the deformable Transformer attention mechanism: key (k) and value (v). Convolution operations can also yield the 'query vector' (abbreviated as q) for the deformable Transformer attention mechanism. Self-attention is then calculated between the 'query vector' q and the 'key-value pair vectors' k and v. The deformable Transformer self-attention mechanism calculates the connection information between features of adjacent video frames. The specific process of the deformable Transformer is as follows: the similarity relationship between the 'query vector' q and the 'key vector' k is calculated using softmax (normalized exponential function), and the 'value vector' v is calculated using distributed weights. Through multiple convolution calculations, the spatial attention weight matrix on the features of the previous and next frame images is obtained. The specific mathematical expression is as follows:

[0030]

[0031] Among them, qk T d represents the similarity matrix between the features of the current frame image and the features of adjacent frames image. k Represents qk T In terms of dimensions, Attention() represents a deformable attention weight distribution.

[0032] Spatially enhanced video features are obtained through an attention weight fusion formula. Its expression is as follows:

[0033]

[0034] in, It is the global feature of frame t after attention enhancement. and yes and The attention weight matrices generated separately will affect the spatially augmented video sequences. Alignment is achieved through deformable convolution, and the data is then fed into the reconstruction module.

[0035] The reconstruction part provides a deformable Transformer-based video super-resolution model, consisting of one upsampling layer, four convolutional layers, and four corresponding deconvolutional layers. To achieve global residual learning and accelerate training, skip connections are added between the second and third convolutional layers and their respective deconvolutional layers. Specifically, the input feature map is first enlarged using an upsampling layer, then features are progressively extracted through the four convolutional layers, followed by the corresponding four deconvolutional layers. A skip connection layer is introduced between the second convolutional layer and the second deconvolutional layer. This skip connection layer directly adds the output of the second convolutional layer to the input of the third deconvolutional layer, allowing lower-level feature information to be directly passed to higher layers, avoiding information loss and achieving global residual learning. Introducing a skip connection layer into the network accelerates training because it helps the network converge faster by directly passing lower-level feature information and mitigates the vanishing gradient problem. This design effectively improves network performance and training efficiency.

[0036] By reconstructing, the target column for reconstruction is obtained. The expression is as follows:

[0037]

[0038] Among them, H pix () represents the reconstruction module, T represents the total number of input video frames, t is the current enhancement frame number, t=2,…T, and the value of T can be set according to the specific implementation. Generally, T≥3.

[0039] For example, such as Figure 2 Taking three low-resolution video frames as an example, video frames 1, 2, and 3 are processed through convolutional layers for global feature extraction, followed by Swin-Transformer for local feature extraction. Local feature cosine similarity is then calculated to obtain video features 1, 2, and 3. Video feature 1 is linearly projected to obtain key-value pair vectors k1 and v1. Video feature 2 is linearly projected to generate a query vector q2. Video feature 3 is linearly projected to obtain key-value pair vectors k3 and v3. The connection information between features of adjacent video frames is calculated using the deformable Transformer self-attention mechanism to obtain the spatial attention weight matrix. Spatially enhanced video features are obtained through attention weight fusion formula, and high-resolution video frames are reconstructed after temporal alignment.

[0040] In practical implementation, a deformable Transformer-based video super-resolution network can be pre-established and trained as described above. The trained network can then be directly used for actual video super-resolution enhancement. Alternatively, a training set can be established based on the actual situation, and testing can be performed after training. In some possible embodiments, a deformable Transformer-based video super-resolution method is provided, including the following steps, see [link to relevant documentation]. Figure 1 :

[0041] Step 1: Collect low-resolution video dataset samples in the field of video super-resolution.

[0042] You can use a raw dataset of video sequences that cover a variety of complex scenes and actions; 80% of it is used as training data and 20% as test data.

[0043] The video sequences provided by the original dataset This is called a true high-resolution video sequence. A true high-resolution video sequence is represented as... Where T represents the total number of high-resolution video frames. Let I represent the image of the t-th frame in the high-resolution video sequence; similarly, the low-resolution video sequence provided by the original dataset is I. L , which is represented as in Let represent the image of the t-th frame in a low-resolution video sequence, where T represents the total number of low-resolution video frames.

[0044] Step 2: Perform convolution operations on the low-resolution video training set samples to obtain video sequence feature maps.

[0045] Step 3: Extract local features from the video sequence feature map from Step 2 and perform downsampling.

[0046] Step 3 includes:

[0047] Step 3.1: Extract the video sequence feature map from Step 2. Local features are extracted using k Swin-Transformer blocks; the extracted local feature maps are then downsampled.

[0048] Step 3.2: The downsampled local feature map output from Step 3.1 is passed through k Swin-Transformer blocks again to extract local features; the extracted local feature map is then downsampled. At this point, the number of channels in the video feature map increases, while the spatial size decreases. This yields the video local feature sequence.

[0049] Step 4: The downsampled local features are fed into a deformable Transformer. The attention mechanism of the deformable Transformer calculates vector similarity. Local features with high similarity to the obtained local features are extracted and fused together. This enhances the spatial information of the local features.

[0050] Step 4 includes:

[0051] Step 4.1: Obtain the downsampled video frame features Adjacent video features Perform cosine similarity calculation. Obtain the n vectors with the highest similarity using cosine similarity calculation;

[0052] Step 4.2: Feature analysis of video frames The n vectors with the highest similarity are convolved to obtain the 'query vector' of the deformable Transformer attention mechanism: query(q);

[0053] Step 4.3: For Adjacent video features The n vectors with the highest similarity are convolved to obtain the 'key-value pair vectors' of the deformable Transformer attention mechanism: key(k) and value(v);

[0054] Step 4.4: Perform self-attention calculation on the 'query vector' q and the 'key-value pair vectors' k and v. The connection information between features of adjacent video frames is calculated using the deformable Transformer self-attention mechanism. While a regular Transformer calculates similarity across the entire video frame, this invention preferably uses a deformable Transformer, which differs from a regular Transformer in that it selects the vector with the highest similarity for attention calculation. The specific process of the deformable Transformer is as follows: the similarity relationship between the 'query vector' q and the 'key vector' k is calculated using softmax, and the 'value vector' v is calculated using distributed weights. The spatial attention weight matrix on the features of the previous and next frame images is obtained through multiple convolution calculations. The specific mathematical expression is as follows:

[0055]

[0056] Step 4.5: This can be obtained through step 4.4. and Spatial attention weight matrix as well as and Spatial attention weight matrix

[0057] Step 4.6: Obtain spatially enhanced video features using the attention weight fusion formula. Its expression is as follows:

[0058]

[0059] Repeat steps 2 to 4 to iterate through the local features of each frame of the video and perform upsampling.

[0060] Step 5: Feed all the enhanced video features into a deformable convolutional network for time alignment. The specific implementation of the deformable convolutional network can employ existing technologies, which will not be elaborated upon in this invention.

[0061] Step 6: Finally, the spatially enhanced and temporally aligned video features are fed into the super-resolution reconstruction module, which then outputs high-resolution video frames.

[0062] Step 6 includes:

[0063] Step 6.1: The key to the super-resolution reconstruction module is to construct a deformable Transformer-based video super-resolution model. The input low-resolution video sequence is reconstructed by this model and outputs a high-resolution video sequence, thus completing the video super-resolution reconstruction.

[0064] Step 6.2: Using Charbonnier Loss, construct the loss function L for the video super-resolution reconstruction model based on deformable Transformer. The specific formula is as follows:

[0065]

[0066] in, I represents a high-resolution video sequence from the original dataset. L This represents a high-resolution video sequence generated by a deformable Transformer-based video super-resolution model; ε is a constant to make the video super-resolution model trained on the deformable Transformer more stable.

[0067] Step 6.3: The constructed deformable Transformer-based video super-resolution model is trained using the original dataset and the training dataset, and evaluated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). The trained deformable Transformer-based video super-resolution model is then obtained, which can be used for the reconstruction output of high-resolution video frames.

[0068] Compared with existing methods, the video super-resolution method using the deformable Transformer provided by this invention yields better PSNR results and better subjective motion compensation.

[0069] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0070] In some possible embodiments, a deformable Transformer-based video super-resolution system is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a deformable Transformer-based video super-resolution method as described above.

[0071] In some possible embodiments, a deformable Transformer-based video super-resolution system is provided, including a readable storage medium on which a computer program is stored, which, when executed, implements a deformable Transformer-based video super-resolution method as described above.

[0072] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A video super-resolution method based on deformable Transformer, characterized in that: After collecting low-resolution video dataset samples in the field of video super-resolution, the following processing is performed: The low-resolution video training set samples are convolved to obtain video sequence feature maps; local features are extracted from the video sequence feature maps and downsampled; the downsampled local features are fed into a deformable Transformer, and the attention mechanism of the deformable Transformer extracts local features with high similarity to the local features in the video through vector similarity calculation and fuses them to enhance the spatial information of the local features. Repeat the above steps to iterate through the local features of each frame of the video and perform upsampling; then feed all the enhanced video features into a deformable convolutional network for time alignment. Finally, the spatially enhanced and temporally aligned video features are fed into the super-resolution reconstruction module, which then outputs high-resolution video frames. in, The downsampled local features are then fed into a deformable Transformer. The Transformer's attention mechanism uses vector similarity calculation to extract and fuse local features in the video that have high similarity to the obtained local features. The implementation method is as follows. Downsampled video frame features Adjacent video features , Perform cosine similarity calculation to obtain the vectors with the highest similarity; Video frame features The query vector q is obtained by convolving several vectors with the highest similarity. right Adjacent video features , Convolution is performed on the n vectors with the highest similarity to obtain the 'key-value pair vectors' k and v of the deformable Transformer attention mechanism; Based on the 'query vector' q and the 'key-value pair vectors' k and v, the connection information of features between adjacent video frames is calculated through the deformable Transformer self-attention mechanism, resulting in... and Spatial attention weight matrix as well as and Spatial attention weight matrix ; Spatially enhanced video features are obtained using the following attention weight fusion formula. , = + （ + ） The spatially enhanced and temporally aligned video features are fed into the super-resolution reconstruction module, which then outputs high-resolution video frames. The implementation method is as follows: A deformable Transformer-based video super-resolution model is constructed. The input low-resolution video sequence is processed by the model to output a high-resolution video sequence, thus completing the video super-resolution reconstruction. Charbonnier Loss is used to construct the loss function L for a video super-resolution reconstruction model based on deformable Transformer; The deformable Transformer-based video super-resolution model is trained using a dataset and evaluated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), ultimately yielding the trained deformable Transformer-based video super-resolution model.

2. The video super-resolution method based on deformable Transformer according to claim 1, characterized in that: The process of extracting local features from the video sequence feature map and then downsampling it is implemented as follows: Extracted video frame features Local features are extracted using several Swin-Transformer blocks; the extracted local feature maps are then downsampled. The downsampled local feature map is then processed again through several Swin-Transformer blocks to extract local features; the extracted local feature map is then downsampled to obtain the video local feature sequence. .

3. The video super-resolution method based on deformable Transformer according to claim 1, characterized in that: The implementation of the deformable Transformer involves calculating the similarity relationship between the 'query vector' q and the 'key vector' k using softmax, calculating the 'value vector' v with distributed weights, and obtaining the spatial attention weight matrix on the features of the previous and next frame images through multiple convolutions. .

4. A video super-resolution system based on a deformable Transformer, characterized in that: This method is used to implement a video super-resolution method based on a deformable Transformer as described in any one of claims 1-3.

5. The video super-resolution system based on deformable Transformer according to claim 4, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a video super-resolution method based on a deformable Transformer as described in any one of claims 1-3.

6. The video super-resolution system based on deformable Transformer according to claim 4, characterized in that: It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements a video super-resolution method based on a deformable Transformer as described in any one of claims 1-3.

Images