Satellite video blind super-resolution system and method based on inter-frame pixel-level information compensation
By extracting and utilizing inter-pixel-level information in satellite video blind super-resolution system, the resolution reduction problem caused by blur in satellite video is solved, and fine compensation for blurred pixels and high-quality super-resolution effects are achieved.
Patent Information
- Application Number
- CN202411071023.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-06
AI Technical Summary
In satellite video, the spatial resolution is reduced due to blur caused by jitter and atmospheric scattering, and the prior art is difficult to effectively achieve inter-frame pixel-level compensation, especially in complex backgrounds, lack of synchronous extraction and smoothing of local and global information features and pixels. compensate.
A satellite video blind super-resolution system with inter-frame pixel-level information compensation was designed. Through the preprocessing module, feature mining fusion module and blur compensation module, local and global information between frames are extracted and utilized to achieve fine compensation of blur pixels. . The system includes a preprocessing module to perform fuzzy estimation and frame alignment of video frames, a feature mining fusion module to extract attention features, and adjust intermediate features through a fuzzy compensation module to achieve high resolution reconstruction.
It effectively improves the super resolution quality of satellite video, suppresses the impact of complex backgrounds on the target, and takes into account the improvement of the overall spatial resolution of the image, especially with high degradation, it shows excellent compensation effect.
Smart Images

Figure CN118587096B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of satellite image processing, and in particular relates to a satellite video blind super-resolution system and method for inter-frame pixel-level information compensation. Background Art
[0002] In the remote sensing imaging process, the jitter of the satellite platform and the atmospheric scattering often produce unpredictable blur in the satellite video, which reduces the spatial resolution of the satellite video. Therefore, the quality of the satellite video is inevitably polluted, resulting in the degradation of the performance of subsequent applications in target tracking, change detection and other fields. Based on this, the video blind super-resolution technology for satellite remote sensing scenarios has received widespread attention. However, achieving high-quality video super-resolution in remote sensing scenarios is a challenging problem: on the one hand, compared with natural videos, satellite videos have more complex backgrounds and lack sufficient texture and detail information, which makes the extraction of inter-frame information in the video poor; on the other hand, there is a lack of fine pixel-level compensation strategies for degraded satellite videos, making it difficult to achieve fine compensation for blurred pixels in degraded videos.
[0003] As a mainstream method in the field of computer vision, deep learning technology also plays an important role in the field of blind super-resolution of satellite videos. In recent years, blind super-resolution methods for natural scenes have achieved great breakthroughs, such as USRNet, IKC, AdaTarget and other convolutional neural networks have been proposed one after another. However, due to the characteristics of satellite videos such as large data volume, rich information and complex image content, directly applying these algorithms to satellite video super-resolution tasks has poor results. Therefore, researchers have conducted a series of studies on blind super-resolution methods for satellite videos.
[0004] Liu et al. proposed a deep joint estimation network in the paper "Deep Joint Estimation Network for Satellite Video Super-Resolution With Multiple Degradations" to combine blur kernel estimation and super-resolution processes in a parallel manner. He et al. proposed a convolutional network model based on Ghost modules in the paper "Blind Superresolution of Satellite Videos by Ghost Module-Based Convolutional Networks" to estimate unknown blur kernels more flexibly. However, when faced with satellite videos with severe degradation and blur, they all focused on estimating blur kernels and ignored the compensation of pixel-level information between frames. To solve this problem, Xiao et al. proposed a blind super-resolution network BSVSR for temporal compensation in the paper "Deep Blind Super-Resolution for Satellite Video" to explore sharper clues by considering the blur level at the pixel level from coarse to fine. However, this method lacks the synchronous extraction of local information and global information features between frames, only considers blurry pixels and ignores the compensation of smooth pixels, pays more attention to local areas in the image, and lacks correlation with static or uniform areas. Therefore, a more efficient method is needed to fully utilize the pixel-level information between frames to achieve high-quality video super-resolution effects. Summary of the invention
[0005] In view of this, the invention aims to provide a satellite video blind super-resolution system and method for inter-frame pixel-level information compensation, which uses a feature mining and fusion module to extract local and global information between frames at the pixel level and obtain clean intermediate features; in order to enable the intermediate features to better perceive the fuzzy features, a fuzzy compensation module is designed to adjust the intermediate features, thereby gradually achieving fine compensation for fuzzy pixels. Starting from local and global pixel-level information compensation, the present invention can better extract and utilize inter-frame information, thereby achieving high-quality super-resolution of satellite videos.
[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows:
[0007] A satellite video blind super-resolution system for inter-frame pixel-level information compensation comprises a preprocessing module, a feature mining and fusion module, a blur compensation module and an image reconstruction module; wherein the preprocessing module performs blur estimation and frame alignment on an input video frame sequence, the alignment features after frame alignment are input into the feature mining and fusion module for extracting attention features, the blur features generated by blur estimation are spliced with the attention features to obtain fusion features; the fusion features are input into the blur compensation module for blur compensation to obtain blur compensation features; the blur compensation features are input into the image reconstruction module to obtain reconstruction features, the reconstruction features are added to the corresponding elements of the bilinear interpolation results of the target frame in the video frame sequence to obtain a high-resolution video frame corresponding to the target frame.
[0008] Furthermore, the preprocessing module includes a parallel blur estimation submodule and a frame alignment submodule; wherein, the target frame in the video frame sequence is processed by the blur estimation submodule to obtain blur features; and the video frame sequence is processed by the frame alignment submodule to obtain alignment features.
[0009] Furthermore, the blur estimation submodule consists of a first fully connected layer and a second fully connected layer, the first fully connected layer is followed by a ReLU activation function, and the second fully connected layer is followed by a Softmax function; the target frame in the video frame sequence passes through the first fully connected layer and the second fully connected layer in sequence to obtain blur features.
[0010] Furthermore, the number of frame alignment submodules is equal to the number of video frames in the video frame sequence minus 1, and the frame alignment submodules correspond one-to-one to the other video frames in the video frame sequence except the target frame; in each frame alignment submodule, the video frame corresponding to the frame alignment submodule and the target frame in the video frame sequence are subjected to residual operation to extract features, and the first feature and the second feature are respectively obtained; the first feature and the second feature are spliced and fused to obtain the first fused feature; the first fused feature is simultaneously subjected to convolution processing of convolution layers with no less than 3 different convolution kernels to obtain three features; the three features are then spliced and fused to obtain the second fused feature; the corresponding elements of the first fused feature and the second fused feature are added to obtain the deviation between the target frame and the video frame input at the same time; the deviation and the second feature are simultaneously input into the variable convolution layer to obtain the common preliminary alignment feature of the target frame and the video frame input at the same time; the preliminary alignment features output by all frame alignment submodules are spliced and fused to obtain the alignment feature.
[0011] Furthermore, the feature mining fusion module includes a hybrid attention submodule and a window self-attention submodule, and the alignment features are simultaneously input into the hybrid attention submodule and the window self-attention submodule; wherein, the alignment features are processed by the window self-attention submodule to obtain the self-attention features; the alignment features are processed by the hybrid attention submodule to obtain the hybrid attention features; and the attention features are obtained by adding the corresponding elements of the self-attention features and the hybrid attention features.
[0012] Furthermore, in the hybrid attention submodule, an efficient channel attention operation is performed on the alignment feature to obtain the channel attention weight; the alignment feature is multiplied by the corresponding element of the channel attention weight to obtain the channel attention feature; the spatial attention operation is performed on the channel attention feature to obtain the spatial attention weight; the channel attention feature is multiplied by the corresponding element of the spatial attention weight to obtain the hybrid attention feature.
[0013] Furthermore, in the blur compensation module,
[0014] Perform the maximum pooling operation and the average pooling operation on the fused features at the same time, and then concatenate and fuse the features after the two pooling operations to obtain the first pooling feature;
[0015] Performing maximum pooling and average pooling operations on the first pooling feature at the same time, and then concatenating and fusing the features after the two pooling operations to obtain the second pooling feature;
[0016] The second pooling feature is upsampled, and the upsampled feature is added to the corresponding element of the first pooling feature; the added feature is then upsampled by bicubic interpolation to obtain the primary blur compensation feature;
[0017] Perform Sigmoid activation operation on the primary fuzzy compensation feature, and multiply the result of the Sigmoid activation operation with the corresponding element of the fusion feature to obtain the secondary fuzzy compensation feature;
[0018] The fusion feature processed by the hybrid attention submodule is multiplied by the secondary blur compensation feature and the corresponding element to obtain the blur compensation feature.
[0019] Furthermore, the image reconstruction module includes a convolution layer and a sub-pixel convolution layer; wherein, the blur compensation features are processed by the convolution layer and the sub-pixel convolution layer in sequence to obtain reconstructed features, and the reconstructed features are added to the corresponding elements of the bilinear interpolation results of the target frame in the video frame sequence to obtain a high-resolution video frame corresponding to the target frame.
[0020] A satellite video blind super-resolution method for inter-frame pixel-level information compensation specifically comprises the following steps:
[0021] S1: Obtain video data, preprocess the video data, obtain training video data, and form a training video data set; then divide each training video data into n groups of video frame sequences along the time axis, and the number of video frames in each video frame sequence is an odd number not less than 3, and the middle frame in each video frame sequence is the target frame to be super-resolution;
[0022] S2: Constructing a satellite video blind super-resolution system for inter-frame pixel-level information compensation as provided by the present invention;
[0023] S3: input the grouped video frame sequence into the blind super-resolution system in step S2, train the blind super-resolution system, and obtain a trained blind super-resolution network model;
[0024] S4: Use the blind super-resolution network model obtained in step S3 to perform super-resolution reconstruction on the video image to be super-resolved.
[0025] Furthermore, in step S1, preprocessing the video data specifically includes:
[0026] S11: capturing the original resolution video at the same position of all video data;
[0027] S12: Degrade each frame of the original resolution video using an isotropic Gaussian blur kernel with random width to obtain a corresponding low-resolution video;
[0028] S13: Perform flipping and rotating data enhancement operations on the low-resolution video and the corresponding original-resolution video, and obtain the data-enhanced low-resolution video and the corresponding original-resolution video as training video data, and constitute a training video data set.
[0029] Compared with the prior art, the invention can achieve the following beneficial effects:
[0030] (1) The present invention creates a satellite video blind super-resolution system and method based on inter-frame pixel-level information compensation, which uses a feature mining and fusion module to extract local and global information between frames at the pixel level and obtain clean intermediate features, while suppressing the influence of complex background on targets of different scales and taking into account the improvement of the overall spatial resolution of the image;
[0031] (2) In the satellite video blind super-resolution system and method based on inter-frame pixel-level information compensation described in the present invention, in order to enable the intermediate features to better perceive the blurred features, a blur compensation module is designed to adjust the intermediate features, thereby gradually achieving fine compensation for the blurred pixels. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:
[0033] Figure 1 A schematic diagram of a satellite video blind super-resolution system for inter-frame pixel-level information compensation according to an embodiment of the present invention;
[0034] Figure 2A schematic diagram of a frame alignment submodule according to an embodiment of the present invention;
[0035] Figure 3 A schematic diagram of a hybrid attention submodule according to an embodiment of the present invention;
[0036] Figure 4 A schematic diagram of the self-attention operation described in the embodiment of the present invention;
[0037] Figure 5 A flowchart of a satellite video blind super-resolution method for inter-frame pixel-level information compensation according to an embodiment of the present invention is provided. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.
[0039] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0040] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.
[0041] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.
[0042] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.
[0043] like Figures 1 to 4 As shown, the satellite video blind super-resolution system with inter-frame pixel-level information compensation described in the present invention comprises a preprocessing module, a feature mining and fusion module, a blur compensation module and an image reconstruction module. The preprocessing module performs blur estimation and frame alignment on the video frame sequence, and the alignment features after frame alignment are input into the feature mining and fusion module for attention feature extraction. The fuzzy features generated by fuzzy estimation Attention characteristics Perform splicing and fusion to obtain fusion features . Fusion features Input the fuzzy compensation module to perform fuzzy compensation and obtain the fuzzy compensation feature . Fuzzy compensation feature The reconstructed features are obtained in the input image reconstruction module, and the reconstructed features are added to the corresponding elements of the bilinear interpolation results of the target frame in the video frame sequence to obtain a high-resolution video frame corresponding to the target frame.
[0044] It should be noted that all the "splicing and fusion" operations mentioned here and below are to first splice multiple input features in the channel direction, and then input the spliced features into the convolution layer for convolution operation to obtain the spliced and fused features.
[0045] The preprocessing module includes a parallel fuzzy estimation submodule and a frame alignment submodule. The target frame in the video frame sequence is processed by the fuzzy estimation submodule to obtain the fuzzy feature ,The video frame sequence is processed by the frame alignment submodule to obtain the alignment features.
[0046] The blur estimation submodule consists of the first fully connected layer and the second fully connected layer. The first fully connected layer is followed by a ReLU activation function, and the second fully connected layer is followed by a Softmax function. The target frame in the video frame sequence passes through the first fully connected layer and the second fully connected layer in turn to obtain blur features. .
[0047] The number of frame alignment submodules is equal to the number of video frames in the video frame sequence minus 1, and the frame alignment submodules correspond to the other video frames in the video frame sequence except the target frame. In the embodiment of the present invention, since the video frame sequence in the embodiment of the present invention includes video frames at 5 consecutive moments, namely, video frame I t-2 , video frame I t-1 , target frame I t , video frame I t+1 And video frame I t+2Therefore, the number of frame alignment submodules is 4, where the input of the first frame alignment submodule is the target frame I t and video frame I t-2 , that is, the first frame alignment submodule and the video frame I t-2 Corresponding; the input of the second frame alignment submodule is the target frame I t and video frame I t-1 , that is, the second frame alignment submodule and the video frame I t-1 Corresponding; The input of the third frame alignment submodule is the target frame I t and video frame I t+1 , that is, the third frame alignment submodule and the video frame I t+1 Correspondingly; the input of the fourth frame alignment submodule is the target frame I t and video frame I t+2 , that is, the fourth frame alignment submodule and the video frame I t+2 correspond.
[0048] The frame alignment submodule in the embodiment of the present invention is derived from the paper "Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection" by Xiao et al. Figure 2 As shown, in each frame alignment submodule, the video frame corresponding to the frame alignment submodule (in the form of video frame I t-1 For example) and the target frame I t After extracting features through residual operation, the first feature and the second feature are obtained respectively. The first feature and the second feature are concatenated and fused to obtain the first fused feature. The first fused feature is convolved by at least three convolution layers with different convolution kernels to obtain three features. The three features are then concatenated and fused to obtain the second fused feature. The corresponding elements of the first fused feature and the second fused feature are added to obtain the target frame I t With video frame I t-1 The deviation between t-1 . The deviation θ t-1 The first feature is input into the variable convolution layer together with the second feature to obtain the common preliminary alignment feature of the target frame and the video frame input at the same time. The preliminary alignment features output by all frame alignment submodules are concatenated and fused to obtain the alignment feature.
[0049] In an embodiment of the present invention, the residual operation includes a convolution operation and three residual operations performed in sequence, wherein a PReLU activation function operation is required after the convolution operation. The three residual operations are all implemented by two convolution layers and a PReLU activation function between the two convolution layers. During each residual operation, the input features are sequentially processed by the convolution layer, the PReLU activation function and the convolution layer, and the output features are then added to the corresponding elements of the input features to obtain the final output features. In addition, it is preferred that the first fusion feature is simultaneously processed by convolution of three convolution layers with different convolution kernels to obtain three features, and the convolution kernel sizes of the three convolution layers are 3×3, 5×5 and 7×7, respectively.
[0050] like Figure 3 As shown in the figure, the feature mining fusion module includes a hybrid attention submodule and a window self-attention submodule. The alignment features are processed by layer normalization and input into the hybrid attention submodule and the window self-attention submodule at the same time. After the alignment features are processed by the window self-attention submodule, the self-attention feature T is obtained. After the alignment features are processed by the hybrid attention submodule, the hybrid attention feature S is obtained. The attention feature is obtained by adding the corresponding elements of the self-attention feature T and the hybrid attention feature S. .
[0051] In the hybrid attention submodule, an efficient channel attention operation is performed on the alignment features of feature size C×H×W (C represents the number of channels of the feature, H represents the height of the feature, and W represents the width of the feature) to obtain a channel attention weight of size C×1×1. After multiplying the alignment features with the corresponding elements of the channel attention weight, a channel attention feature M of feature size C×H×W is obtained. A spatial attention operation is performed on the channel attention feature M to obtain a spatial attention weight of feature size 1×H×W. After multiplying the channel attention feature with the corresponding elements of the spatial attention weight, a hybrid attention feature S of feature size C×H×W is obtained.
[0052] The efficient channel attention operation comes from the paper "ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks" by Wang et al. The input aligned features are adaptively averaged pooled along the channel direction to obtain features with a feature size of C×1×1 ,feature After the convolution layer (convolution kernel size k=3), the output features are activated by the Sigmoid function to obtain the channel attention weight. After multiplying the alignment features with the corresponding elements of the channel attention weight, the channel attention feature M with a feature size of C×H×W is obtained.
[0053] The spatial attention operation comes from the paper "CBAM: Convolutional Block Attention Module" by Woo et al. In the spatial attention operation, the input channel attention feature M is subjected to maximum pooling and average pooling along the spatial dimension to generate features respectively. and Features , the size of both features is 1×H×W. and Features After concatenation and fusion, the Sigmoid activation function is performed to obtain the spatial attention weight. Multiply the spatial attention weight with the corresponding element of the channel attention feature M to obtain the mixed attention feature S. The process of obtaining the mixed attention feature S from the channel attention feature M can be expressed as follows:
[0054]
[0055] in, represents splicing along the channel dimension, represents convolution and Activation function operation, Represents the multiplication of corresponding elements.
[0056] The window self-attention submodule comes from the paper "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" by Liu et al. In the window self-attention submodule, the input alignment feature is divided into n×n windows, and the feature size obtained by the division is C×H / n×W / n. The feature is extended to a feature size of C×1×(HW / ). Then, the feature is subjected to self-attention operation at least 2 times simultaneously. In the embodiment of the present invention, it is preferred to perform self-attention operation 6 times simultaneously. Figure 4 Three of these are shown.
[0057] like Figure 4 As shown in the figure, in each self-attention operation, the channel dimension C of the feature is first expanded to 3×C, and the feature is divided into Q matrix, K matrix and V matrix along the channel dimension. After the Q matrix, K matrix and V matrix are mapped by the linear layer, the scaled dot product attention operation is performed. The scaled dot product attention operation specifically includes: multiplying the Q matrix and the V matrix to obtain the score matrix; scaling the score matrix, that is, dividing it by the square root of the vector dimension; performing the softmax function calculation on the scaled score matrix to obtain the attention weight matrix; finally, multiplying the V matrix by the attention weight matrix to obtain the weighted matrix. The process of each self-attention operation can be expressed by the following formula:
[0058]
[0059] in, represents the weighted matrix, Represents the dimension of the matrix K. By concatenating the weighted matrices output by all self-attention operations and mapping them in the linear layer, we can obtain the self-attention feature T with a feature size of C×H×W.
[0060] The attention feature is obtained by adding the corresponding elements of the self-attention feature T and the mixed attention feature S. ,Right now:
[0061]
[0062] in, Indicates the addition of corresponding elements, at this time the attention feature The characteristic size of is also C×H×W.
[0063] The blur compensation module provided by the embodiment of the present invention adopts a multi-layer iterative pyramid structure to gradually realize the attention feature For fuzzy features perception and compensation.
[0064] Before entering the blur compensation module, the attention feature With the fuzzy feature of feature size C×H×W Perform splicing and fusion to obtain a fusion feature with a feature size of C×H×W ,Right now:
[0065]
[0066] in, Represents a convolution operation.
[0067] In the blur compensation module, the fusion features Perform 2x max pooling and average pooling at the same time to obtain max pooling features of size C×H / 2×W / 2 and average pooling features . The maximum pooling feature and average pooling features The first pooling feature with a feature size of C×H / 2×W / 2 is obtained by splicing and fusion. ,Right now:
[0068]
[0069] in, Represents the first pooling feature The characteristic size is C×H / 2×W / 2.
[0070] For the first pooling feature Perform 2 times the maximum pooling operation and average pooling operation at the same time, and obtain the maximum pooling feature of size C×H / 4×W / 4 and average pooling features , and then the maximum pooling feature and average pooling features Perform splicing and fusion to obtain the second pooling feature ,Right now:
[0071]
[0072] in, Represents the second pooling feature The characteristic size is C×H / 4×W / 4.
[0073] The second pooling feature Perform bicubic interpolation upsampling to double the width and height of the feature, and combine the upsampled feature with the first pooled feature The corresponding elements are added; then the added features are upsampled by bicubic interpolation to obtain the primary blur compensation features ; For primary fuzzy compensation features Perform the Sigmoid activation function operation and combine the result of the Sigmoid activation function operation with the fusion feature Multiply the corresponding elements to get the secondary blur compensation feature , the size of this feature is C×H×W. The fusion feature Features processed by the hybrid attention submodule and secondary blur compensation features After adding the corresponding elements, the fuzzy compensation feature is obtained ,Right now:
[0074]
[0075] in, Represents the fuzzy compensation feature The characteristic dimensions are C×H×W, represents a bicubic interpolation upsampling operation with a scale factor of 2, Represents the Sigmoid activation function operation, Represents the operation of the hybrid attention submodule.
[0076] A residual structure is introduced to the output end of the pyramid structure, and a mixed attention submodule is introduced again in the residual structure, which helps to integrate the C×H×W features. Attention Features Assign more weights to obtain the fuzzy compensation features of the final output .
[0077] The image reconstruction module uses a lightweight super-resolution reconstruction network, which is implemented by convolutional layers and sub-pixel convolutional layers. First, a convolutional layer with a 3×3 convolution kernel is used to compensate the blur of the input features. The number of channels is expanded (r represents the multiple of super resolution), and then the sub-pixel convolution layer is used to The feature maps of the channels are rearranged so that the output reconstructed features are r times larger than the size of the target frame of the input video frame sequence. Finally, the corresponding elements of the bilinear interpolation results of the reconstructed features and the target frame in the video frame sequence are added to obtain the high-resolution video frame corresponding to the target frame, thereby realizing the reconstruction process. For the sub-pixel convolution layer, this is a classic upsampling method. In dealing with super-resolution problems, the reduced feature map can be effectively enlarged. It has been widely used in computer vision problems such as image segmentation. In addition, this method and deconvolution have become the two most commonly used upsampling techniques in neural networks. In comparison, pixel reorganization overcomes the problem of easy generation of checkerboards in deconvolution and is currently the best performing general upsampling technology.
[0078] Based on the satellite video blind super-resolution system for inter-frame pixel-level information compensation provided by the embodiment of the present invention, the embodiment of the present invention also provides a satellite video blind super-resolution method for inter-frame pixel-level information compensation, such as Figure 5 As shown, the specific steps include:
[0079] S1: Obtain video data, pre-process the video data, obtain training video data, and form a training video data set; then divide each piece of training video data into n groups of video frame sequences along the time axis, and the number of video frames in each video frame sequence is an odd number not less than 3, and the middle frame in each video frame sequence is the target frame that needs super resolution. In the embodiment of the present invention, the number of video frames in each video frame sequence is 5 frames, and the middle frame in each video frame sequence is the target frame that needs super resolution.
[0080] The video data in the embodiment of the present invention comes from the dataset Jilin-189, which comes from the paper "Satellite Video Super-Resolution via Multiscale Deformable Convolution Alignment and Temporal Grouping Projection" by Xiao et al., and includes video data of 10 scenes captured by the Jilin-1 satellite.
[0081] In step S1, preprocessing the video data specifically includes:
[0082] S11: Capturing the original resolution video at the same position of all video data.
[0083] Among them, the video data of 8 scenes in the dataset Jilin-189 are selected. The first 100 frames of video frames with a size of 640×640 are intercepted at a random position in the video data of each scene to obtain a video. In the video data of the same scene, the interception position is changed, and the first 100 frames of video frames with a size of 640×640 are intercepted at the current interception position to obtain a video. The overlap rate between two videos in the same scene is 25%. A total of 189 videos are obtained by the same operation in the video data of 8 scenes. The resolution of these 189 videos is 640×640, and the number of video frames is 100, that is, the original resolution video.
[0084] S12: Degrade each frame of the original resolution video using an isotropic Gaussian blur kernel with random width to obtain a corresponding low-resolution video.
[0085] S13: Perform flipping and rotating data enhancement operations on the low-resolution video and the corresponding original-resolution video, and obtain the data-enhanced low-resolution video and the corresponding original-resolution video as training video data, and constitute a training video data set.
[0086] After completing the preprocessing operation on the video data to obtain the training video data, each training video data is divided into n groups of video frame sequences along the time axis. In the embodiment of the present invention, each group of video frame sequences is 5 frames, and the middle frame is selected as the target frame to be super-resolution, denoted as I t , then all video frames in each group of video frame sequences can be represented as I t-2 , I t-1 , I t , I t+1 , I t+2 .
[0087] S2: Constructing a satellite video blind super-resolution system with inter-frame pixel-level information compensation as provided in an embodiment of the present invention.
[0088] S3: Input the grouped video frame sequence into the blind super-resolution system in step S2, train the blind super-resolution system, and obtain a trained blind super-resolution network model.
[0089] In step S3, the fuzzy estimation submodule needs to be pre-trained first, so as to train the fuzzy estimation submodule to produce a fuzzy kernel close to the actual situation. The data set used for pre-training is also a training video data set.
[0090] The pre-training process is as follows: first, an isotropic Gaussian blur kernel with a random blur kernel width is given, which is the initial blur kernel. The high-resolution image is degraded by the initial blur kernel to obtain a low-resolution image, and then the low-resolution target frame in the video frame sequence is input into the blur estimation submodule to obtain the corresponding blur feature. At this time, the blur kernel estimated in the blur estimation submodule is extracted, and the blur estimation submodule is optimized by comparing the gap between the estimated blur kernel and the initial blur kernel until an estimated blur kernel that fits the initial blur kernel is obtained. During the pre-training process, the Adam optimizer is used, and the parameters in the Adam optimizer are , as well as ; The L1 loss function is used to calculate the gap between the estimated blur kernel and the initial blur kernel. The expression of the L1 loss function is:
[0091]
[0092] Where N represents the number of training set inputs, which here represents the number of low-resolution target frames in the video frame sequence. represents the true value, here represents the initial blur kernel, Indicates the estimated value obtained in each training, here it indicates the estimated blur kernel obtained in the training. The learning rate is set to 1e-4, the number of epochs is set to 30, and the number of batchsizes for each training is set to 16.
[0093] After the blur estimation submodule completes pre-training, it now has the ability to obtain a blur kernel that is close to the actual situation. Applying the blur estimation submodule to the blind super-resolution system can obtain more accurate blur features, laying the foundation for better compensating for the blur feature by subsequently mining information between adjacent frames.
[0094] After completing the pre-training of the blur estimation submodule, the overall training of the blind super-resolution system is started with the low-resolution video as input and the corresponding original-resolution video as the true value. In the overall training, the Adam optimizer is also used, and the parameters in the Adam optimizer are also , as well as ; The L1 loss function is also used to calculate the difference between the high-resolution video frame reconstructed in each training and the corresponding video frame in the original resolution video. The learning rate is also set to 1e-4, the number of epochs is changed to 100, and the number of batchsizes per training is changed to 8.
[0095] S4: Use the blind super-resolution network model obtained in step S3 to perform super-resolution reconstruction on the video image to be super-resolved.
[0096] In the embodiment of the present invention, 12 video clips are obtained from the video data of the remaining two scenes in the dataset Jilin-189, and the method of obtaining the video clips is the same as the method of obtaining the 189 video clips in the training video dataset. Then, a test video dataset is obtained from the 12 video clips, and the method of obtaining the test video dataset is the same as the method of obtaining the training video dataset.
[0097] In order to clearly demonstrate the superiority of the satellite video blind super-resolution method based on inter-frame pixel-level information compensation provided by the embodiment of the present invention, the current advanced DBVSR method (from Pan et al.'s paper "Deep Blind Video Super-resolution") and BSVSR method are selected in the embodiment of the present invention to compare the method provided by the present invention with 4 times super-resolution, and the test results of the three networks are compared. In the embodiment of the present invention, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation indicators. Among them, peak signal-to-noise ratio is the most common and widely used objective image evaluation indicator, which is used to measure the quality of compressed images. The unit is usually expressed in decibels (dB). The higher the index, the better the image quality; structural similarity is a full-reference image quality evaluation index, which measures image similarity from three aspects: brightness, contrast, and structure. Its value range is [0,1]. The larger the index value, the smaller the degree of image distortion, indicating that the image quality is better.
[0098] The comparison results are shown in Table 1, where: Represents the width of the blur kernel, different Represents different levels of degradation of video frames, It can be seen that no matter the degradation degree is 1.2 or 2.0, the super-resolution effect of the method provided by the present invention is better than that of the DBVSR method and the BSVSR method, and the higher the degradation degree, the better the super-resolution effect of the method provided by the present invention.
[0099] Table 1:
[0100]
[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.
[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A satellite video blind super-resolution system with inter-frame pixel-level information compensation, characterized by: It includes a preprocessing module, a feature mining and fusion module, a fuzzy compensation module and an image reconstruction module; wherein the preprocessing module includes a parallel fuzzy estimation submodule and a frame alignment submodule, the alignment features after frame alignment are input into the feature mining and fusion module to extract attention features, the fuzzy features generated by fuzzy estimation are spliced with the attention features to obtain fusion features; the fusion features are input into the fuzzy compensation module to perform fuzzy compensation to obtain fuzzy compensation features; the fuzzy compensation features are input into the image reconstruction module to obtain reconstruction features, the reconstruction features are added to the corresponding elements of the bilinear interpolation results of the target frame in the video frame sequence to obtain a high-resolution video frame corresponding to the target frame; The feature mining fusion module includes a hybrid attention submodule and a window self-attention submodule, and the alignment feature is simultaneously input into the hybrid attention submodule and the window self-attention submodule; wherein, after the alignment feature is processed by the window self-attention submodule, a self-attention feature is obtained; after the alignment feature is processed by the hybrid attention submodule, a hybrid attention feature is obtained; and the self-attention feature and the hybrid attention feature are added together by corresponding elements to obtain the attention feature; In the blur compensation module, Performing a maximum pooling operation and an average pooling operation on the fused features simultaneously, and then concatenating and fusing the features after the two pooling operations to obtain a first pooling feature; Performing a maximum pooling operation and an average pooling operation on the first pooling feature at the same time, and then concatenating and fusing the features after the two pooling operations to obtain a second pooling feature; Performing an upsampling operation on the second pooled feature, and adding the upsampled feature to the corresponding element of the first pooled feature; and then performing a bicubic interpolation upsampling operation on the added feature to obtain a primary blur compensation feature; Performing a Sigmoid activation operation on the primary fuzzy compensation feature, and multiplying the result of the Sigmoid activation operation by the corresponding element of the fusion feature to obtain a secondary fuzzy compensation feature; The fusion feature processed by the hybrid attention submodule is multiplied by the secondary fuzzy compensation feature and the corresponding element to obtain the fuzzy compensation feature.
2. The satellite video blind super-resolution system with inter-frame pixel-level information compensation according to claim 1, characterized in that: The target frame in the video frame sequence is processed by the blur estimation submodule to obtain the blur feature; The video frame sequence is processed by the frame alignment submodule to obtain the alignment feature.
3. The satellite video blind super-resolution system with inter-frame pixel-level information compensation according to claim 2, characterized in that: The blur estimation submodule consists of a first fully connected layer and a second fully connected layer, the first fully connected layer is followed by a ReLU activation function, and the second fully connected layer is followed by a Softmax function; the target frame in the video frame sequence passes through the first fully connected layer and the second fully connected layer in sequence to obtain the blur feature.
4. The satellite video blind super-resolution system with inter-frame pixel-level information compensation according to claim 2, characterized in that: The number of the frame alignment submodules is equal to the number of video frames in the video frame sequence minus 1, and the frame alignment submodules correspond to the other video frames in the video frame sequence except the target frame one by one; in each frame alignment submodule, the video frame corresponding to the frame alignment submodule and the target frame in the video frame sequence are subjected to residual operation to extract features, and a first feature and a second feature are obtained respectively; the first feature and the second feature are spliced and fused to obtain a first fused feature; the first fused feature is simultaneously subjected to convolution processing of convolution layers with no less than 3 different convolution kernels to obtain three features; the three features are then spliced and fused to obtain a second fused feature; the first fused feature and the corresponding elements of the second fused feature are added to obtain the deviation between the target frame and the video frame input at the same time; the deviation and the second feature are simultaneously input into the variable convolution layer to obtain a common preliminary alignment feature of the target frame and the video frame input at the same time; The alignment features are obtained by concatenating and fusing the preliminary alignment features output by all frame alignment submodules.
5. The satellite video blind super-resolution system with inter-frame pixel-level information compensation according to claim 1, characterized in that: In the hybrid attention submodule, an efficient channel attention operation is performed on the alignment feature to obtain a channel attention weight; After multiplying the alignment feature with the corresponding element of the channel attention weight, the channel attention feature is obtained; performing a spatial attention operation on the channel attention feature to obtain the spatial attention weight; after multiplying the channel attention feature with the corresponding element of the spatial attention weight, the mixed attention feature is obtained.
6. The satellite video blind super-resolution system with inter-frame pixel-level information compensation according to claim 1, characterized in that: The image reconstruction module includes a convolution layer and a sub-pixel convolution layer; wherein the blur compensation feature is processed by the convolution layer and the sub-pixel convolution layer in sequence to obtain the reconstructed feature, and the reconstructed feature is added to the corresponding elements of the bilinear interpolation result of the target frame in the video frame sequence to obtain a high-resolution video frame corresponding to the target frame.
7. A satellite video blind super-resolution method with inter-frame pixel-level information compensation, characterized in that: The specific steps include: S1: Obtain video data, preprocess the video data, obtain training video data, and form a training video data set; then divide each training video data into n groups of video frame sequences along the time axis, and the number of video frames in each video frame sequence is an odd number not less than 3, and the middle frame in each video frame sequence is the target frame to be super-resolution; S2: Constructing a satellite video blind super-resolution system for inter-frame pixel-level information compensation as described in any one of claims 1 to 6; S3: inputting the grouped video frame sequence into the blind super-resolution system in step S2, training the blind super-resolution system, and obtaining a trained blind super-resolution network model; S4: Use the blind super-resolution network model obtained in step S3 to perform super-resolution reconstruction on the video image to be super-resolved.
8. The satellite video blind super-resolution method with inter-frame pixel level information compensation according to claim 7 is characterized in that: In step S1, preprocessing the video data specifically includes: S11: capturing the original resolution video at the same position of all video data; S12: Degrade each frame of the original resolution video using an isotropic Gaussian blur kernel with random width to obtain a corresponding low-resolution video; S13: Performing data enhancement operations of flipping and rotating the low-resolution video and the corresponding original-resolution video to obtain the data-enhanced low-resolution video and the corresponding original-resolution video, which are the training video data and constitute the training video data set.
Citation Information
Patent Citations
Unmanned aerial vehicle motion blurred image restoration method based on deep learning
CN110648291A
Method of reconstruction of super-resolution of video frame
US20220261959A1
