A satellite video super-resolution reconstruction method and system for multiple degradation processes

Abstract

The present invention provides a satellite video super-resolution reconstruction method and system for multiple degradation processes. The whole process of the method is as follows: the satellite video random degradation module is used to generate corresponding low-resolution video data, and then the feature extraction module is used to extract features of the low-resolution data. The extracted features use a high-order grid bidirectional propagation scheme to transmit the features of each frame of the entire video sequence, and the feature alignment module is used to achieve feature alignment of the video data. Finally, the aligned features are pixel reorganized and added to the up-sampled low-resolution features to obtain a reconstructed high-resolution image. Experiments with three types of satellite video data show that the present invention can effectively restore the high fidelity of satellite videos, significantly improve the reconstruction quality of remote sensing images, and also has a certain denoising capability.

Description

Technical Field

[0001] The invention belongs to the field of high-resolution optical satellite video image processing, and in particular relates to a satellite video super-resolution reconstruction method and system for multiple degradation processes. Background Art

[0002] With the rapid development of remote sensing technology, users can more easily obtain high spatial resolution and high temporal resolution earth observation remote sensing data, and are no longer satisfied with traditional static earth observation data. Video satellite is a new type of earth observation satellite developed in recent years. Compared with traditional earth observation satellites, it can "stare" at a specific target for a long time and capture time-series images of specific targets at a certain frame rate. Compared with traditional static remote sensing images, satellite video data not only has sub-meter high spatial resolution, but also has video-level high temporal resolution, which can provide continuous information of specific targets and realize high-temporal and spatial dynamic earth observation. It is widely used in dynamic scene applications such as change detection, target tracking, and traffic monitoring. Therefore, the processing research of video satellite data is of great significance for land monitoring, disaster relief, national defense security, smart cities, etc. However, the spatial resolution of satellite video data is affected by complex environmental factors in imaging and data transmission, such as atmospheric scattering, sensor vibration, data compression, etc. The high-frequency information in satellite video data may be lost, resulting in blurring, which greatly reduces the performance of subsequent applications. Therefore, improving the spatial resolution of satellite video is crucial for human perception and the realization of downstream tasks. Compared with improving resolution through hardware methods, super-resolution reconstruction technology can greatly save satellite maintenance, transmission and storage costs.

[0003] Super-resolution reconstruction is a very classic underlying task in computer vision. It uses low-resolution images to reconstruct high-resolution images to improve image quality. Video super-resolution reconstruction requires the resolution of each frame in the video and puts forward requirements for image consistency. Broadly speaking, video super-resolution reconstruction can be regarded as an extension of image super-resolution reconstruction, which can be processed frame by frame using a single image super-resolution reconstruction algorithm. However, in practice, the results of processing videos using image super-resolution reconstruction algorithms are difficult to be satisfactory. This is because the temporal information is not considered, which brings artifacts and lags. Videos have richer information and have an additional temporal dimension relative to images, so it is more challenging to design super-resolution reconstruction algorithms for videos. In order to make better use of this temporal information, researchers often introduce frame alignment to eliminate the influence of object or background motion in the video. Compared with natural scene videos, first, satellite videos have lower spatial resolution and lack more texture information; second, the visual range of satellite videos is much larger than that of natural scene videos, the scenes are more diverse, and the information density is greater; third, the scale of moving objects in satellite videos changes and there are more complex motion changes. Therefore, although deep learning-based video super-resolution reconstruction has made great progress in recent years, these general methods are not suitable for direct application to satellite videos, and more efforts are needed to develop a method suitable for satellite video super-resolution reconstruction.

[0004] There are three main ways to obtain low-resolution data: 1) Based on degradation assumptions such as bicubic degradation: synthesize low-resolution images from corresponding high-resolution images. However, when the test low-resolution image does not meet the degradation assumption, the performance of the trained super-resolution reconstruction model will drop sharply. 2) Create low-resolution-high-resolution image pairs from real remote sensing images, where the low-resolution and high-resolution images are collected at the same location. However, this method has some problems, such as land cover changes and spectral gaps between the two images. To address the above problems, a third method using unpaired low-resolution-high-resolution images to train super-resolution reconstruction models is proposed. These methods usually utilize a downgrader and a generator, where the low-resolution image is first processed by the generator, and then the generator can be trained in a self-supervised manner. This makes it possible to use unpaired low-resolution-high-resolution images, making it easier to collect training data. However, due to the domain gap between low-resolution and high-resolution images and the lack of intermediate supervision, the training of these methods is usually unstable. For the application of remote sensing image super-resolution reconstruction in the real remote sensing world, recent studies have explored blind super-resolution considering various degradations in real scenes. AWGN introduces anisotropic Gaussian blur kernel and additive Gaussian white noise to synthesize and train low-resolution-high-resolution image pairs in a blind super-resolution manner. However, the degradation model based on the anisotropic Gaussian blur assumption is too simple in actual scenarios, which limits its application.

[0005] In addition, the alignment of images or features of adjacent frames is a key and difficult problem in the video super-resolution reconstruction algorithm. Due to the movement of ground objects such as vehicles, aircraft, and ships, as well as the change in perspective caused by the movement of the satellite itself, direct fusion will introduce errors and lead to a decrease in effect. Therefore, most current methods introduce an alignment operation, which helps to accurately find the missing information in adjacent frames. There are two types of alignment operations: image alignment and feature alignment. The image alignment method usually uses the optical flow method, which works well for small-scale motion (such as the movement of ground objects), but not for large-scale changes (such as background motion). Feature alignment usually uses a deformable convolutional network and achieves good performance, but the training of the deformable convolutional network is unstable. Neither optical flow nor deformable convolution can be directly applied to satellite videos, because satellite videos have both moving ground objects and moving backgrounds, and the motion features are even less obvious. Although the alignment operation is introduced in the satellite video super-resolution reconstruction, the alignment error cannot be eliminated. At the same time, due to the timed movement of objects in different occluded areas, the information of the relevant areas will be missing. Direct fusion of multi-frame information is likely to produce poor results, and may even be fused into wrong information, resulting in a decrease in effect. Summary of the invention

[0006] Based on this, the present invention proposes a satellite video super-resolution reconstruction method and system for multiple degradation processes. First, a degradation model of satellite video data is constructed. This degradation model takes into account the blur kernel estimated from the real remote sensing image and the blur kernel generated from the predefined distribution. This ensures that the low-resolution data is close to the real scene, and the diversity of the low-resolution data does not completely depend on the diversity of the external data set. Secondly, a bidirectional high-order grid feature propagation network is used to propagate the features extracted from the entire video frame, so that the reconstruction of each frame can utilize the information of all frames. Finally, a feature alignment module based on spatiotemporal fusion is proposed. This method uses the spatiotemporal attention mechanism to fuse the spatiotemporal feature information between frames, improves the utilization of key information, reduces the impact of erroneous information, and uses a deformable convolution to align the spatiotemporal fused features, reducing the error accumulation of long sequence images over time.

[0007] In order to achieve the above objectives and the above technical effects, the present invention adopts a technical solution: a satellite video super-resolution reconstruction method for multiple degradation processes, comprising the following steps:

[0008] Step S1, obtaining a satellite video data set;

[0009] Step S2, constructing a satellite video super-resolution reconstruction model for multiple degradation processes, including a satellite video random degradation module, a feature extraction module, a feature alignment module and a pixel reorganization module;

[0010] Step S3, the satellite video data in the satellite video data set is subjected to a satellite video random degradation module to obtain a low-resolution frame image, and a data enhancement operation is performed to obtain low-resolution frame data;

[0011] Step S4, passing the low-resolution frame data obtained in step S3 through a feature extraction module to obtain frame image feature information;

[0012] Step S5, inputting the frame image feature information obtained in step S4 into a feature alignment module, performing propagation, aggregation and alignment operations on the input features to obtain aligned features;

[0013] Step S6, passing the aligned features obtained in step S5 through a pixel reorganization module to obtain residual features of the high-resolution image;

[0014] Step S7, upsampling the low-resolution frame image obtained in step S3, and then adding the residual features of the high-resolution image obtained in step S6 to obtain a final high-resolution reconstructed image.

[0015] Furthermore, the satellite video random degradation module described in step S3 includes a downsampling module, four random degradation modules and a video compression module; the downsampling module uses a downsampling method selected from nearest neighbor interpolation, bilinear interpolation or bicubic interpolation to downsample the image; each of the four random degradation modules uses a degradation mode selected from random blur, motion blur, random noise or sensor tremor to degrade the image; the video compression module is implemented using intra-frame compression or inter-frame compression, wherein the intra-frame compression includes JPEG compression, JPEG 2000 compression, PNG compression, WEBP compression, BMP compression, TIFF compression, BPG compression and FLIF compression, and the inter-frame compression includes H.263, H.264, H.265, MJPEG, MJPEG-2, MPEG-4, VP8, VP9 and AV1.

[0016] Furthermore, the feature extraction module described in step S4 includes K residual blocks, each residual block includes two convolutional layers and one activation layer, and the residual blocks are connected by skip connections.

[0017] Furthermore, the feature alignment module described in step S5 is used to propagate and aggregate the frame image feature information obtained in step S4 through a high-order grid propagation scheme. This process is divided into three stages: the first stage is to propagate the features forward along the time increment, the second stage is to propagate the features backward along the time decrement, and the third stage is to connect the target frame information with the feature information of the previous and next frames; the feature alignment module includes a residual module, a feature alignment module based on optical flow guidance and a spatiotemporal feature fusion module; the residual module includes K residual blocks, each residual block includes two convolutional layers and an activation layer; the feature alignment module based on optical flow guidance introduces optical flow information to assist deformable convolution in feature alignment operations; the spatiotemporal feature fusion module includes three convolutional layers, a temporal attention mechanism and a spatial attention mechanism.

[0018] Furthermore, the calculation process of the feature alignment module is as follows:

[0019] For backward propagation, a residual module is first used to further extract the extracted feature information:

[0020] (1)

[0021] in, represents the residual module, represents the cascade operation in the channel dimension, Represents the feature information extracted by the feature extraction module. Represents the features calculated by the forward propagation at the i-th time; in addition, the two frames before and after the target frame are aligned:

[0022] (2)

[0023] in, represents the aligned features, , , and Respectively represent the features of the i-2, i-1, i+1 and i+2 moments of the backward propagation, Represents a deformable feature alignment module guided by optical flow; finally, a spatiotemporal feature fusion module is used to obtain the final feature map:

[0024] (3)

[0025] in, represents the final feature map, Represents the deeper feature information extracted by the residual module. Represents the spatiotemporal feature fusion module.

[0026] Furthermore, in the deformable feature alignment module based on optical flow guidance, the optical flow estimation network is first used to calculate the optical flow map, and then , , , Respectively represent the optical flow from , , , Frame to Frame mapping, distorting the two frames before and after the target frame i:

[0027] (4)

[0028] (5)

[0029] (6)

[0030] (7)

[0031] in, , , , Respectively represent , , , The characteristics of the optical flow information at each moment after spatial distortion, Represents the spatial warping operation, and then uses the pre-aligned features to calculate the optical flow residual and deformable convolutional masks :

[0032] (8)

[0033] (9)

[0034] in, Indicates channel connection, and represents the convolution calculation, Represents the activation function, and finally the aligned features are obtained through deformable convolution :

[0035] (10)

[0036] Among them, DCN represents deformable convolution operation.

[0037] Furthermore, in the spatiotemporal feature fusion module, the deep feature information extracted from the current frame is first Feature map obtained after alignment with subsequent frames Perform convolution operation to calculate the similarity distance after embedding :

[0038] (11)

[0039] in, represents the activation function, conv represents the convolution calculation, represents the dot product; then the similar distance is processed by the temporal attention mechanism, and for each spatial position, the temporal attention is spatially heterogeneous;

[0040] (12)

[0041] in, represents the temporal attention map, Represents the temporal attention processing; then the aligned feature map is multiplied pixel by pixel with the temporal attention map, and the feature map of the temporal attention fusion processing is obtained through the convolution layer :

[0042] (13)

[0043] in, represents pixel-by-pixel dot product operation, Represents the fused convolution operation; finally, the fused features are processed by the spatial attention mechanism to enhance the texture information and obtain the final feature map :

[0044] (14)

[0045] Among them, SA represents the spatial attention operation, Represents element-by-element addition.

[0046] Furthermore, the pixel reorganization module described in step S6 includes a reconstruction layer, four convolutional layers, two pixel shuffling layers and three activation layers; the reconstruction layer is composed of K residual blocks with the same structure, each residual block includes two convolutional layers and one activation layer; the two pixel shuffling layers adopt a pixel shuffling strategy; the first and second activation layers both adopt the LeakyReLU activation function, and the third activation layer adopts the ReLU activation function.

[0047] Furthermore, the upsampling operation in step S7 adopts one of nearest neighbor interpolation, bilinear interpolation or bicubic interpolation.

[0048] The present invention also provides a satellite video super-resolution reconstruction system for multiple degradation processes, comprising:

[0049] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a satellite video super-resolution reconstruction method for multiple degradation processes as described in the above technical solution.

[0050] It can be seen from the above technical scheme that the present invention provides a satellite video super-resolution reconstruction method and system for multiple degradation processes, which uses a satellite video random degradation model to generate corresponding low-resolution data, and the acquired low-resolution data passes through a feature extraction layer, and then a high-order grid feature bidirectional propagation scheme is used to propagate the extracted features, and finally a spatiotemporal fusion feature alignment module is used to align the features of the sequence images, thereby reducing the error accumulation of long sequence images, effectively restoring the high fidelity of the satellite video, significantly improving the reconstruction quality of the satellite video, and also having a certain noise removal capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the overall framework of a satellite video super-resolution reconstruction method for multiple degradation processes provided by the present invention.

[0052] Figure 2 This is a schematic diagram of the structure of the satellite video random degradation model constructed in the present invention.

[0053] Figure 3 It is a structural schematic diagram of the feature extraction module in the present invention.

[0054] Figure 4 Schematic diagram of the feature alignment module structure constructed in the present invention.

[0055] Figure 5 Schematic diagram of the structure of the optical flow-guided deformable feature alignment module constructed in the present invention.

[0056] Figure 6 Schematic diagram of the structure of the spatiotemporal feature fusion module constructed in the present invention.

[0057] Figure 7 It is a schematic diagram of the structure of the pixel recombination module constructed in the present invention.

[0058] Figure 8 This is the visualization result of the 2x super-resolution reconstruction performed by the present invention in three satellite video datasets.

[0059] Fig. 9 This is the visualization result of 3x super-resolution reconstruction performed by the present invention in three satellite video datasets.

[0060] Fig.10 This is the visualization result of the 4x super-resolution reconstruction performed by the present invention in three satellite video datasets. DETAILED DESCRIPTION

[0061] In order to make the purpose and technical solution of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments, so that the characteristics and performance of the present invention can be more easily understood by researchers in related fields, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0062] The embodiment of the present invention discloses a satellite video super-resolution reconstruction method for multiple degradation processes, comprising the following steps:

[0063] Step 1: Construct a satellite video dataset, which is cropped from 100 videos taken by video satellites, covering a wide range of ground features, including cities, docks, airports, suburbs, cities, deserts, etc. The videos contain dynamic scenes, such as moving cars, airplanes, ships, etc. We crop the videos into 284 small videos, each of which contains 180 consecutive frames, of which 240 small videos are used as training sets and 44 small videos are used as validation sets. The size of the frame image of each video is 1280×720.

[0064] Step 2: Build Figure 1 The high-resolution optical satellite video super-resolution reconstruction model for multiple degradation processes mainly includes: Figure 2 Satellite video random degradation module, Figure 3 The feature extraction module Figure 4 The feature alignment module and Figure 7 Pixel reorganization module.

[0065] Step 3: Pass the data set obtained in step 1 through the satellite video random degradation module to obtain the corresponding low-resolution video frame data set, and perform data enhancement operations such as rotation, translation and scaling on the data to expand the sample library.

[0066] Specifically, the satellite video random degradation model contains a downsampling module, four random degradation modules and a video compression module, such as Figure 2 As shown. Input the original frame image First, it passes through the downsampling module. The downsampling module contains three downsampling methods: nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. Each time it passes through the downsampling module, a downsampling method is randomly selected to downsample the image. Then the downsampled image passes through four random degradation modules. Each random degradation module contains four degradation modes: random blur, motion blur, random noise and sensor tremor. Each time it passes through the random degradation module, a degradation mode is randomly selected. The image after the random degradation module finally passes through the video compression module. The video compression module contains two parts: intra-frame compression and inter-frame compression. The intra-frame compression is JPEG compression, and the inter-frame compression contains three modes: H.264, H.265 and MEPG-4. Each time it passes through the video compression, an inter-frame compression mode is randomly selected to obtain the final low-resolution frame image. , and Perform data enhancement operations such as rotation, translation, and scaling to expand the sample library.

[0067] In a specific embodiment, the downsampling method adopts nearest neighbor interpolation, bilinear interpolation and bicubic interpolation, and the probability of each downsampling method being selected is equal, which is 1 / 3. Random blurring adopts isotropic blur kernel, anisotropic blur kernel, generalized isotropic blur kernel, generalized anisotropic blur kernel, platform isotropic blur kernel, platform anisotropic blur kernel and sinc function blur kernel, and the probability of each blur kernel type being selected is set to 0.405, 0.225, 0.108, 0.027, 0.108, 0.027 and 0.1 respectively. The standard deviation range of Gaussian blur in the x and y directions is [0.2,3], and the value can be randomly selected between 0.2 and 3 to control the intensity of blur. The angle of rotation of the blur kernel is [-π, π], and the blur kernel can be randomly rotated in the range of -π to π to simulate blur effects in different directions. The range of the generalized Gaussian blur kernel is [0.5, 4]. A larger value means a closer uniform distribution, while a smaller value means a sharper distribution. Motion blur mainly describes the direction and intensity of the moving target. The motion direction includes horizontal motion blur, vertical motion blur, and diagonal motion blur. The probability range of being randomly selected in the three directions is 0.4, 0.4, and 0.2, respectively. The step size of the motion direction change is 1 degree, which can finely control the direction change of motion blur; the random value range of the motion intensity is [2, 10], and the step size of the intensity change is 1, which makes the change of motion blur smoother and more accurate. Random noise mainly includes Gaussian noise and Poisson noise. The probability of selecting both noises is 50%. The standard deviation of Gaussian noise will be randomly selected between [1, 30], and the step size will be adjusted in increments of 0.1. The scaling factor of Poisson noise is randomly selected between [0.05, 3], and the step size will be adjusted in increments of 0.005. Sensor tremor can model a high-frequency vibration, which can usually be expressed as a function of time. The probability of sensor tremor is randomly selected in the range of [0.05, 0.6]. The tremor direction includes horizontal tremor, vertical tremor and random direction tremor, with random probability ranges of [0, 0.3], [0, 0.3] and [0, 0.4] respectively. The tremor intensity is randomly selected in the range of [0.1, 2]. The quality range of intra-frame JPEG compression is [30, 95], and the quality value will be randomly selected in this interval. There are three optional encoders for inter-frame compression, namely H.264, H.265 and MEPG-4, and the probability of each encoder being selected is equal, which is 1 / 3.

[0068] Step 4: The low-resolution frame image obtained in step 3 Through the feature extraction module, the features are obtained The feature extraction module consists of five residual blocks with the same structure, such as Figure 3 As shown in Figure 1, each residual block includes two convolutional layers and one activation layer, and the residual blocks are connected by skip connections.

[0069] In a specific embodiment, the convolution kernel size of the two convolutional layers is 3×3, the step size is 2, the padding is 1, the number of input channels is 3, the number of intermediate features is 64, the number of output channels is 64, and the activation layer uses the ReLU activation function with a slope set to 0.1.

[0070] Step 5: The features obtained in step 4 are propagated and aggregated, mainly through a high-order grid propagation scheme. This process is divided into three stages: the first stage is to propagate the features forward along the time increment, the second stage is to propagate the features backward (decrease along the time), and the third stage is to connect the target frame information with the feature information of the previous and next two frames, such as Figure 1 Through this connection, feature information at different locations can be gathered to improve the effectiveness of the model in the occluded area. During the feature transfer process, the input feature information needs to be aligned, such as Figure 4 As shown in the figure, feature alignment mainly includes a residual module, a deformable feature alignment module based on optical flow guidance, and a spatiotemporal feature fusion module.

[0071] Specifically, taking backward propagation as an example, a residual module is first used to further extract the extracted features:

[0072] (1)

[0073] in, represents the residual module, represents the cascade operation in the channel dimension, Represents the feature information extracted by the feature extraction module. represents the features calculated by the forward propagation at the i-th time. In addition, the two frames before and after the target frame are aligned:

[0074] (2)

[0075] in, represents the aligned features, , , and Respectively represent the features of the i-2, i-1, i+1 and i+2 moments of the backward propagation, It represents the deformable feature alignment operation guided by optical flow. Finally, through a spatiotemporal feature fusion module, the final feature map is obtained:

[0076] (3)

[0077] in, represents the final feature map, Represents the deeper feature information extracted by the residual module. Spatiotemporal feature fusion operation.

[0078] In the optical flow-guided deformable feature alignment module, Figure 5 As shown, the optical flow estimation network is first used to calculate the optical flow map, and then , , , Respectively represent the optical flow from , , , Frame to For frame mapping, we distort the two frames before and after the target frame i:

[0079] (4)

[0080] (5)

[0081] (6)

[0082] (7)

[0083] in , , , Respectively represent , , , The characteristics of the optical flow information at each moment after spatial distortion, Represents the spatial warping operation, and then uses the pre-aligned features to calculate the optical flow residual and deformable convolutional masks :

[0084] (8)

[0085] (9)

[0086] in, Indicates channel connection, and represents the convolution calculation, Represents the activation function. Finally, the aligned features are obtained through deformable convolution .

[0087] (10)

[0088] Among them, DCN represents deformable convolution operation.

[0089] In the spatiotemporal feature fusion module, the temporal attention mechanism and the spatial attention mechanism are used. The attention mechanism redistributes the weights, thereby helping the model to select important information from adjacent frames and reduce erroneous information, thereby making more effective use of longer sequences, such as Figure 6 As shown. First, the deep feature information extracted from the current frame Feature map obtained after alignment with subsequent frames Perform convolution operation to calculate the similarity distance after embedding :

[0090] (11)

[0091] in, represents the activation function, conv represents the convolution calculation, Denotes the dot product. Then the similar distance is processed by the temporal attention mechanism, and for each spatial position, the temporal attention is spatially heterogeneous.

[0092] (12)

[0093] in, represents the temporal attention map, Represents the temporal attention processing. Then the aligned feature map is multiplied pixel by pixel with the temporal attention map, and the feature map of the temporal attention fusion processing is obtained through the convolution layer. .

[0094] (13)

[0095] in, represents pixel-by-pixel dot product operation, Represents the fused convolution operation. Finally, the fused features are processed by the spatial attention mechanism to enhance texture information, etc., to obtain the final feature map .

[0096] (14)

[0097] Among them, SA represents the spatial attention operation, Represents element-by-element addition.

[0098] In a specific embodiment, the residual module in the feature alignment is composed of three residual blocks with the same structure, each residual block includes two convolutional layers and an activation layer, and the residual blocks are connected by jump connections. The convolution kernel size of the two convolutional layers is 3×3, the step size is 2, the padding is 1, the number of input channels is 64, the number of intermediate features is 64, and the number of output channels is 128. The activation layer adopts the ReLU activation function, and the slope is set to 0.1.

[0099] Step 6: Align the features obtained in step 5 After the pixel reorganization module, the residual features of the high-resolution image are obtained .

[0100] Specifically, the pixel reassembly module consists of a reconstruction layer, four convolutional layers, two pixel shuffling layers, and three activation layers, e.g. Figure 7 First, the feature map obtained in step 5 is further extracted through the reconstruction layer. The reconstruction layer consists of five residual blocks with the same structure. It has the same structure as the feature extraction module in step 4, except that the number of input and output channels is different. Then, two convolutions and two pixel shuffles are performed to increase the number of channels and image size. Finally, two more convolutions are performed to obtain the residual features of a high-resolution frame image with 3 channels. .

[0101] In a specific embodiment, the number of input channels of the reconstruction layer is 320, and the number of output channels is 64.

[0102] The first convolutional layer has 64 input channels, 256 output channels, a kernel size of 3×3, a stride of 1, and a padding of 1. This convolutional layer amplifies the number of channels by 4 times to cooperate with the subsequent pixel shuffle operation. The second convolutional layer has 64 input channels, 256 output channels, a kernel size of 3×3, a stride of 1, and a padding of 1 to further amplify the number of channels. The third convolutional layer has 64 input channels, 64 output channels, a kernel size of 3×3, a stride of 1, and a padding of 1. The fourth convolutional layer has 64 input channels, 3 output channels, a kernel size of 3×3, a stride of 1, and a padding of 1. The first and second activation layers use the Leaky ReLU activation function, and the third activation layer uses the ReLU activation function. The slope of the activation function is set to 0.1. The pixel shuffle strategy adopts the pixel shuffle package strategy. The pixel shuffle package is an enhanced version of the pixel shuffle module. The standard pixel shuffle simply redistributes the channel data to the spatial dimension, while the pixel shuffle package also uses convolution for preprocessing before pixel shuffling to improve the expressiveness of features and upsampling quality. The first pixel shuffle has 256 input channels, 64 output channels, a scaling factor of 2, and an upsampling convolution kernel size of 3. The second pixel shuffle has 256 input channels, 64 output channels, a scaling factor of 2, and an upsampling convolution kernel size of 3.

[0103] Step 7: interpolate the input low-resolution frame image and add it to the residual features of the high-resolution frame image obtained in step 6 to obtain the final reconstructed image. .

[0104] In a specific embodiment, bicubic interpolation is used to interpolate low-resolution frame images, and PSNR and SSIM are used as objective evaluation indicators. The evaluation indicators are obtained by averaging all frames of each satellite video tested, and are tested in three satellite video datasets. The objective evaluation results are shown in Table 1, and the visualization results are shown in Table 1. Figure 8 , Fig. 9 and Fig.10 shown.

[0105] Table 1 Objective evaluation results of different super-resolution ratios on three satellite video datasets

[0106]

[0107] Figure 8 The visualization results of 2x super-resolution reconstruction on three satellite video datasets. Figure 8In the figure, A is the 033rd original image of dataset 1, A1 represents the magnified details of the original frame image, and A2 represents the 2x super-resolution reconstruction result of the corresponding position; B is the 000th original frame image of dataset 2, B1 represents the magnified details of the original frame image, and B2 represents the 2x super-resolution reconstruction result of the corresponding position; C is the 001st original frame image of dataset 3, C1 represents the magnified details of the original frame image, and C2 represents the 2x super-resolution reconstruction result of the corresponding position.

[0108] Fig. 9 Visualization results of 3x super-resolution reconstruction on three satellite video datasets. Fig. 9 In the figure, A is the 026th original frame image of dataset 1, A1 represents the magnified details of the original frame image, and A2 represents the 3x super-resolution reconstruction result of the corresponding position; B is the 016th original frame image of dataset 2, B1 represents the magnified details of the original frame image, and B2 represents the 3x super-resolution reconstruction result of the corresponding position; C is the 001th original image of dataset 3, C1 represents the magnified details of the original frame image, and C2 represents the 3x super-resolution reconstruction result of the corresponding position.

[0109] Fig.10 Visualization results of 4x super-resolution reconstruction on three satellite video datasets. Fig.10 In the figure, A is the 030th original frame image of dataset 1, A1 represents the magnified details of the original frame image, and A2 represents the 4x super-resolution reconstruction result of the corresponding position; B is the 001th original frame image of dataset 2, B1 represents the magnified details of the original frame image, and B2 represents the 4x super-resolution reconstruction result of the corresponding position; C is the 005th original image of dataset 3, C1 represents the magnified details of the original frame image, and C2 represents the 4x super-resolution reconstruction result of the corresponding position.

[0110] In summary, the present invention overcomes the problem of poor reconstruction effect caused by multiple degradation factors of satellite video data in actual scenes by considering the diverse noises existing in satellite video data in the satellite video random degradation module; the high-order grid feature bidirectional propagation scheme adopted propagates feature information forward and backward in time in an alternating manner, which can "revisit" information from different frames and use it for feature refinement, thereby improving the expressiveness of features; in the process of feature alignment, the spatiotemporal features are fused, which helps the model to select important information from adjacent frames and reduce erroneous information, thereby making more effective use of longer sequence images; finally, experiments are carried out on three satellite video data sets, and the experimental results prove the effectiveness of the method proposed in the present invention from both qualitative and quantitative aspects.

[0111] On the other hand, an embodiment of the present invention further provides a satellite video super-resolution reconstruction system for multiple degradation processes, comprising:

[0112] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a satellite video super-resolution reconstruction method for multiple degradation processes as described in the above technical solution.

[0113] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A satellite video super-resolution reconstruction method for multiple degradation processes, characterized in that: The steps include: Step S1, obtaining a satellite video data set; Step S2, constructing a satellite video super-resolution reconstruction model for multiple degradation processes, including a satellite video random degradation module, a feature extraction module, a feature alignment module and a pixel reorganization module; Step S3, the satellite video data in the satellite video data set is subjected to a satellite video random degradation module to obtain a low-resolution frame image, and a data enhancement operation is performed to obtain low-resolution frame data; Step S4, passing the low-resolution frame data obtained in step S3 through a feature extraction module to obtain frame image feature information; Step S5, inputting the frame image feature information obtained in step S4 into a feature alignment module, performing propagation, aggregation and alignment operations on the input features to obtain aligned features; The feature alignment module in step S5 is used to propagate and aggregate the frame image feature information obtained in step S4 through a high-order grid propagation scheme. This process is divided into three stages: the first stage is to propagate the features forward along the time increment, the second stage is to propagate the features backward along the time decrement, and the third stage is to connect the target frame information with the feature information of the previous and next frames; the feature alignment module includes a residual module, a feature alignment module based on optical flow guidance, and a spatiotemporal feature fusion module; The residual module contains K residual blocks, each of which includes two convolutional layers and one activation layer. The feature alignment module based on optical flow guidance introduces optical flow information to assist deformable convolution in feature alignment operations. The spatiotemporal feature fusion module consists of three convolutional layers, a temporal attention mechanism, and a spatial attention mechanism; Step S6, passing the aligned features obtained in step S5 through a pixel reorganization module to obtain residual features of the high-resolution image; Step S7, upsampling the low-resolution frame image obtained in step S3, and then adding the residual features of the high-resolution image obtained in step S6 to obtain a final high-resolution reconstructed image.

2. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 1, characterized in that: The satellite video random degradation module described in step S3 includes a downsampling module, four random degradation modules and a video compression module; the downsampling module uses a downsampling method selected from nearest neighbor interpolation, bilinear interpolation or bicubic interpolation to downsample the image; each of the four random degradation modules uses a degradation mode selected from random blur, motion blur, random noise or sensor tremor to degrade the image; the video compression module is implemented by intra-frame compression or inter-frame compression, wherein the intra-frame compression includes JPEG compression, JPEG 2000 compression, PNG compression, WEBP compression, BMP compression, TIFF compression, BPG compression and FLIF compression, and the inter-frame compression includes H.263, H.264, H.265, MJPEG, MJPEG-2, MPEG-4, VP8, VP9 and AV1.

3. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 1, characterized in that: The feature extraction module described in step S4 includes K residual blocks, each residual block includes two convolutional layers and one activation layer, and the residual blocks are connected by skip connections.

4. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 1, characterized in that: The calculation process of the feature alignment module is as follows: For backward propagation, a residual module is first used to further extract the extracted feature information: (1) in, represents the residual module, represents the cascade operation in the channel dimension, Represents the feature information extracted by the feature extraction module. Represents the features calculated by the forward propagation at the i-th time; in addition, the two frames before and after the target frame are aligned: (2) in, represents the aligned features, , , and Respectively represent the features of the i-2, i-1, i+1 and i+2 moments of the backward propagation, It represents the deformable feature alignment module guided by optical flow; finally, a spatiotemporal feature fusion module is used to obtain the final feature map: (3) in, represents the final feature map, Represents the deeper feature information extracted by the residual module. Represents the spatiotemporal feature fusion module.

5. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 4 is characterized in that: In the deformable feature alignment module based on optical flow guidance, the optical flow estimation network is first used to calculate the optical flow map. , , , Respectively represent the optical flow from , , , Frame to Frame mapping, for the target frame i The two images before and after the frame are distorted: (4) (5) (6) (7) in, , , , Respectively represent , , , The characteristics of the optical flow information at each moment after spatial distortion, Represents the spatial warping operation, and then uses the pre-aligned features to calculate the optical flow residual and deformable convolutional masks : (8) (9) in, Indicates channel connection, and represents the convolution calculation, Represents the activation function, and finally the aligned features are obtained through deformable convolution : (10) Among them, DCN represents deformable convolution operation.

6. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 5, characterized in that: In the spatiotemporal feature fusion module, the deep feature information extracted from the current frame is first Feature map obtained after alignment with subsequent frames Perform convolution operation to calculate the similarity distance after embedding : (11) in, represents the activation function, conv represents the convolution calculation, represents the dot product; then the similar distance is processed by the temporal attention mechanism, and for each spatial position, the temporal attention is spatially heterogeneous; (12) in, represents the temporal attention map, Represents the temporal attention processing; then the aligned feature map is multiplied pixel by pixel with the temporal attention map, and the feature map of the temporal attention fusion processing is obtained through the convolution layer : (13) in, represents pixel-by-pixel dot product operation, Represents the fused convolution operation; finally, the fused features are processed by the spatial attention mechanism to enhance the texture information and obtain the final feature map : (14) Among them, SA represents the spatial attention operation, Represents element-by-element addition.

7. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 1, characterized in that: The pixel reorganization module described in step S6 includes a reconstruction layer, four convolution layers, two pixel shuffling layers and three activation layers; the reconstruction layer is composed of K residual blocks with the same structure, and each residual block includes two convolution layers and one activation layer; The two pixel shuffling layers adopt a pixel shuffling strategy; The first and second activation layers both use Leaky ReLU activation function, and the third activation layer uses ReLU activation function.

8. The satellite video super-resolution reconstruction method for multiple degradation processes according to claim 1, characterized in that: The up-sampling operation in step S7 adopts one of nearest neighbor interpolation, bilinear interpolation or bicubic interpolation.

9. A satellite video super-resolution reconstruction system for multiple degradation processes, characterized in that: include: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a satellite video super-resolution reconstruction method for multiple degradation processes as described in any one of claims 1 to 8.

Images