Space target image super-resolution restoration system and method based on deformable convolution

Abstract

The invention discloses a space target image super-resolution restoration system and method based on deformable convolution, relates to the field of space target detection and image processing, and solves the problems that an existing video super-resolution restoration method is high in labeling cost, highly depends on the quality and quantity of labeled data and is difficult to apply a task lacking label data. The method comprises the following steps: constructing a data set of LR space target image data, and carrying out blurring kernel estimation on an LR input sequence by adopting a blurring kernel estimation network; obtaining a degraded LR frame with the same size as the LR input sequence through a main branch module; an auxiliary SR frame with the same size as the LR input sequence is obtained through an auxiliary branch module; and constructing a loss function between the obtained degraded LR frame and the auxiliary SR frame and the LR input sequence to carry out self-supervised training so as to realize space target image restoration. According to the method, the complex process of optical flow estimation and possibly introduced estimation errors are avoided, and the method adapts to complex motion and deformation conditions in a space target image.

Description

Technical Field

[0001] The present invention relates to the field of space target detection and image processing, and in particular to a super-resolution restoration method for sequence space target images, which performs super-resolution restoration on space target images by combining a self-supervised learning structure with deformable convolution. Background Art

[0002] With the establishment of low-orbit constellations like Starlink, the number of low-orbit satellites and space debris has rapidly increased. Due to their proximity and brightness, low-orbit targets are advantageous for observation using small-aperture telescopes. However, low-orbit targets move rapidly, resulting in shorter observation arcs. To improve orbit measurement accuracy, a multi-station network and relay measurement are required.

[0003] Small-aperture telescopes are highly cost-effective space target detection devices. By employing automation technologies, intelligent devices, and related software systems, ground-based small-aperture, wide-field-of-view electro-optical telescopes can be operated automatically, enabling unmanned operation characterized by automated observation, real-time data processing, and networked data transmission.

[0004] Compared to large telescopes, small-aperture telescopes are not only expensive but also less survivable in wartime. Automated small-aperture telescopes can be flexibly deployed, have a high degree of automation, require minimal site conditions, and are low-cost, making them suitable for multi-station networking and complementing large telescopes.

[0005] Automated small-aperture telescopes have the characteristics of intelligence, lightweight, low power consumption, high precision, low cost, and high maneuverability. They can quickly realize distributed measurements, are widely used in space target orbit measurements, and complement large telescopes. It has become a development trend of future telescopes.

[0006] To ensure timely detection of space targets, small-aperture telescopes are needed to detect satellites in medium and high orbits. However, due to the long detection distance and high background noise, the light collected from space targets is relatively small, resulting in low target radiation energy detected by the system. At the same time, to ensure full coverage of space targets, small-aperture telescopes are needed to detect space debris in low-orbit orbits. If the target is small-sized space debris, it will appear as dim and small star images in the image and easily be submerged in the background, making star image extraction extremely difficult. To improve the ability to extract dim and small space target star images, the method of lowering the image segmentation threshold can be used to increase detection sensitivity. However, while this approach improves the visibility of dim targets to a certain extent, it also significantly increases the amount of noise extracted, which can affect subsequent target detection results.

[0007] Improving the spatial resolution of a telescope by increasing the number of pixels occupied by a space target is the most direct way to detect small, dim space targets. Current telescope systems all use CCD or CMOS cameras as sensors, so their pixel size determines the telescope's spatial resolution. By improving the sensor's manufacturing process, reducing the size of the photosensitive cells and increasing the number of pixels in the image, high-resolution images can be obtained. Currently, with technological advances, the resolution of image sensors has indeed increased. However, if the resolution continues to increase indefinitely, the size of the photosensitive cells will become smaller, resulting in a decrease in photosensitivity, a significant reduction in detection range and sensitivity, and an increase in readout noise and dark current. In summary, improving the spatial resolution of an image through a "hardware approach" is difficult.

[0008] Because small-aperture telescopes observe single targets using precision measurement mode, there is information redundancy between the images they capture. Super-resolution restoration can be used to recover high-resolution (HR) images from information-limited low-resolution (LR) images. Consecutive image frames have strong correlations, and inter-frame information is particularly important. Combining information from adjacent frames for restoration has the potential to improve restoration accuracy and enhance the brightness and resolution of small, dim targets.

[0009] Traditional super-resolution restoration models usually rely on a large amount of LR / HR paired data (high resolution) to train the model. In this case, the LR / HR paired data obtained using real observations is the most reliable. However, under the conditions of optical observations of space targets, real LR / HR paired data is usually very expensive or unavailable. In addition, although supervised learning methods using synthetic data simulate certain real-world conditions, there are significant differences between synthetic data and real data.

[0010] Therefore, the method of the present invention establishes a self-supervised sequential space target image super-resolution restoration model based on deformable convolution, abandons the method of calculating the distortion matrix based on optical flow in the traditional algorithm, uses deformable convolution to achieve feature alignment between adjacent image frames, and jointly trains in an end-to-end manner to solve the problem of difficult identification of dim and small targets. Summary of the Invention

[0011] In order to solve the problems of high labeling cost, high dependence on the quality and quantity of labeled data, and difficulty in applying tasks without labeled data in existing video super-resolution restoration methods, the present invention proposes a self-supervised sequential space target image super-resolution restoration system and method based on deformable convolution.

[0012] A spatial target image super-resolution restoration system based on deformable convolution, the restoration system comprising a blur kernel estimation network, a main branch module and an auxiliary branch module;

[0013] The main branch module and the auxiliary branch module are both composed of a video super-resolution module and a post-blur downsampling module;

[0014] The blur kernel estimation network is used to perform blur kernel estimation on the LR input sequence to obtain the blur kernel;

[0015] The post-blur downsampling module uses a blur kernel to perform post-blur downsampling on the LR input sequence to obtain an auxiliary LR sequence; and uses a blur kernel to perform post-blur downsampling on the potential HR image frame to obtain a degraded LR frame of the same size as the LR input sequence;

[0016] The video super-resolution module is used to extract features of the LR input sequence and the auxiliary LR sequence and obtain alignment features of adjacent frames and intermediate frames of the LR input sequence and alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence;

[0017] Sequentially concatenate the alignment features of adjacent frames and intermediate frames of the LR input sequence to obtain a potential HR image frame of the same size as the LR input sequence; sequentially concatenate the alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence to obtain an auxiliary SR frame of the same size as the LR input sequence;

[0018] A loss function is constructed between the degraded LR frame of the same size as the LR input sequence, the auxiliary SR frame of the same size as the LR input sequence, and the LR input sequence for self-supervised training to achieve the restoration of spatial target images.

[0019] The present invention also provides a method for super-resolution restoration of spatial target images based on deformable convolution. The restoration method is implemented by the above-mentioned system for super-resolution restoration of spatial target images based on deformable convolution. The implementation process of the restoration method is as follows:

[0020] Step S1: constructing a dataset of LR space target image data, which consists of an LR input sequence;

[0021] Step S2: Use the blur kernel estimation network to perform blur kernel estimation on the LR input sequence to obtain the blur kernel;

[0022] Step S3: Input the LR input sequence into the main branch module to obtain a degraded LR frame of the same size as the LR input sequence;

[0023] Step S4: Input the LR input sequence into the auxiliary branch module to obtain an auxiliary SR frame of the same size as the LR input sequence;

[0024] Step S5: Construct a loss function between the degraded LR frames and auxiliary SR frames obtained by the main branch module and the auxiliary branch module and the LR input sequence for self-supervised training to achieve spatial target image restoration.

[0025] Beneficial effects of the present invention:

[0026] The restoration method described in this paper derives a blur kernel from the low-resolution spatial target image data itself. Through a main branch and auxiliary branches, it ultimately produces an image of the same size as the input sequence spatial target image. A loss function is then constructed between the image and the input spatial target image, thereby using the input spatial target image itself for self-supervision. Furthermore, a deformable convolutional network is introduced to directly learn alignment and motion information, thus avoiding the complex process of optical flow estimation and the potential for estimation errors. This allows for better adaptation to complex motion and deformation in spatial target images.

[0027] This invention utilizes a video super-resolution reconstruction method based on self-supervised deformable convolution technology, which significantly improves the area, flux, and magnitude of stars and space objects in an image. Therefore, when extracting small, faint space objects, reasonable thresholds can be set for both area and flux. This helps to screen out those of a certain size that are more likely to be stars or objects rather than noise or cosmic rays.

[0028] Furthermore, during the super-resolution reconstruction process, the relative positions of the stars and the target remain unchanged regardless of changes in image resolution. Therefore, this proportional relationship persists in the super-resolution reconstructed image, enabling the use of astronomical positioning algorithms to accurately determine the target's orbital position. Furthermore, compared to the original image sequence, the super-resolution reconstructed image sequence provides a more complete trajectory with no missing points, providing more favorable conditions for subsequent trajectory improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a principle block diagram of a spatial target image super-resolution restoration system based on deformable convolution according to the present invention;

[0030] Figure 2 Schematic diagram of the blur kernel estimation network;

[0031] Figure 3 Schematic diagram of the feature extraction network;

[0032] Figure 4 Schematic diagram of deformable convolutional network;

[0033] Figure 5 It is a schematic diagram of the VSR module;

[0034] Figure 6 The results of 4X and 8X super-resolution reconstruction. DETAILED DESCRIPTION

[0035] Specific implementation method 1. Combination Figures 1 to 5 This embodiment describes a spatial target image super-resolution restoration system based on deformable convolution, which mainly consists of a data acquisition module, a blur kernel estimation network N k The main branch module is mainly used to estimate the blur kernel and potential HR image frame, and use the deformable convolutional network to learn the alignment information and motion information between video frames; the auxiliary branch module mainly uses the auxiliary training data generated by the LR input sequence and the blur kernel to constrain the deformable convolutional network N w and clear frame reconstruction network N r training.

[0036] Wherein, the main branch module and the auxiliary branch module are both composed of a video super-resolution (VSR) module and a blur downsampling (BD) module; and, Figure 5 As shown, the VSR module is mainly composed of feature extraction network N e , Deformable Convolutional Network N w and clear frame reconstruction network N r composition.

[0037] The data acquisition module is used to collect LR space target image data to form a data set consisting of an LR input sequence;

[0038] The blur kernel estimation network N k Used to estimate the blur kernel of the LR input sequence to obtain the blur kernel K i ;

[0039] The BD module is used to use the blur kernel K i The LR input sequence is blurred and then downsampled to obtain the auxiliary LR sequence; it is also used to use the blur kernel K i Blurring the latent HR image frame and then downsampling it to obtain a degraded LR frame of the same size as the LR input sequence;

[0040] The feature extraction network N e It is used to extract features of the LR input sequence to obtain features at different scales; it is also used to extract features of the auxiliary LR sequence to obtain features at different scales;

[0041] The deformable convolutional network N w For the feature extraction network N e The features obtained at different scales are processed separately to obtain the alignment features of adjacent frames and intermediate frames of the LR input sequence and the alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence;

[0042] The clear frame reconstruction network N r For the deformable convolutional network N w The alignment features of adjacent frames and intermediate frames of the LR input sequence are sequentially spliced to obtain potential HR image frames of the same size as the LR input sequence; it is also used to align the features of adjacent frames and intermediate frames of the LR input sequence with the deformable convolutional network N. w The alignment features of adjacent frames and intermediate frames of the obtained auxiliary LR sequence are spliced in sequence to obtain auxiliary SR (Super Resolution) frames of the same size as the LR input sequence; a loss function is constructed between the degraded LR frames and auxiliary SR frames of the same size as the LR input sequence and the original LR spatial target image data for self-supervised training.

[0043] In this implementation, a deformable convolutional network is added to the VSR module to directly learn alignment and motion information. By learning the local deformation of each pixel, the deformable convolution kernel can adaptively adapt to the shape and structure of the target. This better preserves image detail, especially for spatial target images with small motion displacements. It can more accurately capture subtle motion changes, avoiding the complex process of optical flow estimation and the potential for estimation errors.

[0044] like Figure 2 As shown, in this embodiment, the blur kernel estimation network N k It is a neural network with a convolutional layer Conv, a residual channel attention mechanism (RCAB) module, an adaptive average pooling layer, a fully connected layer and a softmax function.

[0045] Input a feature, pass through a convolution layer Conv, then pass through two RCAB modules, adaptive average pooling layer, two more RCAB modules, and then pass through two convolution layers Conv to input into the fully connected layer (Fully connected layer), and finally obtain the blur kernel K after residual analysis. i Among them, the RCAB module combines the channel attention mechanism (CA) and the residual idea. The RCAB module is mainly composed of two CA modules, a global average pooling layer and a Sigmoid function, and the CA module is mainly composed of two convolutional layers Conv and a Relu function.

[0046] The input feature input enters the RCAB module and first passes through the first CA module for a convolution, ReLU, and convolution operation to obtain feature f. After global average pooling, feature f passes through another CA module for a convolution, ReLU, and convolution operation to rescale it to obtain feature x. Feature x is then passed through a sigmoid function and element-wise multiplied with feature f. Finally, the input feature input and the result of the element-wise multiplication are element-wise added. The convolution operation uses a 3×3 convolution kernel.

[0047] In this embodiment, the blur kernel K i The specific calculation formula is as follows:

[0048] K i =N k (C{y i-N ,...,y i ,...,y i+N})

[0049] Among them, N k represents the blur kernel estimation network, {y i-N ,...,y i ,...,y i+N} represents the LR input sequence, and C represents the sequential concatenation operation. In this embodiment, the blur kernel is obtained by the input LR input sequence itself, without using an artificially preset kernel.

[0050] like Figures 3 to 5 As shown in the figure, the feature extraction network adopts a pyramid convolutional neural network, which consists of a convolution layer Conv, a Leaky ReLU function and a residual block ResBlock; the residual block ResBlock consists of a convolution layer Conv+ReLU function; the deformable convolution network consists of multiple layers of deformable convolution layers Dconv and a connection layer Concat; the clear frame reconstruction network consists of a convolution layer, a residual block and a sub-pixel convolution layer.

[0051] Specific implementation method 2: Figures 1 to 5 This embodiment is described as follows. This embodiment is a restoration method implemented by a spatial target image super-resolution restoration system based on deformable convolution described in Specific Embodiment 1. The restoration method is implemented by the following steps:

[0052] Step S1: construct a dataset containing only LR space target image data, which consists of LR input sequences;

[0053] In this embodiment, the selected LR space target image data can be obtained by real shooting with a small-aperture telescope to obtain LR space target image data in different scenes; and the LR space target image data is preprocessed; the preprocessing operation includes: randomly rotating or flipping the LR input sequence, for example, rotating it 90 degrees, 180 degrees, 270 degrees or flipping it horizontally, and then cropping it into a uniform size of 64×64, and finally normalizing the pixels and mapping them to 0~1.

[0054] This embodiment does not require manual labeling of a large amount of labeled data. In some tasks, labeled data may be very limited or expensive.

[0055] The preprocessed LR input sequence is divided into a training set, a validation set, and a test set. The training set includes 20 spatial target images, each of which includes 50 frames of images. The validation set and the test set each include 1 spatial target image, each of which includes 50 frames of images.

[0056] Step S2: Figure 2 As shown, the fuzzy kernel estimation network N k Perform blur kernel estimation on the LR input sequence; obtain blur kernel K i ;

[0057] Step S3: Input the LR input sequence into the main branch module to obtain an image of the same size as the LR input sequence. The specific steps are as follows:

[0058] Step S3-1: Using feature extraction network N e Perform feature extraction on the features of the LR input sequence and finally extract features at different scales. The specific steps are as follows:

[0059] like Figure 3 As shown, first, a convolutional layer Conv+Leaky ReLU function and 5 residual blocks ResBlock are used to extract the adjacent frames y of the LR input sequence j Features at scale p1 (size 1×h×w) (Dimensions are c f ×h×w), where c f is the number of channels, h is the image height, and w is the image width;

[0060] Then, a 2-layer convolutional layer Conv+Leaky ReLU function is used to extract features Features at the p2 scale (Dimensions are c f ×h / 2×w / 2), and uses 2 layers of convolutional layers Conv+Leaky ReLU function to extract features. Features at the p3 scale (Dimensions are c f ×h / 4×w / 4), and finally obtain features at three different scales The specific calculation formula is as follows:

[0061]

[0062] Among them, N e represents the feature extraction network.

[0063] Similarly, follow the above operation method to extract the middle frame y of the LR input sequence i Features f at three different scales i p1 ,f i p2 ,f i p3 .

[0064] Step S3-2: Figure 4 As shown, using a deformable convolutional network N w After processing the features at three different scales, we get the adjacent frames y of the LR input sequence j With the intermediate frame y i Alignment features Among them, the deformable convolutional network N w It is composed of multiple layers of deformable convolutional layers Dconv and connection layers Concat.

[0065] First, at the p3 scale, use the feature f i p3 and features Estimate offset offset3. The specific calculation formula is as follows:

[0066]

[0067] Among them, C represents the concatenation operation in sequence, and conv1 represents a general function composed of several convolutional layers.

[0068] The obtained offset offset3 and feature Input the deformable convolution layer Dconv to get the aligned features The specific calculation formula is as follows:

[0069]

[0070] Among them, conv2 represents a general function composed of several convolutional layers, and Dconv represents a deformable convolution operation.

[0071] Then use bilinear interpolation to add offset3 and Perform 2x upsampling and p2 scale features As input, we align it again and get the offset offset2 and the aligned features. Similarly, for offset2 and alignment features Perform 2x upsampling and p1 scale features As input, we align the offset and the alignment features. The specific calculation formula is as follows:

[0072]

[0073] Align features at the p1 scale After deformable convolution, the final alignment feature w is obtained j→i The specific calculation formula is as follows:

[0074]

[0075] Finally, all the alignment features are obtained

[0076] Step S3-3: Reconstruct network N using clear frames r All the alignment features will be obtained The splicing operation is performed in sequence, and the clear frame reconstruction network N r It mainly consists of convolutional layers, residual blocks and sub-pixel convolutional layers to align all features The input convolution layer and residual block perform feature aggregation to obtain aggregated features, and then use the convolution layer and sub-pixel convolution layer to map the aggregated features into high-resolution features, which are then combined with the LR input sequence intermediate frame L i Perform bicubic interpolation upsampling (4 times) and residual analysis to obtain the potential HR image frame X i The specific calculation formula is as follows:

[0077]

[0078] Among them, N r represents the clear frame reconstruction network, Represents the intermediate frame L of the LR input sequence i Perform bicubic interpolation and 4x upsampling.

[0079] Step S3-4: Using blur kernel K i For the latent HR image frame X i Perform blur downsampling (BD) to obtain a degraded LR frame SK of the same size as the LR input sequence i X i .

[0080] Step S4: Input the LR input sequence into the auxiliary branch module to obtain an image of the same size as the LR input sequence. The specific steps are as follows:

[0081] Step S4-1: First, use the blur kernel K i The LR input sequence is blurred and downsampled (BD) to obtain the auxiliary LR sequence {L i-N ,...,L i ,...,L i+N};

[0082] Step S4-2: Using feature extraction network N e Extract the features of the auxiliary LR sequence and obtain features at three different scales

[0083] Step S4-3: Further, extract features at three different scales Then, a deformable convolutional network N is used w After processing, the alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence are obtained

[0084] Step S4-4: Further, use the clear frame to reconstruct the network N r All the alignment features will be obtained The splicing operation is performed in sequence, and the clear frame reconstruction network N r It mainly consists of convolutional layers, residual blocks and sub-pixel convolutional layers to align all features. The input convolution layer and residual block perform feature aggregation to obtain aggregated features, and then the convolution layer and sub-pixel convolution layer are used to map the aggregated features into high-resolution features. The auxiliary SR frame of the same size as the LR input sequence is obtained by bicubic interpolation upsampling and residual analysis with the intermediate frame of the auxiliary LR sequence.

[0085] Step S5: Construct a loss function between the image data obtained by the main branch module and the auxiliary branch module and the LR input sequence for self-supervised training. The specific steps are as follows:

[0086] In this embodiment, the LR space target image data is input into the blur kernel estimation network N k Get the blur kernel K i Then the LR space target image data is input into the two branches of the overall framework. The main branch module first performs super-resolution restoration (4 times) on the input LR space target image data, and then uses the blur kernel K i After blurring, downsampling (4 times) is performed; the auxiliary branch module first uses the blur kernel K to input the LR space target image data iAfter blurring, downsampling (4 times) is performed, and then super-resolution restoration (4 times) is performed; finally, the degraded LR frame SK obtained by the two branch modules is i X i A loss function is constructed between the auxiliary SR frame and the original LR spatial target image data for self-supervised training;

[0087] Set the initial learning rate to 2.5×10 -5 , the optimizer is Adam (parameter settings β1=0.9,β2=0.999,ε=10 -8 ), the batch size is set to 4, the learning rate is halved every 20 rounds of training, and a total of 60 rounds of training are conducted. Among them, the loss function constructed uses the mean square error loss MSE, which is as follows:

[0088] Main branch module loss function L self ,as follows:

[0089]

[0090] The loss function L of the auxiliary branch module I ,as follows:

[0091]

[0092] The total loss function is: L total =L I +L self ;

[0093] Among them, S represents downsampling, K i represents the fuzzy operation, X i represents the potential HR image frame restored by the original LR spatial target image data through the video super-resolution module VSR, y i Represents the middle frame of the original LR input sequence.

[0094] In this embodiment, the degraded LR frame SK obtained by the main branch module and the auxiliary branch module i X i The auxiliary SR frame is self-supervised with the input original LR spatial target image data itself, without the need for additional high-resolution video data for supervision, which can greatly reduce the cost and workload of data annotation.

[0095] In this embodiment, a testing phase is also included;

[0096] The prediction result can be obtained by inputting the image to be tested into the above-mentioned trained system.

[0097] Star imagery is more complex than visible light images and is difficult to acquire due to equipment and environmental factors. Therefore, it is not suitable for supervised video super-resolution restoration methods that require extensive labeling. The restoration method described in this embodiment eliminates the need for extensive labeling and can generate blur kernels and paired data from the original low-resolution space target image data itself, without requiring additional high-resolution video data for supervision. This makes it well-suited for space target image super-resolution restoration tasks.

[0098] The restoration method described in this embodiment utilizes the APOSOS (Asiapacific Ground-based Optical Space Observation System) telescope at the Changchun Satellite Observatory. This telescope can capture, track, and measure space targets, providing high-precision measurement data for orbital refinement. However, with a diameter of only 150 mm, it collects fewer photons than larger telescopes, making it difficult to detect faint and small targets.

[0099] 5 consecutive frames and 11 frames of images from the sequence of images taken by APOSOS are used as test images, and they are magnified by 4 times and 8 times respectively to generate HR result images of 4096×4096 and 8192×8192, as shown in the figure. Figure 6 shown.

[0100] from Figure 6 The presented content clearly demonstrates that the video super-resolution reconstruction method based on self-supervised deformable convolution technology significantly improves the area, flux, and magnitude of stars and space objects in the image. Therefore, when extracting small, faint space objects, reasonable thresholds can be set for both area and flux. This helps to screen out those of a certain size that are more likely to be stars or objects rather than noise or cosmic rays.

[0101] The centroid coordinates of stars and targets will show a 4x or 8x ratio in images of different resolutions. This means that regardless of how the image resolution changes, the relative positions of stars and targets remain unchanged. Therefore, this ratio still holds in super-resolution reconstructed images, enabling the use of astronomical positioning algorithms to accurately determine the orbital position of the target. Table 1 shows the comparison data of super-resolution reconstruction results;

[0102] Table 1

[0103]

[0104] To more accurately identify and locate faint, small objects, a space target positioning experiment based on super-resolution reconstruction was conducted. In this experiment, a MEO navigation satellite (NORAD ID: 39155) was selected as the observation target. Due to the limited aperture of the APOSOS telescope and its range constraints, observation of this space target fully meets the detection requirements for faint, small celestial bodies. Table 2 shows the trajectory and external symbol accuracy of navigation satellite 39155 at 1X (original resolution). Table 3 shows the trajectory and external symbol accuracy of navigation satellite 39155 at 4X (quadruple super-resolution).

[0105] Table 2

[0106]

[0107] Table 3

[0108]

[0109] After completing astronomical positioning and target identification, the OC values were compared with the target's high-precision ephemeris data and confirmed to meet the standards required for subsequent orbit refinement. Furthermore, compared to the original image sequence, the super-resolution reconstructed HR image sequence trajectory is more complete, with no missing points, providing more favorable conditions for subsequent orbit refinement.

[0110] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A spatial target image super-resolution restoration system based on deformable convolution, characterized by: The restoration system includes a fuzzy kernel estimation network, a main branch module and an auxiliary branch module; The main branch module and the auxiliary branch module are both composed of a video super-resolution module and a post-blur downsampling module; The blur kernel estimation network is used to perform blur kernel estimation on the LR input sequence to obtain the blur kernel; The post-blur downsampling module uses a blur kernel to perform post-blur downsampling on the LR input sequence to obtain an auxiliary LR sequence; and uses a blur kernel to perform post-blur downsampling on the potential HR image frame to obtain a degraded LR frame of the same size as the LR input sequence; The video super-resolution module is used to extract features of the LR input sequence and the auxiliary LR sequence and obtain alignment features of adjacent frames and intermediate frames of the LR input sequence and alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence; Sequentially concatenate the alignment features of adjacent frames and intermediate frames of the LR input sequence to obtain a potential HR image frame of the same size as the LR input sequence; sequentially concatenate the alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence to obtain an auxiliary SR frame of the same size as the LR input sequence; A loss function is constructed between the degraded LR frame of the same size as the LR input sequence, the auxiliary SR frame of the same size as the LR input sequence, and the LR input sequence for self-supervised training to achieve the restoration of spatial target images.

2. The system for super-resolution restoration of spatial target images based on deformable convolution according to claim 1, characterized in that: The video super-resolution module VSR consists of a feature extraction network, a deformable convolutional network and a clear frame reconstruction network; The feature extraction network is used to extract features of the LR input sequence and the auxiliary LR sequence, and obtain features of different scales; The deformable convolutional network is used to separately process the features at different scales obtained by the feature extraction network, and correspondingly obtain the alignment features of adjacent frames and intermediate frames of the LR input sequence and the alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence; The clear frame reconstruction network is used to sequentially splice the alignment features of adjacent frames and intermediate frames of the LR input sequence obtained by the deformable convolutional network to obtain a potential HR image frame of the same size as the LR input sequence; it is also used to sequentially splice the alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence obtained by the deformable convolutional network to obtain an auxiliary SR frame of the same size as the LR input sequence.

3. The system for super-resolution restoration of spatial target images based on deformable convolution according to claim 1, characterized in that: The process of obtaining the blur kernel by the blur kernel estimation network is as follows: The blur kernel estimation network is a neural network consisting of a convolutional layer, multiple residual channel attention mechanism RCAB modules, an adaptive average pooling layer, a fully connected layer and a softmax function; After passing through a convolutional layer, the input features pass through two RCAB modules, an adaptive average pooling layer, and two more RCAB modules, and are input to the fully connected layer through two convolutional layers Conv. Finally, the blur kernel is obtained after residual analysis.

4. The system for super-resolution restoration of spatial target images based on deformable convolution according to claim 3, characterized in that: Each RCAB module consists of two CA modules, a global average pooling layer, and a Sigmoid function; each CA module consists of two convolutional layers Conv and a Relu function; The feature input after the convolution operation of the convolution layer Conv is subjected to a convolution, Relu and convolution operation by the first CA module to obtain the feature f, and then global average pooling is performed by the global average pooling layer. The feature f is then subjected to a convolution, Relu and convolution operation by another CA module to be rescaled to obtain the feature x. The feature x is then multiplied element-wise with the feature f after passing through the Sigmoid function. Finally, the feature input and the element-by-element multiplication result are added element-by-element.

5. The system for super-resolution restoration of spatial target images based on deformable convolution according to claim 1, characterized in that: The feature extraction network adopts a pyramid convolutional neural network, which consists of a convolution layer Conv, a Leaky ReLU function and a residual block ResBlock; the residual block ResBlock consists of a convolution layer Conv+ReLU function; The deformable convolutional network consists of multiple layers of deformable convolutional layers Dconv and connection layers Concat; The clear frame reconstruction network consists of a convolutional layer, a residual block and a sub-pixel convolutional layer.

6. A method for super-resolution restoration of spatial target images based on deformable convolution, characterized by: The restoration method is implemented by a spatial target image super-resolution restoration system based on deformable convolution according to any one of claims 1 to 5; the implementation process of the restoration method is: Step S1: constructing a dataset of LR space target image data, which consists of an LR input sequence; Step S2: Use the blur kernel estimation network to perform blur kernel estimation on the LR input sequence to obtain the blur kernel; Step S3: Input the LR input sequence into the main branch module to obtain a degraded LR frame of the same size as the LR input sequence; Step S4: Input the LR input sequence into the auxiliary branch module to obtain an auxiliary SR frame of the same size as the LR input sequence; Step S5: Construct a loss function between the degraded LR frames and auxiliary SR frames obtained by the main branch module and the auxiliary branch module and the LR input sequence for self-supervised training.

7. The method for super-resolution restoration of spatial target images based on deformable convolution according to claim 6, characterized in that: The specific process of step S3 is: Step S3-1: Use the feature extraction network in the video super-resolution module to extract features of the LR input sequence, and obtain features of three different scales of adjacent frames of the LR input sequence and features of three different scales of the middle frame of the LR input sequence; Step S3-2: Using a deformable convolutional network to process the features of adjacent frames and intermediate frames at three different scales, the alignment features of adjacent frames and intermediate frames of the LR input sequence are obtained; Step S3-3: Use the clear frame reconstruction network to splice the aligned features in order, input the aligned features into the convolution layer and the residual block for feature aggregation to obtain aggregated features, and then use the convolution layer and sub-pixel convolution layer to map the aggregated features into high-resolution features, and then combine them with the LR input sequence intermediate frame y i Perform bicubic interpolation upsampling and residual analysis to obtain the potential HR image frame; Step S3-4: Blur the latent HR image frame using the blur kernel and then downsample it to obtain a degraded LR frame of the same size as the LR input sequence.

8. The method for super-resolution restoration of spatial target images based on deformable convolution according to claim 6, characterized in that: The specific process of step S4 is: Step S4-1: blur the LR input sequence using a blur kernel and then downsample it to obtain an auxiliary LR sequence; Step S4-2: extract features of the auxiliary LR sequence using a feature extraction network to obtain features of adjacent frames of the auxiliary LR sequence at three different scales and features of the intermediate frames at three different scales; Step S4-3: Obtain alignment features of adjacent frames and intermediate frames of the auxiliary LR sequence after processing using a deformable convolutional network; Step S4-4: Use a clear frame reconstruction network to sequentially concatenate the obtained alignment features to obtain an auxiliary SR frame of the same size as the LR input sequence.

9. The method for super-resolution restoration of spatial target images based on deformable convolution according to claim 6, characterized in that: The specific process of step S5 is: First, the LR space target image data is input into the blur kernel estimation network to obtain the blur kernel; Then, the LR space target image data is input into two branch modules. The main branch module first performs super-resolution restoration on the input LR space target image data, and then blurs it with a blur kernel and downsamples it; the auxiliary branch module first blurs the input LR space target image data with a blur kernel and downsamples it, and then performs super-resolution restoration. Finally, a loss function is constructed between the degraded LR frames and auxiliary SR frames obtained by the two branch modules and the LR spatial target image data for self-supervised training.

10. The method for super-resolution restoration of spatial target images based on deformable convolution according to claim 9, characterized in that: The loss function adopts mean square error loss MSE; The loss function is: L total =L I +L self ; Where, L self is the main branch module loss function, L I is the loss function of the auxiliary branch module.

Images