Satellite-borne edge end image reconstruction method
Through an image super-resolution network based on degraded representation learning, the challenges of image compression and recovery in the satellite-borne environment are solved, efficient image compression and high-quality recovery are achieved, and data transmission needs of the satellite-borne environment are adapted.
Patent Information
- Application Number
- CN202411465131.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art cannot effectively compress large-size optical images in a satellite-borne environment, while ensuring high-quality recovery of images, especially during the transmission process, which cannot meet the goals of large-data transmission and high-quality image recovery.
Using an image super-resolution network based on degradation representation learning, low-resolution images are acquired and super-resolution reconstruction is carried out through degradation encoding networks and degradation-perceptual super-resolution networks, reducing the number of pixels and ensuring the recovery of important information.
It realizes efficient image compression and good ground recovery in a satellite-based environment, ensures image details and target acceptability, and can perform reverse super-score and image super-resolution recovery, reduces communication delay and improves channel transmission speed.
Smart Images

Figure CN119006291A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image compression and restoration, and in particular to a satellite-borne edge image reconstruction method. Background Art
[0002] The increasing complexity of satellite platform observation missions and the continuous improvement of detector manufacturing levels have made it possible to use large-field-of-view, high-frame-rate, and high-resolution optical payloads to observe the Earth in orbit. However, optical satellite images have a large field of view, high frame rate, and large data volume. The actual bandwidth of current mature satellite-to-ground communication transmission is only in the hundreds of megabytes, and the massive amount of original satellite images collected cannot be transmitted in real time. Therefore, it is necessary to complete high-reliability real-time processing on the satellite to obtain the required target information of interest. Compression of the original large-size optical image before transmission is a necessary step for real-time image transmission, and it is required to be able to achieve high-quality restoration after reception on the ground.
[0003] At present, most of the existing earth observation satellites store the observation image data on the satellite first, and then slowly transmit it to the ground for extraction, identification and other processing of the target of interest. This model is relatively mature and reliable, but it requires that the on-board storage capacity is large enough and the detector cannot be turned on for a long time. The ground often delays the processing of the image data after receiving it from the satellite by hours, which is not suitable for all-day monitoring and high-time-sensitive target intelligence information acquisition tasks. There are also a few earth observation satellites with on-board image processing capabilities, but due to the limitations of space radiation environment, equipment power consumption and volume, they only realize on-board image compression, blind pixel elimination and other pre-processing and single-frame target candidate point detection processing, and the algorithm software does not support rapid on-orbit reconstruction, which is a certain gap from actual needs. The existing image compression methods are all based on encoding, and the size of the encoded image file is significantly reduced. The size of the decoded image is the same as the size before compression. When preprocessing is done on the satellite, the size of the image is not reduced. The lossless compression characteristics of Huffman coding make the compression ratio relatively low, and the high compression rate characteristics of transform coding easily lead to the loss of key pixels in the image. The coding-based method cannot reduce the number of pixels. Optical satellite images have a large amount of data, and it is difficult to transmit all the massive original satellite images collected in real time. Traditional compression technology cannot simultaneously meet the goals of large data volume transmission and high-quality image restoration. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a satellite-borne edge image reconstruction method in view of the shortcomings of the existing technology, which improves the image compression capability while adapting to the anti-irradiation environment on the satellite, has the ability to be well restored on the ground, and ensures that the restored image details and targets are acceptable, and can realize inverse super-resolution and image super-resolution restoration of large-size optical images.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a satellite-borne edge image reconstruction method, comprising the following steps: S1, obtain low-resolution images of the satellite edge; S2, inputting the low-resolution image into a degradation coding network to obtain a degradation representation vector; S3, inputting the low-resolution image and the degradation representation vector into a degradation-aware super-resolution network to obtain a restored image; The degradation coding network includes a plurality of cascaded convolutional layers, and the last convolutional layer is connected to a global pooling layer; The degradation-aware super-resolution network includes multiple cascaded residual groups, each of the residual groups includes multiple cascaded degradation-aware modules, and all the degradation-aware modules are connected to the global pooling layer through a fully connected layer; the input side of the first residual group is connected to an input convolutional layer, and the output of the input convolutional layer is spliced with the output of the last residual group and then input into the upsampling layer.
[0006] The existing coding-based method cannot reduce the number of pixels, and the interpolation method is used to extract the image pixels to reduce the number of pixels at the data source. Although the low-resolution image obtained by the interpolation method reduces the image pixels, it cannot ensure that the important pixel information will not be lost in the interpolation process and that the important pixel information can be restored later. The image super-resolution network based on degradation representation learning of the present invention can better restore the low-resolution image and ensure that the important pixel information is correctly restored. The image super-resolution network based on degradation representation learning includes two parts: a degradation coding network and a degradation-aware super-resolution network, wherein the degradation coding network learns the degradation representation from the input low-resolution image and processes the low-resolution image into a degradation representation vector. The degradation-aware super-resolution network sends the low-resolution image to the convolution, residual group, degradation perception module, and upsampling layer for super-resolution reconstruction to obtain the final super-resolution result. In this process, a fully connected layer is used to process the degradation representation vector and then sent to the degradation perception module to provide implicit degradation information, thereby completing image reconstruction. The advantage of the present invention is that it can reduce pixel points from the data source. At the same time, the network after learning does not need to be trained again, and the image reconstruction task can be performed once and for all, and it has strong robustness.
[0007] In the present invention, the loss function of the degenerate coding network is It is expressed as: ; in, is the number of negative samples, is the temperature parameter, represents the dot product of vectors, It is a degraded representation vector obtained by further mapping the degraded representation to a certain feature space using a multi-layer perceptron. Representation and Vectors with the same degenerate characteristics, Representation and Vectors with different degradation characteristics.
[0008] In the present invention, in the degradation-aware super-resolution network, two adjacent residual groups are connected via a convolutional layer, and the output of the last residual group is connected to an output convolutional layer.
[0009] In the present invention, the degradation perception module includes a plurality of cascaded convolutional layers.
[0010] In the present invention, the number of the residual groups is 5.
[0011] In the present invention, each of the residual groups includes 5 degradation perception modules.
[0012] In the present invention, the training process of the degradation coding network and the degradation-aware super-resolution network includes: obtaining a true value image-compressed image data pair, taking the compressed image as the input of the degradation coding network and the degradation-aware super-resolution network, taking the true value image as the output of the degradation coding network and the degradation-aware super-resolution network, training the degradation coding network and the degradation-aware super-resolution network, and obtaining the trained degradation coding network and the degradation-aware super-resolution network.
[0013] As an inventive concept, the present invention also provides a satellite-borne edge image reconstruction system, comprising:
[0014] The sending end is used to input the low-resolution image into the degradation coding network to obtain a degradation representation vector;
[0015] The receiving end is used to input the low-resolution image and the degradation representation vector into a degradation-aware super-resolution network to obtain a restored image.
[0016] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses a large amount of data to train the deep information model, uses the distribution law in the massive data, learns the nonlinear mapping from low-resolution images to high-resolution images, realizes information increment, and completes the restoration of the image. This method uses the distribution law in a large number of samples as information increment, and the artificial neural network formed is conducive to the retention and restoration of key information in the image. In the present invention, at the sending end, the original image can be reduced in file size after being processed by the compression algorithm, which is convenient for accelerating the channel transmission speed and reducing communication delay. After receiving the compressed image, the receiving end performs the image restoration task to restore the information lost in the compressed image as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 This is a network structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0019] The image reconstruction process of the embodiment of the present invention includes: S1. Complete the training of the DASR super-resolution network model and reduce the inference error to the minimum; S2. Complete the bicubic interpolation compression of the image to obtain the compressed image. Specifically, the pixel value of the interpolated image is taken as follows: .in is the pixel coordinate of the original image (i.e. the true value image), As the reference point, and is the distance value, is the weight matrix, is the sliding window matrix; S3. Complete the image super-resolution network inference, send the compressed image to the image super-resolution restoration network, and output the restored image. Specifically, the degradation coding network learns the degradation representation from the input low-resolution image and processes the low-resolution image into a degradation representation vector. The degradation-aware super-resolution network first sends the low-resolution image to a layer of 3×3 convolution to extract shallow features, and then sends the extracted features to 5 residual groups to extract deep features, where each residual group includes 5 degradation-aware modules. Finally, the extracted deep features are added to the shallow features and sent to the upsampling layer for super-resolution reconstruction to obtain the final super-resolution result. The degradation representation vector is processed by a fully connected layer and sent to the degradation-aware module to provide implicit degradation information, thereby completing image reconstruction; S4. Perform image quality evaluation, calculate the structural similarity index and peak signal-to-noise ratio index between the original image and the restored image, and manually verify the positions of important pixels between the true image and the reconstructed restored image; S5. Test a large number of large-size images (image size is not less than 2000×2000), and statistically analyze the overall indicators to determine the restoration effect, and verify that the restoration effect is acceptable. Specifically, perform S2., S3., and S4 steps on a large number of large-size images one by one to ensure that the image reconstruction method of the embodiment of the present invention has strong robustness; S6. Execute S2 and S3 on the new input image. Specifically, after the steps S1 to S5 are completed, it can be ensured that the image reconstruction method of the embodiment of the present invention has strong robustness. When the application is deployed later, there is no need to execute S1., S4 and S5, but only the compression and super-resolution network inference steps need to be executed on the new input image, that is, prediction is performed using the trained model.
[0020] The DASR super-resolution network training process includes: S11. Preprocess the data set to obtain a true image-compressed image data pair, where the compressed image is used as the input in the network training process and the true image is used as the output in the network training process. Specifically, perform an image bicubic interpolation compression operation on any true image in the data set (consistent with the S2 process) to obtain a compressed image, save it as a file, and name it the same as the true image and add a suffix character as the mark of the compressed image, thereby obtaining a true image-compressed image data pair; S12. transmitting a set of true value image-compressed image data pairs to the network, and the network calculates the cost function; S13. Feed the obtained loss back to the back propagation to update the network weights; S14. Iteratively repeat steps S12-S13 until the set number of training steps is reached and the weight update is obtained. At this time, the network loss has been reduced to the minimum, and the degradation coding network and the degradation-aware super-resolution network are obtained respectively.
[0021] From the perspective of network model training and reasoning, the image compression and image super-resolution restoration networks are trained to achieve the fitting of compression and restoration algorithms. The training process uses a deep learning optimization framework for back propagation and iterative optimization. The reasoning process uses a deep learning forward propagation operation. The compression and restoration algorithm is based on the interpolation method-based degradation representation learning image super-resolution (Degradation Aware Super Resolution, DASR) network model for image compression and restoration. The specific framework of the neural network in the embodiment of the present invention is as follows: Figure 1 shown.
[0022] Figure 1 In the paper, the single-frame image super-resolution reconstruction algorithm for multiple degradations includes two parts: a degradation coding network and a degradation-aware super-resolution network. The degradation coding network is responsible for extracting degradation representations from low-resolution images, and then the degradation-aware super-resolution network is responsible for using the degradation information in the degradation representation to perform super-resolution reconstruction on the low-resolution image.
[0023] The degradation coding network adopts the momentum contrast unsupervised representation learning network structure. The input low-resolution image passes through a 6-layer convolutional neural network and then a global pooling layer to obtain the final degradation representation vector.
[0024] Contrastive learning does not require high-grained restoration or learning pixel-level features. It uses data enhancement and mutual information for learning without supervised information or data reconstruction. Data enhancement includes color data enhancement, affine transformation, and blurring. Mutual information is used to measure the correlation between two data. The degradation models at different positions in each image are consistent (that is, any two batches in the image are positive samples of each other), while the degradation models between different images are different (negative samples). Afterwards, the degradation coding network extracts the degradation representation vector from these image blocks, and uses a multi-layer perceptron to further map the degradation representation to a feature space, and the encoding is obtained. , and The degraded coding network is optimized using the information noise contrast estimation loss as the loss function: ; In this feature space, to make the loss value approach 0, we need to have the same degradation and should be as close as possible, and x and x− with different degenerates should be as far away from the origin as possible.
[0025] The scale and diversity of negative samples have an important impact on performance. In order to ensure that the degradation representation learning can learn robust degradation representations without being affected by the image content, the negative samples used in the degradation coding network training process need to cover different image degradations and rich image content. To achieve this goal, the negative sample queue constructed during the training process updates the current batch of low-resolution image blocks to the negative sample queue at each iteration, and removes the earliest negative sample that enters the queue. This setting can ensure that the negative sample queue can provide negative samples with image degradation and diverse image content.
[0026] First, the low-resolution image is sent to a layer of 3×3 convolution to extract shallow features; then the extracted features are sent to 5 residual groups to extract deep features, each of which includes 5 degradation-aware modules; finally, the extracted deep features are added to the shallow features and sent to the upsampling layer for super-resolution reconstruction to obtain the final super-resolution result. In order to make full use of the degradation information in the degradation representation, the degradation representation is passed through a fully connected layer to obtain R, and then R is sent to the degradation-aware module to provide implicit degradation information, which enables the network to cope with a variety of different degradations.
[0027] Different image degradations require different filters for processing. In the degradation-aware convolution layer of the degradation-aware module, the degradation representation vector R is processed into a convolution kernel w of a depth-separable convolution. The depth-separable convolution kernel obtained based on the degradation representation vector can be adaptively adjusted according to the degradation information in the degradation representation vector R, so that the network has the ability to handle different image degradations.
[0028] The degradation representation vector R is processed into scaling factors v for different channels. The scaling factors are multiplied by the corresponding channels in the feature F, and the response values of different channels are scaled to obtain the feature F2. The scaling factors obtained based on the degradation representation vector R can adaptively adjust the scaling ratio according to the degradation information therein. Using the dynamic convolution kernel w and the dynamic scaling factor v in the two branches, the degradation-aware convolution can adaptively adjust itself according to the degradation information in the degradation representation R, so that the entire network can handle a variety of different degradations.
[0029] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0030] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A satellite-borne edge image reconstruction method, characterized in that: The following steps are involved: S1, obtain low-resolution images of the satellite edge; S2, inputting the low-resolution image into a degradation coding network to obtain a degradation representation vector; S3, inputting the low-resolution image and the degradation representation vector into a degradation-aware super-resolution network to obtain a restored image; The degradation coding network includes a plurality of cascaded convolutional layers, and the last convolutional layer is connected to a global pooling layer; The degradation-aware super-resolution network includes multiple cascaded residual groups, each of the residual groups includes multiple cascaded degradation-aware modules, and all the degradation-aware modules are connected to the global pooling layer through a fully connected layer; the input side of the first residual group is connected to an input convolutional layer, and the output of the input convolutional layer is spliced with the output of the last residual group and then input into the upsampling layer.
2. The satellite-borne edge image reconstruction method according to claim 1, characterized in that: The loss function of the degenerate coding network is It is expressed as: ; in, is the number of negative samples, is the temperature parameter, represents the dot product of vectors, It is to use a multi-layer perceptron to further map the degraded representation to a feature vector obtained in a certain feature space. Representation and Vectors with the same degenerate characteristics, Representation and Vectors with different degradation characteristics.
3. The satellite-borne edge image reconstruction method according to claim 1, characterized in that: In the degradation-aware super-resolution network, two adjacent residual groups are connected via a convolutional layer, and the output of the last residual group is connected to an output convolutional layer.
4. The satellite-borne edge image reconstruction method according to claim 1, characterized in that: The degradation perception module includes a plurality of cascaded convolutional layers.
5. The satellite-borne edge image reconstruction method according to claim 1, characterized in that: The number of residual groups is 5.
6. The satellite-borne edge image reconstruction method according to claim 1, characterized in that: Each of the residual groups includes 5 degradation perception modules.
7. The satellite-borne edge image reconstruction method according to any one of claims 1 to 6, characterized in that: The training process of the degradation coding network and the degradation-aware super-resolution network includes: A true value image-compressed image data pair is obtained, the compressed image is used as the input of a degradation coding network and a degradation-aware super-resolution network, the true value image is used as the output of the degradation coding network and the degradation-aware super-resolution network, the degradation coding network and the degradation-aware super-resolution network are trained, and the trained degradation coding network and the degradation-aware super-resolution network are obtained.
Citation Information
Patent Citations
Image blind super-resolution method based on learning degradation information
CN116309051A
Image blind super-resolution reconstruction method based on weak supervision contrast learning
CN116721013A
Blind image super-resolution reconstruction method based on diffusion model and attention mechanism
CN118485574A
