A remote sensing image compression method and system based on reference image texture migration

Abstract

The application provides a high-resolution optical remote sensing image compression method and system with reference to historical images, including data preparation, registration of the found reference image and the re-visited image to be compressed, and then cutting the images as training data and test data; setting an optical remote sensing image deep learning compression model based on reference texture migration, adopting a learning-based network architecture of down-sampling first and super-resolution later, taking the reconstruction loss of the original image, the reconstruction loss of the low-resolution image and the entropy of the encoding code as the loss function to train the set network, and obtaining the trained optical remote sensing image compression model; based on the trained optical remote sensing image compression model, inputting the optical remote sensing image to be compressed, extracting the compressed image and performing quality evaluation. For the reference area with weak correlation, the self-texture learning method is used to replace the reference texture migration, which can not only effectively improve the image quality of the compressed high-resolution remote sensing image, but also has more flexible rate-distortion optimization.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image compression, and particularly relates to a high-resolution remote sensing image compression method with reference to historical images, and adaptive rate-distortion optimization according to the similarity of reference images. BACKGROUND

[0002] With the development of remote sensing technology, high spatial resolution remote sensing images are widely used, which makes the amount of data generated by remote sensing satellites larger and larger. However, the limited storage on the satellite and the limited bandwidth of the transmission link between the satellite and the ground limit the real-time application of remote sensing satellites, so efficient image compression methods become more and more important. Image compression is divided into two compression types: lossless compression and lossy compression. For remote sensing images, lossless compression can only achieve 2-5 times compression, which cannot meet the current demand, so the compression methods currently studied are all lossy compression. However, lossy compression will introduce image distortion, in order not to affect the observation and analysis of the compressed image, it is necessary to retain as much key information as possible in the original data while removing all redundant information under high compression ratio.

[0003] In the past 20 years, people have conducted extensive research on remote sensing image compression technology, and various algorithms have been developed. Several typical image compression methods such as differential pulse code modulation (DPCM), adaptive pulse code modulation (ADPCM), discrete cosine transform (DCT) and discrete wavelet transform (DWT) have been applied to remote sensing image compression. Generally, DPCM and ADPCM can be regarded as a prediction coding framework, and due to its simple hardware implementation, it has become one of the most successful RS image coding methods. Their variants have been used in LANDSAT, IKONOS, Worldview, Quickbird and Geoeye satellites.

[0004] With the progress of mathematical tools, transform coding has developed rapidly and become one of the most effective and popular RS image compression methods. The purpose of transform coding is to transform remote sensing images from spatial domain to frequency domain, so as to code more effectively. DCT and DWT are two foundations of the commonly used transform coding framework. Methods developed on this basis include JPEG, AVS and JPEG 2000, which have been applied to "Resource 3" satellite.

[0005] In recent years, image compression based on deep neural networks (DNN) has attracted more and more attention, among which the most typical are convolutional neural networks (CNN) and recurrent neural networks (RNN). The method of CNN mainly uses convolutional autoencoder, and through end-to-end training, the encoder learns the mapping from high-dimensional image space to low-dimensional feature space, so as to realize image compression. The RNN method gradually reduces the residual error by embedding the autoencoder and recycling the residual input network, which can capture high-order and long-range connections between pixels, but the most significant defect is that the time cost is too high. Recently, some researchers have studied adaptive context models for entropy estimation to guide the optimization process of neural network parameters, so as to achieve the best balance between reconstruction error and required bits (entropy). The entropy estimation technique greatly improves the learning-based compression algorithm, among which the most representative methods are hyperprior model and joint model. However, the existing context-based hyperprior and joint model still ignores some problems. First, the probability estimation of the to-be-encoded elements in the latent representation depends on the local module of the previous decoded elements, which limits the accuracy of long-term conditional probability estimation. Secondly, the hyperprior information used is not regularized and utilized, although this part will be encoded in the bit stream, but it cannot be used for image reconstruction.

[0006] High-resolution remote sensing images have rich details and complex textures, and the compression method applied to general images usually has poor performance when compressing remote sensing images, which seriously affects the observation and analysis of the compressed images. The present application proposes that considering the non-periodic revisit of remote sensing satellites to the same area, there is great similarity between these revisit images, which means that the redundant information between them can be further compressed. At present, there is no related paper published at home and abroad on the compression of revisit redundancy of remote sensing images, and relevant research is urgently needed. SUMMARY

[0007] The present application provides a remote sensing image compression method based on reference image texture migration for the compression task of high-resolution remote sensing satellite revisit images of the same area.

[0008] The technical scheme provided by the present application is a high-resolution optical remote sensing image compression method based on reference historical image, comprising the following steps:

[0009] Step A, data preparation, the found reference image is registered with the to-be-compressed revisit image, and then the image is cropped as training data and test data;

[0010] Step B, setting an optical remote sensing image deep learning compression model based on reference texture migration, adopting a learning-based network architecture of down-sampling first and super-resolution later, and including the following processing when corresponding training,

[0011] The extraction of the image texture features comprises extracting texture features from the reference image and the current image respectively by using a learnable texture extractor, constructing a hard attention mapping H and a soft attention mapping S by using a standard inner product as a similarity, and recording the position and confidence of the texture in the reference image related to the current image;

[0012] The coding and decoding of the down-sampling and super-resolution comprises filtering out most of the spatial information by down-sampling the current image, reconstructing a low-resolution image after the down-sampling, and recovering a reconstructed image of the original resolution by introducing a reference image as auxiliary information of a super-resolution network for corresponding super-resolution of the down-sampling;

[0013] Step C: training the network set in step B by using the reconstruction loss of the original image, the reconstruction loss of the low-resolution image, and the entropy of the coding code as a loss function, to obtain a trained optical remote sensing image compression model;

[0014] Step D: inputting an optical remote sensing image to be compressed into the optical remote sensing image compression model trained in step C, extracting a compressed image, and performing quality evaluation.

[0015] Moreover, the learnable texture extractor is a shallow neural network for extracting texture features from an input image, and sequentially comprises a CONV, a CONV, a POOL, a CONV, a CONV, a POOL, and a CONV, wherein the CONV is a convolution layer, and the POOL is a pooling layer.

[0016] Moreover, the similarity of the reference image is used as a judgment basis in step D to perform a reference texture-based adaptive compression method and an adaptive down-sampling method.

[0017] Moreover, the coding and decoding of the down-sampling and super-resolution in the optical remote sensing image deep learning compression model based on reference texture migration has the following implementation principle,

[0018] During coding, the features extracted from the current image and the reference image are Q and K respectively, and c is a dimension, is a real field, the features K and Q are unfolded into vectors unfold(K) and unfold(Q) from the pixel dimension (h x w), and the following formula is used:

[0019]

[0020] Each sub-feature q in Q i Each sub-feature k in K j The correlation between them is calculated, and the calculation formula uses a standard inner product:

[0021]

[0022] From the above correlation, the hard attention map and the soft attention map can be obtained, the i-th component of the hard attention map H is The i-th component of the soft attention map S is

[0023] Where the mean of the soft attention map is As a measure of the similarity between the reference image and the current image, it guides the model to adaptively compress and downsample.

[0024] In decoding, the hard attention map H and the soft attention map S are calculated using the features Q' extracted from the upsampled low-resolution image and the reference image features K, and the implementation is the same as at the encoding end. Then the transferable texture features are constructed, and the hard attention mechanism is used to find the most similar components from the reference texture features, which form the transferable texture features. The formula is as follows,

[0025]

[0026] Where, t i is the i-th component of T, is the feature component at the corresponding position i in the feature K searched according to the hard attention map;

[0027] For the region using the self-texture reference model in adaptive compression, the feature T needs to be modified. The implementation is to extract the texture feature block from the high-resolution image block that has undergone coarse compression, read out the position of the region using the self-texture reference from the code stream, and replace the value of the corresponding position feature component in the original feature T with this texture feature. The replacement area of each block is 1 / 4 of the original feature T in length and width.

[0028] The modified feature T is the transferable texture feature input into the super-resolution generation network. Before super-resolution, feature fusion is also needed, and the formula is as follows:

[0029]

[0030] Where, F out is the output fusion feature, Conv and Concat represent convolution operation and tensor splicing operation respectively, is the operation of multiplying the elements at the corresponding positions of the two matrices;

[0031] Finally, the fusion feature is input into the super-resolution generation network to output the reconstructed image of the original resolution.

[0032] Moreover, in step C, the reconstruction loss of the original image, the reconstruction loss of the low-resolution image, and the entropy of the encoding code are used as the loss function to train the network of step B.

[0033] In another aspect, the present application also provides a high-resolution optical remote sensing image compression system for reference historical images, which is used to implement the high-resolution optical remote sensing image compression method for reference historical images as described above.

[0034] Moreover, the system comprises the following modules,

[0035] The first module is used for data preparation, and the found reference image is registered with the revisit image to be compressed, and then the image is cropped as training data and test data;

[0036] The second module is used for setting an optical remote sensing image deep learning compression model based on reference texture migration, and adopts a learning-based network architecture of down-sampling first and super-resolution later, and includes the following processing when corresponding training is performed,

[0037] Extraction of image texture features, including extracting texture features from the reference image and the current image respectively by using a learnable texture extractor, constructing a hard attention map H and a soft attention map S by using a standard inner product as similarity, and recording the position and confidence of the texture related to the current image in the reference image;

[0038] Coding and decoding of down-sampling first and super-resolution later, including filtering out most of the spatial information by down-sampling the current image, reconstructing a low-resolution image after down-sampling at the time of decoding, and then performing corresponding super-resolution on the down-sampling by introducing a super-resolution network of the reference image as auxiliary information, to restore a reconstructed image of the original resolution size;

[0039] The third module is used for training the network by taking the reconstruction loss of the original image, the reconstruction loss of the low-resolution image, and the entropy of the encoding code as a loss function, to obtain a trained optical remote sensing image compression model;

[0040] The fourth module is used for inputting the optical remote sensing image to be compressed based on the trained optical remote sensing image compression model obtained by the third module, extracting the compressed image, and performing quality evaluation.

[0041] Alternatively, the system comprises a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the high-resolution optical remote sensing image compression method for reference historical images as described above.

[0042] Alternatively, the system comprises a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed to implement the high-resolution optical remote sensing image compression method for reference historical images as described above.

[0043] The present application provides a high-resolution remote sensing image compression method which can use historical revisit images as reference information, and has the following advantages compared with the prior art:

[0044] (1) The method makes full use of the periodic revisit characteristics of remote sensing satellites, and can effectively improve the quality of compressed images.

[0045] (2) The method has strong robustness to reference image changes, and can be applied to image compression of optical remote sensing satellites in most scenarios.

[0046] (3) The adaptive self-texture learning method can not only improve the situation of insufficient reference images, but also realize more flexible rate-distortion optimization. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The encoding flowchart of the embodiment of the present application.

[0048] Figure 2 The decoding flowchart of the embodiment of the present application.

[0049] Figure 3 The network structure diagram of end-to-end training of the embodiment of the present application.

[0050] Figure 4 The schematic diagram of the adaptive compression strategy of the embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions of the present application will be described in detail below in combination with the drawings and embodiments.

[0052] The main contribution of the present application is to propose a method of using historical revisit images as a reference for high-resolution remote sensing image compression. The technical solutions of the present application can be applied to multiple compression methods. Compared with reference-free compression, the present application can significantly improve the clarity of compressed images. In addition, the present application can use the current popular deep learning framework to train the model, including Pytorch, TensorFlow, etc.

[0053] The application provides a high-resolution optical remote sensing image compression scheme with reference to historical images, including data preparation, registration of high-resolution remote sensing images and data preprocessing; texture feature extraction of the images, texture similarity calculation using the texture features of the reference images and the current images to obtain the redundancy information between the two images; a coding and decoding mode of downsampling first and then super-resolution, adjustment of the parameters of the Gaussian blur kernel based on the similarity, 2 times downsampling of the current image using Gaussian blur and bicubic interpolation to filter out most of the spatial information, reconstruction of the downsampled image at the decoding time, then 2 times super-resolution of the downsampled image through the super-resolution network to obtain the final decoded image by introducing the reference image as auxiliary information; rate distortion optimization, using the similarity of the reference image and the current image as the basis, for the reference area with weak correlation, using the self-texture learning method instead of reference texture migration, which can more flexibly adjust the rate distortion of the model. The application can not only effectively improve the compression image quality of high-resolution remote sensing images, but also has more flexible rate distortion optimization.

[0054] The embodiment of the application provides a high-resolution image compression method, which takes reference image texture migration as the main link to illustrate the technical scheme of the application.

[0055] The embodiment of the application provides a remote sensing image compression method based on reference image texture migration, including the following steps:

[0056] A. Data preparation, registration of the found reference image and the to-be-compressed revisit image, and then cutting the images as training data and test data.

[0057] In step A, the high-resolution remote sensing image dataset is divided into a current image and a reference image, the current image refers to the image to be compressed, the reference image refers to the image providing reference information to guide the compression, and the dataset is cut.

[0058] The embodiment uses a high-resolution remote sensing image dataset, first performs coarse registration on the geographic coordinate information of the remote sensing data, finds the historical image of the region where the current image is located, and then performs fine registration on the paired images to obtain the paired current image and reference image. After registration, the data is cut. The training data is cut into 128x128 size image blocks, and the test data is cut into 256x256 size image blocks. The size of the cut image can be adjusted according to the device used for training and testing.

[0059] In the embodiment, remote sensing images from the SPOT-5 satellite are used to make the dataset to evaluate the performance of the model, and the data is optical remote sensing images taken by the satellite in Atlanta, Madrid, Munich and other places from 2005 to 2019. The dataset uses 8bit RGB three-channel TIF format, and the ground resolution is 1m.

[0060] B. Set up the optical remote sensing image deep learning compression model based on reference texture migration, adopt the learning-based network architecture of down-sampling first and super-resolution later. The network architecture of down-sampling first and super-resolution later can significantly improve the compression ratio of the model, and the use of the information of the reference image to assist the super-resolution can restore the high-frequency information lost in the down-sampling process, so that the network can improve the quality of the compressed image while ensuring high compression ratio.

[0061] Firstly, the coding and decoding principle of the embodiment of down-sampling first and super-resolution later is as follows,

[0062] During encoding, the features extracted from the current image and the reference image are Q and K respectively, and c is the dimension, is a real field, the features are unfolded from the pixel dimension (h x w), and the features K and Q are unfolded from the pixel dimension (h x w) to the vector unfold (K) and unfold (Q) respectively, as follows:

[0063]

[0064] Each sub-feature q in Q i Each sub-feature k in K j Calculate the correlation between them, and the calculation formula uses the standard inner product:

[0065]

[0066] The hard attention mapping and the soft attention mapping can be obtained from the above correlation, the i-th component of the hard attention mapping H is The i-th component of the soft attention mapping S is

[0067] Among them, the mean of the soft attention mapping is It can be used as an index to measure the similarity between the reference image and the current image in step D, guiding the model to perform adaptive compression and adaptive down-sampling;

[0068] During decoding, the features Q' extracted from the up-sampled image of the low-resolution image and the reference image features K are used to calculate the hard attention mapping H and the soft attention mapping S, and the method is the same as that at the encoding end. Then construct the transferable texture feature, use the hard attention mechanism to find the most similar components from the reference texture feature, and these components form the transferable texture feature T, as follows:

[0069]

[0070] Among them, t i is the i-th component of T, is the feature component at the corresponding position i searched in the feature K according to the hard attention mapping;

[0071] The obtained feature T is the transferable texture feature input into the super-resolution generation network next. Before super-resolution, feature fusion is needed, and the formula is as follows:

[0072]

[0073] wherein F out is the output fusion feature, F is the low-resolution texture feature extracted from the low-resolution image, T is the transferable texture feature, S is the soft attention mapping matrix, Conv and Concat represent convolution operation and tensor splicing operation respectively, is the operation of multiplying the elements in the corresponding positions of the two matrices;

[0074] Finally, the fusion feature is input into the super-resolution generation network to output the reconstructed image of the original resolution.

[0075] For example, the network in the embodiment specifically includes the following steps in the training process: Figure 3

[0076] B1. Image texture feature extraction. The texture features K and Q are extracted from the reference image Ref and the current image Input respectively by using the learnable texture extractor (LTE), and the standard inner product of them is taken as the similarity to construct the hard attention mapping H and the soft attention mapping S. The two kinds of attention mappings record the position and confidence of the texture in the reference image related to the current image.

[0077] In the embodiment, the implementation of the learnable texture extractor and the attention mapping refers to the document Yang F, Yang H, Fu J, et al. Learning texture transformer network for image super-resolution [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 5791-5800. The network structure of the learnable texture extractor (LTE) is referred to Figure 3 The lower left corner successively includes CONV, CONV, POOL, CONV, CONV, POOL, and CONV, wherein CONV is a convolution layer, and POOL is a pooling layer. The hard attention mapping can be used to find the transferable texture in the reference image, and the mean value of the soft attention mapping can be used to estimate the similarity degree of the current image block and the reference image.

[0078] ​B2. The codec of down-sampling first and super-resolution later, including 2 times down-sampling of the current image to filter out most of the spatial information, reconstructing the low-resolution image after down-sampling in the decoding, and then super-resolving the down-sampling by 2 times through the super-resolution network with the reference image as auxiliary information to restore the reconstructed image of the original resolution.

[0079] In the embodiment, 2 times down-sampling is performed on the current image Input to obtain a low-resolution image LR, and the resolution is reduced from (128×128×3) to (64×64×3). The down-sampling process can filter out most of the spatial information, and then the network encodes and decodes (Codec) the low-resolution image. After encoding and decoding, the reconstructed low-resolution image is obtained, and then the down-sampled image is super-resolved by 2 times through the super-resolution network with the reference image as auxiliary information. Similarly, the low-resolution texture feature F is extracted from the encoded and decoded low-resolution image by the learnable texture extractor LTE. The transferable feature T is obtained from the reference texture feature through the hard attention mechanism, and then the low-resolution texture feature and the transferable feature are fused through the soft attention mechanism. The obtained fused feature is input into the super-resolution generation network G to restore the reconstructed image output of the original resolution (128×128×3). The structure of the super-resolution generation network is shown in Figure 3 The lower right corner refers to the literature Zhang Z, Wang Z, Lin Z, et al. Image super-resolution by neural texture transfer [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019:7982-7991. The embodiment modifies the original network, and the super-resolution multiple of the modified network is changed from 4 times to 2 times, wherein CONV is a convolutional layer, RB is a residual block (ResBlock), and TT is a feature transfer module. For specific implementation, please refer to the related literature, which will not be described herein.

[0080] C. The network set in step B is trained with the reconstruction loss of the original image, the reconstruction loss of the low-resolution image, and the entropy of the encoding code as the loss function, to obtain the trained optical remote sensing image compression model.

[0081] In the embodiment, the learnable texture extractor (LTE) and the super-resolution generation network are trained first, and the reconstruction loss of the original image is used as the loss function. The network used in the training is shown in Figure 3 If the compression method is a learning-based method, the compression module (codec) is trained first, and then the encoding and decoding process of the compression network is added between the down-sampling and the feature extractor in the above network, as shown in Figure 3Finally, the entire network including the texture extractor, super-resolution, and codec is trained with the loss function of the reconstruction loss, low-resolution image reconstruction loss, and entropy of the encoding code.

[0082] D. Based on the optical remote sensing image compression model trained in step C, input the optical remote sensing image to be compressed, and perform quality evaluation on the compressed image. In the test stage, the compression network based on reference image texture migration proposed in the embodiment of the application can be separated for encoding and decoding after training. The encoding process is shown in Figure 1 , and the decoding process is shown in Figure 2 . For the input image, downsampling and encoding are performed first, and then decoding and super-resolution are performed to finally obtain the reconstructed image.

[0083] For the convenience of reference, the codec implementation method after network training is provided.

[0084] As shown in Figure 1 , after the network training is completed, the specific implementation process of encoding for test data is as follows:

[0085] (1) Data processing.

[0086] First, coarse registration is performed using the geographic coordinate information of remote sensing data, the historical image of the region where the current image is located is found, and then fine registration is performed on the paired images to obtain the paired current image and reference image. After registration, the data is cropped, and the cropping size is determined according to the used device. In the test of the embodiment of the application, the test image is cropped into a 256x256 image block.

[0087] (2) Texture feature extraction.

[0088] The trained learnable texture feature extractor (LTE) is used to extract texture features from the input image and the reference image, respectively:

[0089] Q = LTE (I HR )

[0090] K = LTE (I Ref )

[0091] Where I HR and I Ref represent the registered input image block and reference image block, respectively, and Q and K are the extracted texture features, which can be represented by a tensor, where c, h, and w represent the channel number, height, and width of the tensor, respectively.

[0092] (3) Constructing hard attention mapping and soft attention mapping.

[0093] ​Unfold the features K and Q from the pixel dimension (h x w) into per-vector unfold(K), unfold(Q) as follows:

[0094]

[0095] where, is a c-dimensional real number field vector space, j and i are the indexes of the vector components, h is the height of the tensor, and w is the width of the tensor

[0096] Each sub-feature q in Q i may be associated with each sub-feature k in K j Calculate the correlation r between them i,j The calculation formula uses the standard inner product:

[0097]

[0098] The i-th component h of the hard attention mapping H i is calculated by the following formula:

[0099]

[0100] The i-th component s of the soft attention mapping S i is calculated by the following formula:

[0101]

[0102] (4) Downsample and adaptively compress the input image.

[0103] Adjust the blur kernel of the Gaussian blur used in the downsampling process. The larger the size or standard deviation of the blur kernel, the more blurred the processed image. The degree of blur will affect the code rate used for compression and the distortion of the decoded image. The method adjusts the size and standard deviation of the blur kernel according to the similarity of the reference image.

[0104] The mean of the soft attention mapping is used as an indicator to measure the similarity between the reference image and the current image. Adjust the blur kernel of the Gaussian blur used in the downsampling process. The larger the size or standard deviation of the blur kernel, the more blurred the processed image. The degree of blur will affect the code rate used for compression and the distortion of the decoded image.

[0105] In the embodiment, the similarity indicator and the linear relationship between the blur kernel parameters are constructed to adaptively adjust the downsampling method. The linear relationship is as follows:

[0106]

[0107] where a represents the size of the blur kernel and σ represents the standard deviation of the blur kernel.

[0108] The compression model is adaptively adjusted based on the similarity to the reference image. First, the soft attention map S is rearranged according to a (128×128) dimension, and then divided into 32×32 macroblocks, each macroblock corresponding to a 64×64 macroblock in the original image. A threshold t is constructed; in this embodiment, if the similarity index of a certain macroblock... If the value is below a threshold t, it is determined that no suitable reference image block has been found for the macroblock. In this case, self-texture learning compression is used, such as... Figure 4 As shown on the right, macroblocks below a threshold t are cropped out and compressed separately at a multiple higher than the input image; this is called coarse compression. The coarse-compressed coded stream is then transmitted along with the input image's coded stream during the compression encoding process. If... If the value is higher than or equal to the threshold t, then the reference image is determined to have suitable transferable texture features, and there is no need to transfer them from the texture. Figure 4 On the left, only the input image is encoded.

[0109] (5) Low-resolution image coding and self-texture learning region image block coding.

[0110] Various compression methods can be used to compress low-resolution images. This embodiment employs two methods: JPEG 2000 and a learning-based Discrete Gaussian Mixture Coding (GMM) model. JPEG 2000 is an image compression standard based on wavelet transform, proposed in 2000 by the Joint Group of Image Experts. GMM is an image compression method based on deep neural networks, whose main framework includes an autoencoder and a Gaussian mixture entropy coding model, making it suitable for end-to-end training.

[0111] The example uses the JPEG 2000 method to compress image patches that employ self-texture learning, with a compression ratio twice that of the low-resolution image. The row and column numbers of the high-resolution image patch in the original image are appended to the compressed code as position information.

[0112] The final compressed code includes compressed code for low-resolution images and compressed code for high-resolution images using self-texture learning.

[0113] like Figure 2 As shown, the specific implementation process for decoding test data is as follows:

[0114] (1) Low-resolution image decoding and self-texture learning region image block decoding.

[0115] Based on the compression method used during encoding, a corresponding decoder is selected to decode the low-resolution image and the image blocks of the self-texture learning region. In this embodiment, JPEG 2000 and a learning-based compression network (GMM) are selected for compression of the low-resolution image, and the JPEG 2000 method is used for the image blocks of the self-texture learning region.

[0116] (2) Texture feature extraction.

[0117] Feature extraction of low-resolution image. The low-resolution image I LR is up-sampled by 2 to get the up-sampled image I LR↑ , which has the same resolution as the original image. The low-resolution image and the up-sampled image are input into the trained learnable texture feature extractor (LTE) to get the texture features F and Q' of the compressed image and the up-sampled image, respectively:

[0118] F = LTE(I LR )

[0119] Q' = LTE(I LR↑ )

[0120] Feature extraction of reference image. Since the reference image is a historical image stored on the satellite, the information of the reference image can still be used during decoding. The reference image is input into the trained learnable texture feature extractor (LTE) to get the texture features K of the reference image:

[0121] K = LTE(I Ref )

[0122] For the case where the reference image is sufficiently similar, such as Figure 4 the left half, only the decoded low-resolution image and the reference image are input into the super-resolution module; for the case where the reference image is not sufficient, such as Figure 4 the right half, the super-resolution module is also input with the self-reference image block that is the image block I HR_patch from the original high-resolution image, and the texture features P = LTE(I HR_patch ) extracted from this part of the image block are recorded.

[0123] (3) Construction of hard attention mapping and soft attention mapping in the decoding process.

[0124] The features K and Q' are unfolded from the pixel dimension (h x w) to individual vectors, and are recorded as unfold(K) and unfold(Q'), respectively, as follows:

[0125]

[0126] wherein,

[0127] Each sub-feature q' i in Q' can calculate the correlation r j between it and each sub-feature k i,j in K, and the calculation formula uses the standard inner product:

[0128]

[0129] the i-th component h of the hard attention map H i is calculated by the following formula:

[0130]

[0131] the i-th component s of the soft attention map S i is calculated by the following formula:

[0132]

[0133] In the region where self-texture learning is adopted, the hard attention map H and the soft attention map S need to be rearranged. Specifically, the hard attention map and the soft attention map are rearranged according to the dimension of (128x128) to the data.

[0134] Suppose the decoded position of the high-resolution image block in the original image is (b1, b2), and the image block size is 64x64. Then, the sub-matrix of the rearranged matrix H and the matrix S is assigned as follows:

[0135] H(i, j) = 128i + j

[0136] S(i, j) = 1

[0137] i = b1, b1 + 1,..., b1 + 31; j = b2, b2 + 1,..., b2 + 31

[0138] where b1 and b2 represent the coordinates of the image block in the original image, i is the row number of the rearranged matrix H and S, j is the column number of the rearranged matrix H and S, H(i, j) is the i-th row and j-th column of the matrix H, and S(i, j) is the i-th row and j-th column of the matrix S.

[0139] (4) Super-resolution with reference to the low-resolution image.

[0140] Constructing transferable texture features. Using the hard attention mechanism to find the most similar components from the reference texture features, these components form transferable texture features, and the formula is as follows:

[0141]

[0142] where T is the transferable texture feature, unfold() is to expand the feature according to the resolution dimension, t i is the i-th component of T, is the feature component at the corresponding position i searched in the components of the feature K according to the hard attention map. For the region where self-texture learning is adopted, the following modification needs to be made:

[0143] T = resize(T, (128, 128))

[0144] T(a1:a1+31, a2:a2+31) = P

[0145] where resize(T, (128, 128)) is to rearrange the feature vector T into a matrix with the specified dimensions (128, 128).

[0146] This formula replaces the components of feature T in the 32x32 rectangular frame from the top-left corner (a1, a2) to the bottom-right corner (a1+31, a2+31) with the features P of the high-resolution image block. The modified feature T is the transferable texture feature that is input into the super-resolution generation network, but before super-resolution, feature fusion is also needed, and the formula is as follows:

[0147]

[0148] where F out is the output fusion feature, Conv and Concat represent convolution operation and tensor splicing operation respectively, is the operation of multiplying the elements in the corresponding positions of the two matrices.

[0149] Then the fusion feature F out is input into the super-resolution generation network, and the specific structure of the network can be referred to in Figure 3 the lower right part. In actual implementation, reference can be made to the literature Zhang Z, Wang Z, Lin Z, et al. Image super-resolution by neural texture transfer [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 7982-7991. In the figure, CONV represents the convolution layer, RB represents the residual module (ResBlock), which is the residual block without batch normalization layer and the ReLU layer after the jump connection, TT represents texture transfer, which is the feature fusion as in the above formula, and Scale 2x represents 2x upsampling of the feature. Finally, the super-resolution network outputs the reconstructed image of the original resolution image.

[0150] where (4) corresponds to the encoding end, and the decoding end will also correspond to the respective decoding methods of the self-texture learning compression and general compression. For the region using self-texture learning compression, the decoding end will simultaneously decode the low-resolution image of the input image and the cropped self-reference image block, and then replace the corresponding region of the reference image with the self-reference image block. The modified reference image is used as the reference of the super-resolution module, as shown in Figure 4Right side; for general compression, only the low-resolution image of the input image is decoded, and the original reference image is used as the reference of the super-resolution module.

[0151] In order to further improve the quality of the compressed image, the application also proposes an improved encoding method based on the above-mentioned encoding and decoding. The increased operation is as follows:

[0152] (1) In the encoding stage, the hard attention map and the soft attention map can be compressed together and transmitted with the image encoding. For the soft attention map S, it can be first rearranged into a matrix S with a dimension of 128x128. The numerical value of the matrix S is converted from a floating-point type to a UINT8, and then compressed using JPEG 2000, and the compression multiple is set to 30-60 times in the embodiment.

[0153] For the hard attention map, prediction encoding can be used for compression. First, it is rearranged into a matrix H with a dimension of 128x128, and then divided into macroblocks with a size of 16x16. Each macroblock records the element h (1,1) , and the other elements are estimated by the following formula

[0154]

[0155] (2) In the decoding stage, when the encoded matrix S and matrix H are transmitted, the matrix S and matrix H are decoded at the same time as the low-resolution image is decoded. The decoding of S uses the JPEG 2000 decoder; for the decoding of H, the first row and first column element h (1,1) of each macroblock recorded in the encoding is used to decode the complete H by the prediction formula.

[0156] In the embodiment, for the image that selects the transmission of the attention map encoding information, it is not necessary to reconstruct the new attention map by calculating the texture feature similarity during decoding. This method is a method for improving the compression image quality and decoding efficiency under the condition of sacrificing the compression multiple. The application can flexibly adjust the compression method to better optimize the rate-distortion.

[0157] In order to understand the technical effects of the application, the comparison between the application and the traditional method JPEG 2000 and the most advanced neural network compression method is as follows:

[0158] Table I Comparison results on SOPT remote sensing image test data

[0159]

[0160] The results of various models on the SPOT remote sensing image test data are recorded in Table I. The coding rate of the model is controlled at about 0.250bpp, and it is converted from the compression ratio by the following formula:

[0161]

[0162] Where n represents the bit number of the image, c represents the channel number of the image, and r and rate respectively represent the compression ratio and the coding rate.

[0163] The results of comparison are measured by the peak signal-to-noise ratio (PSNR) of the compressed image, and the higher the value, the clearer the compressed image. It can be found that the image quality of remote sensing image compression using the present application is higher than that of the conventional method and other most advanced compression methods using neural networks, and the PSNR can be improved by 0.12% at about 0.250bpp.

[0164] In specific implementation, the technical solutions of the present application can be realized by computer software by those skilled in the art. In the embodiments, the following steps are operated:

[0165] The first step is for data preparation, including registration and cropping of the current image and the reference image;

[0166] The second step is for model training, the super-resolution network (including the learnable texture extractor, the construction of attention mapping, the migration and fusion of features, and the super-resolution generation network) is first trained. Then the compression method is selected, i.e. Figure 3 Codec, the traditional compression method JPEG 2000, JPEG or the latest neural network compression method can be selected. If the neural network compression method is selected as the Codec module, the network of the compression part is trained separately. The third step is to put the trained super-resolution network and the compression network together for training, and the compression network is inserted between the down-sampling and the super-resolution of the super-resolution network, for compressing the low-resolution image.

[0167] The third step is for data testing, the adaptive compression selection mechanism and the compression module of attention mapping of Figure 4 are inserted into the trained network, and the test image and the reference image are inputted to realize the encoding and decoding of the image. Figure 3 It is only the network diagram of training, and therefore Figure 3 there is no adaptive compression selection mechanism and compression module of attention mapping in the network).

[0168] The system device for realizing the method, such as the computer readable storage medium storing the corresponding computer program of the technical solutions of the present application and the computer equipment including the running corresponding computer program, should also be within the protection scope of the present application.

[0169] In some possible embodiments, a high-resolution optical remote sensing image compression system referring to historical images is provided, comprising the following modules,

[0170] A first module for data preparation, registering the found reference images with the revisit images to be compressed, and then cropping the images as training data and test data;

[0171] A second module for setting an optical remote sensing image deep learning compression model based on reference texture migration, adopting a learning-based network architecture of down-sampling first and super-resolution later, and comprising the following processes during corresponding training,

[0172] Extraction of image texture features, including extracting texture features from the reference images and the current images respectively by using a learnable texture extractor, constructing a hard attention map H and a soft attention map S by using a standard inner product as similarity, and recording the positions and confidence of the texture related to the current images in the reference images;

[0173] Coding and decoding of down-sampling first and super-resolution later, including filtering out most of the spatial information of the current image by down-sampling, reconstructing a low-resolution image after down-sampling at the decoding time, and then performing corresponding super-resolution on the down-sampling by introducing a super-resolution network of the reference image as auxiliary information, to restore a reconstructed image of the original resolution;

[0174] A third module for training the network by taking the reconstruction loss of the original image, the reconstruction loss of the low-resolution image, and the entropy of the encoding code as a loss function, to obtain a trained optical remote sensing image compression model;

[0175] A fourth module for inputting an optical remote sensing image to be compressed based on the trained optical remote sensing image compression model obtained by the third module, extracting the compressed image, and performing quality evaluation.

[0176] In some possible embodiments, a high-resolution optical remote sensing image compression system referring to historical images is provided, comprising a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a high-resolution optical remote sensing image compression method referring to historical images as described above.

[0177] In some possible embodiments, a high-resolution optical remote sensing image compression system referring to historical images is provided, which is realized according to functions and comprises the following modules:

[0178] A preprocessing module for registering and cropping the input current image and reference image.

[0179] A feature extraction module for extracting texture features of the current image and the reference image.

[0180] The attention mapping module is used to construct a texture mapping relationship based on the similarity of texture features between two images, including hard attention mapping and soft attention mapping. Hard attention mapping records the index number of the most similar feature components, while soft attention mapping records the orthogonality value of the most similar feature components.

[0181] The downsampling and adaptive compression module calculates the mean of the soft attention map. This serves as an indicator to measure the similarity between the reference image and the current image. The blur kernel of the Gaussian blur used during downsampling is adjusted. Simultaneously, the original image is divided into multiple macroblocks of 64×64 pixels, and the similarity index for each macroblock is... If the value is less than the threshold t, then self-texture learning is used, and the macroblocks of the original image are used to replace the reference image as auxiliary information for super-resolution; otherwise, the reference image is still used as auxiliary information for super-resolution.

[0182] The low-resolution image encoding and decoding module compresses low-resolution images and outputs encoded code, and can also decode encoded code to output compressed low-resolution images.

[0183] The self-texturing encoding / decoding module compresses the macroblock image for regions learned using self-texturing and writes the macroblock's location information into the encoding. It can also decode the encoded code to output the macroblock's location information and the compressed image of the macroblock.

[0184] The attention mapping encoding / decoding module, for applications requiring high-resolution compressed imagery, can compress both the hard and soft attention maps of the original and reference images together during the encoding stage. This module can improve the quality of the compressed image and the decoding speed.

[0185] The low-resolution image super-resolution module finds transferable texture features from the reference image, uses attention mapping to fuse the low-resolution image features with the transferable features, and then outputs a super-resolution image of 2 times the low-resolution image through the super-resolution generation network.

[0186] During the encoding phase, the attention mapping matrix can be compressed as needed. If the encoded attention mapping matrix is ​​found during decoding, the decoded attention mapping is used directly without recalculation. If the attention mapping matrix is ​​not compressed, the low-resolution image is first upsampled by a factor of 2 during decoding, and then the attention mapping between the upsampled image and the reference image is calculated.

[0187] In some possible embodiments, a high-resolution optical remote sensing image compression system based on historical reference images is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements a high-resolution optical remote sensing image compression method based on historical reference images as described above.

[0188] The technical solution provided by the present application is a general compression framework that can integrate various single image compression methods to compress remote sensing images with reference. The specific examples described in the present application are merely illustrative of the spirit of the present application, and those skilled in the art can make various modifications or supplements to the described specific examples without deviating from the spirit of the present application or exceeding the scope defined in the appended claims.

Claims

1. A high-resolution optical remote sensing image compression method with reference to historical images, comprising the following steps: Step A, data preparation, the found reference image is registered with the to-be-compressed revisit image, including first using the geographic coordinate information of remote sensing data to perform coarse registration, finding the historical image of the region where the current image is located, and then performing fine registration on the paired images to obtain the paired current image and reference image; then the image is cropped as training data and test data; Step B, setting an optical remote sensing image deep learning compression model based on reference texture migration, adopting a learning-based network architecture of down-sampling first and super-resolution later, which performs the following processing, The extraction of the image texture features comprises extracting texture features from the reference image and the current image respectively by using a learnable texture extractor, and constructing a hard attention map by using a standard inner product as a similarity and the soft attention map , and recording the positions and confidence of the texture in the reference image related to the current image; wherein the mean of the obtained soft attention map will be used as an index for measuring the similarity between the reference image and the current image, guiding the model to perform adaptive compression and adaptive down-sampling; The coding and decoding of down-sampling first and super-resolution later include that, during encoding, according to the linear relationship between the mean value of soft attention mapping and the blur kernel parameter, the current image is adaptively down-sampled to filter out most of the spatial information, each macro block is divided, and the mean value of the soft attention mapping of each macro block is compared with a threshold value, if it is less than the threshold value, self-texture learning is adopted, the original image macro block is separately compressed to a multiple higher than the input image, and the reference image is replaced as auxiliary information for super-resolution, otherwise the reference image is still used as auxiliary information for super-resolution; The hard attention mapping and the soft attention mapping are compressed together and transmitted with the image encoding; during decoding, the hard attention mapping and the soft attention mapping are calculated using the features extracted from the up-sampled image of the low-resolution image and the reference image features, the down-sampled low-resolution image is reconstructed first, then the corresponding super-resolution network is introduced to the reference image as auxiliary information to perform super-resolution on the down-sampling, and the reconstructed image of the original resolution size is restored, for the region compressed by using self-texture learning, the low-resolution image of the input image and the cropped self-reference image block will be decoded at the decoding end at the same time, then the corresponding region of the reference image is replaced with the self-reference image block, and the modified reference image is used as the reference of the super-resolution module; Step C, training the network set in step B with a loss function to obtain a trained optical remote sensing image compression model; Step D, based on the trained optical remote sensing image compression model in step C, inputting the to-be-compressed optical remote sensing image, extracting the compressed image and performing quality evaluation.

2. The method of claim 1, wherein the high-resolution optical remote sensing image compression method using reference historical images is characterized by: The learnable texture extractor is a shallow neural network for extracting texture features from the input image, sequentially including CONV, CONV, POOL, CONV, CONV, POOL, and CONV, wherein CONV is a convolution layer and POOL is a pooling layer.

3. The method of claim 1, wherein the high-resolution optical remote sensing image is compressed with reference to historical images. In step D, the similarity of the reference image is used as the basis for judgment to perform an adaptive compression method based on the reference texture and an adaptive down-sampling method.

4. The method of claim 1 or 2 or 3, wherein the high-resolution optical remote sensing image is compressed with reference to historical images. In the optical remote sensing image deep learning compression model based on reference texture migration, the coding and decoding of down-sampling first and super-resolution later are implemented as follows, In encoding, let the features extracted from the current image and the reference image be and respectively, and , c is the dimension, is the real field, the features are unfolded from the pixel dimension , and are unfolded from the pixel dimension to the vector dimension , as follows: each sub-feature in with each sub-feature in calculate the correlation between them, the formula of which is the standard inner product: From the above correlation, a hard attention map and a soft attention map can be obtained, the hard attention map has a first component , the soft attention map has a first component , ; wherein the mean of the soft attention map For the purpose of guiding the model to perform adaptive compression and adaptive down-sampling as an index of measuring the similarity between the reference image and the current image; when the hard attention map and the soft attention map are compressed together and transmitted with the image encoding, the hard attention map is compressed using predictive encoding; The final compressed code includes the compression code of the low-resolution image and the image block compression code of the high-resolution image using self-texture learning; At the time of decoding, the features extracted using the upsampled image of the low-resolution image and the reference image features The hard attention map H and the soft attention map S are calculated, which are the same as at the encoding end. Then the transferable texture features are constructed, and the hard attention mechanism is used to find the components with the highest similarity from the reference texture features. These components form the transferable texture features, and the formula is as follows, wherein, is the first component, is a feature component searched in the corresponding position according to the hard attention mapping in the feature component. For the region using the self-texture reference model in adaptive compression, the feature needs to be modified, and the implementation is to extract the texture feature block from the high-resolution image block after rough compression, read the position of the region using the self-texture reference from the code stream, and replace the numerical value of the corresponding position feature component in the original feature with the texture feature. The length and width of each replaced region are 1 / 4 of the original feature . Amended features The transferable texture features of the super-resolution generation network are input next, and feature fusion is needed before super-resolution, and the formula is as follows: wherein, is the output fused feature, Conv and Concat represent convolution operation and tensor concatenation operation respectively, is the operation of multiplying the elements in the corresponding positions of the two matrices. Finally, the fused features are input into the super-resolution generation network to output the reconstructed image of the original resolution image.

5. A high resolution optical remote sensing image compression system with reference to historical images, characterized by: A high-resolution optical remote sensing image compression method with reference to historical images is implemented.

6. The high resolution optical remote sensing image compression system referencing historical images of claim 5, wherein: The method comprises the following modules, The first module is used for data preparation, and the found reference image is registered with the revisit image to be compressed, and then the image is cropped as training data and test data. The second module is used for setting an optical remote sensing image deep learning compression model based on reference texture migration, adopting a learning-based network architecture of down-sampling first and super-resolution later, and including the following processing during corresponding training, The extraction of the image texture features comprises extracting texture features from the distribution of the reference image and the current image by using a learnable texture extractor, and constructing a hard attention map by using a standard inner product as a similarity and the soft attention map , and recording the position and confidence of the texture related to the current image in the reference image The coding and decoding of down-sampling first and super-resolution later include filtering out most of the spatial information by down-sampling the current image, reconstructing a low-resolution image after down-sampling during decoding, and then performing corresponding super-resolution on the down-sampling through a super-resolution network using the reference image as auxiliary information, to restore a reconstructed image of the original resolution size. The third module is used for training the network with a loss function to obtain a trained optical remote sensing image compression model. The fourth module is used for inputting the optical remote sensing image to be compressed based on the trained optical remote sensing image compression model obtained by the third module, extracting the compressed image, and performing quality evaluation.

7. The high resolution optical remote sensing image compression system referencing historical images of claim 5, wherein: The device comprises a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the stored instructions in the memory to execute the reference historical image high-resolution optical remote sensing image compression method according to any one of claims 1-4.

8. The high resolution optical remote sensing image compression system referencing historical images of claim 5, wherein: The readable storage medium stores a computer program, and the computer program is executed to implement the reference historical image high-resolution optical remote sensing image compression method according to any one of claims 1-4.

Images