A multispectral remote sensing image enhancement method based on frequency-space dual-domain learning

Through the frequency domain-space dual-domain learning method, Fourier transform and feature fusion technology are used to solve the problem of spectral and spatial resolution trade-offs in multispectral images, and efficient spatial resolution of low-spatial resolution images is achieved. The rate enhancement improves the precision of the detailed and background information modeling of multispectral images.

CN118587097BActive Publication Date: 2024-11-08INSPUR OPTOELECTRONICS SATELLITE TECHNOLOGY (SHANDONG) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411073631.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-08
Estimated Expiration
2044-08-07

AI Technical Summary

Technical Problem

The existing multispectral image super-resolution technology is difficult to effectively combine spectrum and spatial resolution, making it difficult to obtain multispectral images with high spatial resolution. The deep learning method mainly focuses on spatial domain feature mining, ignoring the utilization of frequency domain features.

Method used

Using the method based on frequency domain-space dual domain learning, the image is decomposed into phase and amplitude components through Fourier transform, feature extraction and processing are performed separately, and frequency-space feature fusion is performed, and equal differential feature mining and two-dimensional The inverse Fourier transform reconstructs features to achieve spatial resolution enhancement of the image.

Benefits of technology

The effective spatial resolution enhancement of low-spatial resolution multispectral images is achieved, which makes up for the shortcomings of existing methods in the multispectral image enhancement task, improves the modeling accuracy of high-frequency structure and low-frequency background information of the image, and significantly improves the Enhancement effect of multispectral images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118587097B_ABST
    Figure CN118587097B_ABST
Patent Text Reader

Abstract

The present invention relates to a multispectral remote sensing image enhancement method based on frequency-domain-space dual-domain learning, and belongs to the technical field of remote sensing image processing. A full-color image and an upsampled low spatial resolution multispectral image are transformed into corresponding phase components and amplitude components by Fourier transform, and are respectively sent to a frequency domain feature extraction module in two branches for phase component learning and amplitude component learning, and then a two-dimensional inverse Fourier transform is used to reconstruct the frequency domain learning features; the full-color image and the upsampled low spatial resolution multispectral image channels are spliced ​​and input into a feature extraction module for feature extraction to obtain spatial domain learning features; the frequency domain learning features and the spatial domain learning features are feature fused, the loss value is calculated for supervised training, and the model is tested with a test set to obtain a final model. The present invention makes up for the shortcomings of existing deep learning-based methods in multispectral remote sensing image enhancement tasks to a certain extent, and achieves superior multispectral image enhancement effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for enhancing the spatial resolution of a multispectral remote sensing image, and in particular to a multispectral remote sensing image enhancement method based on frequency domain-space dual-domain learning, which adopts equal difference feature mining for phase and amplitude representation in frequency domain feature learning and belongs to the technical field of remote sensing image processing. Background Art

[0002] With the development of remote sensing technology, multispectral images can record richer spectral domain information compared to natural RGB images, so they are widely used in many fields, such as environmental monitoring, precision agriculture and urban development. However, due to various limitations in theory and practical applications, there is a trade-off between the spectral and spatial resolution of directly captured multispectral images, that is, it is difficult to obtain multispectral images with high spatial resolution. Multispectral image super-resolution is an important but challenging remote sensing image processing task that aims to generate high spatial resolution multispectral images from coupled low spatial resolution multispectral images and panchromatic images.

[0003] Existing multispectral image super-resolution can be roughly divided into two categories, including model-based methods and deep learning-based methods. Typical model-based methods include component replacement, multi-resolution analysis, variational optimization, etc. These methods rely more on subjective prior assumptions in the super-resolution process, and the model performance and generalization ability in actual scenarios are not strong. Deep learning-based methods are able to directly learn relatively strong prior knowledge and achieve relatively competitive performance thanks to the excellent feature representation under the cascading paradigm.

[0004] The multispectral image enhancement task is dedicated to enhancing the spatial details of low spatial resolution multispectral images, including the high-frequency structural information and low-frequency background information of the image. Existing deep learning-based methods often focus on mining the spatial domain features of images through deep learning modules. However, considering that images are a typical signal representation, using the Fourier frequency domain perspective as a supplement to enhance the spatial information of low spatial resolution multispectral images is a reasonable approach that has been ignored by existing work. Figure 1 A set of panchromatic images P and four-fold upsampled low spatial resolution multispectral images L are shown. 1 The sample pairs composed of the true value-high spatial resolution multispectral image H and the corresponding phase component and amplitude component. In terms of spatial content and details, the full-color image P and the true value-high spatial resolution multispectral image H maintain a high consistency. After the channel Fourier transform, it is not difficult to find that the phase component P of the full-color image P P With the amplitude component P A The standard is similar to that of the standard, while the low spatial resolution multispectral image L is four times upsampled. 1 The phase component L1P With the amplitude component L 1A The phase component of the true value-high spatial resolution multispectral image H is H P , the amplitude component is H A Therefore, whether analyzing from the spatial domain or the frequency domain, it is reasonable to use the high spatial resolution panchromatic image as a guide to enhance the spatial details of the low spatial resolution multispectral image.

[0005] On the other hand, from the perspective of Fourier frequency domain, considering that the phase component corresponds to the high-frequency structural information of the image, while the amplitude component corresponds to the low-frequency background information, the two have different representational meanings. It is appropriate and important to process the phase and amplitude components respectively. In addition, according to the spectral convolution theorem in Fourier theory, the point product in the Fourier frequency domain is equivalent to the convolution in the spatial domain. This good property provides important theoretical support for efficient Fourier frequency domain feature extraction. Summary of the invention

[0006] The purpose of the present invention is to enhance the spatial resolution of a multispectral image with low spatial resolution and finally obtain an ideal multispectral image with high spatial resolution, and to provide a multispectral remote sensing image enhancement method based on frequency domain-spatial dual-domain learning to achieve superior multispectral image enhancement effect.

[0007] The technical solution adopted by the present invention is:

[0008] A multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning comprises the following steps:

[0009] S1. Load the original multispectral remote sensing images and panchromatic images, preprocess the images and divide the data sets;

[0010] S2. The full-color image and the upsampled low spatial resolution multispectral image are transformed into corresponding phase components and amplitude components by Fourier transform. The phase components corresponding to the full-color image and the upsampled low spatial resolution multispectral image are spliced ​​through channels and then sent to the frequency domain feature extraction module for phase component learning. The corresponding amplitude components are spliced ​​through channels and then sent to the same frequency domain feature extraction module for amplitude component learning. The feature maps after the phase and amplitude branches are processed by two-dimensional inverse Fourier transform and reconstructed back to the spatial domain to obtain the features after frequency domain learning.

[0011] S3. perform channel splicing on the full-color image and the upsampled low spatial resolution multispectral image and input them into the feature extraction module for feature extraction to obtain the features after spatial domain learning;

[0012] S4. The features learned in the frequency domain and the features learned in the spatial domain are spliced ​​through channels, and the spliced ​​feature maps are sent to the frequency-space feature fusion module for feature fusion to obtain the spectrum-space feature map;

[0013] S5. Enhance the spatial resolution by fusing the fused feature map with the residual of the original upsampled low spatial resolution multispectral image. Perform supervised training on the output feature map and the true high spatial resolution multispectral image using the mean absolute value loss. Test the model with the test set to obtain the final model. Input the low spatial resolution multispectral remote sensing image to be processed into the obtained model to finally obtain a multispectral image with enhanced spatial resolution.

[0014] In the above method, each data set in step S1 is split into non-overlapping subsets for model training and testing, wherein both the training set and the test set contain low spatial resolution multispectral images, panchromatic images, and true values ​​of high spatial resolution multispectral images.

[0015] The frequency domain feature extraction module described in step S2 includes three consecutive frequency domain feature extraction units. Each frequency domain feature extraction unit uses 1×1 depth convolution to extract features, then performs channel automatic matching replication under the broadcast mechanism, and performs element-by-element addition and fusion of the feature map extracted by convolution and the corresponding phase or amplitude component after Fourier transformation of the original full-color image.

[0016] The feature extraction module in step S3 includes two consecutive feature extraction units consisting of 3×3 depthwise convolution and ReLU activation function.

[0017] The frequency-space feature fusion module described in step S4 is composed of two consecutive groups of feature fusion units, each of which is composed of a 3×3 convolution and a ReLU activation function.

[0018] The beneficial effects of the present invention are:

[0019] The existing image processing methods involving Fourier transform in different task backgrounds do not use the phase and amplitude components of the guide map as guidance in a targeted and detailed manner, and do not learn the phase and amplitude components. The present invention is based on the observation of a group of multispectral image enhancement samples in the spatial domain and the Fourier frequency domain, and adopts the technology of collaborative feature modeling and extraction in the Fourier frequency domain and the spatial domain. In the Fourier frequency domain feature extraction branch, the phase and amplitude representations are further mined for equal difference features, and the effective modeling of image structure information and image background information is completed respectively. Direct and effective feature extraction is performed in the spatial domain. The frequency-space domain feature fusion step completes the fusion and complementarity of the extracted features of the Fourier frequency domain and the spatial domain. Among them, the key phase and amplitude representation parallel feature modeling and frequency-space feature parallel extraction of this method greatly refine the inspiration based on the observation of the multispectral image enhancement task, and the synergistic effect between the overall frequency-space domain feature learning and the frequency-space domain feature fusion guarantees the performance of this method in the multispectral enhancement task. The present invention makes up for the shortcomings of existing deep learning-based methods in multispectral remote sensing image enhancement tasks to a certain extent, and achieves superior multispectral image enhancement effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Comparison of spatial and frequency domain components for different image samples;

[0021] Figure 2 is a flow chart of the method of the present invention;

[0022] Figure 3 This is a network model framework diagram of the method of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0024] Embodiment A multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning comprises the following steps (see Figure 2 )as follows:

[0025] S1. Load the original multispectral remote sensing images and panchromatic images, preprocess the images and divide the data sets:

[0026] The intensity upper limit of each pixel intensity value of all samples in the loaded original image is reduced to accelerate the convergence speed of the proposed image enhancement model during the training process.

[0027] Considering the unavailability of the true value of the image enhancement task, this method uses bicubic interpolation to complete the image downsampling operation for the data captured by each satellite sensor according to the Wald protocol recognized and popular in the field. Each dataset is then further split into non-overlapping subsets for model training and testing, where the training set and test set contain approximately 1000 and 140 sets of low spatial resolution multispectral images, panchromatic images, and high spatial resolution multispectral images. The true value triplets, their image resolutions are 32×32, 128×128, and 128×128, respectively.

[0028] S2. The full-color image and the upsampled low spatial resolution multispectral image are transformed into corresponding phase components and amplitude components by Fourier transform. The phase components corresponding to the full-color image and the upsampled low spatial resolution multispectral image are spliced ​​through channels and sent to the frequency domain feature extraction module for phase component learning. The corresponding amplitude components are spliced ​​through channels and sent to the same frequency domain feature extraction module for amplitude component learning. The feature maps after the phase and amplitude branches are processed by two-dimensional inverse Fourier transform and reconstructed back to the spatial domain to obtain the features after frequency domain learning:

[0029] like Figure 3 As shown in the figure, because the low spatial resolution multispectral image of the input model has a spatial resolution four times different from that of the panchromatic image in the designed remote sensing satellite scene. First, the low spatial resolution multispectral image is upsampled from 32×32 to 128×128 using bicubic interpolation. Then, the frequency domain and spatial domain feature extraction of the overall dual-branch structure is performed.

[0030] For frequency domain feature extraction, firstly, a channel-by-channel two-dimensional Fourier transform is used to convert the full-color image and the upsampled low spatial resolution multispectral image from the spatial domain to the complex components of the Fourier frequency domain. Then, the corresponding real and imaginary parts are calculated based on the complex components, and the corresponding phase and amplitude components are further calculated. Since the Fourier transform is performed channel by channel, the change from the spatial domain to the Fourier frequency domain does not change the channel dimension. Therefore, the number of component channels corresponding to the full-color image is consistent with the number of channels of the full-color image, both of which are 1, and the number of component channels corresponding to the multispectral image is consistent with the number of channels of the multispectral image, both of which are C. For phase and amplitude components with different representational meanings, a more detailed branch processing is used here, and the guidance of the corresponding phase and amplitude components of the original full-color image is realized. As Figure 3 As shown in the left part, the phase components and amplitude components corresponding to the full-color image and the multispectral image are respectively channel-joined and sent to the frequency domain feature extraction module.

[0031] The frequency domain feature extraction module includes three consecutive frequency domain feature extraction units. As shown in the second frequency domain feature extraction unit shown in detail, it uses 1×1 deep convolution to extract features from the input feature map, and then uses the broadcast mechanism (the phase and amplitude component learning units both use the channel concatenation of the corresponding components of the full-color image and the multispectral image to obtain a feature dimension of C+1. Here, when the original full-color image phase component and the feature map are added element by element, the channel dimensions that do not match are 1 and C+1 respectively. In the Python programming language used, its underlying broadcast mechanism can complete the automatic replication and expansion of the full-color image phase component dimension to C+1) to perform element-by-element addition to fuse the feature map extracted by convolution with the corresponding phase or amplitude component of the original full-color image. The channel dimensions of the first two groups of frequency domain feature extraction units used are kept as C+1, and the output channel dimension of the third group of frequency domain feature extraction units is set to C to ensure the convenience of subsequent frequency-space feature fusion processing. The feature maps after processing the two branches of phase and amplitude components are finally reconstructed back to the spatial domain using the corresponding two-dimensional inverse Fourier transform.

[0032] S3. After channel splicing of the full-color image and the upsampled low spatial resolution multispectral image, input them into the feature extraction module for feature extraction to obtain the features after spatial domain learning:

[0033] The original full-color image and the upsampled low spatial resolution multispectral image are selected for channel splicing to obtain a C+1 dimensional image mapping. Then, two consecutive feature extraction units consisting of 3×3 deep convolution and ReLU activation function are used for feature extraction. The output dimension of the second feature extraction unit is C. Such dual-branch feature extraction in frequency domain and spatial domain realizes the mutual complementation of dual-domain feature extraction, and more effectively completes the modeling and extraction of high-frequency structural information and low-frequency background information of multispectral images.

[0034] S4. The features learned in the frequency domain and the features learned in the spatial domain are spliced ​​through channels, and the spliced ​​feature maps are sent to the frequency-space feature fusion module for feature fusion to obtain the spectrum-space feature map:

[0035] For the features processed by the frequency domain and space domain dual branches, they are first spliced ​​through channels, and the feature maps with a spliced ​​channel dimension of 2C are sent to the frequency-space feature fusion module for feature fusion. The frequency-space feature fusion module is composed of two consecutive groups of feature fusion units, each of which is composed of a 3×3 convolution and a ReLU activation function. The output channel dimension of the latter feature fusion unit is C. Finally, the fused feature map is residually fused with the original upsampled low-resolution multispectral image containing basic image information.

[0036] S5. Enhance the spatial resolution by fusing the fused feature map with the residual of the original upsampled low spatial resolution multispectral image. Use the mean absolute value loss to supervise the output feature map and the true high spatial resolution multispectral image. Test the model with the test set to obtain the final model. Input the low spatial resolution multispectral remote sensing image to be processed into the obtained model to finally obtain the multispectral image with enhanced spatial resolution:

[0037] The output feature map and the true high spatial resolution multispectral image are supervised and trained using the mean absolute value loss (i.e., the conventional |A- B| form). Regarding the training settings, the training is about 130 rounds for the 8-band dataset (C=8) and about 1000 rounds for the 4-band dataset (C=4). The Adam optimizer is used as a whole, with two parameters of 0.9 and 0.999. In addition, the initial learning rate is 0.0015, and the drop coefficient of 0.85 is multiplied every 100 rounds of training.

[0038] During model training, the loss value corresponding to the training set is calculated after each round of iteration, and the model after the set iteration is used as the final model. During the model testing phase, the split test set is used to test the model in different multispectral scenarios. For different low spatial resolution multispectral images, spatial resolution enhancement can be adaptively performed to obtain high spatial resolution multispectral images.

[0039] The above description is only a specific implementation of the present application, and the protection scope of the present invention is not limited thereto.

Claims

1. A multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning, characterized in that: The steps include: S1. Load the original multispectral remote sensing images and panchromatic images, preprocess the images and divide the data sets; S2. The full-color image and the upsampled low spatial resolution multispectral image are transformed into corresponding phase components and amplitude components by Fourier transform. The phase components corresponding to the full-color image and the upsampled low spatial resolution multispectral image are spliced ​​through channels and then sent to the frequency domain feature extraction module for phase component learning. The corresponding amplitude components are spliced ​​through channels and then sent to the same frequency domain feature extraction module for amplitude component learning. The feature maps after the phase and amplitude branches are processed by two-dimensional inverse Fourier transform and reconstructed back to the spatial domain to obtain the features after frequency domain learning. S3. perform channel splicing on the full-color image and the upsampled low spatial resolution multispectral image and input them into the feature extraction module for feature extraction to obtain the features after spatial domain learning; S4. The features learned in the frequency domain and the features learned in the spatial domain are spliced ​​through channels, and the spliced ​​feature maps are sent to the frequency-space feature fusion module for feature fusion to obtain the spectrum-space feature map; S5. The spatial resolution is enhanced by fusing the fused feature map with the residual of the original upsampled low spatial resolution multispectral image. The output feature map and the true high spatial resolution multispectral image are supervised trained using the mean absolute value loss. The model is tested with the test set to obtain the final model. The low spatial resolution multispectral remote sensing image to be processed is input into the obtained model to finally obtain a multispectral image with enhanced spatial resolution.

2. The multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning according to claim 1 is characterized in that: In step S1, each data set is split into non-overlapping subsets for model training and testing, where both the training set and the test set contain low spatial resolution multispectral images, panchromatic images, and high spatial resolution multispectral image ground truths.

3. The multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning according to claim 1 is characterized in that: The frequency domain feature extraction module described in step S2 includes three consecutive frequency domain feature extraction units, each of which uses 1×1 deep convolution to extract features, and then uses a broadcast mechanism to perform channel automatic matching replication, and performs element-by-element addition fusion of the feature map extracted by convolution and the corresponding phase or amplitude component after Fourier transformation of the original full-color image.

4. The multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning according to claim 1 is characterized in that: The feature extraction module in step S3 includes two consecutive feature extraction units consisting of 3×3 depthwise convolution and ReLU activation function.

5. The multispectral remote sensing image enhancement method based on frequency domain-spatial dual domain learning according to claim 1 is characterized in that: The frequency-space feature fusion module described in step S4 is composed of two consecutive groups of feature fusion units, each of which is composed of a 3×3 convolution and a ReLU activation function.

Citation Information

Cited By

  • Remote sensing image panchromatic sharpening method based on double-domain collaborative learning

    CN122391020A