Remote sensing image space-spectrum fusion method and system for spectrum uncovered wave band

Abstract

The invention discloses a remote sensing image space-spectrum fusion method for a spectrum uncovered wave band, and the method comprises the steps: firstly carrying out the preprocessing of a panchromatic image and a multispectral image, and constructing a resolution reduction and full resolution sample library for network training; constructing a fusion network of a wave band self-guiding attention module, a cross-resolution cross-channel interactive learning module, a multi-scale residual block and a reconstruction module by adopting a supervision-non-supervision parallel learning strategy, and establishing a loss function comprising supervision constraint, double-domain spectrum constraint and cumulative frequency constraint; and inputting a low-resolution panchromatic-multispectral sample pair and a high-resolution panchromatic-multispectral sample pair for training until the network converges. And fusing the high-resolution panchromatic image to be processed and the low-resolution multispectral image through the trained network to obtain a multispectral image with high spatial resolution. According to the invention, while the fusion effect of the spectrum coverage wave bands is maintained, the spatial detail information after the fusion of the spectrum uncovered wave bands is effectively enhanced, so that all multispectral wave bands can achieve a balanced fusion effect.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing and computer vision technology, and in particular relates to a remote sensing image space-spectrum fusion method based on deep learning, and specifically relates to a remote sensing image space-spectrum fusion method and system for frequency constraints of uncovered spectral bands. Background Art

[0002] Remote sensing imaging is limited by the physical mechanisms of sensors. Acquiring high-spatial-resolution images requires narrowing the instantaneous field of view. Consequently, the sensor receives less electromagnetic radiation, resulting in lower spectral resolution. Spatial-spectral fusion aims to address the interdependent relationship between spatial and spectral resolution in remote sensing imagery. It integrates the complementary information of high-spatial-resolution panchromatic images and low-spatial-resolution multispectral images to produce high-spatial-spectral-resolution images that meet the needs of higher-precision remote sensing.

[0003] Currently, the main fusion methods can be categorized into four main categories: component replacement, multiresolution analysis, variational optimization, and deep learning. Component replacement methods use a forward spectral transform algorithm to separate the spectral and spatial information of low-resolution multispectral images. The fused image is then replaced with a panchromatic image and then inversely transformed to produce the fused image. However, because the separated spatial components still contain some spectral information, these methods suffer from severe spectral distortion. Multiresolution analysis methods decompose the image into multiple scales using methods such as sampled wavelet transforms or Laplace pyramid decomposition, then inject the spatial structure of the panchromatic image into the multispectral image at different scales. This method often achieves good spectral fidelity but lacks spatial information enhancement. Variational optimization methods treat fusion as an ill-posed inverse problem, constructing a relationship between the ideal image and the observed image using Bayesian theory or sparse representation theory, and then optimizing the model using an optimization algorithm. These methods have a solid mathematical foundation and a rigorous logical system, but their effectiveness relies heavily on prior knowledge and reasonable assumptions. Their high time complexity makes it difficult to rapidly solve fused images and apply them on a large scale. Due to its powerful nonlinear feature extraction and learning capabilities, deep learning methods have been widely used in the field of spatial-spectral fusion and have produced a large number of algorithms, achieving relatively satisfactory results overall.

[0004] However, most existing deep learning methods use supervised learning, learning the mapping relationship between the original image and the fused image from a large number of simulated, reduced-resolution training samples. This approach applies the trained model to real images based on the scale-invariant assumption, but is easily affected by scale differences, resulting in blurry fused images. Existing unsupervised learning methods constrain the fused image solely through observed images, resulting in an illogical expression of image degradation relationships and difficulty in establishing constraints. Furthermore, most current spatial-spectral fusion research focuses on multispectral data in the red, green, blue, and near-infrared bands, whose spectral range is similar to that of panchromatic imagery. Fusion methods generally work well in such scenarios. However, for some sensors, there are multispectral bands that are not covered by the spectral range of panchromatic images. For example, the Landsat-8 OLI sensor's panchromatic imagery with a spatial resolution of 15 meters has a spectral coverage range of 500-680 nm, while the near-infrared band (band 5) in its multispectral imagery with a spatial resolution of 30 meters has a spectral coverage range of 850-880 nm. The spectral coverage ranges of the two shortwave infrared bands (bands 6 and 7) are 1570-1650 nm and 2100-2290 nm, respectively. These bands have significant spectral differences from the panchromatic band, and their spectral coverage is significantly lower than that of the panchromatic imagery. Therefore, these bands face difficulty in injecting spatial information into the spatial-spectral fusion process, exhibiting poor spatial enhancement performance, which seriously affects the practical application of fused images in vegetation information extraction, drought monitoring, and other aspects. Existing methods fail to address this issue and employ an equal fusion strategy for each multispectral band. This results in the difficulty of achieving the same satisfactory fusion effect for uncovered bands as for covered visible light bands. Therefore, it is crucial to develop a spatial-spectral fusion method that can account for the characteristics of uncovered bands and achieve balanced spatial enhancement across all multispectral bands. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems of spatial-spectral fusion, the present invention adopts a supervised-unsupervised parallel learning strategy to provide a frequency-constrained remote sensing image spatial-spectral fusion method for uncovered spectral bands, so as to obtain high-spatial-resolution multispectral images with equal fusion effects in each band.

[0006] According to one aspect of the present invention, a method for spatial-spectral fusion of remote sensing images for spectrally uncovered bands is provided, comprising: Obtain the high-resolution panchromatic image and low-resolution multispectral image to be tested; Input the acquired high-resolution panchromatic image and low-resolution multispectral image to be tested into the trained fusion network to output a high-resolution fused image; wherein the training of the fusion network includes: Collect panchromatic and multispectral images acquired by satellite sensors that include spectral uncovered bands, perform image preprocessing, and construct a reduced-resolution and full-resolution training sample library for network training; Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for uncovered spectral bands is constructed, and a loss function is designed that includes supervision constraints, spectral constraints, and frequency accumulation constraints. Use the constructed training data set to train the constructed fusion network until convergence, and output the trained fusion network.

[0007] As a further technical solution, a reduced-resolution and full-resolution training sample library for network training is constructed, including: Geometric registration of the original observed panchromatic image and the multispectral image is performed to ensure strict alignment of the same-named features; Normalize the pixel values ​​of the aligned panchromatic image and multispectral image; According to the ratio of the spatial resolution of the panchromatic image to the multispectral image, the two images are downsampled using the bicubic interpolation algorithm; The original observation image pairs and the reduced-resolution image pairs are cropped to construct a reduced-resolution and full-resolution training sample library.

[0008] As a further technical solution, a frequency-constrained fusion network for the uncovered spectrum bands is constructed, including: Construct a feature extraction module to extract grouping feature maps and frequency feature maps from the input down-resolution panchromatic-multispectral image pairs and full-resolution panchromatic-multispectral image pairs; Construct a band self-guided attention module to obtain low- and high-resolution band self-guided feature maps based on the extracted group feature map and frequency feature map; A cross-resolution and cross-channel interactive module is constructed to interactively learn the two sets of band self-guided feature maps and the two sets of frequency feature maps through several cross-resolution and cross-channel interactive learning modules to obtain interactive learning results; Constructing a multi-scale residual block to obtain low- and high-resolution multi-scale fusion features based on the band self-guided feature map, wherein the low- and high-resolution multi-scale fusion features are used to correct the interactive learning results of their respective resolutions; Construct a fusion result reconstruction module to reconstruct the fusion results of the corrected low- and high-score feature maps.

[0009] As a further technical solution, low- and high-resolution band self-guided feature maps are obtained based on the extracted group feature map and frequency feature map, which also includes: Perform element-wise multiplication of the frequency feature map and each feature group in the grouped feature map, and perform spatial information enhancement on each feature group; Each feature group is element-wise multiplied with the adjacent feature group to achieve step-by-step injection of the spectrum coverage band features into the spectrum non-coverage features; Convolution and ReLU activation functions are used to restore feature channels and obtain low- and high-resolution band self-guided feature maps.

[0010] As a further technical solution, the fused results of the corrected low- and high-score feature maps are reconstructed, including: The corrected low-score feature map is subjected to a reconstructed convolution sequence to obtain a low-score fused image. After upsampling the low-score fused image, it is spliced ​​with the corrected high-score feature map according to the channel dimension, and then subjected to a reconstructed convolution sequence to obtain a high-score fused image.

[0011] As a further technical solution, extracting the grouping feature map and the frequency feature map includes: The convolution operation is performed band by band on the multispectral image with the r-fold resolution upsampled. Each channel is regarded as a group and each group is convolved separately. After completing the convolution calculation of each group, the output feature maps of all groups are spliced ​​together to form the grouped feature maps of the low- and high-resolution multispectral images. The panchromatic image is high-pass filtered using the Laplace operator to obtain its high-frequency texture image, which is then convolved to form frequency feature maps of low- and high-resolution panchromatic images.

[0012] According to one aspect of the present invention, a remote sensing image space-spectrum fusion system for spectrally uncovered bands is provided, wherein the fusion system is used to implement the remote sensing image space-spectrum fusion method for spectrally uncovered bands.

[0013] As a further technical solution, the fusion system includes: Input module, used to obtain the high-resolution panchromatic image and low-resolution multispectral image to be tested; A fusion module is used to input the acquired high-resolution panchromatic image and low-resolution multispectral image to be tested into a trained fusion network and output a high-resolution fused image; wherein the training of the fusion network includes: Collect panchromatic and multispectral images acquired by satellite sensors that include spectral uncovered bands, perform image preprocessing, and construct a reduced-resolution and full-resolution training sample library for network training; Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for uncovered spectral bands is constructed, and a loss function is designed that includes supervision constraints, spectral constraints, and frequency accumulation constraints. Use the constructed training data set to train the constructed fusion network until convergence, and output the trained fusion network.

[0014] According to one aspect of the present invention, a remote sensing image space-spectrum fusion device for uncovered spectral bands is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the remote sensing image space-spectrum fusion method for uncovered spectral bands.

[0015] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the remote sensing image spatial-spectral fusion method for spectrally uncovered bands.

[0016] The method proposed in this paper can effectively take into account the weak spatial enhancement effect of uncovered spectral bands, fuse high-resolution panchromatic images with low-resolution multispectral images, and obtain a fused image with balanced fusion accuracy in each band. Compared with the existing technology, the beneficial effects of this invention are mainly reflected in: (1) The present invention adopts a supervised-unsupervised parallel learning strategy, effectively utilizing the respective advantages of the two learning methods, and directly fuses the processed images in an end-to-end manner to obtain a multispectral image with high spatial resolution.

[0017] (2) The present invention takes into account the characteristics of the bands in multispectral images that are not covered by the spectral range of panchromatic images, embeds the band self-guided attention mechanism and the cross-resolution and cross-channel interaction mechanism in the network, and designs a cumulative high-frequency loss function, making full use of the frequency characteristics of the image to constrain the spatial texture of each band, and constructs a network model that can effectively perform spatial enhancement on the uncovered spectral bands. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of the method provided by an embodiment of the present invention.

[0020] Figure 2 3 is a schematic diagram comparing the fusion of WorldView-2 images according to an embodiment of the present invention with images fused using an advanced method.

[0021] Figure 3 3 is a schematic diagram comparing Landsat-8 image fusion according to an embodiment of the present invention with image fusion using an advanced method. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0023] To address the problem that remote sensing imaging is limited by the physical mechanism of the sensor, resulting in the inability to achieve both spatial and spectral resolution in a single remote sensing image, the present invention fuses high-spatial-resolution panchromatic images and low-spatial-resolution multispectral images of the same ground scene provided by satellites to obtain high-spatial-resolution multispectral images, which are convenient for subsequent applications. To address the problem that multispectral images acquired by some satellite sensors contain bands that are not covered by the spectral range of the panchromatic image, making it difficult to effectively inject the spatial information of the panchromatic image into these bands during the fusion process, the present invention simultaneously adopts supervised and unsupervised learning strategies to construct a frequency-constrained fusion network for spectrally uncovered bands, thereby obtaining high-spatial-resolution multispectral images with equal fusion effects in each band.

[0024] Please see Figure 1 The present invention provides a remote sensing image space-spectrum fusion method for frequency constraints of uncovered spectral bands, comprising the following steps: Step 1: Collect panchromatic and multispectral images acquired by satellite sensors that include spectral uncovered bands, perform image preprocessing, and construct a reduced-resolution and full-resolution sample library for network training.

[0025] Specifically include: Step 1.1: Geometrically register the original observed panchromatic image with the multispectral image to ensure that the features with the same name are strictly aligned; Step 1.2: Normalize the pixel values ​​of the aligned panchromatic and multispectral images to the range [0, 1]. Step 1.3: Based on the spatial resolution ratio of the panchromatic image to the multispectral image, the two images are downsampled using the bicubic interpolation algorithm. Step 1.4: Crop the original observation image pairs and the reduced-resolution image pairs to build a reduced-resolution and full-resolution training sample library.

[0026] Step 2: Based on the supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for the uncovered spectral bands is constructed, and a loss function including supervisory constraints, dual-domain spectral constraints, and cumulative frequency constraints is designed.

[0027] The fusion network takes down-resolution and full-resolution panchromatic-multispectral image pairs as input, constructing down-resolution fusion branches and full-resolution fusion branches, respectively. Each branch contains a band self-guided attention module and a multi-scale residual block. The output features of the band self-guided attention modules of the two branches are passed through a cross-resolution and cross-channel interaction module. The output of the module is then feature-corrected using the outputs of the two multi-scale residual blocks to obtain deep low-resolution features and deep high-resolution features. The feature reconstruction module then produces low-resolution fused images and high-resolution fused images, respectively. The low-resolution fused image is subjected to supervised loss calculation with the original low-resolution multispectral image; the high-resolution fused image is subjected to dual-domain spectral loss calculation with the upsampling result of the original low-resolution multispectral image, and the cumulative frequency loss is calculated with the high-resolution panchromatic image.

[0028] Specifically include: Step 2.1: Input the down-resolution panchromatic-multispectral image pair ( and ) and full-resolution panchromatic-multispectral image pairs ( and ), to extract grouping features and frequency features, where B is the number of bands of the multispectral image, H and W is the number of rows and columns of the full-color image, h and w is the number of rows and columns of the multispectral image, r is the ratio of the spatial resolutions of the two images, that is, H / h .

[0029] Specifically include: Step 2.1.1: r Upsampled multispectral imagery with times higher resolution ( and ) Perform 3×3 convolution operations band by band, treat each channel as a group, and perform convolution on each group separately, resulting in a total of 32× B The grouping characteristics can be expressed as:

[0030] in To output feature maps in groupsg middle( i , j ) position, Yes Group g Middle m The convolution kernel of the input channels, is the local area of ​​the input multispectral image, K represents the number of pixels in each local receptive field of the convolution kernel, and G is the number of groups in group convolution. After completing the convolution calculation of each group, the output feature maps of all groups are spliced ​​together to form the group feature maps of low and high resolution multispectral images. and .

[0031] Step 2.1.2: Use the Laplacian operator to perform high-pass filtering on the panchromatic image to obtain its high-frequency texture image, and then perform 1×1 convolution on it to form the frequency feature map of the low- and high-resolution panchromatic images. and . It can be expressed as:

[0032] Step 2.2: Construct a band self-guided attention module. In this module, the frequency feature map is first multiplied element-wise with each feature group in the grouped feature map to enhance the spatial information of each feature group. Then, each feature group is multiplied element-wise with the adjacent feature group. The first feature group is multiplied with the second feature group, the second feature group is multiplied with the third feature group, and so on. The last feature group is multiplied with the first feature group to achieve the step-by-step injection of the spectral coverage band features into the spectral non-coverage features. Finally, three layers of 3×3 convolution and ReLU activation function are used to restore the features to 32 channels to obtain low and high resolution band self-guided feature maps. and The process can be expressed as:

[0033] Step 2.3: Construct a multi-scale residual block. In this module, the band self-guided feature map will be convolved with kernel sizes of 1, 3, 5, and 7 respectively. Then the multi-scale feature maps are concatenated and passed through 1×1 convolution, ReLU activation function, three residual blocks, and 1×1 convolution to obtain low and high resolution multi-scale fusion features. and .

[0034] Step 2.4: Construct a cross-resolution and cross-channel interactive learning module. In order to enable the features of the low- and high-resolution branches to interact with each other effectively, and to make full use of the frequency feature maps to enhance the effective spatial information, the two sets of band self-guided feature maps and the two sets of frequency feature maps will pass through several cross-resolution and cross-channel interactive modules. In each module, the low-resolution band self-guided feature map and the low-resolution frequency feature map will pass through the high-frequency attention feature fusion module to obtain the low-resolution intermediate layer fusion feature. The high-resolution band self-guided feature map and the high-resolution frequency feature map will also pass through the high-frequency attention feature fusion module to obtain the high-resolution intermediate layer fusion feature. Then, the channel attention mechanism is applied to the two sets of fusion features to obtain the low- and high-resolution channel weights. Subsequently, the low-resolution channel weight is multiplied by the high-resolution intermediate layer fusion feature, and then added to the low-resolution intermediate layer fusion feature after downsampling; the high-resolution channel weight is multiplied by the low-resolution intermediate layer fusion feature, and then added to the high-resolution intermediate layer fusion feature after upsampling. This process can be expressed as:

[0035] in, HPAFF It represents the feature attention fusion module, which is used to deeply couple the frequency feature map with the spectral features; Channel_Att Represents the channel attention mechanism, which is used to obtain the weight of each channel in the feature map; and They are r Downsampling and upsampling process of times spatial resolution; and They are the low- and high-score feature maps output by the first cross-resolution and cross-channel interaction module, and are used as input data together with and Send it to the next cross-resolution and cross-channel interaction module. M This module gets the interactive learning results and .

[0036] Step 2.5: Use low- and high-resolution multi-scale fusion features to further correct the interactive learning results of each resolution, which can be expressed as:

[0037] and That is, the low- and high-score final feature maps before the fusion result is reconstructed.

[0038] Step 2.6: Construct the fusion result reconstruction module. After reconstructing the convolution sequence, a low-resolution fusion image is obtained ,Will After upsampling After splicing according to the channel dimension, the high-resolution fused image is obtained by reconstructing the convolution sequence. The reconstructed convolution sequences for both low and high scores are 3×3 convolution, LeakyReLU activation, 3×3 convolution, LeakyReLU activation, and 1×1 convolution.

[0039] Step 2.7: Construct the loss function.

[0040] Step 2.7.1: Supervision loss. This is the loss between the original low-resolution multispectral image and the low-resolution fused image. It is calculated using the 2-norm and can be expressed as:

[0041] in N Indicates the batch size, 、M (n) They represent the nth low-resolution fused image and low-resolution multispectral image in each batch size respectively.

[0042] Step 2.7.2: Dual-domain spectral loss. This is the loss between the original observed low-resolution multispectral upsampled image and the high-resolution fused image, including constraints in both the image domain and the gradient domain, and can be expressed as:

[0043] in Indicates the gradient operation of the image along the channel dimension, F (n) 、 They represent the nth high-resolution fused image and low-resolution multispectral upsampled image in each batch size respectively.

[0044] Step 2.7.3: Cumulative frequency loss. This is the cumulative loss per band between the Laplacian high-frequency image of the panchromatic image after matching the low-resolution multispectral band-by-band histogram and the Laplacian high-frequency image of the high-resolution fused image. It can be expressed as:

[0045] in, b Indicates the b bands, Indicates that histogram matching is performed on each band of the low-resolution multispectral image. B Band panchromatic image, F represents high-resolution fusion image. The process can be expressed as:

[0046] in, and Represent the mean and standard deviation of the image, M b Represents the bth band of the low-resolution multispectral image.

[0047] Step 2.7.4: Weight the three loss functions to construct the overall loss function, which can be expressed as:

[0048] Represent the weights of the three loss functions respectively.

[0049] Step 3: Use the training data set constructed in step 1 to train the fusion network constructed in step 2 until convergence; In step 4, the fusion network trained and converged in step 3 is used to fuse the high-resolution panchromatic image to be tested with the low-resolution multispectral image. The final high-resolution fused image output is the required high spatial resolution multispectral image.

[0050] Figure 2 A schematic diagram comparing the WorldView-2 image fusion method of an embodiment of the present invention with that of an advanced image fusion method is provided. In the figure, the first row shows the 5-3-2 true color composite visualization effect, and the second row shows the 8-7-1 false color composite visualization effect.

[0051] Figure 3 A schematic diagram comparing Landsat-8 image fusion using an embodiment of the present invention and advanced image fusion methods is provided. In the figure, the first row shows the 4-3-2 true color composite visualization effect, and the second row shows the 7-6-1 false color composite visualization effect.

[0052] The implementation of each embodiment of the present invention is based on programmed processing performed by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, and in addition to the aforementioned embodiments, an embodiment of the present invention provides a remote sensing image spatial-spectral fusion system for uncovered spectral bands. This system is used to implement the remote sensing image spatial-spectral fusion method for uncovered spectral bands described in the aforementioned method embodiments.

[0053] The system includes: an input module for acquiring high-resolution panchromatic images and low-resolution multispectral images to be tested; a fusion module for inputting the acquired high-resolution panchromatic images and low-resolution multispectral images to be tested into a trained fusion network to output a high-resolution fused image; wherein, the training of the fusion network includes: collecting panchromatic images and multispectral images acquired by satellite sensors containing spectrally uncovered bands, performing image preprocessing, and constructing a reduced-resolution and full-resolution training sample library for network training; based on a supervised-unsupervised parallel learning strategy, constructing a frequency-constrained fusion network for spectrally uncovered bands, and designing a loss function including supervisory constraints, spectral constraints, and frequency accumulation constraints; using the constructed training data set, training the constructed fusion network until convergence, and outputting the trained fusion network.

[0054] The embodiment of the present invention provides a remote sensing image space-spectrum fusion system for uncovered spectral bands. It addresses the problem of ignoring the characteristics of uncovered spectral bands in space-spectrum fusion. It adopts the aforementioned modules to effectively enhance the spatial detail information after the fusion of uncovered spectral bands while maintaining the fusion effect of covered spectral bands, so that all multispectral bands can achieve a balanced fusion effect.

[0055] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and its principles are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they will improve the equipment in the above-mentioned system embodiments to obtain corresponding system embodiments for implementing the methods in other method embodiments.

[0056] Based on the same inventive concept as the aforementioned embodiment, an embodiment of the present invention provides a remote sensing image spatial-spectral fusion device for spectrally uncovered bands, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the remote sensing image spatial-spectral fusion method for spectrally uncovered bands.

[0057] In an embodiment of the present invention, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory (volatile memory), such as a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in an embodiment of the present invention may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0058] In the embodiments of the present invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention may be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules within the processor.

[0059] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method for spatial-spectral fusion of remote sensing images for spectrally uncovered bands, comprising: Obtain the high-resolution panchromatic image and low-resolution multispectral image to be tested; Input the acquired high-resolution panchromatic image and low-resolution multispectral image to be tested into the trained fusion network to output a high-resolution fused image; wherein the training of the fusion network includes: Collect panchromatic and multispectral images acquired by satellite sensors that include spectral uncovered bands, perform image preprocessing, and construct a reduced-resolution and full-resolution training sample library for network training; Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for uncovered spectral bands is constructed, and a loss function is designed that includes supervision constraints, spectral constraints, and frequency accumulation constraints. Use the constructed training data set to train the constructed fusion network until convergence, and output the trained fusion network.

[0060] In summary, the present invention discloses a remote sensing image space-spectrum fusion method for frequency constraints of spectrally uncovered bands. First, the panchromatic image and the multispectral image are preprocessed to construct a reduced-resolution and full-resolution sample library for network training. Then, a supervised-unsupervised parallel learning strategy is adopted to construct a fusion network of an embedded band self-guided attention module, a cross-resolution and cross-channel interaction module, a multi-scale residual block, and a reconstruction module. A loss function including supervisory constraints, dual-domain spectral constraints, and cumulative frequency constraints is established. Low-score and high-score panchromatic-multispectral sample pairs are input for training until the network converges. Through the trained network, the high-resolution panchromatic image to be processed is fused with the low-resolution multispectral image to obtain a multispectral image with high spatial resolution. The present invention can overcome the problem of previous methods ignoring the characteristics of spectrally uncovered bands. While maintaining the fusion effect of spectrally covered bands, it effectively enhances the spatial detail information after the fusion of spectrally uncovered bands, so that all multispectral bands can achieve a balanced fusion effect.

[0061] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image spatial-spectral fusion method for spectrally uncovered bands, characterized by: include: Obtain the high-resolution panchromatic image and low-resolution multispectral image to be tested; Input the acquired high-resolution panchromatic image and low-resolution multispectral image to be tested into the trained fusion network to output a high-resolution fused image; wherein the training of the fusion network includes: Collect panchromatic and multispectral images acquired by satellite sensors that include spectral uncovered bands, perform image preprocessing, and construct a reduced-resolution and full-resolution training sample library for network training; Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for uncovered spectral bands is constructed, and a loss function is designed that includes supervision constraints, spectral constraints, and frequency accumulation constraints. Use the constructed training data set to train the constructed fusion network until convergence, and output the trained fusion network.

2. The method for spatial-spectral fusion of remote sensing images for uncovered spectral bands according to claim 1, characterized in that: Build a library of reduced-resolution and full-resolution training samples for network training, including: Geometric registration of the original observed panchromatic image and the multispectral image is performed to ensure strict alignment of the same-named features; Normalize the pixel values ​​of the aligned panchromatic image and multispectral image; According to the ratio of the spatial resolution of the panchromatic image to the multispectral image, the two images are downsampled using the bicubic interpolation algorithm; The original observation image pairs and the reduced-resolution image pairs are cropped to construct a reduced-resolution and full-resolution training sample library.

3. The remote sensing image space-spectrum fusion method for uncovered spectral bands according to claim 1, characterized in that: Construct a frequency-constrained fusion network for uncovered spectral bands, including: Construct a feature extraction module to extract grouping feature maps and frequency feature maps from the input down-resolution panchromatic-multispectral image pairs and full-resolution panchromatic-multispectral image pairs; Construct a band self-guided attention module to obtain low- and high-resolution band self-guided feature maps based on the extracted group feature map and frequency feature map; A cross-resolution and cross-channel interactive module is constructed to interactively learn the two sets of band self-guided feature maps and the two sets of frequency feature maps through several cross-resolution and cross-channel interactive learning modules to obtain interactive learning results; Constructing a multi-scale residual block to obtain low- and high-resolution multi-scale fusion features based on the band self-guided feature map, wherein the low- and high-resolution multi-scale fusion features are used to correct the interactive learning results of their respective resolutions; Construct a fusion result reconstruction module to reconstruct the fusion results of the corrected low- and high-score feature maps.

4. The method for spatial-spectral fusion of remote sensing images for uncovered spectral bands according to claim 3, characterized in that: Based on the extracted group feature map and frequency feature map, low- and high-resolution band self-guided feature maps are obtained, which also includes: Perform element-wise multiplication of the frequency feature map and each feature group in the grouped feature map, and perform spatial information enhancement on each feature group; Each feature group is element-wise multiplied with the adjacent feature group to achieve step-by-step injection of the spectrum coverage band features into the spectrum non-coverage features; Convolution and ReLU activation functions are used to restore feature channels and obtain low- and high-resolution band self-guided feature maps.

5. The method for spatial-spectral fusion of remote sensing images for uncovered spectral bands according to claim 3, characterized in that: The fused results of the corrected low- and high-score feature maps are reconstructed, including: The corrected low-score feature map is subjected to a reconstructed convolution sequence to obtain a low-score fused image. After upsampling the low-score fused image, it is spliced ​​with the corrected high-score feature map according to the channel dimension, and then subjected to a reconstructed convolution sequence to obtain a high-score fused image.

6. The remote sensing image space-spectrum fusion method for uncovered spectral bands according to claim 3, characterized in that: Extract group feature maps and frequency feature maps, including: The convolution operation is performed band by band on the multispectral image with the r-fold resolution upsampled. Each channel is regarded as a group and each group is convolved separately. After completing the convolution calculation of each group, the output feature maps of all groups are spliced ​​together to form the grouped feature maps of the low- and high-resolution multispectral images. The panchromatic image is high-pass filtered using the Laplace operator to obtain its high-frequency texture image, which is then convolved to form frequency feature maps of low- and high-resolution panchromatic images.

7. A remote sensing image space-spectrum fusion system for uncovered spectral bands, characterized by: The fusion system is used to implement the remote sensing image space-spectrum fusion method for uncovered spectral bands as described in any one of claims 1-6.

8. The remote sensing image space-spectrum fusion system for uncovered spectral bands according to claim 7, characterized in that: The fusion system includes: Input module, used to obtain the high-resolution panchromatic image and low-resolution multispectral image to be tested; A fusion module is used to input the acquired high-resolution panchromatic image and low-resolution multispectral image to be tested into a trained fusion network and output a high-resolution fused image; wherein the training of the fusion network includes: Collect panchromatic and multispectral images acquired by satellite sensors that include spectral uncovered bands, perform image preprocessing, and construct a reduced-resolution and full-resolution training sample library for network training; Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for uncovered spectral bands is constructed, and a loss function is designed that includes supervision constraints, spectral constraints, and frequency accumulation constraints. Use the constructed training data set to train the constructed fusion network until convergence, and output the trained fusion network.

9. A remote sensing image space-spectrum fusion device for spectrally uncovered bands, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the remote sensing image space-spectrum fusion method for spectrally uncovered bands as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the remote sensing image spatial-spectral fusion method for spectrally uncovered bands as described in any one of claims 1 to 6.

Images