Spectral un-covered band-oriented remote sensing image space-spectrum fusion method and system

Abstract

The application discloses a kind of remote sensing image space-spectrum fusion methods for spectrum non-covered waveband, first, full color image and multispectral image are preprocessed, construct the sample library for network training of reduced resolution and full resolution;Then using supervised-unsupervised parallel learning strategy, construct waveband self-guided attention module, cross-resolution cross-channel interactive learning module, multiscale residual block and the fusion network of reconstruction module, establish the loss function containing supervision constraint, dual-domain spectral constraint and cumulative frequency constraint, input low and high full color-multispectral sample pair for training until network convergence. Through the network trained, the high spatial resolution multispectral image is obtained by fusing the high-resolution full color image and low-resolution multispectral image to be processed. The application effectively enhances the spatial detail information of the spectrum non-covered waveband after fusion while maintaining the fusion effect of the spectrum covered waveband, so that all multispectral bands can achieve balanced fusion effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing and computer vision, and particularly relates to a remote sensing image space-spectrum fusion method based on deep learning, and in particular to a frequency-constrained remote sensing image space-spectrum fusion method and system for a spectrum-uncovered band. BACKGROUND

[0002] Remote sensing imaging is limited by the physical mechanism of the sensor. To obtain high spatial resolution images, the instantaneous field of view needs to be reduced, and accordingly, the electromagnetic radiation energy received by the sensor will be reduced, resulting in a decrease in spectral resolution. Space-spectrum fusion aims to solve the problem of mutual restriction between spatial resolution and spectral resolution in remote sensing images, and can integrate the complementary information of high spatial resolution panchromatic images and low spatial resolution multispectral images to obtain high space-spectrum resolution images that can meet higher precision remote sensing requirements.

[0003] At present, the main fusion methods can be divided into four categories: component replacement, multi-resolution analysis, variational optimization and deep learning. The component replacement method uses a forward spectral transformation algorithm to separate the spectral information and spatial information of the low-resolution multispectral image, and the fused image can be obtained by replacing the spatial component of the panchromatic image with the inverse transformation. However, this method has serious spectral distortion because the separated spatial component still contains some spectral information. The multi-resolution analysis method uses wavelet transform or Laplace pyramid decomposition to decompose the image into multiple scales, and then injects the spatial structure information of the panchromatic image into the multispectral image at different scales. This method often has good spectral fidelity performance, but the spatial information enhancement is insufficient. The variational optimization method regards fusion as an ill-posed inverse problem, and constructs the relationship between the ideal image and the observed image through the Bayesian theory or the sparse representation theory, and then optimizes the model through the optimization algorithm. This method has a solid mathematical foundation and a rigorous logical system, but the effect is severely dependent on prior knowledge and reasonable assumptions, and the time complexity is high, making it difficult to realize fast solution of the fused image and large-scale application of the method. The deep learning method has been widely applied in space-spectrum fusion due to its strong nonlinear feature extraction and learning ability, and has produced a large number of algorithms, which have achieved satisfactory results overall.

[0004] However, most of the existing deep learning methods adopt a supervised learning method to learn the mapping relationship between the original image and the fused image from a large number of simulated low-resolution training samples. This method applies the trained model to real images based on the scale invariant assumption, but this approach is easily affected by the scale difference, resulting in blurred fused images. Existing unsupervised learning methods only constrain the fused image by observing the image, and the image degradation relationship is unreasonable, making it difficult to construct the constraint relationship. In addition, most of the current multi-spectral fusion research focuses on the red, green, blue and near-infrared four bands of multi-spectral data, and the spectral range is similar to that of the panchromatic image. In such a scene, the fusion method usually works well. However, for some sensors, there are some multi-spectral bands that are not covered by the spectral range of the panchromatic image, such as the Landsat-8 OLI sensor. The spectral coverage range of the 15-meter spatial resolution panchromatic image is 500-680nm, while the spectral coverage range of the near-infrared band (5th band) of the 30-meter spatial resolution multi-spectral image is 850-880nm, and the spectral coverage range of the two short-wave infrared bands (6th and 7th bands) is 1570-1650nm and 2100-2290nm, respectively. They are very different from the spectral range of the panchromatic band, and the correlation between the spectral coverage band and the panchromatic image is significantly reduced, so the spatial information is difficult to inject in the process of multi-spectral fusion, and the spatial enhancement performance is poor, which seriously affects the practical application of the fused image in vegetation information extraction, drought monitoring and other aspects. The existing various methods do not consider this problem, and adopt an equal fusion strategy for each multi-spectral band, which makes it difficult to achieve satisfactory fusion results for spectral non-coverage bands as well as visible light bands. Therefore, it is necessary to develop a multi-spectral fusion method that takes into account the characteristics of spectral non-coverage bands and enables all multi-spectral bands to achieve balanced spatial enhancement. SUMMARY

[0005] To solve the above-mentioned problems of multi-spectral fusion technology, the present application adopts a supervised-unsupervised parallel learning strategy and provides a frequency-constrained remote sensing image multi-spectral fusion method for spectral non-coverage bands to obtain high spatial resolution multi-spectral images with equal fusion effects.

[0006] According to one aspect of the present application, a frequency-constrained remote sensing image multi-spectral fusion method for spectral non-coverage bands is provided, comprising:

[0007] obtaining high-resolution panchromatic images and low-resolution multi-spectral images to be tested;

[0008] inputting the obtained high-resolution panchromatic images and low-resolution multi-spectral images to be tested into a trained fusion network to output high-resolution fused images; wherein the training of the fusion network comprises:

[0009] Collecting panchromatic images and multispectral images acquired by satellite sensors containing spectral un-covered bands, performing image preprocessing, and constructing a reduced resolution and full resolution training sample library for network training;

[0010] Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network for spectral un-covered bands is constructed, and a loss function containing a supervised constraint, a spectral constraint, and a frequency accumulation constraint is designed;

[0011] Using the constructed training data set, the constructed fusion network is trained to convergence, and the trained fusion network is output.

[0012] As a further technical solution, the reduced resolution and full resolution training sample library for network training includes:

[0013] The original observed panchromatic images and multispectral images are geometrically registered to ensure that the same named features are strictly aligned;

[0014] The pixel values of the aligned panchromatic images and multispectral images are normalized;

[0015] According to the ratio of the spatial resolution of the panchromatic image to the multispectral image, a bicubic interpolation algorithm is used to down-sample the two images;

[0016] The original observed image pair and the reduced resolution image pair are cropped to construct a reduced resolution and full resolution training sample library.

[0017] As a further technical solution, the frequency-constrained fusion network for spectral un-covered bands includes:

[0018] A feature extraction module is constructed to extract grouped feature maps and frequency feature maps from the input reduced resolution panchromatic-multispectral image pair and full resolution panchromatic-multispectral image pair;

[0019] A band self-guided attention module is constructed to obtain low and high resolution band self-guided feature maps based on the extracted grouped feature maps and frequency feature maps;

[0020] A cross-resolution and cross-channel interaction module is constructed to interactively learn the two sets of band self-guided feature maps and the two sets of frequency feature maps through several cross-resolution and cross-channel interaction learning modules to obtain an interaction learning result;

[0021] A multi-scale residual block is constructed to obtain low and high resolution multi-scale fusion features based on the band self-guided feature maps, which are used to correct the respective resolution interaction learning results;

[0022] A fusion result reconstruction module is constructed to reconstruct the fused results of the corrected low-resolution feature map and the high-resolution feature map.

[0023] As a further technical solution, the low-resolution and high-resolution band self-guided feature maps are obtained based on the extracted grouping feature map and the frequency feature map, and further include:

[0024] The frequency feature map and each feature group in the grouping feature map are respectively subjected to element-wise multiplication operation, and the spatial information of each feature group is enhanced;

[0025] Each feature group is subjected to element-wise multiplication operation with an adjacent feature group, and the spectral coverage band feature is gradually injected into the spectral non-coverage feature;

[0026] A convolution and a ReLU activation function are used to recover the feature channel, and the low-resolution and high-resolution band self-guided feature maps are obtained.

[0027] As a further technical solution, the fused results of the corrected low-resolution feature map and the high-resolution feature map are reconstructed, including:

[0028] The corrected low-resolution feature map is subjected to a reconstruction convolution sequence to obtain a low-resolution fusion image, the low-resolution fusion image is upsampled, and then is spliced with the corrected high-resolution feature map in the channel dimension, and then is subjected to a reconstruction convolution sequence to obtain a high-resolution fusion image.

[0029] As a further technical solution, the grouping feature map and the frequency feature map are extracted, including:

[0030] The multispectral image of r times resolution is subjected to convolution operation band by band, each channel is regarded as a group, and each group is subjected to convolution, and after the convolution calculation of each group is completed, the output feature maps of all groups are spliced together to form the grouping feature map of the low-resolution and high-resolution multispectral image.

[0031] The panchromatic image is subjected to high-pass filtering processing using a Laplacian operator to obtain a high-frequency texture image, and then the high-frequency texture image is subjected to convolution processing to form the frequency feature map of the low-resolution and high-resolution panchromatic image.

[0032] According to an aspect of the present application, a spectral-unsampled-band-oriented remote sensing image space-spectrum fusion system is provided, and the fusion system is used to implement the spectral-unsampled-band-oriented remote sensing image space-spectrum fusion method.

[0033] As a further technical solution, the fusion system includes:

[0034] An input module is configured to acquire a high-resolution panchromatic image and a low-resolution multispectral image to be tested.

[0035] The fusion module is used for inputting the acquired high-resolution panchromatic image to be tested and low-resolution multispectral image into the trained fusion network, and outputting a high-resolution fused image; wherein the training of the fusion network comprises:

[0036] The panchromatic image and the multispectral image acquired by the satellite sensor containing the spectral non-covered band are collected, preprocessed, and used to construct a reduced resolution and full resolution training sample library for network training;

[0037] Based on the supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network facing the spectral non-covered band is constructed, and a loss function containing a supervised constraint, a spectral constraint and a frequency accumulation constraint is designed.

[0038] The constructed fusion network is trained to convergence using the constructed training data set, and a trained fusion network is output.

[0039] According to an aspect of the present application, a remote sensing image space-spectrum fusion device facing a spectral non-covered band is provided, comprising a memory and a processor, the memory storing program instructions executed by the processor, and the processor calling the program instructions to execute the remote sensing image space-spectrum fusion method facing the spectral non-covered band.

[0040] According to an aspect of the present application, a non-transitory computer readable storage medium is provided, which stores computer instructions for executing the remote sensing image space-spectrum fusion method facing the spectral non-covered band.

[0041] The method can effectively consider the characteristics of weak spatial enhancement effect of the spectral non-covered band, fuse the high-resolution panchromatic image and the low-resolution multispectral image, and obtain a fused image with balanced fusion precision of each band.

[0042] (1) The present application adopts a supervised-unsupervised parallel learning strategy, effectively utilizes the advantages of the two learning methods, and directly fuses the image to be processed to obtain a high spatial resolution multispectral image through an end-to-end method.

[0043] (2) The present application considers the characteristics of the non-covered band in the multispectral image which is not covered by the spectral range of the panchromatic image, embeds a band self-guided attention mechanism and a cross-resolution cross-channel interaction mechanism in the network, and designs a cumulative high-frequency loss function, fully utilizes the frequency characteristics of the image to constrain the spatial texture of each band, and constructs a network model which can effectively enhance the spatial of the spectral non-covered band. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0045] Figure 1 is a method flowchart provided by the embodiment of the present application.

[0046] Figure 2 is a comparison diagram of the fusion image based on the WorldView-2 image and the advanced method according to the embodiment of the present application.

[0047] Figure 3 is a comparison diagram of the fusion image based on the Landsat-8 image and the advanced method according to the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form new technical solutions. Such combination is not restricted by the order of steps and / or structure mode, but should be based on the realization by those skilled in the art. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope of the present application.

[0049] In view of the problem that remote sensing imaging is limited by the physical mechanism of the sensor, resulting in that the spatial-spectral resolution of a single remote sensing image cannot be compatible, the present application can obtain a high spatial resolution multispectral image by fusing the high spatial resolution panchromatic image and the low spatial resolution multispectral image of the same scene on the ground provided by the satellite, which is convenient for subsequent application. In view of the problem that the multispectral image obtained by some satellite sensors has a band not covered by the spectral range of the panchromatic image, resulting in that it is difficult to effectively inject the spatial information of the panchromatic image into the band in the fusion process, the present application simultaneously adopts supervised and unsupervised learning strategies to construct a fusion network facing the frequency constraint of the spectral non-covered band, so as to obtain a high spatial resolution multispectral image with equal fusion effect of each band.

[0050] See Figure 1The application provides a spectral-uncovered-waveband-oriented frequency-constrained remote sensing image space-spectrum fusion method, which comprises the following steps:

[0051] Step 1: Collect panchromatic images and multispectral images obtained by satellite sensors containing spectral-uncovered wavebands, perform image preprocessing, and construct a reduced resolution and full resolution sample library for network training.

[0052] Specifically, the method comprises the following steps:

[0053] Step 1.1: Geometrically register the original observed panchromatic images and multispectral images to ensure that the same named ground objects are strictly aligned;

[0054] Step 1.2: Normalize the pixel values of the aligned panchromatic images and multispectral images, and transform the pixel values to the [0, 1] interval;

[0055] Step 1.3: According to the ratio of the spatial resolutions of the panchromatic images and the multispectral images, perform downsampling on the two images by using a bicubic interpolation algorithm;

[0056] Step 1.4: Crop the original observed image pairs and the reduced resolution image pairs to construct a reduced resolution and full resolution training sample library.

[0057] Step 2: Based on a supervised-unsupervised parallel learning strategy, construct a spectral-uncovered-waveband-oriented frequency-constrained fusion network, and design a loss function comprising a supervised constraint, a dual-domain spectral constraint and a cumulative frequency constraint.

[0058] The fusion network respectively takes the panchromatic-multispectral image pairs of reduced resolution and full resolution as inputs, respectively constructs a reduced resolution fusion branch and a full resolution fusion branch, and each branch comprises a band self-guided attention module and a multi-scale residual block. The output features of the band self-guided attention modules of the two branches will pass through a cross-resolution cross-channel interaction module, and then the output of the two multi-scale residual blocks is used to correct the output of the module, to obtain deep low-resolution features and deep high-resolution features, which are respectively obtained through a feature reconstruction module to obtain a low-resolution fusion image and a high-resolution fusion image. The low-resolution fusion image and the original low-resolution multispectral image are used to calculate the supervised loss; the high-resolution fusion image and the up-sampled result of the original low-resolution multispectral image are used to calculate the dual-domain spectral loss; and the high-resolution panchromatic image is used to calculate the cumulative frequency loss.

[0059] Specifically, the method comprises the following steps:

[0060] Step 2.1: input the reduced resolution panchromatic-multispectral image pair (P1, M1) and the full resolution panchromatic-multispectral image pair (P2, M2); and Step 2.2: the band self-guided attention module of the reduced resolution fusion branch is used to extract the features of the input image pair (P1, M1) to obtain the low-resolution feature map (F1); and ), and extract group features and frequency features, where B The number of bands in a multispectral image. H and W The number of rows and columns in the panchromatic image. h and w The number of rows and columns in the multispectral image. r The ratio of the spatial resolution of the two images, i.e. H / h .

[0061] Specifically, it includes:

[0062] Step 2.1.1: For r Multispectral images with upsampling resolution (x times) and Perform 3×3 convolution operations band by band, treating each channel as a group, and convolving each group separately, resulting in a channel count of 32×3. B The grouping characteristics can be expressed as:

[0063]

[0064] in To output feature maps in groups g middle( i , j The value of the position. It is a group g The Middle m Convolutional kernels for each input channel, This represents a local region of the input multispectral image, where K represents the number of pixels within each local receptive field of the convolution kernel, and G is the number of groups in the grouped convolution. After completing the convolution calculation for each group, the output feature maps of all groups are concatenated to form the grouped feature maps of the low- and high-resolution multispectral images. and .

[0065] Step 2.1.2: Apply a Laplacian operator to the panchromatic image for high-pass filtering to obtain its high-frequency texture image, and then perform a 1×1 convolution to form the frequency feature maps of the low- and high-resolution panchromatic images. and This can be represented as:

[0066]

[0067] Step 2.2: Constructing the band self-guided attention module. In this module, the frequency feature map is first multiplied element by element with each feature group in the grouping feature map to enhance the spatial information of each feature group; then each feature group is multiplied element by element with the adjacent feature group, the first feature group is multiplied with the second feature group, the second feature group is multiplied with the third feature group, and so on, the last feature group is multiplied with the first feature group, realizing the gradual injection of spectral coverage band features to spectral non-coverage features, and finally using three layers of 3x3 convolution and ReLU activation function to restore the features to 32 channels, obtaining low and high resolution band self-guided feature maps With The process can be represented as:

[0068]

[0069] Step 2.3: Constructing a multi-scale residual block. In this module, the band self-guided feature map will be convolved with a convolution kernel size of 1, 3, 5, and 7, respectively, then the multi-scale feature maps are spliced and passed through 1x1 convolution, ReLU activation function, three residual blocks, and 1x1 convolution to obtain low and high resolution multi-scale fusion features With .

[0070] Step 2.4: Constructing a cross-resolution and cross-channel interaction learning module. To enable effective information interaction between the low and high resolution branches, and to make full use of the frequency feature map to enhance the spatial information, the two sets of band self-guided feature maps and the two sets of frequency feature maps will pass through several cross-resolution and cross-channel interaction modules. In each module, the low-band self-guided feature map and the low-frequency feature map will pass through a high-frequency attention feature fusion module to obtain a low intermediate layer fusion feature, and the high-band self-guided feature map and the high-frequency feature map will also pass through a high-frequency attention feature fusion module to obtain a high intermediate layer fusion feature. Then, the channel attention mechanism is applied to the two sets of fusion features to obtain low and high channel weights. Subsequently, the low channel weight is multiplied by the high intermediate layer fusion feature, and after downsampling, it is added to the low intermediate layer fusion feature; the high channel weight is multiplied by the low intermediate layer fusion feature, and after upsampling, it is added to the high intermediate layer fusion feature. The process can be represented as:

[0071]

[0072] wherein, HPAFF represents a feature attention fusion module for deep coupling of frequency feature maps and spectral features; Channel_Att represents a channel attention mechanism for obtaining the weight of each channel in the feature map; With are respectively rDown-sampling and up-sampling processes with double spatial resolution; With are the low and high resolution feature maps output by the first cross-resolution cross-channel interaction module, respectively, and are taken as input data together with With into the next cross-resolution cross-channel interaction module. After M modules, the interaction learning results With .

[0073] Step 2.5: The low and high resolution multi-scale fusion features are used to further correct the interaction learning results of each resolution, which can be expressed as:

[0074]

[0075] With are the low and high resolution final feature maps before the fusion result reconstruction.

[0076] Step 2.6: Construct the fusion result reconstruction module. For After the reconstruction convolution sequence, the low resolution fusion image is obtained. After up-sampling, the high resolution fusion image is obtained. The reconstruction convolution sequence of the low and high resolutions is 3x3 convolution, LeakyReLU activation, 3x3 convolution, LeakyReLU activation, and 1x1 convolution.

[0077] Step 2.7: Construct the loss function.

[0078] Step 2.7.1: Supervised loss. The loss between the original observed low resolution multispectral and low resolution fusion image is calculated using the 2-norm, which can be expressed as:

[0079]

[0080] wherein N denotes the batchsize size, , M (n) denote the nth low resolution fusion image and low resolution multispectral image in each batchsize, respectively.

[0081] Step 2.7.2: Dual-domain spectral loss. The loss between the up-sampled image of the original observed low resolution multispectral and the high resolution fusion image, including the constraints of the image domain and the gradient domain, can be expressed as:

[0082]

[0083] in F represents the gradient operation along the channel dimension of the image. (n) , These represent the nth high-resolution fused image and the low-resolution multispectral upsampled image within each batchsize, respectively.

[0084] Step 2.7.3: Cumulative Frequency Loss. This refers to the band-by-band cumulative loss between the Laplacian high-frequency image of the panchromatic image after band-by-band histogram matching of the low-resolution multispectral image and the Laplacian high-frequency image of the high-resolution fused image. It can be expressed as:

[0085]

[0086] in, b Indicates the first b Each band, This indicates that histogram matching is performed separately for each band of the low-resolution multispectral image. B Band panchromatic image, F represents high-resolution fused image. This process can be represented as:

[0087]

[0088] in, and Let M represent the mean and standard deviation of the image, respectively. b This represents the b-th band of the low-resolution multispectral image.

[0089] Step 2.7.4: Weight the three loss functions to construct the overall loss function, which can be expressed as:

[0090]

[0091] These represent the weights of the three loss functions.

[0092] Step 3: Using the training dataset constructed in Step 1, train the fusion network constructed in Step 2 until convergence;

[0093] Step 4: Using the converged fusion network trained in Step 3, fuse the high-resolution panchromatic image and the low-resolution multispectral image to be tested. The final output high-resolution fused image is the required high spatial resolution multispectral image.

[0094] Figure 2 A comparative schematic diagram of the embodiment of the present invention based on WorldView-2 imagery and image fusion using advanced methods is provided. In the diagram, the first row shows the visualization effect of 5-3-2 true color synthesis, and the second row shows the visualization effect of 8-7-1 false color synthesis.

[0095] Figure 3The application embodiment gives a comparison diagram of the Landsat-8 image and the fused image of the advanced method, wherein the first row is the 4-3-2 true color synthesis visualization effect, and the second row is the 7-6-1 false color synthesis visualization effect.

[0096] The implementation basis of each embodiment of the application is realized by the programmed processing of the device with the processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the application are packaged into various modules. Based on the above-mentioned embodiments, the embodiment of the application provides a remote sensing image space-spectrum fusion system for a spectrum-uncovered wave band, which is used to execute the remote sensing image space-spectrum fusion method for a spectrum-uncovered wave band in the above-mentioned method embodiment.

[0097] The system comprises an input module configured to acquire high-resolution panchromatic images and low-resolution multispectral images to be tested; and a fusion module configured to input the acquired high-resolution panchromatic images and low-resolution multispectral images to be tested into a trained fusion network and output high-resolution fused images; wherein the training of the fusion network comprises: collecting panchromatic images and multispectral images acquired by satellite sensors containing spectrum-uncovered wave bands, performing image preprocessing, and constructing a reduced resolution and full resolution training sample library for network training; based on a supervised-unsupervised parallel learning strategy, constructing a frequency-constrained fusion network for a spectrum-uncovered wave band, and designing a loss function containing a supervised constraint, a spectrum constraint, and a frequency accumulation constraint; using the constructed training data set, training the constructed fusion network to convergence, and outputting the trained fusion network.

[0098] The remote sensing image space-spectrum fusion system for a spectrum-uncovered wave band provided by the embodiment of the application faces the problem that the spectrum-uncovered wave band characteristics are ignored in space-spectrum fusion, adopts the foregoing several modules, effectively enhances the spatial detail information after the spectrum-uncovered wave band fusion while maintaining the spectrum-covered wave band fusion effect, and enables all multispectral wave bands to achieve balanced fusion effect.

[0099] It should be noted that the system embodiment provided by the application is used to implement the method in the above-mentioned method embodiment, and is also used to implement the method in other method embodiments provided by the application, the difference is only that the corresponding function modules are set, the principle is basically the same as that of the above-mentioned system embodiment provided by the application, as long as the person skilled in the art improves the device in the above-mentioned system embodiment on the basis of the above-mentioned system embodiment, refers to the specific technical solutions in other method embodiments, obtains the corresponding technical means by combining the technical features, and the technical solutions composed of these technical means, on the premise of ensuring the practicability of the technical solutions, obtains the corresponding system class embodiment, which is used to implement the method in other method class embodiments.

[0100] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application provide a spectral-uncovered-waveband-oriented remote sensing image space-spectrum fusion device, comprising a memory and a processor, the memory stores program instructions executed by the processor, and the processor invokes the program instructions to execute the spectral-uncovered-waveband-oriented remote sensing image space-spectrum fusion method.

[0101] In the embodiments of the present application, the memory can be a non-volatile memory such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., and can also be a volatile memory such as a random-access memory (RAM). The memory can be any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, for storing program instructions and / or data.

[0102] In the embodiments of the present application, the processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can realize or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0103] According to an aspect of the present application, a non-transitory computer-readable storage medium is provided, which stores computer instructions, and the computer instructions cause the computer to execute the spectral-uncovered-waveband-oriented remote sensing image space-spectrum fusion method, comprising:

[0104] Obtaining high-resolution panchromatic images and low-resolution multispectral images to be tested;

[0105] Inputting the obtained high-resolution panchromatic images and low-resolution multispectral images to be tested into the trained fusion network to output high-resolution fused images; wherein the training of the fusion network comprises:

[0106] Collecting panchromatic images and multispectral images obtained by satellite sensors containing spectral-uncovered-wavebands, performing image preprocessing, and constructing a low-resolution and full-resolution training sample library for network training;

[0107] Based on a supervised-unsupervised parallel learning strategy, a fusion network for frequency constraint of the spectral non-covered band is constructed, and a loss function containing a supervised constraint, a spectral constraint and a frequency accumulation constraint is designed.

[0108] Using the constructed training data set, the constructed fusion network is trained to convergence, and a trained fusion network is output.

[0109] In summary, the present application discloses a remote sensing image space-spectrum fusion method for frequency constraint of the spectral non-covered band. Firstly, the panchromatic image and the multispectral image are preprocessed, and a reduced resolution and full resolution sample library for network training is constructed. Then, a supervised-unsupervised parallel learning strategy is adopted to construct a fusion network embedded with a band self-guided attention module, a cross-resolution cross-channel interaction module, a multi-scale residual block and a reconstruction module. A loss function containing a supervised constraint, a dual-domain spectral constraint and an accumulated frequency constraint is established. The low-resolution and high-resolution panchromatic-multispectral sample pairs are input for training until the network converges. Through the trained network, the high-resolution panchromatic image and the low-resolution multispectral image to be processed are fused to obtain a high spatial resolution multispectral image. The present application can overcome the problem that the previous method ignores the characteristics of the spectral non-covered band, and effectively enhances the spatial detail information after fusion of the spectral non-covered band while maintaining the fusion effect of the spectral covered band, so that all multispectral bands can achieve balanced fusion effect.

[0110] It should be understood that parts not elaborated in the specification are all prior art.

[0111] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the present application.

Claims

1. A method for spectral un-covered band oriented remote sensing image space-spectrum fusion, characterized in that, The application relates to a method for training a fusion network for a satellite sensor with a spectral non-covered band, comprising the following steps: acquiring high-resolution panchromatic images and low-resolution multispectral images to be tested; inputting the acquired high-resolution panchromatic images and low-resolution multispectral images to be tested into a trained fusion network to output high-resolution fused images; wherein the training of the fusion network comprises the following steps: collecting panchromatic images and multispectral images acquired by a satellite sensor containing a spectral non-covered band, performing image preprocessing, and constructing a low-resolution and full-resolution training sample library for network training; based on a supervised-unsupervised parallel learning strategy, constructing a frequency-constrained fusion network for the spectral non-covered band, and designing a loss function containing a supervised constraint, a spectral constraint and a frequency accumulation constraint; the fusion network takes a panchromatic-multispectral image pair of low resolution and a panchromatic-multispectral image pair of full resolution as inputs, respectively, constructs a low-resolution fusion branch and a full-resolution fusion branch, and each branch contains a band self-guided attention module and a multi-scale residual module; the output features of the band self-guided attention modules of the two branches pass through a cross-resolution cross-channel interaction module, and then the output features of the two multi-scale residual modules are used to correct the output features of the module, so as to obtain deep low-resolution features and deep high-resolution features; the low-resolution fused images and the high-resolution fused images are obtained through a feature reconstruction module respectively; using the constructed training data set, the constructed fusion network is trained to convergence, and a trained fusion network is output.

2. The method according to claim 1, wherein, The low-resolution and full-resolution training sample library for network training comprises the following steps: performing geometric registration on the original observed panchromatic images and multispectral images to ensure that the same named ground objects are strictly aligned; normalizing the pixel values of the aligned panchromatic images and multispectral images; adopting a bicubic interpolation algorithm to down-sample the two images according to the ratio of the spatial resolutions of the panchromatic images and the multispectral images; cropping the original observed image pair and the low-resolution image pair to construct the low-resolution and full-resolution training sample library.

3. The method according to claim 1, wherein, The frequency-constrained fusion network for the spectral non-covered band comprises the following steps: constructing a feature extraction module for extracting grouped feature maps and frequency feature maps from the input low-resolution panchromatic-multispectral image pair and full-resolution panchromatic-multispectral image pair; constructing a band self-guided attention module for obtaining low-resolution and high-resolution band self-guided feature maps based on the extracted grouped feature maps and frequency feature maps; constructing a cross-resolution cross-channel interaction module for interacting the two groups of band self-guided feature maps and the two groups of frequency feature maps through a plurality of cross-resolution cross-channel interaction learning modules to obtain an interaction learning result; constructing a multi-scale residual module for obtaining low-resolution and high-resolution multi-scale fusion features based on the band self-guided feature maps, wherein the low-resolution and high-resolution multi-scale fusion features are used for correcting the respective resolution interaction learning results; constructing a fusion result reconstruction module for reconstructing the low-resolution and high-resolution features after correction.

4. The method according to claim 3, wherein the method is characterized in that, The low-resolution and high-resolution band self-guided feature maps obtained based on the extracted grouped feature maps and frequency feature maps further comprise the following steps: performing an element-by-element multiplication operation on each feature group in the frequency feature maps and the grouped feature maps to enhance the spatial information of each feature group. Each feature group is multiplied by an adjacent feature group to achieve spectral coverage band feature injection into spectral non-coverage features; Convolution and ReLU activation function are used to recover feature channels to obtain low and high resolution band self-guided feature maps.

5. The method according to claim 3, wherein the method is characterized in that, The corrected low and high resolution feature maps are fused to reconstruct the result, including: The corrected low resolution feature map is obtained through a reconstruction convolution sequence to obtain a low resolution fusion image, which is up-sampled and concatenated with the corrected high resolution feature map in the channel dimension, and then a reconstruction convolution sequence is used to obtain a high resolution fusion image.

6. The method according to claim 3, wherein the method is characterized in that, The grouped feature map and the frequency feature map are extracted, including: Each channel is regarded as a group, and each group is convolved to obtain a grouped feature map of low and high resolution multispectral images. The panchromatic image is processed using a Laplacian operator to obtain a high-frequency texture image, and then the high-frequency texture image is convolved to obtain a frequency feature map of low and high resolution panchromatic images.

7. A space-spectrum fusion system for remote sensing images facing a spectral non-covered band, characterized in that, The fusion system is used to implement the spectral non-coverage band oriented remote sensing image space-spectrum fusion method of any one of claims 1-6.

8. The system for spectral un-covered band oriented remote sensing image space-spectrum fusion according to claim 7, characterized in that, The fusion system includes: An input module for obtaining a high-resolution panchromatic image and a low-resolution multispectral image to be tested; A fusion module for inputting the obtained high-resolution panchromatic image and low-resolution multispectral image to be tested into the trained fusion network to output a high-resolution fusion image; wherein the training of the fusion network includes: Collecting panchromatic images and multispectral images obtained by satellite sensors containing spectral non-coverage bands, preprocessing the images, and constructing a reduced resolution and full resolution training sample library for network training; Based on a supervised-unsupervised parallel learning strategy, a frequency-constrained fusion network oriented to spectral non-coverage bands is constructed, and a loss function containing a supervised constraint, a spectral constraint, and a frequency accumulation constraint is designed; the fusion network takes a panchromatic-multispectral image pair of reduced resolution and full resolution as input, respectively, to construct a reduced resolution fusion branch and a full resolution fusion branch, each branch containing a band self-guided attention module and a multi-scale residual module, and the output features of the band self-guided attention modules of the two branches are processed through a cross-resolution and cross-channel interaction module, and then the output of the two multi-scale residual modules is used to correct the output of the module to obtain deep low resolution features and deep high resolution features, which are respectively obtained through a feature reconstruction module to obtain a low resolution fusion image and a high resolution fusion image. The constructed training data set is used to train the constructed fusion network to convergence, and the trained fusion network is output.

9. A spectral-uncovered-waveband-oriented remote sensing image space-spectrum fusion device, characterized in that, A memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the spectral non-coverage band oriented remote sensing image space-spectrum fusion method of any one of claims 1-6.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions cause the computer to perform the remote sensing image space-spectrum fusion method for a spectral uncovered wave band as claimed in any one of claims 1 to 6.

Images