A remote sensing image fusion method and system based on a double-coupled deep neural network

Abstract

The application provides a remote sensing image fusion method and system based on a double-coupled deep neural network, which comprises the following steps: obtaining a hyperspectral remote sensing image and a multispectral remote sensing image; inputting the hyperspectral remote sensing image and the multispectral remote sensing image into a double-coupled deep neural network model for image fusion; the double-coupled deep neural network model comprises a spatial information extraction module, a spectral information extraction module and a fusion module; the spatial information extraction module is used for extracting spatial information from the multispectral remote sensing image; the spectral information extraction module is used for extracting spectral information from the hyperspectral remote sensing image; and the fusion module is used for performing image fusion on the extracted spatial information and spectral information. The spatial information and the spectral information can be accurately fused, and the application can be used in remote sensing fields such as target identification, ground object classification and environment monitoring.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a remote sensing image fusion method and system based on a double-coupled deep neural network. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Hyperspectral imaging is a narrowband spectral imaging technology that can obtain hundreds of different wavelengths, which combines imaging technology and spectral technology to obtain information including not only two-dimensional spatial information of a target scene, but also spectral radiation information distributed with wavelengths. Due to its strong hyperspectral coverage, it can accurately identify objects, so it has been widely used in many fields, such as target detection, environmental monitoring, image classification, image segmentation, national defense and military affairs, etc. However, in reality, it is unrealistic to guarantee high-quality spectral resolution and spatial resolution at the same time, because under the condition of narrow spectral bandwidth, obtaining higher spectral resolution means lower spatial resolution. Similarly, in order to obtain a suitable signal-to-noise ratio, it is necessary to widen the spectral bandwidth, so that greater spectral resolution cannot be obtained, and the two are mutually restrictive. In a remote sensing system, a satellite-borne sensor can acquire two different images, i.e. a hyperspectral image (HSI) and a multispectral image (MSI). The hyperspectral image contains rich spectral information, but relatively lacks spatial information. On the contrary, the multispectral image contains rich spatial information, but lacks spectral information. Hyperspectral image fusion (i.e. fusion of hyperspectral image and multispectral image) aims to generate a high-resolution hyperspectral image (HR-HSI) by combining the spectral information of the hyperspectral image and the spatial information of the multispectral image, which provides a good solution to alleviate the above problems.

[0004] The methods of HS I and MSI fusion can be roughly divided into four kinds: matrix decomposition-based method, tensor representation-based method, pan-sharpening-based method and deep learning-based method. 1) The matrix decomposition-based method considers unfolding the three-dimensional HS I along the spectral dimension, reconstructing it into a two-dimensional matrix, one representing the spatial dimension and the other the number of bands. The obtained matrix is usually decomposed into a spectral product multiplied by a coefficient matrix, and the spectral product and the coefficient matrix are often estimated by solving the corresponding optimization problem to obtain the corresponding spatial information and spectral information, and finally obtain the fused HR-HSI. 2) Unlike the matrix decomposition method, the low-resolution hyperspectral image (LR-HSI) and the high-resolution multispectral image (HR-MSI) are three-dimensional data themselves, so they can be represented by three-dimensional tensors. In this way, the three-dimensional structure of the remote sensing image can be maintained, and the structural information can be well preserved, which effectively solves some problems existing in the fusion of LR-HSI and HR-MSI. 3) The pan-sharpening-based method refers to the method of fusing panchromatic images and multispectral images. The panchromatic image has high spatial resolution but only one band, so the spectral information is relatively scarce. The multispectral image has rich spectral information, but the spatial resolution is low. The purpose of fusing the two is to make use of the high spatial resolution of the panchromatic image and the rich spectral information of the multispectral image, realize the complementary advantages, and then obtain the required fused image. 4) With the development of deep learning in recent years, various convolutional neural networks are used for LR-HSI and HR-MSI fusion methods, and excellent performance is shown. Convolutional neural networks are data-driven and can effectively learn various image features through training data to reconstruct images such as UAL, PZRes-Net and SSR-NET. However, the existing methods still have the problem of insufficient spectral and spatial information mining, resulting in unsatisfactory fusion effect. SUMMARY

[0005] To overcome the above-mentioned deficiencies of the prior art, the present application provides a remote sensing image fusion method and system based on a double-coupled deep neural network, which places the hyperspectral image and the multispectral image in a double-coupled network for step-by-step processing, and extracts spectral information and spatial information respectively, so that the present application can accurately capture spectral and spatial information, which is conducive to the use of subsequent image processing techniques and improves the utilization rate of spectral and spatial information.

[0006] To achieve the above object, one or more embodiments of the present application provide the following technical scheme: a remote sensing image fusion method based on a double-coupled deep neural network, comprising:

[0007] obtaining a hyperspectral remote sensing image and a multispectral remote sensing image;

[0008] input the hyperspectral remote sensing image and the multispectral remote sensing image into a double-coupled deep neural network model for image fusion; the double-coupled deep neural network model comprises a spatial information extraction module, a spectral information extraction module, and a fusion module;

[0009] extract spatial information from the multispectral remote sensing image through the spatial information extraction module;

[0010] extract spectral information from the hyperspectral remote sensing image through the spectral information extraction module;

[0011] perform image fusion on the extracted spatial information and spectral information through the fusion module.

[0012] A second aspect of the present application provides a remote sensing image fusion system based on a double-coupled deep neural network, comprising:

[0013] an image acquisition module: acquire a hyperspectral remote sensing image and a multispectral remote sensing image;

[0014] a fusion output module: input the hyperspectral remote sensing image and the multispectral remote sensing image into a double-coupled deep neural network model for image fusion; the double-coupled deep neural network model comprises a spatial information extraction module, a spectral information extraction module, and a fusion module;

[0015] extract spatial information from the multispectral remote sensing image through the spatial information extraction module;

[0016] extract spectral information from the hyperspectral remote sensing image through the spectral information extraction module;

[0017] perform image fusion on the extracted spatial information and spectral information through the fusion module.

[0018] A third aspect of the present application provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above method.

[0019] A fourth aspect of the present application provides an electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, which, when executed by the processor, complete the steps of the above method.

[0020] The above one or more technical solutions have the following beneficial effects:

[0021] The present application places the hyperspectral image and the multispectral image in a double-coupled network for step-by-step processing, extracts spectral information and spatial information respectively, so that the present application can accurately capture spectral and spatial information, which is conducive to the use of subsequent image processing techniques and improves the utilization rate of spectral and spatial information.

[0022] In the present application, there should be interaction or communication in the learning or training process between different resolution images, so that the learned features are more complementary. The above purpose is achieved through the fusion module, which complements the extracted spectral and spatial information, and is further strengthened through the refinement module and the adaptive module, so that the reconstructed high-resolution hyperspectral image contains more details, textures and color information.

[0023] The advantages of the additional aspects of the present application will be given in part in the following description, part will become apparent from the following description, or be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings constituting a part of the specification of the present application are used to provide further understanding of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitation on the present application.

[0025] Figure 1 The flow chart for the implementation in the first embodiment of the present application is shown in Figure 1.

[0026] Figure 2 The network detail diagram of the spatial information extraction module in the first embodiment of the present application is shown in Figure 2.

[0027] Figure 3 The network detail diagram of the spectral information extraction module in the first embodiment of the present application is shown in Figure 3.

[0028] Figure 4 The network detail diagram of the fusion module in the first embodiment of the present application is shown in Figure 4.

[0029] Figure 5 The network detail diagram of the refinement module in the first embodiment of the present application is shown in Figure 5.

[0030] Figure 6 The network detail diagram of the adaptive module in the first embodiment of the present application is shown in Figure 6.

[0031] Figure 7 The network detail diagram of the channel dimension upsampling module in the first embodiment of the present application is shown in Figure 7.

[0032] Figure 8 The comparison diagram of the fusion result of the low-resolution hyperspectral image and the high-resolution multispectral image in the first embodiment of the present application is shown in Figure 8. DETAILED DESCRIPTION

[0033] It should be pointed out that the following detailed description is exemplary, and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0034] It is to be noted that the terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0035] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0036] Embodiment one

[0037] As Figure 1 shown, the embodiment discloses a remote sensing image fusion method based on a double-coupled deep neural network, comprising:

[0038] obtaining a hyperspectral remote sensing image and a multispectral remote sensing image;

[0039] inputting the hyperspectral remote sensing image and the multispectral remote sensing image into a double-coupled deep neural network model for image fusion; the double-coupled deep neural network model comprises a spatial information extraction module, a spectral information extraction module, and a fusion module;

[0040] extracting spatial information from the multispectral remote sensing image through the spatial information extraction module;

[0041] extracting spectral information from the hyperspectral remote sensing image through the spectral information extraction module;

[0042] performing image fusion on the extracted spatial information and spectral information through the fusion module.

[0043] In the embodiment, the data set adopts a CAVE data set containing 32 indoor scenes, which is cropped to the required size as needed, and the low-resolution hyperspectral image LR-HSI Y∈R 64×64×31 and the high-resolution multispectral image HR-MSI X∈R 256×256×3 .

[0044] A four-time upsampling method using Bicubic interpolation is used to upsample the low-resolution hyperspectral image to the same size as the high-resolution multispectral image, i.e. the low-resolution hyperspectral image LR-HSI Y∈R 256×256×64 , the high-resolution multispectral image HR-MSI X∈R 256×256×64 , forming paired training data.

[0045] The four-time upsampled low-resolution hyperspectral image and high-resolution multispectral image are input into the double-coupled network model, and first, convolution operation is performed to raise both to the same dimension, LR-HSI Y∈R 256×256×64 and HR-MSI X∈R 256 ×256×64 .

[0046] In the present example, the double-coupling network model comprises a spatial information extraction module, a spectral information extraction module, a fusion module, a refinement module, an adaptive module, and a channel dimension upsampling module.

[0047] As shown in the spatial information extraction module (SPA-IEM) in the present embodiment, the spatial information in the HR-MSI is very rich, and therefore a spatial graph convolution is used to obtain the features thereof from the HR-MSI. A convolution kernel of 3x3 and a stride of 1 are applied to a convolution operation to increase the channel from 3 to 64 before feature extraction. Figure 2

[0048] The spatial information extraction module mainly comprises two parts: a high-frequency spatial information extraction module and a low-frequency spatial information extraction module.

[0049] In the high-frequency spatial information extraction module, the input is first subjected to a convolution operation with a convolution kernel of 3x3 and a stride of 1, so as to convert the multi-spectral remote sensing input image into two independent representations, i.e., a first feature representation and a second feature representation. The high-frequency spatial information extraction module comprises two branches, one of which is used to construct a graph connection, and the other of which is designed as a graph node. In the process of constructing the first spatial graph connection, the first feature representation after the convolution operation is unfolded to obtain N image blocks arranged in an array. Next, the obtained image blocks are tiled into a first feature vector A1∈R N ×65536 , and the first feature vector is multiplied by its transpose. In this way, the pair-wise similarity can be effectively calculated by dot product, and a first similarity matrix L 1H ∈R N×N is obtained. The corresponding weight is calculated by a softmax function, and the graph connection is obtained. In constructing the first spatial graph node, the obtained second feature representation is subjected to a convolution operation with a convolution kernel of 2x2 and a stride of 2, a LeakyRelu, and a deconvolution operation with a convolution kernel of 2x2 and a stride of 2 in sequence. Next, the N image blocks arranged in an array are obtained by unfolding operation, and are multiplied by the corresponding weight obtained from the other branch, and the graph convolution is performed on the image blocks. Finally, the result after multiplication is subjected to an aggregation operation, i.e., the inverse operation of the unfolding operation, to realize the extraction of high-frequency spatial information, and X H ∈R 256×256×64 is obtained.

[0050] In the low-frequency spatial information extraction module, the input X H ∈R 256×256×64 ​respectively. The low-frequency spatial information extraction module still uses the same method of constructing graph connection as the high-frequency spatial information extraction module, the second spatial graph connection construction: the third feature representation after the convolution operation is unfolded to obtain N neatly arranged image blocks, and then the obtained image blocks are tiled into the second feature vector A2∈R N×65536 , and let the second feature vector multiply its transpose, so that the pair-wise similarity can be effectively calculated by dot product, and the second similarity matrix L 1L ∈R N ×N , and the corresponding weight is calculated by the softmax function, that is, the graph connection is obtained. The difference is that when constructing the graph node, no convolution and deconvolution operation is needed, but the obtained representation is directly unfolded to obtain N neatly arranged image blocks, and the image blocks are tiled into the third feature vector A3∈R N×65536 , to obtain the second spatial graph node, and then the obtained second spatial graph node is multiplied by the constructed second spatial graph connection, and the graph convolution is performed pixel by pixel, and then the result obtained by multiplication is subjected to aggregation operation to obtain the low-frequency spatial information X L ∈R 256×256×64 . Finally, the extracted spatial information X L is subjected to a convolution operation with a convolution kernel of 3*3 and a step of 1 to output, that is, F spa ∈R 256×256×64 .

[0051] As shown in Figure 3 , in the spectral information extraction module (SPE-IEM) of the present embodiment, the spectral information in the LR-HSI is rich, so the spectral graph convolution is used to extract the features of the LR-HIS. Before feature extraction, a convolution operation with a convolution kernel of 3*3 and a step of 1 is applied to increase the channel from 31 to 64. The spectral information extraction module mainly consists of two parts: a high-frequency spectral information extraction module and a low-frequency spectral information extraction module.

[0052] Compared with the construction of the spatial graph, the construction of the spectral graph adopts a different method, but the construction of the spectral graph also includes graph connection and graph node. First, for the high-frequency spectral information extraction module, the input hyperspectral remote sensing image LR-HSI Y∈R 256×256×64These are represented as two independent features: the fifth feature representation and the sixth feature representation. One branch obtains the first spectral map node through a convolution sequence. The fifth feature representation is obtained by performing convolution operations with a 3×3 kernel and a stride of 1, a 1×1 kernel and a stride of 1, LeakyReLU, and a 1×1 kernel and a stride of 1, respectively. In the other branch, the sixth feature representation undergoes average pooling, and the result of average pooling is multiplied by its transpose to obtain the third similarity matrix L. 2H ∈R N×N The corresponding weights are calculated using the softmax function and used as the first spectral map connection. These weights are then multiplied with the nodes of the first spectral map obtained from the previous branch, and graph convolution is performed block by block to obtain the high-frequency spectral information Y. H ∈R 256×256×64 .

[0053] The low-frequency spectral information extraction module extracts low-frequency spectral information in the same way as the high-frequency spectral information extraction module. First, the input Y... H ∈R 256×256×6 4 are represented as two independent features, namely the seventh feature representation and the eighth feature representation. The eighth feature representation is subjected to average pooling, and the result of average pooling is multiplied by its transpose to obtain the fourth similarity matrix L. 2L ∈R N×N Next, the corresponding weights are calculated using the softmax function to obtain the second spectral map connection. The only difference is that the branches for constructing the spectral map nodes do not need to go through a convolution sequence. Instead, the seventh feature representation is directly used as the second spectral map node through a reshape operation, multiplied with the constructed second spectral map connection, and graph convolution is performed pixel by pixel to obtain the spectral low-frequency information Y. L ∈R 256×256×64 Finally, Y L After the reshape operation, the extracted spectral information F can be obtained. spe ∈R 256×256×64 .

[0054] like Figure 4 As shown, in the fusion module (CFM) of this embodiment, in order to fully fuse the obtained spatial and spectral information, a 0.5x FF is used. spe and 0.5 times F spa The information is added together, and a bias b is introduced to balance the fusion of information, resulting in a coarse, high-resolution hyperspectral image HR-HSI F. coarse ∈R 256×256×64 .

[0055] In the reconstruction process, spatial and spectral information will inevitably be lost, which makes the fusion effect worse. In order to make up for the information loss in the fusion process, the refinement module is designed to solve this problem. The fusion module and the refinement module are connected in sequence, and the output of the fusion module is the input of the refinement module. The specific process is to divide the input into two branches first, and then pass through the convolution kernel of 3x3 and the convolution operation with a step of 1 and LeakyRelu in turn. The outputs of the two branches are respectively subtracted from the original LR-HSI and the original HR-MSI to find the lost information, and then added together, and then output through the convolution kernel of 3x3 and the convolution operation with a step of 1 and LeakyRelu. Finally, the jump connection is used to introduce the fusion result F coarse of the previous stage, and add the obtained output to obtain the refined result F R ∈R 256×256×64 .

[0056] As Figure 5 shown, the adaptive module (AM) of the embodiment refines the specific details of the image by designing a mapping block and a self-guiding block, and only contains a small amount of parameters, effectively avoiding overfitting and greatly reducing the computational amount. We stack the outputs of all stage refinement modules, and then output after passing through the convolution kernel of 3x3 and the convolution operation with a step of 1 and LeakyRelu, denoted as F CR , and take the obtained output F CR as the input of AM. The function of the mapping block is realized by the following way: the input F CR passes through the convolution kernel of 3x3 and the convolution operation with a step of 1, Relu, the convolution kernel of 3x3 and the convolution operation with a step of 1, Relu, and the convolution kernel of 3x3 and the convolution operation with a step of 1 in turn. Then the output obtained by the mapping block is sent to three branches. The first branch passes through Relu and the convolution kernel of 3x3 and the convolution operation with a step of 1 in turn; the second branch passes through Relu, the convolution kernel of 3x3 and the convolution operation with a step of 1, and the Sigmoid function in turn; the third branch is not processed. First, multiply the output of the first branch with the output of the second branch, and then subtract the output of the first branch; second, multiply the output of the second branch with the output of the third branch; add the output results of the above two to obtain the output as the guiding block; finally, the jump connection is used to introduce F CR , and add the obtained output to obtain the refined HR-HSI F A ∈R 256×256×64 .

[0057] As Figure 7 shown, in the channel dimension upsampling module (CAM) of the embodiment, in order to further utilize the information in the HR-MSI, the obtained F A ∈R256×256×64 Based on this, spatial and spectral information needs to be introduced again. Upsampling is performed along the channel dimension. This mainly includes two branches: one branch converts the original HR-MSI X∈R... 256×256×3 Reconstructing matrix M∈R 65536×3 and with the degenerate matrix S∈R 31×3 The degenerate matrix is ​​multiplied by its transpose, where the degenerate matrix is ​​given beforehand and is a known quantity. Another branch, after multiplying the degenerate matrix by its transpose, adds a 0.001 times identity matrix to prevent the matrix inverse from not existing, and then performs an inversion operation on this matrix. Finally, the two branches are multiplied, and the reconstructed result F is obtained. C ∈R 256×256×31 Spectral information is obtained through LR-HSI Y∈R after direct four-fold upsampling. 256×256×31 The introduced, subsequently output F of the AM module A Reconstructing F using a convolution operation with a 3×3 kernel and a stride of 1 A ∈R 256 ×256×31 And give the LR-HSI Y∈R after four times upsampling 256×256×64 and CAM output F C ∈R 256×256×31 Multiply all three by the same coefficient 0.5, and then add the three results together to obtain the reconstruction result F∈R. 256×256×31 .

[0058] In this embodiment, a dual-coupled neural network model is trained to fuse the low-resolution hyperspectral image LR-HSI and the high-resolution multispectral image HR-MSI to obtain a high-quality high-resolution hyperspectral image HR-HSI. The fused high-resolution hyperspectral image HR-HSI is F∈R. 256×256×31 Input: Low-resolution hyperspectral image LR-HSI Y∈R 64 ×64×31 and high-resolution multispectral images HR-MSI X∈R 256×256×3 First, the LR-HSI image is upsampled four times using Bicubic interpolation to obtain the image Y∈R. 256×256×31 Then LR-HSI Y∈R 256×256×31 and HR-MSI X∈R 256×256×3 Upgraded to the same dimension, i.e., LR-HSI Y∈R 256×256×64 and HR-MSI X∈R 256×256×64 Next, these are input into the spectral information extraction module and the spatial information extraction module, respectively. After passing through the spectral high-frequency information extraction module and the spatial high-frequency information extraction module, Y is obtained. H ∈R 256×256×64 and X H ∈R256×256×64 Then, the two are sent into the spectral low-frequency information extraction module and the spatial low-frequency information extraction module respectively to obtain Y L ∈R 256×256×64 and X L ∈R 256×256×64 The two output results F spe ∈R 256 ×256×64 and F spa ∈R 256×256×64 are taken as the input of the fusion module, and the rough fusion result F coarse ∈R 256 ×256×64 After obtaining the preliminary fusion result, in order to make up for the loss of information, the obtained result is taken as the input of the refinement module, and the output obtained after processing is F R ∈R 256×256×64 In order to perform better feature extraction, the above process is iterated three times in this paper, so that the spectral and spatial information can be fully mined. The result after iteration is input into the adaptive module after stacking, and the image details are further refined through the mapping block and the self-guiding block therein, and the output result is F A ∈R 256×256×64 Finally, F A is adjusted to the same size as the target HR-HSI, which is F A ∈R 256×256×31 , and added to the result F C ∈R 256×256×31 of the channel dimension upsampling module and the result Y∈R 256×256×31 of the upsampled LR-HSI to obtain the expected HR-HSI:

[0059] F=F A +0.5F c +0.5Y (1)

[0060] The loss function of the double-coupled neural network model in this embodiment is L1Loss:

[0061]

[0062] Wherein, Z∈R 256×256×31 is the reference high-resolution hyperspectral image HR-HSI, and F∈R 256×256×31 is the fused high-resolution hyperspectral image HR-HSI.

[0063] The simulation environment of the scheme of this embodiment: PyCharm Community Edition 2022.1.2x64, NVIDIA GEFORCE RTX3090, Ubuntu 18.04.

[0064] The simulation content of this embodiment: simulation 1: the CAVE data set containing 32 indoor scenes of the application is fused with low-resolution hyperspectral images and high-resolution multispectral images, and the results are as shown in Figure 3 , wherein:

[0065] Figure 8 (a) is a low-resolution hyperspectral image, with a size of 64x64x31,

[0066] Figure 8 (b) is a high-frequency resolution multispectral image, with a size of 256x256x3,

[0067] Figure 8 (c) is Ground-Truth, that is, the reference image of the fusion result, with a size of 256x256x31

[0068] Figure 8 (d) is a high-resolution hyperspectral image obtained by fusing (a) and (b) using the application, with a size of 256x256x31. Figure 8 Figure 8 It can be seen that the spatial detail information of (d) is obviously improved compared to (a), and the edges and textures of the image are clearer, and compared with (b),

[0069] (d) has more rich color information, so it can be seen that the application can better fuse (a) and (b). Figure 8 Figure 8 Figure 8 Figure 8 Figure 8 Figure 8 Figure 8

[0070] Simulation 2: In order to prove the effect of the application, the images of (a) and (b) are fused by the method of the application and the existing CNMF, HySure, FUSE, ICCV15 method and other deep neural network methods PZRes-Net, SSR-NET, UAL and UDALN method, and the fusion results are evaluated by objective indicators, and the evaluation indicators are as follows: Figure 8 Figure 8

[0071] 1) Peak signal-to-noise ratio PSNR, which can evaluate the spatial quality of the reconstructed HR-HSI in units of bands, the minimum value of PSNR is 0, the larger the PSNR is, the smaller the difference between the two images is, indicating that the fusion result is more similar to the reference image.

[0072] 2) Spectral Angle Mapping SAM, which is usually used to evaluate the degree of preservation of spectral information at each pixel, the closer to 0, the better the fusion result is. ​​​​​​​​​​

[0073] 3) Relative global dimensionless error ERGAS, which is specially designed for evaluating the quality of high-resolution synthetic images, which measures the global statistical quality of the estimated HR-HSI, and the smaller the value, the better the fusion result.

[0074] 4) Mean structural similarity MSSIM, which is an index for measuring the similarity between two images, and the value of MSSIM ranges from 0 to 1, and the more similar the two images are, the closer to 1 the MSSIM is.

[0075] 5) Universal image quality index UIQI, which represents the closeness of the fused image to the reference image, and the closer to 1, the better the fusion result.

[0076] According to the above evaluation indexes, the fusion results of the present application and the prior art are evaluated from the objective evaluation indexes, and the results are shown in Table 1.

[0077] Table 1. Objective evaluation of fusion results of various methods

[0078]

[0079] As can be seen from Table 1, the peak signal-to-noise ratio PSNR, the average structural similarity MSSIM and the universal image quality index UIQI of the present application are all greater than the evaluation values of the prior art, and the spectral angle mapping SAM and the relative global dimensionless error ERGAS are all less than the evaluation values of the prior art, and the above evaluation values are all better than the evaluation values of the prior art, so it can be seen that most of the objective evaluation indexes of the present application are better than the objective evaluation indexes of the prior art.

[0080] Example 2

[0081] The purpose of this embodiment is to provide a computing device, including a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the steps of the above method.

[0082] Example 3

[0083] The purpose of this embodiment is to provide a computer-readable storage medium.

[0084] A computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to perform the steps of the above method.

[0085] Example 4

[0086] The purpose of this embodiment is to provide a remote sensing image fusion system based on a double-coupled deep neural network, which comprises:

[0087] An image acquisition module: acquiring a hyperspectral remote sensing image and a multispectral remote sensing image;

[0088] a fusion output module, configured to input the hyperspectral remote sensing image and the multispectral remote sensing image into a dual-coupled deep neural network model to perform image fusion, wherein the dual-coupled deep neural network model comprises a spatial information extraction module, a spectral information extraction module, and a fusion module;

[0089] extract spatial information from the multispectral remote sensing image through the spatial information extraction module;

[0090] extract spectral information from the hyperspectral remote sensing image through the spectral information extraction module;

[0091] perform image fusion on the extracted spatial information and spectral information through the fusion module.

[0092] The steps involved in the devices of the above embodiments two, three and four correspond to the method embodiment one, and the specific implementation can be seen from the relevant description part of embodiment one. The term “computer readable storage medium” should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0093] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively manufactured into each integrated circuit module, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0094] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A method for remote sensing image fusion based on double-coupled deep neural network, characterized in that, The method comprises the following steps: obtaining a hyperspectral remote sensing image and a multispectral remote sensing image; inputting the hyperspectral remote sensing image and the multispectral remote sensing image into a double-coupled deep neural network model for image fusion; the double-coupled deep neural network model comprises a spatial information extraction module, a spectral information extraction module, a fusion module, a refinement module, an adaptive module, and a channel dimension upsampling module; extracting spatial information from the multispectral remote sensing image through the spatial information extraction module; the spatial information extraction module comprises a high-frequency spatial information extraction module and a low-frequency spatial information extraction module, and the input multispectral remote sensing image is converted into a first feature representation and a second feature representation through convolution operation; in the high-frequency spatial information extraction module, the steps are as follows: the first feature representation is unfolded and tiled into a first feature vector, the first feature vector is multiplied by its transpose to obtain a first similarity matrix, and the corresponding weight is calculated through a softmax function to obtain a first spatial graph connection; the second feature representation is sequentially subjected to 2*2 convolution operation, LeakyRelu activation function, and 2*2 deconvolution operation, and then the obtained feature representation is unfolded to obtain a first spatial graph node; after the first spatial graph node is multiplied by the first spatial graph connection, low-frequency spatial information is obtained through an aggregation operation; in the low-frequency spatial information extraction module, the high-frequency spatial information output by the high-frequency spatial information extraction module is converted into a third feature representation and a fourth feature representation through convolution operation; the steps are as follows: the third feature representation is unfolded and tiled into a second feature vector, the second feature vector is multiplied by its transpose to obtain a second similarity matrix, and the corresponding weight is calculated through a softmax function to obtain a second spatial graph connection; the fourth feature representation is unfolded and tiled into a third feature vector to obtain a second graph node; after the second graph node is multiplied by the second spatial graph node, low-frequency spatial information is obtained through an aggregation operation; extracting spectral information from the hyperspectral remote sensing image through the spectral information extraction module; performing image fusion on the extracted spatial information and spectral information through the fusion module.

2. The remote sensing image fusion method based on double-coupled deep neural network according to claim 1, wherein, The method further comprises upsampling the hyperspectral remote sensing image to make it have the same size as the multispectral remote sensing image. 3.The remote sensing image fusion method based on double-coupled deep neural network according to claim 1, wherein, extracting spectral information from the hyperspectral remote sensing image through the spectral information extraction module; the spectral information extraction module comprises a high-frequency spectral information extraction module and a low-frequency spectral information extraction module, and the input hyperspectral remote sensing image is divided into an independent fifth feature representation and a sixth feature representation; in the high-frequency spectral information extraction module, the steps are as follows: the fifth feature representation is sequentially subjected to 3*3 convolution operation, 1*1 convolution operation, LeakyRelu, and 1*1 convolution operation to obtain a first spectral graph node; the sixth feature representation is subjected to average pooling operation respectively, the results after the average pooling operation are transposed to obtain a third similarity matrix, the weight of the obtained similarity matrix is calculated through a softmax function as a first spectral graph connection, the first spectral graph connection is multiplied by the first spectral graph node to obtain high-frequency spectral information; In the low-frequency spectral information extraction module, the high-frequency spectral information output by the high-frequency spectral information extraction module is divided into a seventh feature representation and an eighth feature representation, specifically: The seventh feature representation is obtained by reshaping operation to obtain a second spectral graph node; The eighth feature representation is respectively subjected to average pooling operation, and the results after the average pooling operation are transposed and multiplied to obtain a fourth similarity matrix, and the weight of the obtained similarity matrix is calculated by a softmax function as a second spectral graph connection; The second spectral graph connection is multiplied by the second spectral graph node to obtain low-frequency spectral information. 4.The remote sensing image fusion method based on double-coupled deep neural network according to claim 1, wherein, The extracted spatial information and the extracted spectral information are added after being multiplied by coefficients, and bias balancing information is introduced to obtain a high-resolution hyperspectral image.

5. The remote sensing image fusion method based on double-coupled deep neural network according to claim 4, wherein, In the refinement module, specifically: the high-resolution hyperspectral image is divided into two branches after convolution and LeakyRelu operation, and is subtracted from the multispectral remote sensing image and the hyperspectral remote sensing image, respectively. After adding the results of the subtraction, the output is obtained after convolution operation and LeakyRelu. The output result obtained is added to the high-resolution hyperspectral image by using a jump connection to output.

6. The remote sensing image fusion method based on double-coupled deep neural network according to claim 5, wherein, The input of the adaptive module is the output of the refinement module, and in the adaptive module, specifically: The input of the adaptive module is first subjected to a mapping block, and the output of the mapping block is taken as the input of a guide block. The guide block includes three branches. The first branch includes Relu and 3x3 convolution in sequence; the second branch includes Relu, 3x3 convolution and Sigmoid function connected in sequence; The output of the first branch is multiplied by the output of the second branch, and then the output of the first branch is subtracted; The output of the second branch is multiplied by the output of the third branch; The two results obtained above are added and output; The input of the mapping block is added to the output of the guide block by using a jump connection to obtain the output of the adaptive module; The channel dimension upsampling module includes two branches, one branch reconstructs the output of the adaptive module into a matrix multiplied by the transpose of the degenerate matrix ; the other branch multiplies the degenerate matrix by the transpose of the degenerate matrix, adds a unit matrix multiplied by 0.001, and then performs an inverse operation; the outputs of the two branches are added to obtain the output of the channel dimension upsampling module.

7. A remote sensing image fusion system based on a double-coupled deep neural network, characterized in that, It includes: An image acquisition module: acquiring a hyperspectral remote sensing image and a multispectral remote sensing image; A fusion output module: inputting the hyperspectral remote sensing image and the multispectral remote sensing image into a double-coupled deep neural network model for image fusion; the double-coupled deep neural network model includes a spatial information extraction module, a spectral information extraction module, a fusion module, a refinement module, an adaptive module and a channel dimension upsampling module; The spatial information extraction module extracts spatial information from the multispectral remote sensing image; The spatial information extraction module includes a high-frequency spatial information extraction module and a low-frequency spatial information extraction module. The input multispectral remote sensing image is subjected to convolution operation to convert into a first feature representation and a second feature representation. In the high-frequency spatial information extraction module, specifically: The first feature representation is unfolded and tiled into a first feature vector, and the first feature vector is multiplied by its transpose to obtain a first similarity matrix and the corresponding weight is calculated by a softmax function to obtain a first spatial graph connection; The second feature representation is sequentially processed through a 2×2 convolution operation, a LeakyReLU activation function, and a 2×2 deconvolution operation to obtain the feature representation expanded to obtain the first spatial graph node. The first spatial graph node is connected and multiplied with the first spatial graph, and then the high-frequency spatial information is obtained through an aggregation operation. In the low-frequency spatial information extraction module, the high-frequency spatial information output from the high-frequency spatial information extraction module is converted into a third feature representation and a fourth feature representation through a convolution operation, specifically: The third feature representation is expanded and flattened into a second feature vector. The second feature vector is multiplied by its transpose to obtain a second similarity matrix. The corresponding weights are calculated using the softmax function to obtain the second spatial graph connection. The fourth feature representation is expanded and flattened into the third feature vector to obtain the second graph node. The second graph node is multiplied with the second spatial graph node and then aggregated to obtain the low-frequency spatial information. The spectral information is extracted from the hyperspectral remote sensing image using the spectral information extraction module. The fusion module performs image fusion on the extracted spatial and spectral information.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the steps in the remote sensing image fusion method based on a dual-coupled deep neural network as described in any one of claims 1-6.

9. A processing device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the remote sensing image fusion method based on a dual-coupled deep neural network as described in any one of claims 1-6.

Images