A hyperspectral and SAR image fusion recognition method and system

Through the combination of graph convolution network and dual-branch graph convolution network, the joint spatial features and spectral features of high-spectrum and SAR images are extracted, and the problem of insufficient utilization of nonlinear edge feature representation and context information in the prior art is solved, and the problem of insufficient utilization of nonlinear edge feature representation and context information is realized. Higher remote sensing image classification accuracy.

CN118762262BActive Publication Date: 2024-11-15HUNAN UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411247727.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-15
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Existing multimode remote sensing image classification technology is difficult to effectively represent highly nonlinear remote sensing image edge features, and has not fully utilized the rich context information between adjacent keys.

Method used

The graph convolution network is used to combine the dual-branch graph convolution network and a fusion module based on cross-context attention to extract the joint spatial and spectral features of high-spectral and SAR images, and connect and cross-context attention mechanisms through cross-modal residual jumps. Perform feature fusion.

Benefits of technology

The recognition accuracy of different geographic categories in remote sensing images is improved, and the edge features of geographic objects in complex remote sensing images can be better simulated and more accurate geographic classification can be achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762262B_ABST
    Figure CN118762262B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for fusion recognition of hyperspectral and SAR images. The method of the present invention comprises inputting hyperspectral images and SAR images into a pre-trained graph convolutional fusion network to obtain remote sensing image classification results. The graph convolutional fusion network for remote sensing image classification comprises the following steps: extracting spectral features of the hyperspectral image; extracting joint spatial features of the hyperspectral image and the SAR image; fusing spectral features and joint spatial features to obtain fusion features; and classifying and identifying the fusion features to obtain remote sensing image classification results. The present invention aims to utilize the advantages of graph convolutional networks in modeling high-order nonlinear structures to accurately identify different ground object categories in complex remote sensing images based on hyperspectral and SAR images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multi-mode remote sensing image classification, and in particular to a hyperspectral and SAR image fusion recognition method and system. Background Art

[0002] The intelligent interpretation of multimodal remote sensing images has attracted more and more attention in the remote sensing community. According to different data acquisition methods, remote sensing can be divided into two categories: passive remote sensing and active remote sensing. Passive remote sensing technology includes visible light remote sensing, infrared remote sensing and hyperspectral remote sensing. Among them, hyperspectral images contain rich spectral information and have been widely used in applications such as land object classification, environmental monitoring, and resource survey. Active remote sensing mainly includes synthetic aperture radar (SAR) and light detection and ranging (LiDAR). Among them, SAR data contains information about the shape, structure, and texture of ground objects. Hyperspectral images (HSI) provide rich spatial spectral information about ground objects. Therefore, the fusion of hyperspectral and SAR data can comprehensively utilize the advantages of both and improve the accuracy and reliability of scene interpretation.

[0003] In order to utilize the complementary information between multimodal remote sensing images, a large number of works have been proposed. According to the different feature extraction methods, these research works can be roughly divided into three categories: methods based on artificial features, methods based on convolutional neural networks, and methods based on attention networks. Methods based on artificial features usually use traditional mathematical and statistical models to extract features of multiple data. This type of method can effectively improve the performance of scene interpretation by fusing multimodal remote sensing images. However, these methods rely on subjectively designed features. The method based on convolutional neural networks achieves the fusion classification of multi-source images by constructing appropriate network structures and loss functions. Compared with methods based on artificial features, deep learning models can better represent multimodal remote sensing images. The method based on attention networks imitates the mechanism of human brain processing information and gives greater weight to the part with more information in the data. Unlike the CNN model, the attention network uses attention mechanisms in both feature extraction and fusion modules, and pays more attention to the interaction and fusion between different modalities.

[0004] Although the above methods can improve the classification performance, deep learning techniques for multimodal remote sensing image classification still face two challenges: 1) Edge features in remote sensing images are usually nonlinear and irregular. CNN-based methods cannot represent highly nonlinear structures well; 2) Recent works often use attention-based modules for feature fusion, but they fail to exploit the rich contextual information between adjacent keys. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for fusion recognition of hyperspectral and SAR images are provided. The present invention aims to utilize the advantages of graph convolutional networks in modeling high-order nonlinear structures to accurately identify different types of objects in complex remote sensing images based on hyperspectral and SAR images.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A hyperspectral and SAR image fusion recognition method includes inputting a hyperspectral image and a SAR image into a pre-trained graph convolutional fusion network to obtain a remote sensing image classification result, wherein the graph convolutional fusion network performs remote sensing image classification including the following steps:

[0008] S101, extracting spectral features of the hyperspectral image; extracting joint spatial features of the hyperspectral image and the SAR image;

[0009] S102, fusing the spectral feature and the feature joint spatial feature to obtain a fusion feature;

[0010] S103, classifying and identifying the fused features to obtain a remote sensing image classification result;

[0011] Extracting the joint spatial features of the hyperspectral and SAR images in step S101 refers to using a dual-branch graph convolutional network, and extracting the joint spatial features of the hyperspectral and SAR images using the dual-branch graph convolutional network includes:

[0012] S201, hyperspectral image The RGB image is obtained by dimensionality reduction using the PCA algorithm ;

[0013] S202, for SAR images and RGB images , respectively use the linear discriminant analysis algorithm LDA and the simple linear iterative clustering algorithm SLIC to perform superpixel segmentation operations to obtain the SAR image superpixel segmentation results And RGB image superpixel segmentation results ;

[0014] S203, superpixel segmentation result of SAR image through the first branch Multi-level cross-modal residual skip connections are performed to obtain pixel-level spatial features of SAR images , through the second branch to the RGB image superpixel segmentation result Perform multi-level cross-modal residual jump connections to obtain pixel-level spatial features of RGB images ;

[0015] S204, transforming the pixel-level spatial features of the SAR image and RGB image pixel-level spatial features The two are fused to obtain the joint spatial features of hyperspectral images and SAR images.

[0016] Optionally, extracting the spectral features of the hyperspectral image in step S101 refers to using a spectral residual network, the spectral residual network is composed of a spectral conversion module ST consisting of a convolution layer and a plurality of cascaded spectral residual modules, and extracting the spectral features of the hyperspectral image using the spectral residual network includes: firstly converting the hyperspectral image using the spectral conversion module ST, and then processing the hyperspectral image through a plurality of cascaded spectral residual modules to obtain the output spectral features, and the function expression of the spectral residual module is:

[0017] ,

[0018] In the above formula, is the output of the ith level spectral residual module, is the output of the i-1th level spectral residual module or the input of the 1st level spectral residual module, represents a normalized convolutional activation layer.

[0019] Optionally, the function expression of the normalized convolution activation layer is:

[0020] ,

[0021] In the above formula, R represents the activation function, C represents the 1×1 convolution operation, and B represents the normalization operation.

[0022] Optionally, in the first branch in step S203, the cross-modal residual jump connection function expression is:

[0023]

[0024] In the above formula, is the output of the i-th level cross-modal residual skip connection in the first branch, represents a graph convolution operation, The output of the i-1th level cross-modal residual skip connection in the second branch or the input of the 1st level cross-modal residual skip connection, is the output of the i-1th level cross-modal residual jump connection in the first branch or the input of the 1st level cross-modal residual jump connection; in the second branch, the cross-modal residual jump connection function expression is:

[0025] ,

[0026] In the above formula, is the output of the i-th level cross-modal residual skip connection in the second branch, represents a graph convolution operation, is the output of the i-1th level cross-modal residual jump connection in the first branch or the input of the 1st level cross-modal residual jump connection, It is the output of the i-1th level cross-modal residual skip connection in the second branch or the input of the 1st level cross-modal residual skip connection.

[0027] Optionally, in step S204, the SAR image pixel-level spatial features and RGB image pixel-level spatial features The function expression of the joint spatial features of the hyperspectral image and SAR image obtained by fusing the two is:

[0028] ,

[0029] In the above formula, It is the joint spatial feature of hyperspectral image and SAR image.

[0030] Optionally, in step S102, fusing the spectral feature and the feature joint spatial feature to obtain the fused feature means using a fusion module based on cross-context attention to fuse the spectrum and the feature joint spatial feature to obtain the fused feature, and the fusion module based on cross-context attention to fuse the spectrum and the feature joint spatial feature to obtain the fused feature includes:

[0031] S301, extracting the cross-context features of the spectral features and the feature joint spatial features respectively according to the following formula:

[0032] ,

[0033] ,

[0034] In the above formula, From the spectral characteristics Cross-context features extracted from From the joint space feature Cross-context features extracted from represents a cross-context feature extraction operation, and the function expression of the cross-context feature extraction operation is:

[0035] ,

[0036] ,

[0037] In the above formula, , and Spectral characteristics The query, key and value of the cross-context attention, , and The joint spatial features The query, key and value of the cross-context attention, is a 3×3 convolution operation, and C is a 1×1 convolution operation;

[0038]

[0039]

[0040] In the above formula, and is the attention matrix, C is a 1×1 convolution operation;

[0041]

[0042]

[0043] In the above formula, From the spectral characteristics The cross-context features extracted from From the joint space feature Cross-context features extracted from

[0044] S302, for the cross-context features of the spectral features and the feature joint spatial features, calculate the initial weight according to the following formula:

[0045] ,

[0046] ,

[0047] In the above formula, and is the initial weight, C is a 1×1 convolution operation;

[0048] S303, the initial weight and Use the Softmax activation function to normalize the mapping to between 0 and 1 to obtain the normalized weight and ;

[0049] S303, obtain the fusion feature by calculating according to the following formula:

[0050]

[0051] In the above formula, To fusion features, and is the normalized weight.

[0052] In addition, the present invention also provides a hyperspectral and SAR image fusion recognition system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the hyperspectral and SAR image fusion recognition method.

[0053] In addition, the present invention also provides a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the hyperspectral and SAR image fusion recognition method through a processor.

[0054] In addition, the present invention also provides a computer program product, including a computer program / instruction, wherein the computer program / instruction is programmed or configured to execute the hyperspectral and SAR image fusion recognition method through a processor.

[0055] Compared with the prior art, the present invention mainly has the following advantages: the method of the present invention comprises inputting a hyperspectral image and a SAR image into a pre-trained graph convolutional fusion network to obtain a remote sensing image classification result, and the graph convolutional fusion network for remote sensing image classification comprises the following steps: extracting spectral features of the hyperspectral image; extracting joint spatial features of the hyperspectral image and the SAR image; fusing spectral features and joint spatial features to obtain fusion features; classifying and identifying the fusion features to obtain a remote sensing image classification result. The present invention utilizes the advantages of the graph convolutional network in modeling high-order nonlinear structures, and by introducing the graph convolutional network to extract spatial features, it can better simulate the edges of different objects in the remote sensing image to obtain better object classification accuracy. The present invention can accurately identify different object categories in complex remote sensing images based on hyperspectral and SAR images. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0057] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0058] Figure 2 It is a schematic diagram of the principle structure of the graph convolutional fusion network in an embodiment of the present invention.

[0059] Figure 3 Schematic diagram of the structure of the spectral residual module in an embodiment of the present invention.

[0060] Figure 4 Schematic diagram of the structure of the fusion module based on cross-context attention in an embodiment of the present invention.

[0061] Figure 5Schematic diagram of hyperspectral images, SAR images and labels in the Berlin dataset in an embodiment of the present invention, where (a) is a hyperspectral image, (b) is a SAR image, and (c) is a schematic diagram of labels.

[0062] Figure 6 3 is a comparison chart of the classification results of the method according to the embodiment of the present invention and the existing method on the Berlin dataset, wherein (a) is the classification result of MSE, (b) is the classification result of S2FL, (c) is the classification result of EndNet, (d) is the classification result of DHViT, (e) is the classification result of MFT, and (f) is the classification result of the method according to the embodiment of the present invention.

[0063] Figure 7 Schematic diagram of hyperspectral images, SAR images and labels in the Augsburg dataset in an embodiment of the present invention, where (a) is a hyperspectral image, (b) is a SAR image, and (c) is a label schematic diagram.

[0064] Figure 8 1 is a comparison chart of the classification results of the method according to the embodiment of the present invention and the existing method on the Augsburg dataset, wherein (a) is the classification result of MSE, (b) is the classification result of S2FL, (c) is the classification result of EndNet, (d) is the classification result of DHViT, (e) is the classification result of MFT, and (f) is the classification result of the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be explained and described in detail in combination with the flow chart and the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] like Figure 1 As shown, the hyperspectral and SAR image fusion recognition method of this embodiment includes inputting the hyperspectral image and the SAR image into a pre-trained graph convolution fusion network to obtain a remote sensing image classification result, and the graph convolution fusion network performs remote sensing image classification including the following steps:

[0067] S101, extracting spectral features of the hyperspectral image; extracting joint spatial features of the hyperspectral image and the SAR image;

[0068] S102, fusing the spectral feature and the feature joint spatial feature to obtain a fusion feature;

[0069] S103, classifying and identifying the fused features to obtain remote sensing image classification results.

[0070] In this embodiment, step S101 of extracting the spectral features of the hyperspectral image refers to using a spectral residual network to implement the spectral features of a given hyperspectral image. As input, enter , first use the spectral residual network to extract spectral features It can be expressed as:

[0071] ,

[0072] in, represents the spectral residual network.

[0073] like Figure 2 As shown, the spectral residual network in this embodiment is composed of a spectral conversion module ST composed of a convolution layer and a plurality of cascaded spectral residual modules. As an optional implementation, this embodiment specifically includes three cascaded spectral residual modules. The spectral residual network is used to extract the spectral features of the hyperspectral image, including: firstly, the hyperspectral image is converted by the spectral conversion module ST, and then the spectral features are obtained by processing the output through a plurality of cascaded spectral residual modules, and the function expression of the spectral residual module is:

[0074] ,

[0075] In the above formula, is the output of the ith level spectral residual module, is the output of the i-1th level spectral residual module or the input of the 1st level spectral residual module, represents a normalized convolutional activation layer.

[0076] like Figure 3 As shown, the normalized convolution activation layer in this embodiment includes a normalization module (BatchNorm2d), a convolution module (Conv2d) and an activation function module connected in sequence, and the function expression of the normalized convolution activation layer is:

[0077] ,

[0078] In the above formula, R represents an activation function, C represents a 1×1 convolution operation, and B represents a normalization operation. As an optional implementation, the activation function in this embodiment specifically adopts a LeakyReLU activation function.

[0079] like Figure 2 As shown, in step S101 of this embodiment, the joint spatial features of the hyperspectral and SAR images are extracted by using a dual-branch graph convolutional network. For another given SAR image As input, enter , using a two-branch spatial graph convolutional network to extract spatial features It can be expressed as:

[0080] ,

[0081] Among them, the input RGB image Hyperspectral images After the PCA algorithm dimensionality reduction, Represents a dual-branch spatial graph convolutional network. The dual-branch spatial graph convolutional network consists of a superpixel segmentation module and a dual-branch graph network with cross-modal residual skip connections. The dual-branch graph convolutional network is used to extract joint spatial features of hyperspectral and SAR images including:

[0082] S201, hyperspectral image The RGB image is obtained by dimensionality reduction using the PCA algorithm ;

[0083] S202, for SAR images and RGB images , respectively use the linear discriminant analysis algorithm LDA and the simple linear iterative clustering algorithm SLIC to perform superpixel segmentation operations to obtain the SAR image superpixel segmentation results And RGB image superpixel segmentation results ; If each pixel of the input image is regarded as a graph node in the graph convolution, it will incur huge computational costs. To solve this problem, a common approach is to model the graph network at the superpixel level rather than the pixel level. Therefore, a superpixel segmentation operation is required before feature extraction, which can be expressed as:

[0084] ,

[0085] in, Represents a splicing operation, Indicates the use of linear discriminant analysis (LDA) and simple linear iterative clustering (SLIC) for superpixel segmentation. and They are and The result of superpixel segmentation;

[0086] S203, superpixel segmentation result of SAR image through the first branch Multi-level cross-modal residual skip connections are performed to obtain pixel-level spatial features of SAR images , through the second branch to the RGB image superpixel segmentation result Perform multi-level cross-modal residual jump connections to obtain pixel-level spatial features of RGB images ;

[0087] S204, transforming the pixel-level spatial features of the SAR image and RGB image pixel-level spatial features The two are fused to obtain the joint spatial features of hyperspectral images and SAR images.

[0088] In the first branch of step S203 of this embodiment, the cross-modal residual jump connection function expression is:

[0089]

[0090] In the above formula, is the output of the i-th level cross-modal residual skip connection in the first branch, represents a graph convolution operation, The output of the i-1th level cross-modal residual skip connection in the second branch or the input of the 1st level cross-modal residual skip connection, is the output of the i-1th level cross-modal residual jump connection in the first branch or the input of the 1st level cross-modal residual jump connection; in the second branch, the cross-modal residual jump connection function expression is:

[0091] ,

[0092] In the above formula, is the output of the i-th level cross-modal residual skip connection in the second branch, represents a graph convolution operation, is the output of the i-1th level cross-modal residual jump connection in the first branch or the input of the 1st level cross-modal residual jump connection, It is the output of the i-1th level cross-modal residual skip connection in the second branch or the input of the 1st level cross-modal residual skip connection.

[0093] In step S204 of this embodiment, the pixel-level spatial features of the SAR image are and RGB image pixel-level spatial features The function expression of the joint spatial features of the hyperspectral image and SAR image obtained by fusing the two is:

[0094] ,

[0095] In the above formula, is the joint spatial feature of the hyperspectral image and the SAR image. and They are the super-pixel-level spatial features of the last layer of cross-modal residual jump connections of RGB and SAR; and are the pixel-level spatial features of RGB and SAR, respectively. and The superpixel inverse transform is obtained. It is the joint spatial feature of RGB and SAR.

[0096] After extracting the spectral features and spatial features of multimodal remote sensing images, how to fuse the features to improve the accuracy of object classification is still an important issue. In order to make full use of the rich contextual information in the heterogeneous features when performing feature fusion, thereby further improving the classification ability of the model, the fusion of spectral features and feature joint spatial features in step S102 of this embodiment refers to the use of a fusion module based on cross-context attention to fuse the spectrum and feature joint spatial features to obtain the fusion feature, which can be expressed as:

[0097] ,

[0098] in, is the final fusion feature, Indicates that the fusion operation is performed using a fusion module based on cross-context attention. Figure 4 As shown, in this embodiment, the fusion module based on cross-context attention fuses the spectrum and the feature joint spatial features to obtain the fusion feature, including:

[0099] S301, extracting the cross-context features of the spectral features and the feature joint spatial features respectively according to the following formula:

[0100] ,

[0101] ,

[0102] In the above formula, From the spectral characteristics Cross-context features extracted from From the joint space feature Cross-context features extracted from represents a cross-context feature extraction operation, and the function expression of the cross-context feature extraction operation is:

[0103] ,

[0104] ,

[0105] In the above formula, , and Spectral characteristics The query, key and value of the cross-context attention, , and The joint spatial features The query, key and value of the cross-context attention, is a 3×3 convolution operation, and C is a 1×1 convolution operation;

[0106]

[0107]

[0108] In the above formula, and is the attention matrix, C is a 1×1 convolution operation;

[0109]

[0110]

[0111] In the above formula, From the spectral characteristics The cross-context features extracted from From the joint space feature Cross-context features extracted from

[0112] S302, in order to fuse the cross-context features, a complementary feature fusion operation is used. First, the extracted cross-context features are connected, and then two parallel convolution operations are used to obtain the initial weights. Specifically, in this embodiment, for the cross-context features of the spectral features and the feature joint spatial features, the initial weights are calculated according to the following formula:

[0113] ,

[0114] ,

[0115] In the above formula, and is the initial weight, C is a 1×1 convolution operation;

[0116] S303, the initial weight and Use the Softmax activation function to normalize the mapping to between 0 and 1 to obtain the normalized weight and ;

[0117] S303, obtain the fusion feature by calculating according to the following formula:

[0118]

[0119] In the above formula, To fusion features, and is the normalized weight.

[0120] In this embodiment S103, when the fused features are classified and identified to obtain the remote sensing image classification result, the fused features are classified and predicted using a fully connected layer, which can be expressed as:

[0121] ,

[0122] in, Indicates the use of a fully connected layer for classification prediction. This is the final classification result graph.

[0123] In order to verify the graph convolutional fusion network in this embodiment, two public data sets, Berlin and Augsburg, were used for verification experiments in this embodiment. Each data set contains a hyperspectral image, a SAR image, and a label image. In the experiment, five multimodal data fusion classification methods were compared, and three classification evaluation indicators were used, namely, overall accuracy (OA), average accuracy (AA), and Kappa coefficient. The larger the values ​​of the three, the higher the classification accuracy.

[0124] Figure 5 and Figure 6 Shown are the classification results of the Berlin dataset and five representative fusion classification methods, MSE, S2FL, EndNet, DHViT, MFT, and the method proposed in this embodiment on the Berlin dataset. Figure 5 (a), (b) and (c) are schematic diagrams of hyperspectral images, SAR images and labels in the Berlin dataset. Figure 6The figure is a comparison chart of the classification results of the existing methods (including MSE, S2FL, EndNet, DHViT, MFT) and the method of the embodiment of the present invention on the Berlin dataset. For details of MSE, please refer to the literature: Duan P, Kang X, Ghamisi P, et al. Multilevel structure extraction-based multi-sensor data fusion. Remote Sensing, 2020, 12(24):4034; for details of S2FL, please refer to the literature: Fang S, Li K, Li Z. S²ENet: Spatial-spectral cross-modal enhancement network for classification of hyperspectral and LiDAR data. IEEE Geoscience and Remote Sensing Letters, 2021, 19: 1-5; for details of EndNet, please refer to the literature: Hong D,Gao L, Hang R, et al. Deep encoder-decoder networks for classification of hyperspectral and LiDAR data. IEEE Geoscience and Remote Sensing Letters,2020, 19: 1-5; For details on DHViT, please refer to the literature: Xue Z, Tan X, Yu and Remote Sensing,2023, 61, 5515620. Figure 7 and Figure 8Shown are the classification results of the Augsburg dataset and five representative fusion classification methods MSE, S2FL, EndNet, DHViT, MFT and the method proposed in this embodiment on the Augsburg dataset. Figure 7 (a), (b) and (c) are schematic diagrams of hyperspectral images, SAR images and labels in the Augsburg dataset. Figure 8 The figure is a comparison chart of the classification results of the method of the embodiment of the present invention and the existing methods (including MSE, S2FL, EndNet, DHViT, MFT) on the AugsburgBerlin dataset. On this basis, Table 1 shows the objective evaluation indicators and calculation time of five representative fusion classification methods (MSE, S2FL, EndNet, DHViT, MFT) and the method proposed in this embodiment on the Berlin dataset. Table 2 shows the objective evaluation indicators and calculation time of five representative fusion classification methods (MSE, S2FL, EndNet, DHViT, MFT) and the method proposed in this embodiment on the Augsburg dataset.

[0125] Table 1: Objective performance indicators and calculation time of the method in this embodiment and the existing method on the Berlin dataset

[0126]

[0127] It can be seen from Table 1 that the objective evaluation indicators of the hyperspectral and SAR image fusion recognition method (this method) of this embodiment on the Berlin dataset are better than those of other methods. MSE and S2FL are traditional machine learning models without complex feature extractors, and the calculation time used is less, but the classification accuracy is relatively low. The calculation time used by the method of this embodiment is acceptable and is the least among all deep learning methods.

[0128] Table 2: Objective performance indicators and computation time of the method in this embodiment and the existing method on the Augsburg dataset

[0129]

[0130] It can be seen from Table 2 that the objective evaluation indicators of the hyperspectral and SAR image fusion recognition method (this method) of this embodiment on the Augsburg dataset are better than those of other methods. MSE and S2FL are traditional machine learning models without complex feature extractors, and the calculation time used is less, but the classification accuracy is relatively low. The calculation time used by the method of this embodiment is acceptable and is the least among all deep learning methods.

[0131] In summary, the hyperspectral and SAR image fusion recognition method of this embodiment uses a spectral residual network to extract the spectral features of the hyperspectral image, and uses a spatial graph convolutional network to extract the joint spatial features of the hyperspectral and SAR images. In addition, the hyperspectral and SAR image fusion recognition method of this embodiment also designs a dual-branch spatial graph convolutional network to extract the spatial features of multimodal remote sensing images, and cross-modal residual jump connections are applied in the network to extract fused spatial features of different modalities. In addition, the hyperspectral and SAR image fusion recognition method of this embodiment also designs a fusion module based on cross-context attention to fuse heterogeneous features, which can mine the contextual information between adjacent keys of different features and effectively fuse the spatial and spectral features of multimodal remote sensing data. The hyperspectral and SAR image fusion recognition method of this embodiment obtains better results in time and classification results, and has higher practical value.

[0132] In addition, this embodiment also provides a hyperspectral and SAR image fusion recognition system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the hyperspectral and SAR image fusion recognition method.

[0133] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program / instruction is stored. The computer program / instruction is programmed or configured to execute the hyperspectral and SAR image fusion recognition method through a processor.

[0134] In addition, this embodiment also provides a computer program product, including a computer program / instruction, which is programmed or configured to execute the hyperspectral and SAR image fusion recognition method through a processor.

[0135] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A hyperspectral and SAR image fusion recognition method, characterized in that: The method includes inputting a hyperspectral image and a SAR image into a pre-trained graph convolutional fusion network to obtain a remote sensing image classification result. The graph convolutional fusion network performs remote sensing image classification including the following steps: S101, extracting spectral features of the hyperspectral image; extracting joint spatial features of the hyperspectral image and the SAR image; S102, fusing the spectral feature and the feature joint spatial feature to obtain a fusion feature; S103, classifying and identifying the fused features to obtain a remote sensing image classification result; Extracting the joint spatial features of the hyperspectral and SAR images in step S101 refers to using a dual-branch graph convolutional network, and extracting the joint spatial features of the hyperspectral and SAR images using the dual-branch graph convolutional network includes: S201, hyperspectral image The RGB image is obtained by dimensionality reduction using the PCA algorithm ; S202, for SAR images and RGB images , respectively use the linear discriminant analysis algorithm LDA and the simple linear iterative clustering algorithm SLIC to perform superpixel segmentation operations to obtain the SAR image superpixel segmentation results And RGB image superpixel segmentation results ; S203, superpixel segmentation result of SAR image through the first branch Multi-level cross-modal residual skip connections are performed to obtain pixel-level spatial features of SAR images , through the second branch to the RGB image superpixel segmentation result Perform multi-level cross-modal residual jump connections to obtain pixel-level spatial features of RGB images ; S204, transforming the pixel-level spatial features of the SAR image and RGB image pixel-level spatial features The fusion of the two obtains the joint spatial features of the hyperspectral image and the SAR image; In the first branch of step S203, the cross-modal residual jump connection function expression is: In the above formula, is the output of the i-th level cross-modal residual skip connection in the first branch, represents a graph convolution operation, The output of the i-1th level cross-modal residual skip connection in the second branch or the input of the 1st level cross-modal residual skip connection, is the output of the i-1th level cross-modal residual jump connection in the first branch or the input of the 1st level cross-modal residual jump connection; in the second branch, the cross-modal residual jump connection function expression is: , In the above formula, is the output of the i-th level cross-modal residual skip connection in the second branch, represents a graph convolution operation, is the output of the i-1th level cross-modal residual jump connection in the first branch or the input of the 1st level cross-modal residual jump connection, is the output of the i-1th level cross-modal residual jump connection in the second branch or the input of the 1st level cross-modal residual jump connection; In step S204, the pixel-level spatial features of the SAR image are and RGB image pixel-level spatial features The function expression of the joint spatial features of the hyperspectral image and SAR image obtained by fusing the two is: , In the above formula, It is the joint spatial feature of hyperspectral image and SAR image.

2. The hyperspectral and SAR image fusion recognition method according to claim 1, characterized in that: Extracting the spectral features of the hyperspectral image in step S101 refers to using a spectral residual network, which is composed of a spectral conversion module ST composed of a convolution layer and multiple cascaded spectral residual modules. Using the spectral residual network to extract the spectral features of the hyperspectral image includes: firstly converting the hyperspectral image using the spectral conversion module ST, and then processing it through multiple cascaded spectral residual modules to obtain the output spectral features, and the function expression of the spectral residual module is: , In the above formula, is the output of the ith level spectral residual module, is the output of the i-1th level spectral residual module or the input of the 1st level spectral residual module, represents a normalized convolutional activation layer.

3. The hyperspectral and SAR image fusion recognition method according to claim 2 is characterized in that: The function expression of the normalized convolution activation layer is: , In the above formula, R represents the activation function, C represents the 1×1 convolution operation, and B represents the normalization operation.

4. The hyperspectral and SAR image fusion recognition method according to claim 1 is characterized in that: In step S102, fusing the spectral feature and the feature joint spatial feature to obtain the fused feature means using a fusion module based on cross-context attention to fuse the spectrum and the feature joint spatial feature to obtain the fused feature, and the fusion module based on cross-context attention to fuse the spectrum and the feature joint spatial feature to obtain the fused feature includes: S301, extracting the cross-context features of the spectral features and the feature joint spatial features respectively according to the following formula: , , In the above formula, From the spectral characteristics Cross-context features extracted from From the joint space feature Cross-context features extracted from represents a cross-context feature extraction operation, and the function expression of the cross-context feature extraction operation is: , , In the above formula, , and Spectral characteristics The query, key and value of the cross-context attention, , and The joint spatial features The query, key and value of the cross-context attention, is a 3×3 convolution operation, and C is a 1×1 convolution operation; In the above formula, and is the attention matrix, C is a 1×1 convolution operation; In the above formula, From the spectral characteristics The cross-context features extracted from From the joint space feature Cross-context features extracted from S302, for the cross-context features of the spectral features and the feature joint spatial features, calculate the initial weight according to the following formula: , , In the above formula, and is the initial weight, C is a 1×1 convolution operation; S303, the initial weight and Use the Softmax activation function to normalize the mapping to between 0 and 1 to obtain the normalized weight and ; S303, obtain the fusion feature by calculating according to the following formula: In the above formula, To fusion features, and is the normalized weight.

5. A hyperspectral and SAR image fusion recognition system, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the hyperspectral and SAR image fusion recognition method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program / instruction stored therein, characterized in that: The computer program / instruction is programmed or configured to execute the hyperspectral and SAR image fusion recognition method according to any one of claims 1 to 4 through a processor.

7. A computer program product comprising a computer program / instructions, characterized in that The computer program / instruction is programmed or configured to execute the hyperspectral and SAR image fusion recognition method according to any one of claims 1 to 4 through a processor.

Citation Information

Patent Citations

  • Hyperspectral and laser radar multimode image spatial-spectral fusion ground object identification method and device

    CN117036879A

  • Method for classifying hyperspectral images on basis of adaptive multi-scale feature extraction model

    WO2022160771A1

  • Hyperspectral imaging method and system based on double RGB image fusion, and medium

    WO2024027095A1

  • Method and device for fusion and classification of remote sensing hyperspectral image and laser radar image

    WO2024040828A1