Method and device for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images

Abstract

The present invention relates to the technical field of remote sensing image processing, and specifically refers to a method and device for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images, including: obtaining a hyperspectral image and a synthetic aperture radar image with the same spatial size of the current remote sensing scene, constructing a neighborhood block for each hyperspectral pixel, and a neighborhood block for each synthetic aperture radar pixel; inputting the neighborhood block of the current hyperspectral pixel into the hyperspectral pixel processing branch to output the target feature, reflection feature, and illumination feature of the current hyperspectral pixel; inputting the neighborhood block of the synthetic aperture radar pixel with the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch to output the target feature of the current synthetic aperture radar pixel; inputting all the obtained features into a splicing layer to output a fused feature, and inputting it into a fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel. The present invention improves the accuracy of pixel ground object classification in remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method and device for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images. Background Art

[0002] In the field of remote sensing technology, hyperspectral images and synthetic aperture radar images are commonly used data in various studies; hyperspectral images contain rich spatial information and spectral information. The spatial information refers to the spatial position of pixels at each wavelength, and the spectral information refers to the spectral curve formed by the spectral reflectance of a single pixel at different wavelengths; synthetic aperture radar images can record the elevation information of target ground objects. Deeply fusing hyperspectral images and synthetic aperture radar images can achieve information complementarity and help to learn and model the complete information of ground objects; in addition, fusing and classifying these two types of remote sensing images, namely hyperspectral images and synthetic aperture radar images, can fully exploit the embedded characteristics in pixels at the same position in the two remote sensing images, improve the recognition accuracy of classification research, so as to obtain the ground object classification results accurately corresponding to each pixel, and provide important data support for subsequent applications in multiple fields such as agriculture, forestry, geology, and urban planning. For example, according to the ground object classification results accurately corresponding to each pixel in the urban planning remote sensing image, not only can the distribution and proportion of different ground objects such as buildings, roads, green spaces, and water areas in the city be clarified to optimize the urban spatial layout, but also it can help to plan the layout of infrastructure such as urban transportation networks, water and electricity facilities, and communication base stations, and at the same time monitor the operation status of urban infrastructure, promptly detect facility damage or failures, and ensure the normal operation of the city.

[0003] In existing hyperspectral image and synthetic aperture radar image fusion classification methods in the field of remote sensing, two independent branches are mostly used to mine information and extract features from the two types of images respectively, and then the fusion of multi-source information is completed through simple connection and other methods, ignoring the correlation between different branches, resulting in the failure to consider the joint ability and collaborative ability between the two types of images, making it difficult to achieve an effective balance of multi-source information, unable to comprehensively grasp the complete information of pixels, and thus unable to give full play to the advantages of multi-modal remote sensing images in pixel classification and recognition; secondly, during the image fusion process, after the hyperspectral image is intrinsically decomposed into a reflection image and an illumination image, only the reflection image is used for fusion classification with the synthetic aperture radar image, discarding the decomposed illumination image, resulting in incomplete information in the fused image, making it impossible to fully consider the influence of illumination factors on the image in the subsequent classification process, thereby affecting the accuracy and reliability of classification; at the same time, the internal connection between the illumination information of the hyperspectral image and the elevation information of the synthetic aperture radar image is not fully considered, making it difficult to achieve deep fusion, resulting in the inability to obtain complete and accurate feature descriptions of various types of ground objects in the fused image, unable to deeply mine the true attributes of ground objects, and reducing the accuracy of pixel classification and recognition. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the deficiencies in the prior art, which do not consider the correlation between the two types of images, do not consider incorporating the illumination image obtained by intrinsic decomposition of the hyperspectral image into the image fusion process, and do not consider the internal connection between the illumination information of the hyperspectral image and the elevation information of the synthetic aperture radar image, resulting in low integrity and accuracy of image fusion and low precision of pixel classification and recognition.

[0005] To solve the above technical problems, the present invention provides a method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images, including:

[0006] Obtain a hyperspectral image and a synthetic aperture radar image with the same spatial size of the current remote sensing scene, construct a neighborhood block for each hyperspectral pixel, and a neighborhood block for each synthetic aperture radar pixel;

[0007] Input the neighborhood block of the current hyperspectral pixel into the spectral distribution correction sub-branch of the hyperspectral pixel processing branch: after being processed by the convolutional module, input it into the dynamic update graph convolutional layer and the Mamba network layer respectively; add the output of the dynamic update graph convolutional layer and the output of the Mamba network layer to obtain the target feature of the current hyperspectral pixel;

[0008] Input the neighborhood block of the current hyperspectral pixel into the cross-domain eigen-decomposition sub-branch of the hyperspectral pixel processing branch: After being processed by the convolution module, it is respectively input into two parallel convolution modules. After obtaining the reflection component and the illumination component, they are respectively input into two parallel Mamba network layers; Add the reflection component to the output of its corresponding Mamba network layer to obtain the reflection feature of the current hyperspectral pixel; Use the output of the Mamba network layer corresponding to the illumination component as the illumination feature of the current hyperspectral pixel.

[0009] Input the neighborhood block of the synthetic aperture radar pixel with the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch. After being processed by the sequentially connected convolution module and the dynamic update graph convolutional layer, output the target feature of the current synthetic aperture radar pixel.

[0010] Input the target feature, reflection feature, and illumination feature of the current hyperspectral pixel, as well as the target feature of the current synthetic aperture radar pixel into the splicing layer, output the fusion feature, and input it into the fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel.

[0011] Preferably, the expression of the output of the dynamic update graph convolutional layer is:

[0012] ;

[0013] Among them, is the input of the dynamic update graph convolutional layer; is the vector of the th row of the input of the dynamic update graph convolutional layer; is the vector of the th row of the input of the dynamic update graph convolutional layer; is the intermediate parameter of the dynamic update graph convolutional layer, and its expression is: ; is the vector of the th row of the intermediate parameter of the dynamic update graph convolutional layer; is the output of the dynamic update graph convolutional layer; is the vector of the th row of the output of the dynamic update graph convolutional layer; is the activation function; is the degree matrix corresponding to the time-step based adjacency matrix in the graph convolution; is the identity matrix; is the trainable matrix; is the time step; is the custom threshold; is the time-step based adjacency matrix in the graph convolution. The value of the th row and the th column of this adjacency matrix is The expression is:

[0014] ;

[0015] Among them, is the first coefficient; is the second coefficient; is the Euclidean norm; is the exponential operation; is the number of channels of the hyperspectral image; is the index variable in the graph convolutional layer.

[0016] Preferably, each convolution module includes a two-dimensional convolutional layer, a Tanh activation function layer, and a batch normalization layer.

[0017] Preferably, each Mamba network layer includes two parallel branches, and its output is the sum of the outputs of the two parallel branches; the first parallel branch includes a fully connected layer, a one-dimensional convolutional layer, a SiLU activation function layer, and a state space model connected in sequence; the second parallel branch includes a fully connected layer and a SiLU activation function layer connected in sequence.

[0018] Preferably, an initial deep neural network is constructed, including a first branch and a second branch;

[0019] The first branch includes a hyperspectral pixel processing branch, a synthetic aperture radar pixel processing branch, a splicing layer, and a fully connected layer;

[0020] The second branch includes a hyperspectral pixel processing branch and a synthetic aperture radar pixel processing branch;

[0021] Among them, the hyperspectral pixel processing branch includes a spectral distribution correction sub-branch and a cross-domain eigen decomposition sub-branch;

[0022] The initial deep neural network is trained to obtain a trained deep neural network, and the trained first branch is used as the target deep neural network for pixel ground object classification and recognition.

[0023] Preferably, the training of the initial deep neural network includes:

[0024] Obtain a training set, randomly select two hyperspectral pixels in the hyperspectral image under the current remote sensing scene from the training set, and obtain the first synthetic aperture radar pixel with the same position as the first hyperspectral pixel in its corresponding hyperspectral image and the second synthetic aperture radar pixel with the same position as the second hyperspectral pixel in its corresponding hyperspectral image from the synthetic aperture radar image under the current remote sensing scene;

[0025] Using the first branch, obtain the target feature, reflection feature, and illumination feature of the first hyperspectral pixel, the target feature of the first synthetic aperture radar pixel, and the ground object classification result corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel;

[0026] Using the second branch, obtain the target feature, reflection feature, and illumination feature of the second hyperspectral pixel, and the target feature of the second synthetic aperture radar pixel;

[0027] Based on the output feature of the first convolutional module in the cross-domain eigen decomposition sub-branch of the first branch, the reflection feature and illumination feature of the first hyperspectral pixel, the output feature of the first convolutional module in the cross-domain eigen decomposition sub-branch of the second branch, the reflection feature and illumination feature of the second hyperspectral pixel, as well as the output feature of the convolutional module in the synthetic aperture radar pixel processing sub-branch of the first branch and the output feature of the convolutional module in the synthetic aperture radar pixel processing sub-branch of the second branch, construct the eigen decomposition loss term, reflection component loss term, and illumination component loss term;

[0028] Based on the ground object classification result corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel and its corresponding ground object category label, construct the cross-entropy loss term;

[0029] Based on the eigen decomposition loss term, reflection component loss term, illumination component loss term, and cross-entropy loss term, construct the loss function of the deep neural network;

[0030] Using the training set, train the deep neural network by minimizing the loss function of the deep neural network until the preset number of iterations is reached, and obtain the trained deep neural network.

[0031] Preferably, the loss function of the deep neural network has the following expression:

[0032] ;

[0033] where is the eigen decomposition loss term; is the reflection component loss term; is the illumination component loss term; is the cross-entropy loss term; is the loss function weight.

[0034] Preferably,

[0035] the eigen decomposition loss term has the following expression:

[0036] ;

[0037] where is the output feature of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the first branch; is the reflection component of the first hyperspectral pixel; is the illumination component of the first hyperspectral pixel; is the output feature of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the second branch; is the reflection component of the second hyperspectral pixel; is the illumination component of the second hyperspectral pixel; is the number of channels of the hyperspectral image; is the index variable in the loss function; is the global average pooling operation;

[0038] The reflection component loss term has the following expression:

[0039] ;

[0040] where, is the gradient information of the reflection component, ; is the gradient information of the th channel of the reflection component, ; is the feature dimension; is the Euclidean distance;

[0041] The illumination component loss term has the following expression:

[0042] ;

[0043] where, is the gradient information of the illumination component, ; is the gradient information of the th channel of the illumination component, ; is the output feature of the convolutional module in the synthetic aperture radar pixel processing branch of the first branch; is the output feature of the convolutional module in the synthetic aperture radar pixel processing branch of the second branch;

[0044] The cross-entropy loss term has the following expression:

[0045] ;

[0046] where, is the total number of ground object categories; is the ground object category index; is the One-hot label of ground objects. If the ground object category label corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel is the category of ground objects, then , otherwise ; is the probability of the category of ground objects output by the deep neural network.

[0047] Preferably, the obtaining of the hyperspectral image and the synthetic aperture radar image with the same spatial size of the current remote sensing scene, and the construction of the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel include:

[0048] Based on the current remote sensing scene, the size of the obtained hyperspectral image is , and the size of the synthetic aperture radar image is ; where is the spatial size of the hyperspectral image and the synthetic aperture radar image; is the number of channels of the hyperspectral image; is the number of channels of the synthetic aperture radar image;

[0049] Average the channels of the synthetic aperture radar image to obtain a single-channel synthetic aperture radar image with a size of ;

[0050] Normalize the hyperspectral image and the single-channel synthetic aperture radar image respectively to obtain the preprocessed hyperspectral image and synthetic aperture radar image;

[0051] Based on a preset neighborhood size of , select the neighborhood with a size of around each pixel in the preprocessed hyperspectral image as the neighborhood block of the corresponding pixel, and the neighborhood with a size of around each pixel in the preprocessed synthetic aperture radar image as the neighborhood block of the corresponding pixel, to obtain the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel; where the size of the neighborhood block of each hyperspectral pixel is , and the size of the neighborhood block of each synthetic aperture radar pixel is .

[0052] The present invention also provides a remote sensing hyperspectral image and synthetic aperture radar image fusion classification device, including:

[0053] Neighborhood block construction module: Obtain the hyperspectral image and the synthetic aperture radar image with the same spatial size of the current remote sensing scene, and construct the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel;

[0054] Hyperspectral pixel target feature acquisition module: Input the neighborhood block of the current hyperspectral pixel into the spectral distribution correction sub-branch of the hyperspectral pixel processing branch. After being processed by the convolutional module, it is respectively input into the dynamically updated graph convolutional layer and the Mamba network layer. Add the output of the dynamically updated graph convolutional layer and the output of the Mamba network layer to obtain the target feature of the current hyperspectral pixel.

[0055] Hyperspectral pixel reflection feature and illumination feature acquisition module: Input the neighborhood block of the current hyperspectral pixel into the cross-domain eigen-decomposition sub-branch of the hyperspectral pixel processing branch. After being processed by the convolutional module, it is respectively input into two parallel convolutional modules. After obtaining the reflection component and the illumination component, they are respectively input into two parallel Mamba network layers. Add the reflection component and the output of its corresponding Mamba network layer to obtain the reflection feature of the current hyperspectral pixel. Use the output of the Mamba network layer corresponding to the illumination component as the illumination feature of the current hyperspectral pixel.

[0056] Synthetic aperture radar pixel target feature acquisition module: Input the neighborhood block of the synthetic aperture radar pixel at the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch, and after being processed by the sequentially connected convolutional module and the dynamically updated graph convolutional layer, output the target feature of the current synthetic aperture radar pixel.

[0057] Fusion classification module: Input the target feature, reflection feature and illumination feature of the current hyperspectral pixel, and the target feature of the current synthetic aperture radar pixel into the splicing layer, output the fusion feature, and input it into the fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel.

[0058] The above technical solutions of the present invention have the following beneficial effects compared with the prior art:

[0059] A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to the present invention is directed to hyperspectral images and synthetic aperture radar images in a remote sensing scenario, and corresponding hyperspectral pixel processing branches and synthetic aperture radar pixel processing branches are designed in a deep neural network respectively; a spectral distribution correction sub-branch and a cross-domain eigen-decomposition sub-branch are designed in the hyperspectral pixel processing branch, which can not only extract the overall information of hyperspectral pixels by using the spectral distribution correction sub-branch, but also extract the reflection information and illumination information obtained after the eigen-decomposition of hyperspectral pixels through the cross-domain eigen-decomposition sub-branch. At the same time, the overall information of synthetic aperture radar pixels is extracted through the synthetic aperture radar pixel processing branch, and all the information is fused to improve the correlation between the hyperspectral image information and the synthetic aperture radar information in the final fused features; secondly, graph convolutional layers that are dynamically updated with time steps are designed in the hyperspectral pixel processing branch and the synthetic aperture radar pixel processing branch, which can continuously adjust the form of the adjacency matrix, so as to effectively learn global and local information, and through the iteration of time steps, more detailed learning of highly similar features is achieved, which can significantly improve the global and local information search capabilities of large-scale remote sensing images, and thus improve the accuracy of ground object classification; a Mamba network layer based on a state space model is introduced in the hyperspectral pixel processing branch, which can compress large-scale remote sensing images into one-dimensional linear data for scanning, significantly improving the capture efficiency of key features, as well as effectively distinguishing the data differences between reflection images and illumination images, and finally fully enhancing the internal interaction between hyperspectral images and synthetic aperture radar images, thereby improving the accuracy of the ground object classification results of each pixel in the two images.

[0060] A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to the present invention, during the training process, uses the gradient information of the hyperspectral image and the synthetic aperture radar image to constrain the eigen-decomposition process, that is: based on the output features, reflection components and illumination components of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the first branch, the output features, reflection components and illumination components of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the second branch, the output features of the convolutional module in the synthetic aperture radar pixel processing branch of the first branch and the output features of the convolutional module in the synthetic aperture radar pixel processing branch of the second branch, construct an eigen-decomposition loss term, a reflection component loss term and an illumination component loss term, so as to explore the interaction between the hyperspectral image and the synthetic aperture radar and the images obtained by eigen-decomposition. At the same time, explore the internal connection between the illumination image obtained by decomposing the hyperspectral image and the synthetic aperture radar image, improve the eigen-decomposition accuracy, fully tap the potential of multi-modal remote sensing images, provide high-accuracy eigen-decomposition features for the subsequent classification process, and thus improve the accuracy of the final classification result. Description of the Drawings

[0061] To make the content of the present invention easier to be clearly understood, the following further details the present invention according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0062] Figure 1 is a schematic flowchart of a method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images provided by the present invention;

[0063] Figure 2 is a structural diagram of a deep neural network;

[0064] Figure 3 is a schematic diagram of a device for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images provided by the present invention. Specific Embodiments

[0065] The following further describes the present invention in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it, but the examples given are not intended to limit the present invention.

[0066] Refer to Figure 1 as shown, Figure 1 is a schematic flowchart of a method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images provided by the present invention; specifically including:

[0067] S1: Obtain a hyperspectral image and a synthetic aperture radar image with the same spatial size of the current remote sensing scene, and construct a neighborhood block for each hyperspectral pixel and a neighborhood block for each synthetic aperture radar pixel, including:

[0068] Based on the current remote sensing scene, the size of the obtained hyperspectral image is , and the size of the synthetic aperture radar image is ; wherein, is the spatial size of the hyperspectral image and the synthetic aperture radar image; is the number of channels of the hyperspectral image; is the number of channels of the synthetic aperture radar image; the number of ground object categories in the hyperspectral image and the synthetic aperture radar image with the same spatial size of the current remote sensing scene is the same;

[0069] Average the channels of the synthetic aperture radar image to obtain a single-channel synthetic aperture radar image, the size of which is ;

[0070] Normalize the hyperspectral image and the single-channel synthetic aperture radar image respectively, so that the value range of the image is between -1 and 1, and obtain the preprocessed hyperspectral image and synthetic aperture radar image; wherein, the expression of the normalization is:

[0071] ;

[0072] Among them, represents the normalized image; represents the image to be processed, which is a hyperspectral image or a single-channel synthetic aperture radar image; represents the maximum value of the image to be processed; represents the minimum value of the image to be processed;

[0073] Based on a preset neighborhood size of , select the neighborhood with a size of around each pixel in the preprocessed hyperspectral image as the neighborhood block corresponding to the pixel, and the neighborhood with a size of around each pixel in the preprocessed synthetic aperture radar image as the neighborhood block corresponding to the pixel, to obtain the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel; among them, the size of the neighborhood block of each hyperspectral pixel is , and the size of the neighborhood block of each synthetic aperture radar pixel is ;

[0074] S2: Input the neighborhood block of the current hyperspectral pixel into the spectral distribution correction sub-branch of the hyperspectral pixel processing branch of the target neural network: After being processed by the convolution module, input it into the dynamic update graph convolutional layer and the Mamba network layer respectively; Add the output of the dynamic update graph convolutional layer and the output of the Mamba network layer to obtain the target feature of the current hyperspectral pixel;

[0075] S3: Input the neighborhood block of the current hyperspectral pixel into the cross-domain eigen-decomposition sub-branch of the hyperspectral pixel processing branch of the target neural network: After being processed by the convolution module, input it into two parallel convolution modules respectively. After obtaining the reflection component and the illumination component, input them into two parallel Mamba network layers respectively; Add the reflection component and the output of its corresponding Mamba network layer to obtain the reflection feature of the current hyperspectral pixel; Use the output of the Mamba network layer corresponding to the illumination component as the illumination feature of the current hyperspectral pixel;

[0076] S4: Input the neighborhood block of the synthetic aperture radar pixel with the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch of the target neural network, and after being processed by the sequentially connected convolution module and the dynamic update graph convolutional layer, output the target feature of the current synthetic aperture radar pixel;

[0077] S5: Input the target feature, reflection feature, and illumination feature of the current hyperspectral pixel, and the target feature of the current synthetic aperture radar pixel into the splicing layer, output the fusion feature, and input it into the fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel;

[0078] Among them, the structures of all dynamically updated graph convolutional layers are the same, and the expression for the output of the dynamically updated graph convolutional layer is:

[0079] ;

[0080] Among them, is the input of the dynamically updated graph convolutional layer; is the vector of the th row of the input of the dynamically updated graph convolutional layer; is the vector of the th row of the input of the dynamically updated graph convolutional layer; is the intermediate parameter of the dynamically updated graph convolutional layer, and its expression is: ; is the vector of the th row of the intermediate parameter of the dynamically updated graph convolutional layer; is the output of the dynamically updated graph convolutional layer; is the vector of the th row of the output of the dynamically updated graph convolutional layer; is the activation function; is the degree matrix corresponding to the adjacency matrix based on the time step in the graph convolution; is the identity matrix; is the trainable matrix; is the time step; is the custom threshold; is the adjacency matrix based on the time step in the graph convolution. The value of the th row and th column of this adjacency matrix has the following expression:

[0081] ;

[0082] Among them, is the first coefficient; is the second coefficient; is the Euclidean norm; is the exponential operation; is the number of channels of the hyperspectral image; is the index variable in the graph convolutional layer;

[0083] The structures of all convolutional modules are the same. Each convolutional module includes a two-dimensional convolutional layer, a Tanh activation function layer, and a batch normalization layer; the convolutional kernel size of all two-dimensional convolutional layers is , the convolutional kernel sliding step is , the number of convolutional kernels is , and the padding parameters of all two-dimensional convolutional layers are the same; the expression of the Tanh activation function is:

[0084] ;

[0085] Among them, represents the input of the Tanh activation function layer; represents the output of the Tanh activation function layer;

[0086] All Mamba network layers have the same structure. Each Mamba network layer includes two parallel branches, and its output is the sum of the outputs of the two parallel branches; the first parallel branch includes a fully connected layer, a one-dimensional convolutional layer, a SiLU activation function layer, and a state space model connected in sequence; the second parallel branch includes a fully connected layer and a SiLU activation function layer connected in sequence; among them, if the input feature needs to be stretched, after the input Mamba network layer, the input feature is first stretched into a one-dimensional vector and then simultaneously and separately transmitted to the first parallel branch and the second parallel branch. If stretching is not required, the input feature is directly simultaneously and separately transmitted to the first parallel branch and the second parallel branch; finally, the shape of the output of the Mamba network layer is transformed so that its shape is the same as the input shape of this layer.

[0087] The process of obtaining the target neural network includes:

[0088] Construct an initial deep neural network as Figure 2 shown, including a first branch and a second branch;

[0089] The first branch includes a hyperspectral pixel processing branch, a synthetic aperture radar pixel processing branch, a splicing layer, and a fully connected layer;

[0090] The second branch includes a hyperspectral pixel processing branch and a synthetic aperture radar pixel processing branch;

[0091] Among them, the hyperspectral pixel processing branch includes a spectral distribution correction sub-branch and a cross-domain eigen decomposition sub-branch;

[0092] Train the initial deep neural network to obtain a trained deep neural network, and use the trained first branch as the target deep neural network for pixel ground object classification and recognition.

[0093] The training of the initial deep neural network includes:

[0094] Step 1: Obtain hyperspectral images and synthetic aperture radar images with the same spatial dimensions for different remote sensing scenes, construct neighborhood blocks for each hyperspectral pixel and each synthetic aperture radar pixel in different remote sensing scenes, and combine the ground object category labels corresponding to each hyperspectral pixel and the synthetic aperture radar pixel with the same position in different remote sensing scenes to construct a dataset, and divide the dataset into a training set and a test set; among them, the total number of ground object categories in the hyperspectral image and the synthetic aperture radar image in the same remote sensing scene is the same;

[0095] Step 2: Randomly select two hyperspectral pixels in the hyperspectral image of the current remote sensing scene from the training set, and use them as the first hyperspectral pixel and the second hyperspectral pixel respectively, and obtain, from the synthetic aperture radar image of the current remote sensing scene, the first synthetic aperture radar pixel with the same position as the first hyperspectral pixel in its corresponding hyperspectral image, and the second synthetic aperture radar pixel with the same position as the second hyperspectral pixel in its corresponding hyperspectral image; among them, the label of the first hyperspectral pixel is , and the label of the second hyperspectral pixel is ;

[0096] Step 3: Use the first branch to obtain the target feature, reflection feature and illumination feature of the first hyperspectral pixel, the target feature of the first synthetic aperture radar pixel, and the ground object classification result corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel, that is: input the neighborhood block of the first hyperspectral pixel into the hyperspectral pixel processing branch of the first branch, and output the target feature, reflection feature and illumination feature of the first hyperspectral pixel, which are respectively represented as , and ; input the neighborhood block of the first synthetic aperture radar pixel into the synthetic aperture radar pixel processing branch of the first branch, and output the target feature of the first synthetic aperture radar pixel, which is represented as ;

[0097] Use the second branch to obtain the target feature, reflection feature and illumination feature of the second hyperspectral pixel, and the target feature of the second synthetic aperture radar pixel, that is: input the neighborhood block of the second hyperspectral pixel into the hyperspectral pixel processing branch of the second branch, and output the target feature, reflection feature and illumination feature of the second hyperspectral pixel, which are respectively represented as , and ; input the neighborhood block of the second synthetic aperture radar pixel into the synthetic aperture radar pixel processing branch of the second branch, and output the target feature of the second synthetic aperture radar pixel, which is represented as ;

[0098] Among them, in the actual process, before the output features of the first convolutional module in the cross-domain eigen-decomposition sub-branches of each branch are respectively input into the dynamically updated graph convolutional layer and the Mamba network layer in the spectral distribution correction sub-branch, it is also necessary to change the shape of the output features of the first convolutional module in the cross-domain eigen-decomposition sub-branches of each branch. Change its shape from to ; before obtaining the target features of the hyperspectral pixel, it is also necessary to change the shape of the sum of the outputs of the dynamically updated graph convolutional layer and the Mamba network layer in the spectral distribution correction sub-branch. Change the shape of the output features at this time from to , that is, the shape of the target features of the hyperspectral pixel is ;

[0099] Before the output features of the convolutional module in the synthetic aperture radar pixel processing branch of each branch are input into the dynamically updated graph convolutional layer in the synthetic aperture radar pixel processing branch of each branch, it is also necessary to change the shape of the output features of the convolutional module in the synthetic aperture radar pixel processing branch of each branch. Change the shape of the initial features of the synthetic aperture radar pixel from to ;

[0100] Step 4: Based on the output features of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the first branch, the reflection features and illumination features of the first hyperspectral pixel, the output features of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the second branch, the reflection features and illumination features of the second hyperspectral pixel, and the output features of the convolutional module in the synthetic aperture radar pixel processing branch of the first branch and the output features of the convolutional module in the synthetic aperture radar pixel processing branch of the second branch, construct an eigen-decomposition loss term, a reflection component loss term, and an illumination component loss term;

[0101] Among them, the eigen-decomposition loss term The expression is:

[0102] ;

[0103] Among them, is the output feature of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the first branch; is the reflection component of the first hyperspectral pixel; is the illumination component of the first hyperspectral pixel; is the output feature of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the second branch; is the reflection component of the second hyperspectral pixel; is the illumination component of the second hyperspectral pixel; is the number of channels of the hyperspectral image; is the index variable in the loss function; is the global average pooling operation;

[0104] The reflection component loss term has the following expression:

[0105] ;

[0106] where, is the gradient information of the reflection component, ; is the gradient information of the th channel of the reflection component, ; is the feature dimension; is the Euclidean distance; represents the Sigmoid activation function, has the following expression:

[0107] ;

[0108] The illumination component loss term has the following expression:

[0109] ;

[0110] where, is the gradient information of the illumination component, ; is the gradient information of the th channel of the illumination component, ; is the output feature of the convolutional module in the synthetic aperture radar pixel processing branch of the first branch; is the output feature of the convolutional module in the synthetic aperture radar pixel processing branch of the second branch;

[0111] Based on the ground object classification result corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel and its corresponding ground object category label, a cross-entropy loss term is constructed, and its expression is:

[0112] ;

[0113] where, is the total number of ground object categories; is the ground object category index; is the one-hot label of the th ground object. If the ground object category label corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel is the th ground object, then , otherwise ; The probability of the th type of ground object output by the deep neural network;

[0114] Based on the eigen - decomposition loss term, reflection component loss term, illumination component loss term, and cross - entropy loss term, construct the loss function of the deep neural network, and its expression is:

[0115] ;

[0116] where, is the eigen - decomposition loss term; is the reflection component loss term; is the illumination component loss term; is the cross - entropy loss term; is the loss function weight;

[0117] Step 5: Use the training set to train the deep neural network by minimizing the loss function of the deep neural network until the preset number of iterations is reached, and obtain the trained deep neural network. Use the trained first branch as the target deep neural network for pixel ground object classification and recognition; among them, during the training process, select the Adam optimizer with a step size of , set the number of iterations to 1000 generations, and after the model reaches stability, input all pixels into the deep network model for testing to obtain the final classification result.

[0118] In order to verify the classification accuracy of a remote - sensing hyperspectral image and synthetic aperture radar image fusion classification method provided by the present invention, the hyperspectral image and synthetic aperture radar image used are taken from a part of Houston. Among them, the size of the hyperspectral image is , and by averaging different channels of the synthetic aperture radar, the size of the synthetic aperture radar image is . Therefore, the hyperspectral image obtained in this embodiment is an image with a size of , and the input synthetic aperture radar image is an image with a size of ;

[0119] Set the neighborhood size to 11, perform neighborhood window selection, and for each pixel, obtain neighborhood blocks with sizes of and , and input the neighborhood blocks of each pixel into the initial deep network for training;

[0120] Randomly select 1% of the sample neighborhood blocks from a total of 358,196 groups of sample neighborhood blocks for training the initial deep network model, randomly sort and pack these sample neighborhood blocks, the number of small - batch sample neighborhood blocks is 256, and only use one sample package for each training. After training, obtain the trained deep neural network;

[0121] Using all 358,196 sample neighborhood blocks, the trained deep neural network is tested, and finally all the predicted labels are obtained;

[0122] In summary, the overall classification accuracy and the average classification accuracy are selected as the data analysis indicators for the experiment; the overall classification accuracy is the result obtained by dividing the correctly classified samples by the total number of samples; the average classification accuracy is the result obtained by dividing the number of correctly classified samples in each class by the total number of classes and then averaging.

[0123] The classification results obtained by using the hyperspectral image and synthetic aperture radar image fusion classification method and device in the remote sensing field proposed by the present invention and the currently commonly used MVHN network model are shown in Table 1 below.

[0124] Table 1 Comparison table of classification accuracies

[0125]

[0126] It can be seen that the method of the present invention can better fuse and classify hyperspectral images and synthetic aperture radar images, with fewer misclassified samples; in summary, the method of the present invention can effectively improve the classification accuracy in the field of remote sensing multimodal fusion.

[0127] The advantages of a hyperspectral image and synthetic aperture radar image fusion classification method provided by the present invention are as follows:

[0128] (1) By establishing various cross-domain constraints, the present invention decomposes the hyperspectral image into a reflection image and an illumination image with the help of a neural network, and deeply explores the internal relationship between the illumination image and the synthetic aperture radar image; on the basis of respectively connecting the hyperspectral information with the reflection image and the synthetic aperture radar image with the illumination image, the classification accuracy of ground object information is significantly improved.

[0129] (2) The present invention uses the gradient information of the hyperspectral image and the synthetic aperture radar image to constrain the eigen-decomposition module, so as to explore the interaction between the hyperspectral image and the synthetic aperture radar and the images obtained by eigen-decomposition; practice shows that this method is very effective in improving the eigen-decomposition accuracy of the two.

[0130] (3) The present invention designs a mechanism for decomposing hyperspectral information using elevation information, and uses a graph convolutional network and a Mamba network to extract and analyze the features of objects with different materials, scales, and elevations, fully showing the spectral components and spatial components in the hyperspectral image, which helps to unify the spectral curves with different illumination attributes and obtain more accurate spatial-spectral features.

[0131] (4) The present invention proposes a graph convolutional network that dynamically updates with each time step. This network can continuously adjust the form of the adjacency matrix, thereby effectively learning global-local information. For highly similar features, more detailed learning can be achieved through the iteration of time steps. Combining the dynamic graph convolutional network with other methods can significantly improve the search ability for global-local information in large-scale remote sensing images, and thus improve the accuracy of ground object classification.

[0132] (5) The present invention proposes a Mamba network based on the State Space Model (SSM). This network compresses large-scale remote sensing images into one-dimensional linear data for scanning, significantly improving the capture efficiency of key features. Through this Mamba network, the present invention can effectively distinguish the data differences between reflected images and illumination images, and finally fully reflect the internal interaction between hyperspectral images and synthetic aperture radar images.

[0133] Refer to Figure 3 as shown in Figure 3 is a schematic diagram of a fusion classification device for remote sensing hyperspectral images and synthetic aperture radar images provided by the present invention; specifically including:

[0134] Neighborhood block construction module 100: Obtain hyperspectral images and synthetic aperture radar images with the same spatial size of the current remote sensing scene, and construct neighborhood blocks for each hyperspectral pixel and each synthetic aperture radar pixel;

[0135] Hyperspectral pixel target feature acquisition module 200: Input the neighborhood block of the current hyperspectral pixel into the spectral distribution correction sub-branch of the hyperspectral pixel processing branch: After being processed by the convolution module, input it into the dynamically updated graph convolutional layer and the Mamba network layer respectively; Add the output of the dynamically updated graph convolutional layer and the output of the Mamba network layer to obtain the target feature of the current hyperspectral pixel;

[0136] Hyperspectral pixel reflection feature and illumination feature acquisition module 300: Input the neighborhood block of the current hyperspectral pixel into the cross-domain eigen decomposition sub-branch of the hyperspectral pixel processing branch: After being processed by the convolution module, input it into two parallel convolution modules respectively. After obtaining the reflection component and the illumination component, input them into two parallel Mamba network layers respectively; Add the reflection component to the output of its corresponding Mamba network layer to obtain the reflection feature of the current hyperspectral pixel; Use the output of the Mamba network layer corresponding to the illumination component as the illumination feature of the current hyperspectral pixel;

[0137] Synthetic aperture radar pixel target feature acquisition module 400: Input the neighborhood block of the synthetic aperture radar pixel with the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch, and after being processed by the sequentially connected convolution module and the dynamically updated graph convolutional layer, output the target feature of the current synthetic aperture radar pixel;

[0138] Fusion classification module 500: Input the target feature, reflection feature, and illumination feature of the current hyperspectral pixel, as well as the target feature of the current synthetic aperture radar pixel, into the splicing layer, output the fusion feature, and input it into the fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel.

[0139] The device of this embodiment is used to implement the aforementioned remote sensing hyperspectral image and synthetic aperture radar image fusion classification method. Therefore, the specific implementation manners in the remote sensing hyperspectral image and synthetic aperture radar image fusion classification device can be seen in the embodiment part of the remote sensing hyperspectral image and synthetic aperture radar image fusion classification method in the previous text. For example, the neighborhood block construction module 100, the hyperspectral pixel target feature acquisition module 200, the hyperspectral pixel reflection feature and illumination feature acquisition module 300, the synthetic aperture radar pixel target feature acquisition module 400, and the fusion classification module 500 are respectively used to implement S1 to S5 in the above-mentioned remote sensing hyperspectral image and synthetic aperture radar image fusion classification method. Therefore, its specific implementation manners can refer to the descriptions of the corresponding individual part embodiments and will not be elaborated here.

[0140] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0142] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions in the process Figure 1One or more processes and / or blocks Figure 1 The functions specified in one or more blocks.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 One or more processes and / or blocks Figure 1 The steps of the functions specified in one or more blocks.

[0144] Obviously, the above embodiments are only examples given for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. And the obvious changes or variations derived therefrom still fall within the protection scope of the present invention.

Claims

1. A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images, characterized in that, Including: Obtain a hyperspectral image and a synthetic aperture radar image with the same spatial size as the current remote sensing scene, construct a neighborhood block for each hyperspectral pixel, and a neighborhood block for each synthetic aperture radar pixel; Input the neighborhood block of the current hyperspectral pixel into the spectral distribution correction sub-branch of the hyperspectral pixel processing branch: after being processed by the convolutional module, input it into the dynamic update graph convolutional layer and the Mamba network layer respectively; add the output of the dynamic update graph convolutional layer and the output of the Mamba network layer to obtain the target feature of the current hyperspectral pixel; The expression of the output of the dynamic update graph convolutional layer is: ； Among them, and are the vectors of the th row and the th row of the input of the dynamically updated graph convolutional layer, respectively; is the vector of the th row of the intermediate parameter of the dynamically updated graph convolutional layer; ; is the vector of the th row of the output of the dynamically updated graph convolutional layer; is the identity matrix; is a trainable matrix; is the time step; is a custom threshold; is the adjacency matrix based on the time step in graph convolution. The value of the th row and the th column in this adjacency matrix is as follows: ； is the first coefficient; is the second coefficient; is the Euclidean norm; is the exponential operation; is the number of channels of the hyperspectral image; is the index variable in the graph convolutional layer; Each Mamba network layer includes two parallel branches, and its output is the sum of the outputs of the two parallel branches; the first parallel branch includes a fully connected layer, a one-dimensional convolutional layer, a SiLU activation function layer, and a state space model connected in sequence; the second parallel branch includes a fully connected layer and a SiLU activation function layer connected in sequence; Input the neighborhood block of the current hyperspectral pixel into the cross-domain eigen decomposition sub-branch of the hyperspectral pixel processing branch: after being processed by the convolutional module, input it into two parallel convolutional modules respectively, and after obtaining the reflection component and the illumination component, input them into two parallel Mamba network layers respectively; add the reflection component and the output of its corresponding Mamba network layer to obtain the reflection feature of the current hyperspectral pixel; use the output of the Mamba network layer corresponding to the illumination component as the illumination feature of the current hyperspectral pixel; Input the neighborhood block of the synthetic aperture radar pixel with the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch, and after being processed by the convolutional module and the dynamic update graph convolutional layer connected in sequence, output the target feature of the current synthetic aperture radar pixel; Input the target feature, reflection feature, and illumination feature of the current hyperspectral pixel, and the target feature of the current synthetic aperture radar pixel into the splicing layer, output the fusion feature, and input it into the fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel.

2. A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to claim 1, characterized in that, Each convolutional module includes a two-dimensional convolutional layer, a Tanh activation function layer, and a batch normalization layer.

3. A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to claim 1, characterized in that, Construct an initial deep neural network, including a first branch and a second branch; The first branch includes a hyperspectral pixel processing branch, a synthetic aperture radar pixel processing branch, a splicing layer, and a fully connected layer; The second branch includes a hyperspectral pixel processing branch and a synthetic aperture radar pixel processing branch; Among them, the hyperspectral pixel processing branch includes a spectral distribution correction sub-branch and a cross-domain eigen decomposition sub-branch; Train the initial deep neural network to obtain a trained deep neural network, and use the trained first branch as the target deep neural network for pixel ground object classification and recognition.

4. A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to claim 3, characterized in that, The training of the initial deep neural network includes: Obtain a training set, randomly select two hyperspectral pixels in the hyperspectral image under the current remote sensing scene from the training set, and obtain the first synthetic aperture radar pixel with the same position as the first hyperspectral pixel in its corresponding hyperspectral image, and the second synthetic aperture radar pixel with the same position as the second hyperspectral pixel in its corresponding hyperspectral image from the synthetic aperture radar image under the current remote sensing scene; Using the first branch, obtain the target feature, reflection feature, and illumination feature of the first hyperspectral pixel, the target feature of the first synthetic aperture radar pixel, and the ground object classification result corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel; Using the second branch, obtain the target feature, reflection feature, and illumination feature of the second hyperspectral pixel, and the target feature of the second synthetic aperture radar pixel; Based on the output feature of the first convolutional module in the cross-domain eigen decomposition sub-branch of the first branch, the reflection feature and illumination feature of the first hyperspectral pixel, the output feature of the first convolutional module in the cross-domain eigen decomposition sub-branch of the second branch, the reflection feature and illumination feature of the second hyperspectral pixel, as well as the output feature of the convolutional module in the synthetic aperture radar pixel processing branch of the first branch and the output feature of the convolutional module in the synthetic aperture radar pixel processing branch of the second branch, construct the eigen decomposition loss term, reflection component loss term, and illumination component loss term; Based on the ground object classification result corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel and its corresponding ground object category label, construct the cross-entropy loss term; Based on the eigen decomposition loss term, reflection component loss term, illumination component loss term, and cross-entropy loss term, construct the loss function of the deep neural network; Using the training set, train the deep neural network by minimizing the loss function of the deep neural network until the preset number of iterations is reached, and obtain the trained deep neural network.

5. A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to claim 4, characterized in that The loss function of the deep neural network has the following expression: ； Among them, is the eigenvalue decomposition loss term; is the reflection component loss term; is the illumination component loss term; is the cross-entropy loss term; is the loss function weight.

6. The method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to claim 5, characterized in that The eigen-decomposition loss term has the following expression: ； Among them, is the output feature of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the first branch; is the reflection component of the first hyperspectral pixel; is the illumination component of the first hyperspectral pixel; is the output feature of the first convolutional module in the cross-domain eigen-decomposition sub-branch of the second branch; is the reflection component of the second hyperspectral pixel; is the illumination component of the second hyperspectral pixel; is the number of channels of the hyperspectral image; is the index variable in the loss function; is the global average pooling operation; The reflection component loss term has the following expression: ； Among them, is the gradient information of the reflection component, ; is the gradient information of the th channel of the reflection component, ; is the feature dimension; is the Euclidean distance; The light component loss term has the following expression: ； Among them, is the gradient information of the illumination component, ; is the gradient information of the -th channel of the illumination component, ; is the output feature of the convolution module in the synthetic aperture radar pixel processing branch of the first branch; is the output feature of the convolution module in the synthetic aperture radar pixel processing branch of the second branch; The cross-entropy loss term has the following expression: ； Among them, is the total number of feature classes; is the feature class index; is the one-hot label of the -th feature class. If the feature class label corresponding to the first hyperspectral pixel and the first synthetic aperture radar pixel is the -th feature class, then , otherwise ; is the probability of the -th feature class output by the deep neural network.

7. A method for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images according to claim 1, characterized in that, The obtaining of the hyperspectral image and the synthetic aperture radar image with the same spatial size of the current remote sensing scene and the construction of the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel include: Based on the current remote sensing scene, the size of the hyperspectral image obtained is , and the size of the synthetic aperture radar image is ; where is the spatial size of the hyperspectral image and the synthetic aperture radar image; is the number of channels of the hyperspectral image; is the number of channels of the synthetic aperture radar image; Average the channels of the synthetic aperture radar image to obtain a single-channel synthetic aperture radar image with a size of ; Normalize the hyperspectral image and the single-channel synthetic aperture radar image respectively to obtain the preprocessed hyperspectral image and synthetic aperture radar image; Based on a preset neighborhood size of , select the neighborhood with a size of around each pixel in the preprocessed hyperspectral image as the neighborhood block corresponding to the pixel, and the neighborhood with a size of around each pixel in the preprocessed synthetic aperture radar image as the neighborhood block corresponding to the pixel, to obtain the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel; where the size of the neighborhood block of each hyperspectral pixel is , and the size of the neighborhood block of each synthetic aperture radar pixel is .

8. A device for fusing and classifying remote sensing hyperspectral images and synthetic aperture radar images, characterized in that, including: Neighborhood block construction module: Obtain the hyperspectral image and the synthetic aperture radar image with the same spatial size of the current remote sensing scene, and construct the neighborhood block of each hyperspectral pixel and the neighborhood block of each synthetic aperture radar pixel; Hyperspectral pixel target feature acquisition module: Input the neighborhood block of the current hyperspectral pixel into the spectral distribution correction sub-branch of the hyperspectral pixel processing branch: After being processed by the convolutional module, input them into the dynamic update graph convolutional layer and the Mamba network layer respectively; Add the output of the dynamic update graph convolutional layer and the output of the Mamba network layer to obtain the target feature of the current hyperspectral pixel; The expression of the output of the dynamic update graph convolutional layer is: ； Among them, and are respectively the vectors of the th row and the th row of the input of the dynamically updated graph convolutional layer; is the vector of the th row of the intermediate parameter of the dynamically updated graph convolutional layer; ; is the vector of the th row of the output of the dynamically updated graph convolutional layer; is the activation function; is the degree matrix corresponding to the time-step based adjacency matrix in graph convolution; is the identity matrix; is the time step; is the custom threshold; is the time-step based adjacency matrix in graph convolution. The value of the th column in the th row of this adjacency matrix is: ； is the first coefficient; is the second coefficient; is the Euclidean norm; is the exponential operation; is the number of channels of the hyperspectral image; is the index variable in the graph convolutional layer; Each Mamba network layer includes two parallel branches, and its output is the sum of the outputs of the two parallel branches; the first parallel branch includes a fully connected layer, a one-dimensional convolutional layer, a SiLU activation function layer, and a state space model connected in sequence; the second parallel branch includes a fully connected layer and a SiLU activation function layer connected in sequence; Hyperspectral pixel reflection feature and illumination feature acquisition module: Input the neighborhood block of the current hyperspectral pixel into the cross-domain eigen-decomposition sub-branch of the hyperspectral pixel processing branch: After being processed by the convolutional module, input them into two parallel convolutional modules respectively. After obtaining the reflection component and the illumination component, input them into two parallel Mamba network layers respectively; Add the reflection component to the output of its corresponding Mamba network layer to obtain the reflection feature of the current hyperspectral pixel; Use the output of the illumination component corresponding Mamba network layer as the illumination feature of the current hyperspectral pixel; Synthetic aperture radar pixel target feature acquisition module: Input the neighborhood block of the synthetic aperture radar pixel at the same position as the current hyperspectral pixel into the synthetic aperture radar pixel processing branch, and output the target feature of the current synthetic aperture radar pixel after being processed by the convolutional module and the dynamic update graph convolutional layer connected in sequence; Fusion classification module: Input the target feature, reflection feature and illumination feature of the current hyperspectral pixel, and the target feature of the current synthetic aperture radar pixel into the splicing layer, output the fusion feature, and input it into the fully connected layer to obtain the ground object classification result corresponding to the current hyperspectral pixel and the current synthetic aperture radar pixel.

Images