A Multimodal Remote Sensing Classification Method and Device Based on Spatial Covariance

Abstract

This invention discloses a multimodal remote sensing classification method and apparatus based on spatial covariance, belonging to the field of remote sensing image processing technology. The method includes: acquiring remote sensing images of multiple modalities of the same area and performing image registration on the acquired images; extracting the spatial covariance of each modal remote sensing image to quantitatively characterize the spatial correlation features of ground objects in the image; constructing a channel attention mechanism for remote sensing images with rich spectral information; extracting multimodal features using hierarchical encoding for each modal remote sensing image; extracting covariance features, fusing them separately, and finally fusing the fused multimodal features with the spatial covariance features; performing convolution operations on the fused features to reduce the number of channels and achieve classification output; employing an optimizer with weighted decay, using the cross-entropy loss function as a classification error metric, and adding a regularization term to control model complexity. This invention improves the accuracy of ground object classification in complex scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, in particular to a multi-modal remote sensing classification method and device based on spatial covariance. BACKGROUND

[0002] In recent years, remote sensing data has been growing at an "exploding" rate: high-resolution optical, hyperspectral, thermal infrared, SAR, night light, and other multi-modal images are available simultaneously, providing unprecedented information sources for fine feature classification. Methodologically, classification techniques have evolved from early shallow models such as support vector machines and random forests to deep learning frameworks such as convolutional neural networks and Transformers, significantly improving overall accuracy. However, existing methods generally treat multi-modal data as "stacked channels" input into the network, lacking explicit and interpretable modeling means for the strong and complex spatial dependencies between "pixel-pixel", "pixel-neighborhood", and "pixel-region", resulting in insufficient utilization of spatial structure priors by the model, blurred classification boundaries, high error rates for small targets and confused classes; the spatial correlations learned by the network are difficult for humans to understand, resulting in poor interpretability; simply increasing the number of parameters yields diminishing returns in terms of accuracy gains, making it difficult to meet the needs of fine and business-oriented Earth observation.

[0003] Therefore, there is an urgent need to design an interpretable and scalable multi-modal spatial correlation modeling mechanism to significantly improve the accuracy of remote sensing feature classification in the context of data explosion. SUMMARY

[0004] To solve the above technical problems, the present application provides a multi-modal remote sensing classification method and device based on spatial covariance, which improves the interpretability of spatial features in deep learning models through spatial covariance modeling, effectively extracts complementary features between multi-modal remote sensing data, and improves the accuracy of feature classification in complex scenes.

[0005] To achieve the above purpose, the present application adopts the following technical solution:

[0006] A multi-modal remote sensing classification method based on spatial covariance, comprising the following steps:

[0007] S1, obtaining multiple modal remote sensing images of the same area and performing image registration to make each image correspond one-to-one in space;

[0008] S2, extracting spatial covariance for each modal image after registration to quantitatively represent the spatial correlation of features;

[0009] S3, constructing a channel attention mechanism for modal with rich spectral information to adaptively enhance key spectral features;

[0010] S4, each modality image adopts a layered encoding mode, and the multi-modal features of each layer are extracted;

[0011] S5, the spatial covariance of each modality is input into a single-layer network to extract the corresponding covariance features;

[0012] S6, the multi-modal features of each modality are fused, the covariance features of each modality are fused, and the multi-modal fusion features and the covariance fusion features are finally fused to form comprehensive features;

[0013] S7, the comprehensive features are convolved and reduced in dimension, and the classification results corresponding to the number of categories are output;

[0014] S8, the optimizer with weight attenuation and cross-entropy loss are used for training, and the model complexity is controlled by a regular term.

[0015] The application also provides a multi-modal remote sensing classification device based on spatial covariance, which is used to realize the above method, and includes the following modules:

[0016] The registration module acquires remote sensing images of multiple modalities in the same area and performs image registration, so that each image is one-to-one corresponding in space;

[0017] The quantification module extracts spatial covariance of each modality image after registration to quantitatively represent the spatial correlation of the ground object;

[0018] The enhancement module constructs a channel attention mechanism for the modality with rich spectral information to adaptively enhance the key spectral features;

[0019] The multi-modal feature extraction module adopts a layered encoding mode to extract the multi-modal features of each layer of each modality image;

[0020] The covariance feature extraction module inputs the spatial covariance of each modality into a single-layer network to extract the corresponding covariance features;

[0021] The fusion module fuses the multi-modal features of each modality, fuses the covariance features of each modality, and finally fuses the multi-modal fusion features and the covariance fusion features to form comprehensive features;

[0022] The output module convolves and reduces the comprehensive features in dimension, and outputs the classification results corresponding to the number of categories;

[0023] The training module uses an optimizer with weight attenuation and cross-entropy loss for training, and controls the model complexity through a regular term.

[0024] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-modal remote sensing classification method based on spatial covariance when executing the program.

[0025] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the multi-modal remote sensing classification method based on spatial covariance.

[0026] Advantages:

[0027] (1) Traditional deep learning methods rely on convolution operations to extract spatial features, but lack explicit modeling of spatial correlation. The application quantifies the spatial dependence of features by constructing spatial covariance, enabling the model to more intuitively express the feature correlation of different regions and enhancing the model's interpretability.

[0028] (2) For multi-modal remote sensing images, the application uses a hierarchical encoding method to extract unique features of each modality, and combines a channel attention mechanism to adaptively enhance key spectral information, effectively suppressing redundant features and improving the fusion quality of multi-modal data, enabling the model to maintain high classification accuracy in complex scenarios.

[0029] (3) By fusing multi-modal features and spatial covariance features, the application comprehensively utilizes spectral, texture, and spatial structure information, significantly improving the classification accuracy of complex features such as mixed pixels and shadow regions. At the same time, using an optimizer with weighted decay and a regularization strategy effectively prevents overfitting and ensures stable performance of the model in different regions and datasets.

[0030] (4) The application uses a single-layer network structure to extract covariance features, reducing computational complexity while ensuring classification accuracy, making it suitable for remote sensing images of different resolutions. This method can be widely applied to land use classification, target detection, and other practical scenarios, providing reliable technical support for high-precision remote sensing image analysis. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of a multi-modal remote sensing classification method based on spatial covariance according to the application;

[0032] Figure 2 A network architecture diagram of a hierarchical encoding method for extracting multi-modal features according to an embodiment of the application;

[0033] Figure 3 A schematic diagram of a multi-modal remote sensing classification device based on spatial covariance according to the application. DETAILED DESCRIPTION

[0034] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0035] As shown in Figure 1 The present application provides a multi-modal remote sensing classification method based on spatial covariance, comprising the following steps:

[0036] S1, acquiring multiple modal remote sensing images of the same area, and performing image registration on the acquired remote sensing images;

[0037] S2, extracting spatial covariance of each modal remote sensing image to quantitatively represent the spatial correlation feature of the ground object in the remote sensing image;

[0038] S3, for the remote sensing image with rich spectral information, a channel attention mechanism is constructed to learn the importance of each channel, enhance the key features and suppress the redundant information;

[0039] S4, for each modal remote sensing image, a hierarchical encoding method is used to extract multi-modal features;

[0040] S5, the spatial covariance of each modal remote sensing image is extracted by using a single-layer network structure;

[0041] S6, the multi-modal features and spatial covariance are fused respectively, and finally the fused multi-modal features and spatial covariance features are fused;

[0042] S7, the fused features are subjected to convolution operation to reduce the number of channels, and classification output is realized;

[0043] S8, an Adam optimizer with weight decay is used, a cross-entropy loss function is used as a classification error measure, and a regularization term is added to control the model complexity.

[0044] Embodiment:

[0045] The embodiment of the present application provides a multi-modal remote sensing classification method based on spatial covariance. The method improves the explainability of spatial features in a deep learning model through spatial covariance modeling, effectively mines the complementary features between multi-modal remote sensing data, and improves the ground object classification accuracy in complex scenes. The specific implementation of the method comprises the following steps:

[0046] S1, obtain three kinds of modal remote sensing images in Beijing area, select the same date 10 meter resolution multispectral, 30 meter resolution thermal infrared and low light remote sensing images obtained by sustainable development science satellite 1 (SDGSAT-1) satellite, wherein the low light image is subjected to image fusion processing of 10 meter panchromatic band and 40 meter color band, to obtain 10 meter resolution color low light image, the resolution of the thermal infrared image is resampled to 10 meters, and then the three kinds of modal remote sensing images are subjected to image registration.

[0047] S2, extract the spatial covariance of the three kinds of modal remote sensing images to quantitatively represent the spatial correlation characteristics of the features in the remote sensing images.

[0048] S3, since the multispectral remote sensing image has rich spectral information, a channel attention mechanism is constructed for the multispectral image to learn the importance of each channel, enhance the key features and suppress the redundant information.

[0049] S4, for the three kinds of modal remote sensing images, a hierarchical coding method is used to extract the features of each modal.

[0050] S5, the spatial covariance of the three kinds of modal remote sensing images is extracted by using a single layer network structure.

[0051] S6, the multi-modal features and the spatial covariance features are fused respectively, and finally the fused multi-modal features and the spatial covariance features are fused.

[0052] S7, the fused features are subjected to convolution operation to reduce the number of channels and realize classification output.

[0053] S8, an optimizer with weight decay is used, a cross entropy loss function is used as a classification error measure, and a regularization term is added to control the complexity of the model.

[0054] Specifically, the S2 comprises:

[0055] S21, slice the input three kinds of modal remote sensing images to obtain image slices , wherein is the number of channels of the modal remote sensing image, and the number of channels of the multispectral, thermal infrared and low light remote sensing images is 7, 3 and 3 respectively, is the side length of the image slice, the side length of the neighborhood window is equal to the side length of the image slice, and is set to , the coordinate of the center position of the neighborhood window is , wherein represents the floor function, so the center coordinate is ;

[0056] S22, set the spatial covariance of the image slice as , whose shape is , where is the lag order of spatial covariance, represents the count of each lag order, is the number of channels of the modal remote sensing image, represents the count of channels, defined as , represents the pixel value at the center position of the neighborhood window, i.e. the pixel value at lag order 0 ;

[0057] S23, define the set of all pixel coordinates in the window that have a Manhattan distance of lag order from the center position, i.e.

[0058] ;

[0059] where and represent the coordinate points of any pixel in the window, for each lag order (from 1 to , i.e. 1 to 16) of channel , the weighted average of the remote sensing image pixel values in the neighborhood window that have a distance of from the center position is calculated as the covariance value at lag order :

[0060] ;

[0061] where is the weight at the th lag order, is the matrix multiplication, which is an array related to the lag order , denoted as , and the weight array is calculated as follows:

[0062] Generate an array from 0 to and divide it by :

[0063] ;

[0064] Reverse the array:

[0065] ;

[0066] Modify the last element:

[0067] ;

[0068] That is,

[0069] ;

[0070] Therefore, for the lag order (from 0 to , i.e. 0 to 15), the weight is:

[0071] ;

[0072] S24, for each lag order , the spatial covariance is averaged along the channel dimension to obtain the average covariance :

[0073] ;

[0074] The final spatial covariance can describe the spatial correlation of the ground objects contained in the image slice.

[0075] Specifically, the S3 includes:

[0076] S31, for a multi-spectral remote sensing image with rich spectral information, the image slice size is , and a dual-path network structure of average pooling and maximum pooling is used:

[0077] ;

[0078] wherein, is a channel attention operation, is a Sigmoid activation function, is a linear rectifier unit activation function, is an average pooling that extracts the global statistical information of the image slice ; is a maximum pooling that emphasizes the strongest response of the image slice , and the combination of the two can capture different types of channel features; and are two shared fully connected layer weight matrices, respectively.

[0079] The first fully connected layer uses a ReLU activation function to compress the channel number to times of the original channel number , wherein is a channel compression ratio, and is set to , i.e. the original channel is compressed from 7 to 2:

[0080] ;

[0081] wherein, The first shared-weight fully connected layer, represents a real number set;

[0082] The second fully connected layer recovers the original channel number using a Sigmoid activation function:

[0083] ;

[0084] wherein, The second shared-weight fully connected layer;

[0085] S32, the output of the two paths is added and multiplied with the original feature map channel by channel to enhance the significant channels:

[0086] ;

[0087] wherein, is a multispectral remote sensing image slice after channel attention, is a channel-by-channel multiplication, and the channel weighting operation can highlight the channel features that contribute to classification.

[0088] Specifically, the S4 includes:

[0089] As Figure 2 shown in the network architecture diagram for extracting multi-modal features in a hierarchical coding manner, it is a six-layer convolutional neural network: from the input, the first layer first extracts features using a 3x3 convolution kernel, then performs batch normalization, and then uses ReLU activation; the second layer uses a 3x3 convolution kernel, batch normalization, and ReLU, followed by the first maximum pooling dimension reduction; then the third, fourth, and fifth layers repeat the same convolution-batch normalization-ReLU-maximum pooling dimension reduction process; the sixth layer continues 3x3 convolution, batch normalization, and ReLU, and finally outputs a global feature vector after average pooling, forming a repeated pattern of "convolution-batch normalization-ReLU-pooling".

[0090] S41, for the image slices of thermal infrared and low-light remote sensing , and the multispectral remote sensing image slices after channel attention in S3 , hierarchical coding is used to extract multi-modal features, wherein the first layer convolution includes a convolution layer, batch normalization, and an activation layer:

[0091] ;

[0092] ;

[0093] wherein, represents or , represents a batch normalization operation, wherein It is the linear output after the first convolutional operation. and These are the convolutional kernel weights and bias terms that need to be trained in the first layer. It is the final output feature map of layer 1, which serves as the input for the next layer. This represents a convolution operation. All three modal data branches use a 3×3 convolution kernel, with 16 output channels and a spatial stride of 1.

[0094] S42. Perform convolutional layers 2 through 5, including convolutional layers, batch normalization, activation layers, and max pooling layers:

[0095] ;

[0096] ;

[0097] ;

[0098] in, It is the first The linear output after a convolutional operation. and They are the first The corresponding layers contain the convolutional kernel weights and bias terms that need to be trained. It is the first The result of max pooling of the layer, It is the first The final output feature map of the layer is used as the input of the next layer. All three modal data branches use 3×3 convolution kernels. The number of output channels of layers 2 to 5 are 32, 64, 128 and 128 respectively, with a spatial stride of 1, a pooling size of 2×2 and a pooling stride of 2.

[0099] S43, the 6th convolutional layer, including convolutional layers, batch normalization, activation layers, and average pooling layers, yields multimodal features:

[0100] ;

[0101] ;

[0102] ;

[0103] in, It is the linear output after the 6th convolutional layer. and These are the convolutional kernel weights and bias terms that need to be trained in the 6th layer. This is the average pooling result of layer 6. is the output feature map of the 6th layer. The three modal data branches all use a 3x3 convolution kernel, with 64 output channels, a spatial step of 1, a pooling size of 2x2, and a pooling step of 2.

[0104] Specifically, the S5 comprises:

[0105] S51, average covariance of each modal remote sensing image After linear transformation, batch normalization and activation:

[0106] ;

[0107] ;

[0108] wherein, is the output after linear transformation, and are the weight matrix and bias term of the spatial covariance that need to be trained and learned, is the final output feature map after the activation layer. The output channel number of the three modal data is 64, and the spatial step is 1.

[0109] Specifically, the S6 comprises:

[0110] S61, multi-modal feature fusion:

[0111] ;

[0112] wherein, represents a multi-modal feature fusion layer, which splices the features extracted from the three modal remote sensing images in the channel dimension, i.e. , wherein represents the channel output shape of each modal after the 6th layer convolution after S4, according to the output shape parameters of the 6th layer convolution =64, and after splicing, a fused encoding layer output is obtained, with an output shape of 192.

[0113] S62, spatial covariance feature fusion:

[0114] ;

[0115] wherein, represents the spatial covariance feature of the three modal remote sensing images, which splices the spatial covariance features of the three modal remote sensing images and flattens them into a one-dimensional tensor feature for use in multi-modal feature fusion, wherein represents the channel output shape of each modal spatial covariance after single-layer convolution after S5 = 64, thus The output shape of the output layer is 192.

[0116] S63, final fusion:

[0117] ;

[0118] For the final fusion layer, the multi-modal features and the spatial covariance features are spliced, and the output shape is 384. The spliced feature vector contains multi-modal features and spatial covariance features, which provides more comprehensive support for classification.

[0119] Specifically, the S7 includes:

[0120] S71, using 1x1 convolution to avoid destroying the spatial structure while reducing the channel number to the class number, realizing the classification output layer:

[0121] ;

[0122] wherein is the classification output layer, is the number of classes of the classification task, is the linear output after the final convolution operation, and are the convolution kernel weights and bias items that need to be trained and learned, respectively.

[0123] S72, flattening the output into a prediction vector:

[0124] ; wherein is the number of samples in the batch, which is set to 64;

[0125] Specifically, the S8 includes:

[0126] S81, using cross-entropy loss function as the classification error measure:

[0127] ;

[0128] wherein, is the one-hot encoding of the true label of the i-th sample, the prediction probability is obtained by function.

[0129] S82, adding L2 regularization term to control the model complexity, suppress overfitting, and improve the generalization ability of the model:

[0130] ; ​

[0131] where, is a regularization coefficient, set to 0.001, is the model parameters (all weights and biases), represents a single trainable parameter in the model parameter set .

[0132] S83, the total loss function is composed of the cross-entropy loss and the L2 regularization term :

[0133] ;

[0134] S84, the Adam optimizer update is performed, the learning rate adopts an exponential decay strategy, the training process introduces batch normalization and Dropout layers to improve the model generalization ability:

[0135] ;

[0136] The total number of training iteration rounds is 200, wherein is the model parameters (all weights and biases) at the i-th iteration, corresponding to in the regularization. is the decay learning rate, set to 0.001, used to control the update step size. is the first-order moment estimate of the gradient (momentum term), the corrected mean, and the calculation formula is: ;

[0137] wherein, the first-order moment estimate of the gradient of the uncorrected mean ,

[0138] is the gradient of the i-th iteration, is the momentum decay coefficient, set to 0.9. is the second-order moment estimate of the gradient (adaptive learning rate term), the corrected variance, and the calculation formula is:

[0139] ;

[0140] ;

[0141] wherein, the second-order moment estimate of the gradient of the uncorrected variance , is the decay coefficient of the second-order moment estimate, set to 0.999; is a very small constant, set to , to prevent division by zero.

[0142] ​​In a specific embodiment of the present invention, taking SDGSAT-1 remote sensing images of Beijing in three modalities as an example, land use classification is carried out in complex urban areas, including 14 land use categories: commercial land, industrial land, elevated and grade-separated traffic facilities, public management and public service land, roads, urban residential land, rural residential areas, temporary land, large transportation hub land, cultivated land, forest land, grassland, lakes / reservoirs, and rivers. This method is compared with classification methods based solely on multi-spectral remote sensing data (single-modal), classification methods that synthesize three modalities of data through channel fusion (simple multi-modal fusion), and classification methods that do not incorporate spatial covariance features (multi-modal + no spatial covariance). The F1 score and overall precision are used to evaluate classification accuracy. The F1 score is the harmonic mean of precision and recall, ranging from 0 to 1; a higher value indicates better classification performance and evaluates the classification quality of each category. Overall precision refers to the proportion of correctly classified samples to the total number of samples, directly reflecting the overall classification accuracy of the model. As shown in Table 1, the classification accuracy of this method is higher than that of the other three classification methods.

[0143] Table 1 shows the F1 scores and overall accuracy (%) for various classification methods.

[0144]

[0145] In summary, by utilizing the technical solution described above, the spatial covariance quantification of land cover dependencies enables the model to more intuitively express the feature correlations between different regions, enhancing the model's interpretability. Furthermore, the adaptive enhancement of key spectral information using a channel attention mechanism improves the fusion quality of multimodal data, allowing the model to maintain high classification accuracy even in complex scenarios. This method can be widely applied to practical scenarios such as land use classification and target detection, providing reliable technical support for high-precision remote sensing image analysis.

[0146] like Figure 3 As shown, the present invention also provides a multimodal remote sensing classification device based on spatial covariance to implement the above method, comprising the following modules:

[0147] The registration module acquires remote sensing images of multiple modalities in the same area and performs image registration, so that each image corresponds one-to-one in space.

[0148] The quantization module extracts the spatial covariance of each registered modal image to quantify the spatial correlation of ground features.

[0149] The enhancement module constructs a channel attention mechanism for modes rich in spectral information to adaptively enhance key spectral features;

[0150] The multi-modal feature extraction module adopts a hierarchical coding manner to each modality image, and extracts multi-modal features of each layer by layer;

[0151] The covariance feature extraction module inputs the spatial covariance of each modality into a single-layer network, and extracts the corresponding covariance features;

[0152] The fusion module fuses the multi-modal features of each modality, fuses the covariance features of each modality, and finally fuses the multi-modal fusion features and the covariance fusion features to form comprehensive features;

[0153] The output module performs convolution dimension reduction on the comprehensive features, and outputs a classification result corresponding to the number of categories;

[0154] The training module uses an optimizer with weight decay and cross-entropy loss for training, and controls the model complexity through a regularization term.

[0155] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-modal remote sensing classification method based on spatial covariance when executing the program.

[0156] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the multi-modal remote sensing classification method based on spatial covariance.

[0157] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. The solutions in the embodiments of the application can be implemented in various computer languages, such as object-oriented programming languages Java and interpreted scripting language JavaScript.

[0158] The application is described with reference to flowcharts and / or block diagrams according to the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a machine that implements the functions described in the flowcharts and / or block diagrams.Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0162] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A spatial covariance-based multi-modal remote sensing classification method, characterized in that, The method comprises the following steps: S1, acquiring remote sensing images of multiple modalities in the same area and performing image registration, so that each image corresponds to each other in space; S2, extracting spatial covariance of each modality image after registration to quantitatively represent the spatial correlation of the ground object; S3, constructing a channel attention mechanism for the modality with rich spectral information to adaptively enhance the key spectral features; comprising: S31, for multispectral remote sensing images, the image slice size is 32*32, and a double-path network structure of average pooling and maximum pooling is adopted; S32, the outputs of the double paths are added and then multiplied with the image slice in each channel to enhance the significant channels; S4, using a hierarchical encoding method to extract the respective multi-modal features layer by layer; comprising: S41, for the image slices of thermal infrared and low-light remote sensing and the multispectral remote sensing image slices enhanced by S3, the multi-modal features are extracted by hierarchical encoding, wherein the first layer convolution comprises a convolution layer, a batch normalization and an activation layer; S42, the second to fifth layer convolution is performed, comprising a convolution layer, a batch normalization, an activation layer and a maximum pooling layer; S43, the sixth layer convolution is performed, comprising a convolution layer, a batch normalization, an activation layer and an average pooling layer, to obtain the multi-modal features; S5, inputting the spatial covariance of the multi-modal features into a single layer network to extract the corresponding covariance features; S6, fusing the multi-modal features and the covariance features, and finally fusing the multi-modal fusion features and the covariance fusion features to form comprehensive features; S7, performing convolution dimension reduction on the comprehensive features to output classification results corresponding to the number of categories; S8, training by using an optimizer with weight decay and cross-entropy loss, and controlling the model complexity by a regularization term.

2. The multi-modal remote sensing classification method based on spatial covariance according to claim 1, characterized in that, In the S2, the spatial covariance is calculated by a neighborhood window, the window size matches the image slice size of the input network, a variable related to the lag order and the channel number is established to measure the spatial correlation of the ground object at different distances from the center point of the neighborhood window in each channel.

3. The multi-modal remote sensing classification method based on spatial covariance according to claim 1, characterized in that, In the S3, the channel attention mechanism adopts a double-path structure of average pooling and maximum pooling, and generates channel weights through a fully connected layer.

4. The multi-modal remote sensing classification method based on spatial covariance according to claim 1, characterized in that, In the S5, the single layer network is sequentially connected by linear transformation, batch normalization and activation function.

5. The spatial covariance-based multi-modal remote sensing classification method according to claim 1, wherein, In the S6, the final fusion is channel dimension splicing, so that the multi-modal features and the covariance features are aligned in a unified vector space.

6. The spatial covariance-based multi-modal remote sensing classification method according to claim 1, wherein, In the S8, the optimizer is Adam, the learning rate is exponentially decayed, and a Dropout layer is introduced in the training process.

7. A spatial covariance based multi-modal remote sensing classification apparatus, characterized in that, The method comprises the following modules: A registration module for acquiring remote sensing images of multiple modalities in the same area and performing image registration, so that each image corresponds to each other in space; A quantification module for extracting spatial covariance of each modality image after registration to quantitatively represent the spatial correlation of the ground object; An enhancement module for constructing a channel attention mechanism for the modality with rich spectral information to adaptively enhance the key spectral features; for multispectral remote sensing images, the image slice size is 32*32, and a double-path network structure of average pooling and maximum pooling is adopted; the outputs of the double paths are added and then multiplied with the image slice in each channel to enhance the significant channels; The multi-modal feature extraction module extracts multi-modal features of each modality image layer by layer by using hierarchical encoding; for the image slices of thermal infrared and low-light remote sensing and the enhanced multi-spectral remote sensing image slices, the multi-modal features are extracted by hierarchical encoding, wherein the first layer convolution includes a convolution layer, a batch normalization and an activation layer; the second to fifth layer convolution is performed, including a convolution layer, a batch normalization, an activation layer and a maximum pooling layer; the sixth layer convolution is performed, including a convolution layer, a batch normalization, an activation layer and an average pooling layer, to obtain the multi-modal features; The covariance feature extraction module inputs the spatial covariance of each modality into a single-layer network to extract corresponding covariance features; The fusion module fuses the multi-modal features of each modality, fuses the covariance features of each modality, and finally fuses the multi-modal fusion features and the covariance fusion features to form comprehensive features; The output module performs convolution dimension reduction on the comprehensive features and outputs classification results corresponding to the number of categories; The training module is trained by using an optimizer with weight decay and cross-entropy loss, and the model complexity is controlled by a regularization term.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the multi-modal remote sensing classification method based on spatial covariance according to any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the multi-modal remote sensing classification method based on spatial covariance according to any one of claims 1 to 6 when executed by the processor.

Images