Multi-modal remote sensing data classification method based on linear time sequence selective state space model

Abstract

The invention discloses a multi-modal remote sensing data classification method based on a linear time sequence selective state space model, and belongs to the technical field of image processing. Compared with a CNN-based multi-modal remote sensing data classification method, the CNN-based multi-modal remote sensing data classification method relieves modeling constraints of the CNN through a global receptive field and dynamic weighting. The method comprises the following steps: establishing mapping layers of different modals based on a multilayer perceptron, dividing input multi-modal remote sensing data into sample blocks, and superposing position codes to obtain a multi-modal remote sensing representation vector of a sequence; constructing a cross-modal space fusion module, and obtaining the output of the cross-modal space fusion module through space information interaction; and averaging the output of all the cross-modal space fusion modules, and outputting a classification result by using a multi-layer perceptron. The modeling capability of the multi-modal remote sensing data classification model is effectively improved, and the method can be applied to remote sensing data processing.

Description

Technical Field

[0001] The invention relates to a multimodal remote sensing data classification method, in particular to a multimodal remote sensing data classification method based on a linear time series selective state space model, and belongs to the technical field of image processing. Background Art

[0002] Multimodality in the field of remote sensing usually refers to the imaging results of scenes and targets obtained by different sensors, such as multispectral, hyperspectral, synthetic aperture radar and lidar. Reasonable use of multimodal remote sensing data can provide more comprehensive description information for ground objects in terms of spectrum, time and space, thereby improving the interpretation ability of remote sensing data to meet the needs of practical applications such as military reconnaissance and smart agriculture.

[0003] In order to promote the in-depth and extensive application of multimodal remote sensing data in the above applications, effective information processing means are needed. Classification, as one of the important links in remote sensing data processing technology, has always been a hot topic in current research. In recent years, due to its powerful feature extraction ability, deep learning has become the mainstream method for multimodal remote sensing data classification. Among many deep learning methods, multimodal remote sensing data methods based on CNN models have attracted much attention, but lack the extraction of long-range dependencies. Therefore, a multimodal remote sensing data classification method with efficient global feature extraction capabilities is needed. Summary of the invention

[0004] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0005] In view of this, in order to solve the deficiencies of the prior art multimodal remote sensing data classification method in terms of global feature extraction capability, the present invention provides a multimodal remote sensing data classification method based on a linear time series selective state space model.

[0006] The technical solution is as follows: A multimodal remote sensing data classification method based on a linear time series selective state space model comprises the following steps:

[0007] S1. Acquire multimodal remote sensing data;

[0008] S2. Establish a mapping layer based on a multi-layer perceptron of different modalities, input multi-modal remote sensing data, divide the output of the multi-modal remote sensing data into different sample blocks, add the position code to the sample block, and obtain a multi-modal remote sensing representation vector of the sequence;

[0009] S3. Construct a cross-modal spatial fusion module, input the multi-modal remote sensing representation vector of the sequence, and obtain the output of the cross-modal spatial fusion module through the interaction of spatial information of different modalities;

[0010] S4. Iterate step S3 N times, take the average of the outputs of all cross-modal spatial fusion modules, and use a multi-layer perceptron to output the classification results.

[0011] Furthermore, multimodal remote sensing data includes hyperspectral data D HSI and ground surface model D LiDAR , ground surface model D LiDAR It is obtained by denoising and rasterizing the data collected by the lidar. is a set of real numbers with length H, width W and spectral number L, is a set of real numbers with length H and width W.

[0012] Furthermore, the output of multimodal remote sensing data includes hyperspectral data D HSI Output O HSI and ground surface model D LiDAR Output O LiDAR , the multimodal remote sensing representation vector of the sequence includes the final hyperspectral data D HSI Output O HSI' And the final ground surface model D LiDAR Output O LiDAR' .

[0013] Furthermore, a cross-modal spatial fusion module based on the linear time series selective state space model is constructed to transform the final hyperspectral data D HSI Output O HSI' And the final ground surface model D LiDAR Output O LiDAR' Input the one-dimensional convolution and SiLU activation function of the cross-modal spatial fusion module respectively to obtain the first output O' HSI and the second output O' LiDAR ;

[0014] O' HSI =σ(Conv(MLP(O HSI' ))) (1)

[0015] O' LiDAR =σ(Conv(MLP(O LiDAR' ))) (2)

[0016] Among them, Conv is a one-dimensional convolution operation, MLP is a multi-layer perceptron, σ represents the SiLU activation function, and z HSI Represents the final hyperspectral data DHSI Output O HSI' The output, z LiDAR Represents the final ground surface model D LiDAR Output O LiDAR' Output:

[0017] The first output O' HSI and the second output O' LiDAR They are input into the linear time series selective state space block, and the first output O' HSI After passing through the linear time series selective state space block, the first parameter B is output HSI 、The second parameter C HSI and the time scale parameter Δ of the hyperspectral mode HSI , the second output O' LiDAR After passing through the linear time series selective state space block, the third parameter B is output LiDAR 、The fourth parameter C LiDAR and the time scale parameter Δ of the lidar modality LiDAR , for the first output O' HSI and the second output O' LiDAR The output obtained after the linear time series selective state space block is used to interact with spatial information of different modes;

[0018] The spatial information interaction process of different modalities is expressed as:

[0019] B HSI ,C HSI ,Δ HSI =MLP(O' HSI ) (3)

[0020] B LiDAR ,C LiDAR ,Δ LiDAR =MLP(O' LiDAR ) (4)

[0021]

[0022]

[0023]

[0024] y=(y HSI ⊙z HSI )⊙(y LiDAR ⊙σ(MLP(O LiDAR )) (8)

[0025] in, is used to extract the corresponding discrete parameters in the state equation of HSI, For BHSI The corresponding discrete parameters, ZOH is the zero-order holder, A is the continuous variable, y LiDAR is the hybrid feature of LiDAR and HSI based on the selective state space block output of linear time series, It is used to extract the corresponding discrete parameters in the state equation of LiDAR, h t-1 is the system state at time t-1, For B LiDAR The corresponding discrete parameter, y HSI is the mixed feature of HSI and LiDAR based on the selective state space block output of the linear time series, y is the output of the cross-modal spatial fusion module, ⊙ is the Hadamard product, HSI is the hyperspectral image, and LiDAR is the laser radar data.

[0026] The beneficial effects of the present invention are as follows: compared with the multimodal remote sensing data method based on the CNN model, the present invention alleviates the modeling constraints of the convolutional neural network through global receptive field and dynamic weighting, and improves the global modeling ability of the multimodal remote sensing data classification model; the present invention uses a linear time series modeling method on the basis of a state space model, uses selective state space support to more efficiently and effectively capture relevant information across long sequences, and uses shared weights to learn channel dependencies to achieve efficient channel information interaction between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 It is a flowchart of a multimodal remote sensing data classification method based on a linear time series selective state space model;

[0029] Figure 2 The present invention is a flowchart of an embodiment of a multimodal remote sensing data classification method based on a linear time series selective state space model. DETAILED DESCRIPTION

[0030] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0031] refer to Figure 1 and Figure 2The present embodiment is described in detail, a multimodal remote sensing data classification method based on a linear time series selective state space model, specifically comprising the following steps:

[0032] S1. Acquire multimodal remote sensing data;

[0033] S2. Establish a mapping layer based on a multi-layer perceptron of different modalities, input multi-modal remote sensing data, divide the output of the multi-modal remote sensing data into different sample blocks, add the position code to the sample block, and obtain a multi-modal remote sensing representation vector of the sequence;

[0034] S3. Construct a cross-modal spatial fusion module, input the multi-modal remote sensing representation vector of the sequence, and obtain the output of the cross-modal spatial fusion module through the interaction of spatial information of different modalities;

[0035] S4. Iterate step S3 N times, take the average of the outputs of all cross-modal spatial fusion modules, and use a multi-layer perceptron to output the classification results;

[0036] Specifically, position encoding provides position information to the model by adding a random vector of integer values ​​between 0, 1, 2, ..., N to each sample block, where N is the number of sample blocks.

[0037] Furthermore, multimodal remote sensing data includes hyperspectral data D HSI and ground surface model D LiDAR , ground surface model D LiDAR It is obtained by denoising and rasterizing the data collected by the lidar. is a set of real numbers with length H, width W and spectral number L, is a set of real numbers with length H and width W.

[0038] Furthermore, the output of multimodal remote sensing data includes hyperspectral data D HSI Output O HSI and ground surface model D LiDAR Output O LiDAR , the multimodal remote sensing representation vector of the sequence includes the final hyperspectral data D HSI Output O HSI' And the final ground surface model D LiDAR Output O LiDAR' .

[0039] Furthermore, a cross-modal spatial fusion module based on the linear time series selective state space model is constructed to transform the final hyperspectral data D HSI Output O HSI' And the final ground surface model D LiDAR Output O LiDAR'Input the one-dimensional convolution and SiLU activation function of the cross-modal spatial fusion module respectively to obtain the first output O' HSI and the second output O' LiDAR ;

[0040] O' HSI =σ(Conv(MLP(O HSI' ))) (1)

[0041] O' LiDAR =σ(Conv(MLP(O LiDAR' ))) (2)

[0042] Among them, Conv is a one-dimensional convolution operation, MLP is a multi-layer perceptron, σ represents the SiLU activation function, and z HSI Represents the final hyperspectral data D HSI Output O HSI' The output of z LiDAR Represents the final ground surface model D LiDAR Output O LiDAR' Output:

[0043] The first output O' HSI and the second output O' LiDAR They are input into the linear time series selective state space block, and the first output O' HSI After passing through the linear time series selective state space block, the first parameter B is output HSI 、The second parameter C HSI and the time scale parameter Δ of the hyperspectral mode HSI , the second output O' LiDAR After passing through the linear time series selective state space block, the third parameter B is output LiDAR 、The fourth parameter C LiDAR and the time scale parameter Δ of the lidar modality LiDAR , for the first output O' HSI and the second output O' LiDAR The output obtained after the linear time series selective state space block is used to interact with spatial information of different modes;

[0044] The spatial information interaction process of different modalities is expressed as:

[0045] B HSI ,C HSI ,Δ HSI =MLP(O' HSI ) (3)

[0046] B LiDAR ,C LiDAR ,Δ LiDAR =MLP(O'LiDAR ) (4)

[0047]

[0048]

[0049]

[0050] y=(y HSI ⊙z HSI )⊙(y LiDAR ⊙σ(MLP(O LiDAR )) (8)

[0051] in, is used to extract the corresponding discrete parameters in the state equation of HSI, For B HSI The corresponding discrete parameters, ZOH is the zero-order holder, A is the continuous variable, y LiDAR is the hybrid feature of LiDAR and HSI based on the selective state space block output of linear time series, It is used to extract the corresponding discrete parameters in the state equation of LiDAR, h t-1 is the system state at time t-1, For B LiDAR The corresponding discrete parameter, y HSI is the hybrid feature of HSI and LiDAR based on the output of the linear time series selective state space block, y is the output of the cross-modal spatial fusion module, ⊙ is the Hadamard product, HSI is the hyperspectral image, and LiDAR is the laser radar data;

[0052] Specifically, refer to Figure 2 , extract the hyperspectral image HSI and the laser radar data LiDAR and divide them into sample blocks respectively. After the sample blocks are superimposed with the corresponding position encoding, they are input into the cross-modal spatial fusion module. In the cross-modal spatial fusion process, Liner means to maintain linearity. The hyperspectral image HSI and the laser radar data LiDAR are respectively subjected to the one-dimensional convolution operation conv and the activation function SiLU to obtain the time scale parameter Δ of the hyperspectral modality. HSI 、The first parameter B HSI 、The second parameter C HSI , the time scale parameter Δ of the lidar mode LiDAR 、The third parameter B LiDAR 、The fourth parameter C LiDAR , Represents the outer product of the matrix, repeats the above process N times, and outputs the classification result graph.

[0053] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A multimodal remote sensing data classification method based on a linear time series selective state space model, characterized in that: The following steps are involved: S1. Acquire multimodal remote sensing data; S2. Establish a mapping layer based on a multi-layer perceptron of different modalities, input multi-modal remote sensing data, divide the output of the multi-modal remote sensing data into different sample blocks, add the position code to the sample block, and obtain a multi-modal remote sensing representation vector of the sequence; S3. Construct a cross-modal spatial fusion module, input the multi-modal remote sensing representation vector of the sequence, and obtain the output of the cross-modal spatial fusion module through the interaction of spatial information of different modalities; S4. Iterate step S3 N times, take the average of the outputs of all cross-modal spatial fusion modules, and use a multi-layer perceptron to output the classification results.

2. The multimodal remote sensing data classification method based on a linear time series selective state space model according to claim 1 is characterized in that: In S1, the multimodal remote sensing data includes hyperspectral data D HSI and ground surface model D LiDAR , ground surface model D LiDAR It is obtained by denoising and rasterizing the data collected by the lidar. is a set of real numbers with length H, width W and spectral number L, is a set of real numbers with length H and width W.

3. The multimodal remote sensing data classification method based on the linear time series selective state space model according to claim 2 is characterized in that: In S2, the output of the multimodal remote sensing data includes hyperspectral data D HSI Output O HSI and the ground surface model S LiDAR Output O LiDAR , the multimodal remote sensing representation vector of the sequence includes the final hyperspectral data D HSI Output O HSI′ And the final ground surface model D LiDAR Output O LiDAR' .

4. The multimodal remote sensing data classification method based on a linear time series selective state space model according to claim 3 is characterized in that: In S3, a cross-modal spatial fusion module based on a linear time series selective state space model is constructed to transform the final hyperspectral data D HSI Output O HSI′ And the final ground surface model D LiDAR Output O LiDAR′ Input the one-dimensional convolution and SiLU activation function of the cross-modal spatial fusion module respectively to obtain the first output O′ HSI and the second output O′ LiDAR ; THE' HSI =σ(Conv(MLP(O HSI′ ))) (1) THE' LiDAR =σ(Conv(MLP(O LiDAR ′))) (2) Among them, Conv is a one-dimensional convolution operation, MLP is a multi-layer perceptron, σ represents the SiLU activation function, and Z HSI Represents the final hyperspectral data D HSI Output O HSI′ The output, Z LiDAR Represents the final ground surface model D LiDAR Output O LiDAR′ Output: The first output O' HSI and the second output O′ LiDAR They are input into the linear time series selective state space block respectively, and the first output O′ HSI After passing through the linear time series selective state space block, the first parameter B is output HSI 、The second parameter C HSI and the time scale parameter Δ of the hyperspectral mode HSI , the second output O′ LiDAR After passing through the linear time series selective state space block, the third parameter B is output LiDAR 、The fourth parameter C LiDAR and the time scale parameter Δ of the lidar modality LiDAR , for the first output O′ HSI and the second output O′ LiDAR The output obtained after the linear time series selective state space block is used to interact with spatial information of different modes; The spatial information interaction process of different modalities is expressed as: B HSI ,C HSI ,Δ HSI =MLP(O′ HSI ) (3) B LiDAR ,C LiDAR ,Δ LiDAR =MLP(O′ LiDAR ) (4) and=(and HSI ⊙z HSI )⊙(and LiDAR ⊙σ(MLP(O LiDAR )) (8) in, is used to extract the corresponding discrete parameters in the state equation of HSI, For B HSI The corresponding discrete parameters, ZOH is the zero-order holder, A is the continuous variable, y LiDAR is the hybrid feature of LiDAR and HSI based on the selective state space block output of linear time series, It is used to extract the corresponding discrete parameters in the state equation of LiDAR, h t-1 is the system state at time t-1, For B LiDAR The corresponding discrete parameter, y HSI is the mixed feature of HSI and LiDAR based on the selective state space block output of the linear time series, y is the output of the cross-modal spatial fusion module, ⊙ is the Hadamard product, HSI is the hyperspectral image, and LiDAR is the laser radar data.

Images