A multi-source remote sensing satellite data intelligent interpretation and classification method

Abstract

The application discloses a kind of multi-source remote sensing satellite data intelligent interpretation and classification method, introduces graph convolution network on the basis of traditional convolutional neural network, obtains the topological structure information hidden in data by graph convolution operation, reduces the dependence degree on sample while mining more meaningful features, in feature fusion stage, utilize global dependence fusion module to capture long-distance dependence in data, better carry out multi-source data feature fusion.The application can realize higher classification accuracy by introducing additional feature coordination in target scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-source remote sensing satellite data intelligent interpretation and classification, and particularly relates to a multi-source remote sensing satellite data intelligent interpretation and classification method, which is suitable for accurately interpreting and classifying hyperspectral and multispectral remote sensing data obtained on a satellite-borne sensor. BACKGROUND

[0002] The intelligent interpretation and classification of remote sensing data for a target scene is one of the important research directions in the field of remote sensing, which extracts and uses feature information in the target scene using a certain algorithm, and assigns a class label to each pixel point in the scene. The intelligent interpretation and classification of remote sensing data for a target scene has attracted a lot of attention in the fields of wetland monitoring, smart agriculture, urban planning, etc.

[0003] In the intelligent interpretation and classification task of remote sensing data, due to the complexity of the scene and the limitation of observation, the classification performance of single modal remote sensing data is severely restricted. Therefore, it is difficult to meet the growing demand of practical application by using hyperspectral images (HSI), multispectral images (MSI) or synthetic aperture radar (SAR) data alone. Therefore, collaborative interpretation and classification of multi-source remote sensing data has become a new hotspot in the current research field. In recent years, the rapid development of deep learning has provided new impetus for the progress of various industries, and the excellent feature extraction and representation ability of convolutional neural network has attracted people's attention and is widely used in the intelligent interpretation and classification task of remote sensing data. However, as we all know, convolutional neural network needs a large number of labeled samples to optimize network parameters, and it is time-consuming and laborious to obtain a large number of labeled data, which is difficult to meet in practical application. In addition, the features that can be obtained by convolutional neural network only include Euclidean structure features, ignoring other abstract features hidden in the data, making it more difficult to fully optimize the model parameters.

[0004] At the same time, due to the different emphases of multi-source remote sensing data, how to fully integrate the complementary advantages of multi-source information is also a problem that must be considered, which means that a feature fusion module needs to be introduced into the network to perform feature fusion. However, due to the fact that convolution operation can only process local neighborhood building blocks in data at a time, the dependency relationship between long-distance positions in complementary data cannot be transmitted. SUMMARY

[0005] The present application provides a new multi-source remote sensing satellite data intelligent interpretation and classification method to solve the defects in the prior art.

[0006] The application fully considers the data characteristics in the intelligent interpretation and classification task of multi-source remote sensing data, and proposes a multi-source remote sensing satellite data intelligent interpretation and classification method based on the convolutional neural network, aiming at how to extract more meaningful features from limited labeled samples, and the need to capture long-distance dependence relationship for global fusion of multi-source features in the feature fusion stage.

[0007] In order to achieve the above application purposes, the technical solutions adopted by the application are as follows:

[0008] A multi-source remote sensing satellite data intelligent interpretation and classification method, comprising the following steps:

[0009] Step 1: Construct a spectral-spatial feature extraction network. Since the traditional convolutional neural network is efficient and effective in extracting spectral and spatial features, the application uses two convolutional neural networks to extract spectral and spatial features from hyperspectral and multispectral data respectively.

[0010] Step 2: Construct a graph feature extraction model. By introducing a graph convolutional network to obtain the topological structure information hidden in the data, a graph convolution operation is used to perform relationship learning to capture the topological structure information. A graph dependence fusion module is designed to enhance the interaction between multi-source graph features, so as to better explore the complementary advantages of multi-source data.

[0011] Step 3: On the basis of obtaining the graph convolutional features and the spectral-spatial features, a global dependence fusion module is introduced to capture long-distance dependence relationship in the process of fully fusing various features.

[0012] Step 4: The fused features are sent to a classifier, and a cross-entropy loss function is used to minimize the error as the target to train the classification model. After training, the model is loaded, the remote sensing image of the target area to be interpreted is input, and the final classification result is obtained, and a visual classification map is output.

[0013] Further, step 1 comprises the following substeps:

[0014] Step 1.1: Take the hyperspectral and multispectral remote sensing data of the same area as input, wherein the size of the hyperspectral image is h x w x c1, and the size of the multispectral image is h x w x c2, wherein h represents the height of the image, w represents the width of the image, and c1 and c2 represent the number of channels respectively. Because the context information is particularly important for robust feature representation, a spatial block with a size of r centered at position (i,j) is extracted from the data and sent to the spectral-spatial feature extraction network to obtain the corresponding hyperspectral feature X H and multispectral feature X MThe network consists of two two-dimensional convolutional neural network branches with three convolutional layers, which are parameterized with training samples, followed by batch normalization (BN) and activation layers, and the specific structure is shown as follows:

[0015]

[0016] where ConvBN(c,c') is a convolutional layer followed by batch normalization and activation layers, and c and c' represent the number of channels of input and output data, respectively.

[0017] Further, step 2 includes the following sub-steps:

[0018] Step 2.1: Map the features extracted by the spectral-spatial feature extraction network in step 1 to a high-dimensional graph space. In the graph space G=(V,E), V represents nodes and E represents edges. The extracted features X H,M ∈R H×W×C are mapped into a graph representation G N×D ∈R . Where C is the dimension of the feature vector, H and W are the height and width of the feature vector, respectively, N represents the number of nodes in the graph, corresponding to the number of categories in the scene, and D is the dimension of the node.

[0019] Step 2.2: Construct a transition graph G T to enhance the feature association and interaction of hyperspectral and multispectral data. G H ∈R N×D represents the hyperspectral graph features, and G M ∈R N×D represents the multispectral graph features.

[0020] G T =α(G H +F·G M ·W M )+(1-α)(G M +F T ·G H ·W H )

[0021] where α is a balance parameter that can be learned during training. W M ,W H ∈R D×D is a weight matrix, and F is a fusion matrix used to better combine heterogeneous graph features.

[0022] Step 2.3: For a given graph feature representation, use graph convolution to propagate information between nodes on G through matrix multiplication to obtain an evolved feature space G p

[0023] Gp = σ r (A·G p-1 ·W+b)p=1,2,3

[0024] where W∈R D×D and b are weight matrix and bias, σ r is ReLU activation function, and p is the number of graph convolution layers. In this process, the node adjacency weight matrix A is used to learn and capture the relationship between different categories.

[0025] Step 2.4: Graph dependency fusion is used to fuse three types of graph features G H , G M and G T . Specifically, the three types of graph features are concatenated, and then the dimension of the concatenated features is restored to D using a fully connected layer, thereby generating the fused graph feature G F .

[0026] G F = FC(con(G H , G T , G M ))

[0027] where con(·) represents the concatenation operation, and FC() is a fully connected layer.

[0028] Step 2.5: G F is mapped back to the convolution space using a de-mapping function to obtain the feature X G , which is used to supplement and enhance the classification representation of the spectral spatial feature.

[0029] Further, step 3 includes the following sub-steps:

[0030] Step 3.1: Construct a cross-modal fusion block (CFblock). Define some linear functions f1(x) = W1x, f2(x) = W2x, f3(x) = W3x and f4(x) = W4x implemented by 1x1 convolution. They are used to change the shape of the input feature from HxWxC to PxC, where P < HxW, so as to reduce the parameters and complexity of the model.

[0031] CFblock(K, V, Q) = f4(softmax(f1 T (K)·f2(V))·f3 T (Q))

[0032] where K, V and Q are the inputs of the fusion block.

[0033] Step 3.2: Implementation of the global dependency fusion module (GDFM). After the construction of the cross-modal fusion module, the hyperspectral features, multispectral features, and graph convolutional features are respectively applied to this module, and the combination of the three obtains the final global dependency fusion module.

[0034] GDFM = con(X H + CFblock(X M , X G , X H ), X M + CFblock(X H , X G , X M ), X G + CFblock(X H , X M , X G )

[0035] Further, step 4 includes the following sub-steps:

[0036] Step 4.1: After obtaining the fusion features processed by the global dependency fusion module, the features are sent to the classifier to obtain the prediction results, and the corresponding loss is obtained using the cross-entropy loss function, and the network parameters are optimized according to the back propagation. After multiple rounds of training, the classification model is saved.

[0037] Step 4.2: Collect and input the remote sensing image of the target area to be interpreted, use the saved deep learning classification model learned in step 4.1 to predict the class label of each pixel point in the scene, and obtain the final classification result and visualize the classification map.

[0038] Compared with the prior art, the advantages of the present application are:

[0039] The characteristics of multi-source remote sensing images are fully considered, the model is supplemented with features for the problem of insufficient feature extraction of traditional convolutional neural networks, the sample dependence is reduced, and the long-distance dependency relationship between the information in the feature fusion stage is enhanced. On the basis of using convolutional neural networks to extract hyperspectral and multispectral features, a graph convolutional network is introduced to extract the topological structure information hidden in the data, which reduces the dependence on samples while mining more meaningful features, and the graph dependency fusion also enhances the interaction between the multi-source data graph features; the traditional feature fusion method is optimized, and the global dependency fusion module is used to capture the long-distance dependency relationship in the data. A method for intelligent interpretation and classification of multi-source remote sensing satellite data is proposed, which realizes the intelligent classification of target scene remote sensing data. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a flowchart of the multi-source remote sensing satellite data intelligent interpretation and classification method of the present application;

[0041] Figure 2 is the structure diagram of the graph feature extraction model constructed by the present application;

[0042] Figure 3 is the structure diagram of the global dependency fusion module constructed by the present application;

[0043] Figure 4 is the target scene classification flowchart of the multi-source remote sensing satellite data intelligent interpretation and classification method proposed in the embodiment of the present application on the remote sensing satellite data set. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the present application is further described in detail below according to the drawings and examples.

[0045] As shown in Figure 1 , a multi-source remote sensing satellite data intelligent interpretation and classification method comprises the following steps:

[0046] Step 1: Construct a spectral-spatial feature extraction network. Due to the high efficiency and effectiveness of traditional convolutional neural networks in extracting spectral and spatial features, the present application uses two convolutional neural networks to extract spectral and spatial features from hyperspectral and multispectral data respectively;

[0047] Step 2: Construct a graph feature extraction model. By introducing a graph convolutional network to obtain the topological structure information implied in the data, a graph convolution operation is used to perform relationship learning to capture the topological structure information. A graph dependency fusion module is designed to enhance the interaction between multi-source graph features, so as to better explore the complementary advantages of multi-source data;

[0048] Step 3: On the basis of obtaining the graph convolutional features and the spectral-spatial features, a global dependency fusion module is introduced to capture long-distance dependency relationships in the process of fully fusing various features;

[0049] Step 4: The fused features are sent to a classifier, and a cross-entropy loss function is used to minimize the error as the goal to train the classification model. After training, the model is loaded, the remote sensing image of the target area to be interpreted is input, and the final classification result is obtained. The classification map is visualized and output.

[0050] Specifically, step 1 comprises the following steps:

[0051] Step 1.1: Take hyperspectral and multispectral remote sensing data of the same region as input. The hyperspectral image has dimensions h×w×c1, and the multispectral image has dimensions h×w×c2, where h represents the image height, w represents the image width, and c1 and c2 represent the number of channels, respectively. Because contextual information is particularly important for robust feature representation, a spatial block of size r centered at position (i,j) is extracted from the data and fed into the spectral-spatial feature extraction network to obtain the corresponding hyperspectral feature X. H and multispectral features X M The network consists of two 2D convolutional neural network branches, each with three convolutional layers. These convolutional layers use training samples for parameterization, followed by batch normalization (BN) and activation layers. The specific structure is shown below:

[0052]

[0053] In the formula, ConvBN(c,c') is a convolutional layer that follows a batch normalization and activation layer, where c and c' represent the number of channels in the input and output data, respectively.

[0054] In the constructed graph feature extraction network, the specific network structure is as follows: Figure 2 As shown.

[0055] Step 2 includes the following steps:

[0056] Step 2.1: Map the features extracted by the spectral-spatial feature extraction network in Step 1 to a high-dimensional graph space. In the graph space G = (V, E), V represents a node, and E represents an edge. The extracted features X... H,M ∈R H×W×C Mapped into a graph representation G∈R N×D Where C is the dimension of the feature vector, H and W are the height and width of the feature vector, respectively, N represents the number of nodes in the graph, corresponding to the number of categories in the scene, and D is the dimension of the node.

[0057] Step 2.2: First, construct a transition graph G. T Used to enhance feature correlation and interaction in hyperspectral and multispectral data. Using G... H ∈R N×D G represents the characteristics of the hyperspectral image. M ∈R N×D This indicates the characteristics of the multispectral image.

[0058] G T =α(G H +F·G M ·W M )+(1-α)(G M +F T ·G H ·WH ) (2)

[0059] where a is a balance parameter, which can be learned during training. W M ,W H ∈R D×D is a weight matrix. And F is a fusion matrix, which is used to better combine the heterogeneous graph features.

[0060] Step 2.3: For a given graph feature representation, graph convolution is used to propagate information between nodes on G by matrix multiplication to obtain the evolved feature space G p

[0061] G p = σ r (A·G p-1 ·W+b)p=1,2,33)

[0062] where W∈R D×D and b are weight matrix and bias, σ r is ReLU activation function, and p is the number of graph convolution layers. In this process, the node adjacency weight matrix A is used to learn and capture the relationship between different categories.

[0063] Step 2.4: Graph dependency fusion is used to fuse three types of graph features G H , G M and G T . Specifically, the three types of graph features are spliced, and then the dimension of the spliced feature is restored to D by using a fully connected layer, thereby generating the fused graph feature G F .

[0064] G F = FC(con(G H ,G T ,G M )) (4)

[0065] where con(·) represents the splicing operation, and FC() is a fully connected layer.

[0066] Step 2.5: G F is remapped back to the convolution space by using a back mapping function to obtain the feature X G , which is a fused graph feature representation used to enhance the classification representation of the spectral space feature.

[0067] The structure of the global dependency fusion module constructed is shown in Figure 3 .

[0068] Step 3 includes the following steps:

[0069] Step 3.1: Construction of cross-modal fusion module (CFblock). Define some linear functions f1(x) = W1x, f2(x) = W2x, f3(x) = W3x and f4(x) = W4x implemented by 1x1 convolution. They are used to transform the shape of input features from HxWxC to PxC, where P < HxW, so as to reduce the parameters and complexity of the model.

[0070] CFblock(K, V, Q) = f4(softmax(f1 T (K)·f2(V))·f3 T (Q)) (5)

[0071] In the formula, K, V and Q are the inputs of the fusion module.

[0072] Step 3.2: Implementation of global dependency fusion module (GDFM). After the construction of the cross-modal fusion module, the module is applied to the hyperspectral features, multispectral features and graph convolution features respectively, and the three are combined to obtain the final global dependency fusion module.

[0073]

[0074] Step 4 includes the following steps:

[0075] Step 4.1: After obtaining the fusion features processed by the global dependency fusion module, the features are sent to the classifier to obtain the prediction results, and the corresponding loss is obtained by using the cross-entropy loss function, and the network parameters are optimized according to the back propagation. After multiple rounds of training, save the classification model.

[0076] Step 4.2: Collect and input the remote sensing image of the target area to be interpreted, use the saved deep learning classification model learned in step 4.1 to predict the class label of each pixel point in the scene, obtain the final classification result, and visualize the classification map.

[0077] Example 1

[0078] The example realizes the classification of multiple targets in multi-source remote sensing satellite data in wetland scenes, and the specific implementation process is as shown in Figure 4

[0079] ​Step 1: Collect multi-source remote sensing satellite data in wetland scenarios, label the categories of pixel points in the scene, preprocess and data augment the training data in the dataset, increase the diversity of training samples, and construct a multi-source remote sensing satellite data classification dataset. Collect data of the target wetland area through the data publication website of the remote sensing satellite, and label the categories of sample points according to the results of field investigation and professional knowledge to obtain data and corresponding labels for training the multi-source remote sensing data classification model. The label labels the category c to which the corresponding pixel point belongs. The collected data needs to be preprocessed, such as radiation correction, spectral correction, registration, etc. After that, take the image block centered on the pixel point as the training sample to construct the dataset.

[0080] Step 2: Construct a spectral-spatial feature extraction network to extract spectral and spatial information contained in multi-source remote sensing data using a three-layer convolutional neural network. The image block centered on the pixel point is taken as the input sample and sent to the spectral-spatial feature extraction network. The convolution operation is used to restore the feature dimension of multi-source remote sensing data to consistency and extract the abstract features contained in the image block.

[0081] Step 3: Construct a graph feature extraction network to obtain the topological structure information implied in the data. First, perform a mapping operation on the abstract convolutional features obtained in Step 2 to map them to a high-dimensional graph feature space. Then construct a graph feature representation and use multi-source data to construct an additional transition graph, thereby enhancing the feature association and interaction between different data. Then use graph convolution operation to propagate information between nodes in the graph, and then perform graph feature fusion using graph dependency fusion to obtain the graph features after multi-source data fusion. Finally, the graph features are mapped back to the convolution space.

[0082] Step 4: Construct a global dependency fusion module. The convolutional features of multi-source remote sensing data obtained in Step 2 and the graph features obtained in Step 3 are sent to this fusion module for effective combination of features. The entire global dependency fusion consists of three cross-modal fusion modules. In each cross-modal fusion module, different types of features are combined in different ways, with attention weighting through the formed weight matrix, enhancing the fusion strength between features.

[0083] Step 5: Load the dataset, train the classification model, and use the cross-entropy loss function to minimize the loss for model training. After training, load the trained model, input the remote sensing satellite data of the target wetland area to be interpreted and classified, and obtain the final classification result through model prediction and output.

[0084] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and understanding of the methods of practicing the present application and are not intended to limit the scope of the present application in any way. Various modifications and alterations of this application can be made by those skilled in the art without departing from the scope and spirit of this application, which is measured by the appended claims and their equivalents.

Claims

1. A multi-source remote sensing satellite data intelligent interpretation and classification method, characterized in that, Comprising the following steps: Step 1: Constructing a spectral-spatial feature extraction network; Spectral and spatial features are extracted from hyperspectral and multispectral data respectively by using two convolutional neural networks; Step 2: Constructing a graph feature extraction model; A graph convolutional network is introduced to obtain the topological structure information hidden in the data, and a graph convolution operation is used to perform relationship learning to capture the topological structure information; a graph dependency fusion module is introduced to enhance the interaction between multi-source graph features; Comprising the following sub-steps: Step 2.1: Map the features extracted by the spectral-spatial feature extraction network in step 1 to a high-dimensional graph space; wherein the graph space is a matrix wherein each row represents a node, and each column represents an edge; the extracted features are mapped into the graph representation ; wherein C is the dimension of the feature vector, H and W are the height and width of the feature vector respectively, N represents the number of nodes in the graph, which corresponds to the number of classes in the scene, and D is the dimension of the nodes. Step 2.2: Constructing a transition graph to enhance feature association and interaction of hyperspectral and multispectral data; to represent hyperspectral image features, represent multispectral image features; Step 2.3: For a given graph feature representation, information between nodes is propagated by matrix multiplication using graph convolution to obtain an evolved feature space :​ Step 2.4: Graph dependency fusion is used to fuse the three types of graph features , and ; the three types of graph features are concatenated, and then the dimension of the concatenated features is restored to using a fully connected layer, thereby generating the fused graph features ; Step 2.5: Re-mapping back to convolution space using the inverse mapping function to get the feature This fused image feature representation is used to complement the classification representation of the spectral space features;​ Step 3: On the basis of obtaining the graph convolutional features and the spectral-spatial features, a global dependency fusion module is introduced to capture long-distance dependency relationships in the process of fully fusing various features; Step 4: The fused features are sent to a classifier, and a cross-entropy loss function is used to minimize errors to train the classification model; After training, the model is loaded, the remote sensing image of the target area to be interpreted is input, and the final classification result is obtained, and a visual classification map is output.

2. The intelligent interpretation and classification method of multi-source remote sensing satellite data according to claim 1, characterized in that, Step 1 comprises the following sub-steps: Hyperspectral and multispectral remote sensing data of the same region are taken as input, where the size of the hyperspectral image is , and the size of the multispectral image is , where represents the height of the image, represents the width of the image, and represent the number of channels, respectively; The size of the spatial block centered at position is The spatial block is extracted from the data and fed into the spectral-spatial feature extraction network to obtain the corresponding hyperspectral features and multispectral features ; the network consists of two two-dimensional convolutional neural network branches with three convolutional layers, which are parameterized using the training samples, followed by batch normalization BN and activation layers.

3. The method of claim 2, wherein the method further comprises: Network specific structure: ； wherein, is a convolution layer followed by batch normalization and activation layer, and represent the number of channels of the input and output data, respectively.

4. The intelligent interpretation and classification method of multi-source remote sensing satellite data according to claim 1, characterized in that, Transition diagram in step 2.2 is: ； wherein is a balance parameter, learned during training; is a weight matrix; is a fusion matrix, used to combine heterogeneous graph features; Step 2.3 Feature space : ； wherein and are weight matrices and bias, is a ReLU activation function, and is the number of graph convolution layers; In this process the node adjacency weight matrix to learn and capture the relationships between different classes; Features of the fusion in step 2.4 ; ； In the formula denotes a concatenation operation, is a fully connected layer.

5. The intelligent interpretation and classification method of multi-source remote sensing satellite data according to claim 1, characterized in that, Step 3 comprises the following sub-steps: Step 3.1 : Constructing the cross-modal fusion module ; Step 3.2: Global dependency fusion module The implementation of the global dependency fusion module; after the construction of the cross-modal fusion module, the hyperspectral features, multispectral features, and graph convolution features are respectively applied to this module, and the combination of the three obtains the final global dependency fusion module.

6. The intelligent interpretation and classification method of multi-source remote sensing satellite data according to claim 5, characterized in that, The cross-modal fusion module in step 3.1 : The definition is by Linear functions implemented by convolutions , , and are used to transform the shape of the input features from to where ; ； wherein , and are the inputs of the fusion module.

7. The intelligent interpretation and classification method of multi-source remote sensing satellite data according to claim 6, characterized in that, The global dependency fusion module in step 3.2 is: 。 8. The intelligent interpretation and classification method of multi-source remote sensing satellite data according to claim 1, characterized in that, Step 4 comprises the following sub-steps: Step 4.1: After obtaining the fusion features processed by the global dependency fusion module, the features are sent to a classifier to obtain a prediction result, and a cross-entropy loss function is used to obtain the corresponding loss, and the network parameters are optimized according to back propagation; after multiple rounds of training, the classification model is saved; Step 4.2: Collect and input the remote sensing image of the target area to be interpreted, use the deep learning classification model learned and saved in step 4.1 to predict the class label of each pixel point in the scene, obtain the final classification result, and visualize the classification map.

Images