Land feature classification method based on fusion of satellite remote sensing multispectral and panchromatic images

Abstract

The present invention relates to a method for classifying land features based on the fusion of satellite remote sensing multispectral and panchromatic images. The method specifically comprises the following steps: S1. Acquire multispectral and panchromatic images of a target area, extract EMAP spatial features from the multispectral image, and concatenate them with the multispectral image to obtain shallow spatial spectral features; S2. Use the shallow spatial spectral features as input to a deep network and the panchromatic image as the reconstructed output, and obtain multi-resolution hidden layer features through network training; S3. Extract low-resolution hidden layer features, medium-resolution hidden layer features, and high-resolution hidden layer features, respectively, and perform feature fusion and classification training through a convolutional neural network to obtain land feature classification results within the target area. Compared with the prior art, the present invention combines shallow features at the global scale of the image with deep features at the local scale, fully exploiting the multi-resolution hidden layer feature expression between MS and PAN images, and has the advantages of high classification accuracy and high robustness.

Description

Technical Field

[0001] The present invention relates to the field of feature-level fusion and classification of multi-source remote sensing images, and in particular to a ground feature classification method based on the fusion of satellite remote sensing multispectral and panchromatic images. Background Art

[0002] With the increasing diversity of remote sensing platforms and sensor types, a single sensor can now simultaneously acquire multispectral (MS) and panchromatic (PAN) images. Multispectral images typically contain four or more bands but suffer from low spatial resolution. Panchromatic images, on the other hand, have high spatial resolution but a single band. Effectively fusing multispectral images, rich in spectral information, with panchromatic images, which provide precise object boundary information, for high-precision identification of ground features holds enormous potential.

[0003] Currently, the main methods for fusing multispectral and panchromatic image data can be divided into: 1) pixel-level fusion; 2) feature-level fusion. The main method for pixel-level fusion is panchromatic sharpening, but this process introduces some spectral and spatial distortions, and the advantages of multispectral and panchromatic images are not fully utilized, especially the cross-resolution implicit features that are beneficial for feature recognition. Feature-level fusion extracts features from multispectral and panchromatic images separately, and then fuses these features for ground feature recognition, thus avoiding the shortcomings of pixel-level fusion. However, the main challenge is how to effectively extract significant features from multispectral and panchromatic images and build a fusion model. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defect of the above-mentioned existing technology that a single multispectral image or panchromatic image will lead to low recognition accuracy of land feature elements and to provide a land feature element classification method based on the fusion of satellite remote sensing multispectral and panchromatic images.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for ground feature classification based on the fusion of satellite remote sensing multispectral and panchromatic images specifically includes the following steps:

[0007] S1. Obtain multispectral and panchromatic images of the target area collected by satellite remote sensing, extract the extended multi-attribute profiles (EMAP) spatial features of the multispectral images, and concatenate the EMAP features with the original multispectral images to obtain the global shallow spatial spectrum features.

[0008] S2. The shallow spatial spectrum features are used as the input of a deep network similar to an autoencoder, and the full color image is used as the reconstructed output of the deep network similar to an autoencoder, and multi-resolution hidden layer features between the input and output are obtained through network training;

[0009] S3. Extract the second hidden layer features, the fifth hidden layer features and the eighth hidden layer features from the multi-resolution hidden layer features as low-resolution hidden layer features, medium-resolution hidden layer features and high-resolution hidden layer features, respectively, and perform feature fusion and classification training on the low-resolution hidden layer features, medium-resolution hidden layer features and high-resolution hidden layer features through a convolutional neural network to obtain the classification results of the land features in the target area.

[0010] In step S2, both the input and output are patches. The size of the input patch is R×R×d, and the size of the output patch is nR×nR×1, where R is the side length of the patch and n is the multiple of the resolution of the panchromatic image relative to the multispectral image.

[0011] The low-resolution hidden layer features, medium-resolution hidden layer features and high-resolution hidden layer features are used as fusion features of the multispectral image and the panchromatic image.

[0012] The resolution of the second hidden layer feature is low resolution, which is the same as the resolution of the multispectral image; the resolution of the fifth hidden layer feature is medium resolution, which is between the resolutions of the multispectral image and the panchromatic image; the resolution of the eighth hidden layer feature is high resolution, which is the same as the resolution of the panchromatic image.

[0013] The size of the multispectral image is H×W×c, where H, W and c are the height, width and number of bands of the multispectral image respectively.

[0014] Furthermore, the dimension of the extended multi-attribute profile space feature is H×W×d, where d is the dimension of the extended multi-attribute profile space feature.

[0015] The extended multi-attribute profile spatial feature extraction uses four attributes: regional area, regional border diagonal length, moment of inertia, and standard deviation of pixel grayscale values ​​within the region. Appropriate parameters are selected for each attribute based on actual data.

[0016] The network structure similar to the autoencoder deep network includes, in sequence, an input layer (Input), a convolution block 1 (Conv_Block1), a convolution block 1, an upsampling layer (Upsampling), a convolution block 1, a convolution block 1, an upsampling layer, a convolution block 2 (Conv_Block2), a convolution block 2, a total convolution layer (Conv), and an output layer (Output). The upsampling factor of the upsampling layer is 2×2. The optimizer used in network training is Adam, the learning rate is 0.0001, and the loss function is the MSE loss function.

[0017] Furthermore, the convolution block 1 includes a convolution layer, a batch normalization layer and an activation layer, and the convolution block 2 includes a convolution layer, a batch normalization layer and an activation layer.

[0018] Furthermore, the convolution kernel size of the convolution layer in the convolution block 1 is 3×3, and the number of filters is 128; the convolution kernel size of the convolution layer in the convolution block 2 is 5×5, and the number of filters is 128; the convolution kernel size of the total convolution layer is 3×3, and the number of filters is 1.

[0019] The convolutional neural network corresponding to the low-resolution hidden layer features includes convolution block 3 (Conv_Block3), convolution block 4 (Conv_Block4), convolution block 5 (Conv_Block5), global maximum pooling layer (GlobalMaxPool), fully connected layer (Dense) and Softmax classifier in sequence; the convolutional neural network corresponding to the medium-resolution hidden layer features includes convolution block 3, maximum pooling layer (MaxPooling), convolution block 4, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence; the convolutional neural network corresponding to the high-resolution hidden layer features includes convolution block 3, maximum pooling layer, convolution block 4, maximum pooling layer, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence.

[0020] Furthermore, the optimizer used in the single-branch convolutional neural network training is SGD (Stochastic Gradient Descent), the learning rate is 0.001, and the loss function is the categorical_crossentropy loss function.

[0021] Furthermore, the convolution block 3 includes a convolution layer, a batch normalization layer and an activation layer, the convolution kernel size of the convolution layer is 3×3, and the number of filters is 128; the convolution block 4 includes a convolution layer, a batch normalization layer and an activation layer, the convolution kernel size of the convolution layer is 3×3, and the number of filters is 256; the convolution block 5 includes a convolution layer, a batch normalization layer and an activation layer, the convolution kernel size of the convolution layer is 3×3, and the number of filters is 512.

[0022] Furthermore, step S3 also includes removing all layers after the global maximum pooling layer in the three-branch convolutional neural network corresponding to the low-resolution hidden layer features, the medium-resolution hidden layer features and the high-resolution hidden layer features, cascading the global maximum pooling layer of the three-branch convolutional neural network, and fixing the network weights, and then connecting the fully connected layer, the fully connected layer and the Softmax classifier in sequence to perform fine-tuning fusion classification training and prediction. The optimizer used in the network training is SGD, the learning rate is 0.0001, and the loss function is the categorical_crossentropy loss function.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. Before the autoencoder deep network cuts the image patches, the present invention extracts the global shallow spatial features of the multispectral image to make up for the disadvantage that the complete objects may be separated during the image patch cutting process, but the spatial connectivity and consistency of the objects are destroyed due to the internal inconsistency of the objects. It realizes the combination of shallow and deep, global and local features, thereby improving the classification performance of the deep network.

[0025] 2. The present invention eliminates the differences in spatial and spectral resolution information between multispectral and panchromatic images through a deep network similar to an autoencoder, extracts multi-resolution hidden features between multispectral and panchromatic images in an unsupervised manner, and fuses and classifies the hidden features through a multi-branch convolutional neural network, thereby improving the accuracy of ground object classification.

[0026] 3. The present invention directly uses the original multispectral image for operation, rather than upsampling the multispectral image to the same resolution as the panchromatic image in advance, reducing the error and calculation time caused by upsampling interpolation; in addition, the present invention adopts a combination of unsupervised features and supervised features, making the classification results more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the process of the present invention;

[0028] Figure 2 Schematic diagram of the structure of the neural network of the present invention;

[0029] Figure 3 is a true color composite image of a multispectral image in an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of a full-color image according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of a ground truth reference image in an embodiment of the present invention;

[0032] Figure 6 Schematic diagram of the classification results of the SVM (Support Vector Machine) method on MS*PAN in an embodiment of the present invention, where MS*PAN is a pan-sharpened image of multispectral image and panchromatic image data;

[0033] Figure 7 Schematic diagram of the classification results of the RF (Random Forest) method on MS*PAN in an embodiment of the present invention;

[0034] Figure 8 Schematic diagram of the classification results of the VGG-Like method on MS*PAN in an embodiment of the present invention;

[0035] Figure 9 Schematic diagram of the classification results of the SVM method on MS+PAN in an embodiment of the present invention, where MS+PAN is the result of concatenating multispectral image and panchromatic image data;

[0036] Figure 10 Schematic diagram of the classification result of the RF method on MS+PAN in an embodiment of the present invention;

[0037] Figure 11 Schematic diagram of the classification results of the VGG-Like method on MS+PAN in an embodiment of the present invention;

[0038] Figure 12 Schematic diagram of the classification results of the DMIL (Deep Multiple Instance Learning) method in an embodiment of the present invention;

[0039] Figure 13 Schematic diagram of the classification results of the MultiResoLCC (Multi-Resolution Land Cover Classification) method in an embodiment of the present invention;

[0040] Figure 14 Schematic diagram of the classification results of the method proposed in the present invention on MSemap_PAN in an embodiment of the present invention, where MSemap_PAN is the feature-level fusion of EMAP features extracted from multispectral images and panchromatic images. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0042] Example

[0043] like Figure 1 As shown in FIG, a ground feature classification method based on the fusion of satellite remote sensing multispectral and panchromatic images includes the following steps:

[0044] S1. Obtain multispectral and panchromatic images of the target area collected by satellite remote sensing, extract the extended multi-attribute profiles (EMAP) spatial features of the multispectral images, and concatenate the EMAP features with the original multispectral images to obtain the global shallow spatial spectrum features.

[0045] S2: The shallow spatial spectrum features are used as the input of a deep network similar to an autoencoder, and the full color image is used as the reconstructed output of a deep network similar to an autoencoder. The multi-resolution hidden layer features between the input and output are obtained through network training.

[0046] S3. Extract the second hidden layer features, the fifth hidden layer features and the eighth hidden layer features from the multi-resolution hidden layer features as low-resolution hidden layer features, medium-resolution hidden layer features and high-resolution hidden layer features respectively, and perform feature fusion and classification training on the low-resolution hidden layer features, medium-resolution hidden layer features and high-resolution hidden layer features respectively through convolutional neural networks to obtain the classification results of the ground features in the target area.

[0047] In step S2, both the input and output are patches. The input patch size is R×R×d, and the output patch size is nR×nR×1, where R is the side length of the patch and n is the multiple of the resolution of the panchromatic image relative to the multispectral image.

[0048] Low-resolution hidden layer features, medium-resolution hidden layer features and high-resolution hidden layer features are used as fusion features of multispectral images and panchromatic images.

[0049] The resolution of the second hidden layer feature is low resolution, which is the same as the resolution of the multispectral image; the resolution of the fifth hidden layer feature is medium resolution, which is between the resolution of the multispectral image and the panchromatic image; the resolution of the eighth hidden layer feature is high resolution, which is the same as the resolution of the panchromatic image.

[0050] The size of the multispectral image is H×W×c, where H, W, and c are the height, width, and number of bands of the multispectral image, respectively.

[0051] The dimension of the extended multi-attribute profile space feature is H×W×d, where d is the dimension of the extended multi-attribute profile space feature.

[0052] The extended multi-attribute profile spatial feature extraction uses four attributes: regional area, regional border diagonal length, moment of inertia, and standard deviation of pixel grayscale values ​​within the region. Appropriate parameters are selected for each attribute based on actual data.

[0053] like Figure 2 As shown in the figure, the network structure similar to the autoencoder deep network includes the input layer (Input), convolution block 1 (Conv_Block1), convolution block 1, upsampling layer (Upsampling), convolution block 1, convolution block 1, upsampling layer, convolution block 2 (Conv_Block2), convolution block 2, total convolution layer (Conv), and output layer (Output). The upsampling factor of the upsampling layer is 2×2. The optimizer used in the network training is Adam, the learning rate is 0.0001, and the loss function is the MSE loss function.

[0054] Convolutional block 1 includes convolutional layers, batch normalization layers, and activation layers, and convolutional block 2 includes convolutional layers, batch normalization layers, and activation layers.

[0055] The convolution kernel size of the convolution layer in convolution block 1 is 3×3, and the number of filters is 128; the convolution kernel size of the convolution layer in convolution block 2 is 5×5, and the number of filters is 128; the convolution kernel size of the total convolution layer is 3×3, and the number of filters is 1.

[0056] The convolutional neural network corresponding to the low-resolution hidden layer features includes convolution block 3 (Conv_Block3), convolution block 4 (Conv_Block4), convolution block 5 (Conv_Block5), global maximum pooling layer (GlobalMaxPool), fully connected layer (Dense) and Softmax classifier in sequence; the convolutional neural network corresponding to the medium-resolution hidden layer features includes convolution block 3, maximum pooling layer (MaxPooling), convolution block 4, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence; the convolutional neural network corresponding to the high-resolution hidden layer features includes convolution block 3, maximum pooling layer, convolution block 4, maximum pooling layer, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence.

[0057] The optimizer used in the single-branch convolutional neural network training is SGD (Stochastic Gradient Descent), the learning rate is 0.001, and the loss function is the categorical_crossentropy loss function.

[0058] Convolutional block 3 includes a convolutional layer, a batch normalization layer, and an activation layer. The convolution kernel size of the convolutional layer is 3×3, and the number of filters is 128. Convolutional block 4 includes a convolutional layer, a batch normalization layer, and an activation layer. The convolution kernel size of the convolutional layer is 3×3, and the number of filters is 256. Convolutional block 5 includes a convolutional layer, a batch normalization layer, and an activation layer. The convolution kernel size of the convolutional layer is 3×3, and the number of filters is 512.

[0059] Step S3 also includes removing all layers after the global maximum pooling layer in the three-branch convolutional neural network corresponding to the low-resolution hidden layer features, the medium-resolution hidden layer features, and the high-resolution hidden layer features, cascading the global maximum pooling layer of the three-branch convolutional neural network, and fixing the network weights, followed by connecting the fully connected layer, the fully connected layer and the Softmax classifier in sequence for fine-tuning fusion classification training and prediction. The optimizer used in the network training is SGD, the learning rate is 0.0001, and the loss function is the categorical_crossentropy loss function.

[0060] During the specific implementation, the Gaofen-2 remote sensing image obtained in a certain area in January 2015 was selected. Figures 3 to 5 The dataset includes true color composite images of MS data, PAN images, and ground truth reference images. The dataset contains five surface feature categories: buildings, roads, water bodies, trees, and grass. Table 1 provides detailed information on the reference samples for each surface feature.

[0061] Table 1. Introduction to the total number of samples and experimental training and test samples

[0062]

[0063] In order to compare the performance of different methods, the overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa) are selected as evaluation indicators.

[0064] like Figures 6 to 14 As shown in Table 2, the average results of 10 evaluations of different methods are given, and each method uses the same training sample. In order to compare the robustness of different methods, the standard deviation of the 10 results is given. The value after "±" in Table 2 represents the standard deviation. It can be seen that the method proposed in this invention has the smallest standard deviation and is more robust. The details of Table 2 are as follows:

[0065] Table 2 Accuracy evaluation results of different methods

[0066]

[0067] The total accuracy of the proposed method MRHFF (MSemap_PAN) is 98.12%, which is 4.53%, 9.33%, and 2.56% higher than the deep learning methods VGG-Like (MS+PAN), DMIL, and MultiResoLCC, respectively, and 15.83% and 12.85% higher than the traditional machine learning methods SVM (MS+PAN) and RF (MS+PAN), respectively.

[0068] The hidden layer features of the method proposed in the present invention have different resolutions and can focus on different aspects of the object category. For example, in low-resolution features, the interior of the same object is more uniform, which is more conducive to the internal category judgment of the object. In high-resolution features, the boundary of the object is more precise, which can ensure the classification accuracy of the object boundary. Figures 6 to 14The classification diagrams for different methods are presented, along with the accuracy closest to the overall classification accuracy among the nine sub-accuracies. It can be seen that the traditional machine learning models SVM and RF perform poorly, with many misclassifications and a significant salt-and-pepper phenomenon. The proposed method (MS_PAN) also performs better than currently used deep learning methods. The classification diagram shows that the proposed method produces relatively few misclassifications of buildings and roads, and also provides relatively accurate classification of interiors and boundaries.

[0069] In summary, the results of a series of qualitative and quantitative experimental analyses show that the unsupervised multi-resolution hidden layer feature fusion and classification method for MS and PAN data proposed in this invention can achieve higher-precision high-resolution surface feature classification compared with other methods, and has obvious advantages in accurately depicting the geometric boundaries of surface elements and the uniformity of internal spectra, and is highly robust.

[0070] In addition, it should be noted that the specific embodiments described in this specification may be named differently, and the above content described in this specification is merely an example of the structure of the present invention. Any equivalent changes or simple changes made based on the structure, features and principles of the present invention are included in the protection scope of the present invention. Those skilled in the art of the present invention may make various modifications or supplements to the specific examples described or adopt similar methods, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for ground feature classification based on the fusion of satellite remote sensing multispectral and panchromatic images, characterized in that: The specific steps include: S1. Obtain multispectral and panchromatic images of the target area collected by satellite remote sensing, extract the extended multi-attribute profile spatial features of the multispectral image, and cascade the extended multi-attribute profile spatial features with the original multispectral image to obtain the global shallow spatial spectrum features; S2. The shallow spatial spectrum features are used as the input of a deep network similar to an autoencoder, and the full color image is used as the reconstructed output of the deep network similar to an autoencoder, and multi-resolution hidden layer features between the input and output are obtained through network training; S3, respectively extracting the second hidden layer features, the fifth hidden layer features, and the eighth hidden layer features from the multi-resolution hidden layer features as low-resolution hidden layer features, medium-resolution hidden layer features, and high-resolution hidden layer features, and performing feature fusion and classification training on the low-resolution hidden layer features, the medium-resolution hidden layer features, and the high-resolution hidden layer features through a convolutional neural network to obtain a classification result of ground features in the target area; The network structure similar to the autoencoder deep network includes, in sequence, an input layer, a convolution block 1, a convolution block 1, an upsampling layer, a convolution block 1, a convolution block 1, an upsampling layer, a convolution block 2, a convolution block 2, a total convolution layer, and an output layer; The convolution block 1 includes a convolution layer, a batch normalization layer and an activation layer, and the convolution block 2 includes a convolution layer, a batch normalization layer and an activation layer; The convolution kernel size of the convolution layer in the convolution block 1 is 3×3, and the number of filters is 128; the convolution kernel size of the convolution layer in the convolution block 2 is 5×5, and the number of filters is 128; the convolution kernel size of the total convolution layer is 3×3, and the number of filters is 1.

2. The method for ground feature classification based on satellite remote sensing multispectral and panchromatic image fusion according to claim 1, characterized in that: The size of the multispectral image is H×W×c, where H, W and c are the height, width and number of bands of the multispectral image respectively.

3. The method for ground feature classification based on satellite remote sensing multispectral and panchromatic image fusion according to claim 2, characterized in that: The extended multi-attribute profile spatial feature extraction uses four attributes: regional area, regional border diagonal length, moment of inertia, and standard deviation of pixel grayscale values ​​within the region.

4. The method for ground feature classification based on satellite remote sensing multispectral and panchromatic image fusion according to claim 1, characterized in that: The extended multi-attribute profile spatial features are extracted based on the attributes of the target region, where the attributes of the target region include the region area, the diagonal length of the region border, the moment of inertia, and the standard deviation of the grayscale values ​​of pixels within the region.

5. The method for ground feature classification based on satellite remote sensing multispectral and panchromatic image fusion according to claim 1, characterized in that: The convolutional neural network corresponding to the low-resolution hidden layer features includes convolution block 3, convolution block 4, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence; the convolutional neural network corresponding to the medium-resolution hidden layer features includes convolution block 3, maximum pooling layer, convolution block 4, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence; the convolutional neural network corresponding to the high-resolution hidden layer features includes convolution block 3, maximum pooling layer, convolution block 4, maximum pooling layer, convolution block 5, global maximum pooling layer, fully connected layer and Softmax classifier in sequence.

6. The method for ground feature classification based on satellite remote sensing multispectral and panchromatic image fusion according to claim 5, characterized in that: The convolution block 3 includes a convolution layer, a batch normalization layer and an activation layer, the convolution kernel size of the convolution layer is 3×3, and the number of filters is 128; the convolution block 4 includes a convolution layer, a batch normalization layer and an activation layer, the convolution kernel size of the convolution layer is 3×3, and the number of filters is 256; the convolution block 5 includes a convolution layer, a batch normalization layer and an activation layer, the convolution kernel size of the convolution layer is 3×3, and the number of filters is 512.

7. The method for ground feature classification based on satellite remote sensing multispectral and panchromatic image fusion according to claim 5, characterized in that: The step S3 also includes removing all layers after the global maximum pooling layer in the three-branch convolutional neural network corresponding to the low-resolution hidden layer features, the medium-resolution hidden layer features and the high-resolution hidden layer features, cascading the global maximum pooling layers of the three branches, fixing the network weights, and then sequentially connecting the fully connected layer, the fully connected layer and the Softmax classifier to perform fine-tuning fusion classification training and prediction.

Images