Spatial neighborhood-based multi-scale spectral feature fusion classification method for domestic satellite hyperspectral data

Abstract

The invention provides a domestic satellite hyperspectral data-oriented multi-scale spectral feature fusion classification method based on a spatial neighborhood, and the method comprises the following specific operation steps: firstly, carrying out the cutting of processed satellite data in a spatial dimension, and selecting a label of a central pixel as a real label of a corresponding image block; input data firstly passes through the multi-scale spectral feature extraction module. The extracted spectral features are transmitted to a spatial domain information fusion module, and spatial neighborhood information and spectral information are further extracted and fused in the spatial domain information fusion module. And inputting the fused features into a classification head to obtain a classification result. According to the method, a preliminary effect is achieved on satellite hyperspectral data. The method can effectively reduce the adverse effects caused by the problems of spectrum mixing and spatial errors caused by low spatial resolution of the satellite, practically and effectively improves the precision level of pixel-level classification, and provides an efficient and accurate solution for domestic hyperspectral satellite data classification.

Description

Technical Field

[0001] The invention relates to the technical field of hyperspectral remote sensing, and in particular to a classification method for domestic satellite hyperspectral data based on multi-scale spectral feature fusion in spatial neighborhood. Background Art

[0002] Hyperspectral Sensing Imagery (HSI) is different from traditional color images. It records the reflection or radiation information on hundreds of continuous spectral bands. Each band corresponds to a different wavelength range of objects on the surface, and has a high spectral resolution, including visible light, infrared light, and ultraviolet light. This enables objects to draw almost continuous spectral curves through the imaging information of hyperspectral images, thereby characterizing their essential characteristics. Therefore, hyperspectral images can distinguish some special objects that cannot be recognized by traditional images, such as minerals and vegetation. Hyperspectral image classification is the basis for hyperspectral object recognition, and its related methods have been applied to many practical fields, such as military target detection, mineral exploration, and agricultural production.

[0003] The Resource-1 02E (ZY1F) satellite was launched on December 26, 2021. It is a medium-to-low resolution hyperspectral satellite with a high spectral resolution and a revisit period of only two days. Its performance is sufficient to support the needs of a variety of downstream tasks. Its hyperspectral sensor has a spatial resolution of 30 meters and a total of 166 bands. It has a spectral resolution of 10nm and 20nm in the visible near-infrared band and short-wave infrared, respectively, and can obtain a width of 60 kilometers. The satellite operates in a sun-synchronous orbit, which can realize all-day earth observation and has broad application prospects. However, as the latest hyperspectral satellite launched, Resource-1 has a spatial resolution of 30m, which can adapt to some large-scale tasks, but the low spatial resolution leads to more mixed pixels. At the same time, it is more likely to produce spatial errors when the image is corrected, which causes the spectra between adjacent pixels to affect each other, and ultimately affects the classification results. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a classification method based on multi-scale spectral feature fusion of spatial neighborhood for domestic satellite hyperspectral data, which can effectively extract the spectral dimension of hyperspectral data, and at the same time, by considering the spectral information of the neighborhood pixels around the central pixel as a supplement, the weight of each pixel is adaptively assigned, thereby reducing the impact of spectral mixing and spatial errors and improving the classification accuracy.

[0005] To achieve the above purpose, the present invention adopts the following technical solution: a classification method based on multi-scale spectral feature fusion of spatial neighborhood for domestic satellite hyperspectral data, which specifically includes the following steps:

[0006] Step S1: For the hyperspectral satellite image data obtained by the satellite, preliminary processing operations of water vapor noise removal and geometric correction are carried out; after completing the above preliminary processing operations, the data is cut into blocks of 3×3 in the spatial dimension, and the label of the central pixel is set as the real label of the image block, which is used as the input data of step S2;

[0007] Step S2: design a multi-scale spectral feature extraction module to obtain feature data;

[0008] Step S3: Based on the feature data obtained in step S2, a spatial neighborhood feature fusion operation is performed;

[0009] Step S4: Convert the classification features fused in step S3 into a one-dimensional vector and send them to two classification heads consisting of a linear layer and an activation layer for classification, and finally obtain an accurate classification result.

[0010] In a preferred embodiment: the step S2 first performs a multi-scale spectral feature extraction task on the input image block; in the initial stage, a 3D convolution layer with a spectral dimension step size of 2 is used for extraction to refine and reduce the spectral features to 1 / 2 of the original, and at the same time, the number of 3D feature image blocks is expanded to 24, so as to obtain a diverse spectral feature representation; then, further extraction operations are carried out through three groups of convolution layers of different sizes; finally, the features of the three scales are added and sent to the last 3D convolution layer with a convolution kernel size equal to the number of bands of the feature image block, so as to globally integrate all bands and obtain a more comprehensive feature representation, and step S2 is repeated twice, and then the obtained feature data is sent to step S3.

[0011] In a preferred embodiment: in the step S3, first, adaptive weighting processing is performed on each pixel through an attention module, and then the weighted data is input into two 3D convolutional layers, and the feature image blocks are increased to 16 to improve the diversity of feature extraction; then, the feature image block dimension and the spectral dimension are reshaped into a new 2D feature dimension through a data reshaping operation, so as to effectively blend the spatial-spectral features; then, a 1×1 dot product operation is used to reduce the size of the feature dimension to refine complex features and reduce the amount of calculation; finally, all features are fused through two 2D convolutional layers to obtain features that can be used for classification.

[0012] In a preferred embodiment: in the step S3, an additional 1×1 dot product operation is added after the 3D convolution, and the padding operation is removed in the last layer of convolution, so that the spatial and spectral features are mapped into a one-dimensional vector, thereby achieving efficient fusion of spatial and spectral information.

[0013] Compared with the prior art, the present invention has the following beneficial effects: the present invention can effectively deal with the characteristics of high spectral dimension and high redundant information of hyperspectral images, and at the same time consider the spatial neighborhood information, which can effectively reduce the influence of spectral confusion and spatial error under low spatial resolution. In particular, it has been initially applied to the hyperspectral data of the domestic satellite resource No. 1 02E satellite, and has higher classification accuracy than other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the overall principle architecture of the preferred embodiment of the present invention.

[0015] Figure 2 It is a schematic diagram of the principle of multi-scale spectral feature extraction in a preferred embodiment of the present invention.

[0016] Figure 3 It is a schematic diagram of the spatial neighborhood feature fusion principle of a preferred embodiment of the present invention.

[0017] Figure 4 It is a schematic diagram of the principle of the spatial attention module used in the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0020] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0021] like Figure 1 , 2 As shown in , 3, and 4, this embodiment provides a hyperspectral classification method based on multi-scale spectral feature extraction of spatial neighborhood for domestic satellite data, taking the Resource-1 02E hyperspectral data in the domestic satellite as an example, including the following steps:

[0022] Step S1: For the hyperspectral satellite image data obtained by Resource 102E, the preliminary processing operations of water vapor noise removal and geometric correction are first carried out. After completing the above processing, the data is cut into blocks of 3×3 in the spatial dimension, and the label of the central pixel is set as the real label of the image block, which is used as the input data of step S2.

[0023] Step S2: In this step, the multi-scale spectral feature extraction task is first performed on the input image block. In order to effectively reduce the redundancy of adjacent bands, a 3D convolution layer with a spectral dimension step size of 2 is used for extraction in the initial stage, which can refine and reduce the spectral features to 1 / 2 of the original, and expand the number of 3D feature image blocks to 24, so as to obtain a diverse spectral feature representation. Subsequently, further extraction operations are carried out through three groups of convolution layers of different sizes. Finally, the features of the three scales are added and sent to the last 3D convolution layer with a convolution kernel size equal to the number of bands of the feature image block, so as to globally integrate all bands and obtain a more comprehensive feature representation. Repeat step S2 twice to ensure that the spectral features are fully extracted, and then send them to the next step.

[0024] Step S3: Based on the data obtained in step S2, a spatial neighborhood feature fusion operation is performed. First, an attention module is used to perform adaptive weighting processing on each pixel, and then the weighted data is input into two 3D convolutional layers to increase the feature image blocks to 16 to improve the diversity of feature extraction. Next, the feature image block dimension and the spectral dimension are reshaped into a new 2D feature dimension through a data reshaping operation, thereby effectively blending the spatial-spectral features. After that, a 1×1 dot product operation is used to reduce the size of the feature dimension to refine complex features and reduce the amount of calculation. Finally, all features are fused through two 2D convolutional layers to obtain features that can be used for classification.

[0025] Step S4: Convert the classification features fused in step S3 into a one-dimensional vector and send them to two classification heads consisting of a linear layer and an activation layer for classification, and finally obtain an accurate classification result.

[0026] In this embodiment, the spatial block size of step S1 is fixed to 3×3, which is designed based on the spatial resolution and image characteristics of the Resource No. 1 02E satellite image. Compared with the method of directly using the spectral classification of pixel points, this method can consider the spectral characteristics of other pixels in the spatial neighborhood, and will not be affected by the redundant information of other distant pixels due to the large block size. Using the spectral information of neighboring pixels as a supplement to reduce the influence of spectral mixing

[0027] In this embodiment, a spatial attention mechanism is added in step S3 to adaptively assign weights to each pixel. The weight of each pixel can be adaptively adjusted to highlight the contribution of the central pixel and adapt to the spatial deviation of the edge of the classified object to obtain a more accurate classification result.

[0028] In this embodiment, in step S3, 3D convolution is used to extract and expand spatial spectrum information, 2D dot product is used to reduce the refined features and reduce the amount of calculation, and the subsequent 2D convolution is used to integrate the spatial spectrum features. Except for the last layer of 2D convolution operation, the remaining convolution layers use padding operation in space, which can not change the original spatial size of the data, and fully integrate the spectral information of the extracted neighborhood pixels. The last time the padding is discarded, the spatial-spectral information can be finally integrated into a one-dimensional vector.

[0029] Preferably, the multi-scale spectral feature extraction module in step S2 of this example can extract spectral features through multiple 3D convolution layers of different sizes, and select convolution kernel features of different sizes for extraction locally according to the spectral curve representation of different ground objects, which can cope with the characteristics of high spectral dimension and more redundant information in hyperspectral data.

[0030] The specific steps include:

[0031] Preferably, in this embodiment, the hyperspectral classification method based on multi-scale spectral feature extraction of spatial neighborhood for the Resource-1 02E satellite data can effectively improve the classification accuracy of the Resource-1 02E satellite hyperspectral data.

[0032] Step S1: For the hyperspectral satellite image data obtained by Resource 102E, the preliminary processing operations of water vapor noise removal and geometric correction are first carried out. After completing the above processing, the data is cut into blocks of 3×3 in the spatial dimension, and the label of the central pixel is set as the real label of the image block, which is used as the input data of step S2.

[0033] Step S2: In this step, the multi-scale spectral feature extraction task is first performed on the input image block. In order to effectively reduce the redundancy of adjacent bands, a 3D convolution layer with a spectral dimension step size of 2 is used for extraction in the initial stage, which can refine and reduce the spectral features to 1 / 2 of the original, and expand the number of 3D feature image blocks to 24, so as to obtain a diverse spectral feature representation. Subsequently, further extraction operations are carried out through three groups of convolution layers of different sizes. Finally, the features of the three scales are added and sent to the last 3D convolution layer with a convolution kernel size equal to the number of bands of the feature image block, so as to globally integrate all bands and obtain a more comprehensive feature representation. Repeat step S2 twice to ensure that the spectral features are fully extracted, and then send them to the next step.

[0034] Step S3: Based on the data obtained in step S2, a spatial neighborhood feature fusion operation is performed. First, an attention module is used to perform adaptive weighting processing on each pixel, and then the weighted data is input into two 3D convolutional layers to increase the feature image blocks to 16 to improve the diversity of feature extraction. Next, the feature image block dimension and the spectral dimension are reshaped into a new 2D feature dimension through a data reshaping operation, thereby effectively blending the spatial-spectral features. After that, a 1×1 dot product operation is used to reduce the size of the feature dimension to refine complex features and reduce the amount of calculation. Finally, all features are fused through two 2D convolutional layers to obtain features that can be used for classification.

[0035] Step S4: Convert the classification features fused in step S3 into a one-dimensional vector and send them to two classification heads consisting of a linear layer and an activation layer for classification, and finally obtain an accurate classification result.

[0036] The present invention has the following beneficial effects: the present invention can effectively deal with the characteristics of high spectral dimension and much redundant information of domestic satellite hyperspectral images, and at the same time considers the spatial neighborhood information, and can effectively reduce the influence of spectral confusion and spatial error under low spatial resolution. In particular, it has been initially applied to the hyperspectral data of Resource-1 02E satellite, and has higher classification accuracy than other methods.

[0037] In summary, the characteristics of low spatial resolution and high spectral resolution of domestic satellite hyperspectral image data. It is easy to have problems such as insufficient spectral feature extraction, pixel spectral mixing and spatial error. The hyperspectral classification method based on multi-scale spectral feature extraction of spatial neighborhood for domestic satellite satellite data proposed in this embodiment. The present invention first spatially blocks the processed hyperspectral data, and then sends it to the multi-scale spectral feature extraction module, extracts the spectral information through multi-scale convolution, and then sends the extracted spectral features to the spatial neighborhood feature fusion module. The module uses a combination of 3D convolution and 2D convolution to extract the spectral information of the center and the neighborhood at the same time, and adds a spatial attention mechanism, which can adaptively adjust the weight of each pixel, and finally obtain the classification feature, which is sent to the MLP classifier to obtain the classification result. The experimental data is the data of Resource No. 1 02E in two experimental areas of Pucheng County and Zhenghe County in Fujian Province to extract the rice planting area, and it has been verified on the public vegetation data sets Indian Pines and Salinas. The present invention extracts spectral features through multi-scale convolution and considers the spectral information in the spatial neighborhood at the same time. It can effectively deal with the characteristics of high spectral dimension and high redundant information of hyperspectral images and reduce the influence of spectral confusion and spatial error under low spatial resolution. Compared with existing methods, the present method can obtain higher classification results under the same experimental conditions.

[0038] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A classification method based on multi-scale spectral feature fusion of spatial neighborhood for domestic satellite hyperspectral data, characterized in that: The specific steps include: Step S1: For the hyperspectral satellite image data obtained by the satellite, preliminary processing operations of water vapor noise removal and geometric correction are carried out; after completing the above preliminary processing operations, the data is cut into blocks of 3×3 in the spatial dimension, and the label of the central pixel is set as the real label of the image block, which is used as the input data of step S2; Step S2: design a multi-scale spectral feature extraction module to obtain feature data; Step S3: Based on the feature data obtained in step S2, a spatial neighborhood feature fusion operation is performed; Step S4: Convert the classification features fused in step S3 into a one-dimensional vector and send them to two classification heads consisting of a linear layer and an activation layer for classification, and finally obtain an accurate classification result.

2. The classification method of domestic satellite hyperspectral data based on multi-scale spectral feature fusion in spatial neighborhood according to claim 1 is characterized by: The step S2 first performs a multi-scale spectral feature extraction task on the input image block; In the initial stage, a 3D convolutional layer with a spectral dimension step size of 2 is used to extract the spectral features and reduce them to 1 / 2 of the original size. At the same time, the number of 3D feature image blocks is expanded to 24 to obtain a diverse spectral feature representation. Subsequently, further extraction operations are performed through three groups of convolutional layers of different sizes; finally, the features of the three scales are added and sent to the last 3D convolutional layer with a convolution kernel size equal to the number of bands of the feature image block, so as to globally integrate all bands and obtain a more comprehensive feature representation. Step S2 is repeated twice, and then the obtained feature data is sent to step S3.

3. The classification method based on multi-scale spectral feature fusion of spatial neighborhood for domestic satellite hyperspectral data according to claim 1 is characterized by: In step S3, first, adaptive weighting processing is performed on each pixel through an attention module, and then the weighted data is input into two 3D convolutional layers, and the feature image blocks are increased to 16 to improve the diversity of feature extraction; then, the feature image block dimension and the spectral dimension are reshaped into a new 2D feature dimension through a data reshaping operation, so as to effectively blend the spatial-spectral features; then, a 1×1 dot product operation is used to reduce the size of the feature dimension to refine complex features and reduce the amount of calculation; finally, all features are fused through two 2D convolutional layers to obtain features that can be used for classification.

4. The classification method based on multi-scale spectral feature fusion based on spatial neighborhood for domestic satellite hyperspectral data according to claim 1 is characterized by: In step S3, an additional 1×1 dot product operation is added after the 3D convolution, and the padding operation is removed in the last layer of convolution, so that the spatial and spectral features are mapped into a one-dimensional vector, thereby achieving efficient fusion of spatial and spectral information.

Images