A hyperspectral image classification method for complex environments

Abstract

The present application relates to the technical field of hyperspectral remote sensing image processing and classification, and discloses a hyperspectral image classification method for complex environment; the present application is based on a multiscale attention mechanism reconstruction network, which combines an information redistribution module, a local and global attention mechanism and a residual connection, and through multi-level information redistribution, feature selection and multiscale fusion, effectively improves the accuracy and robustness in the image classification task. The network can automatically learn important features and perform information fusion at global and local levels, so that it has better expression ability and generalization ability when processing complex image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hyperspectral remote sensing image processing and classification, and particularly relates to a hyperspectral image classification method for complex environment. BACKGROUND

[0002] At present, the Yellow River wetland is an important ecological protection area in China, and has extremely rich biodiversity and ecological system service functions. However, with the intensification of human activities and climate change, the ecological environment of the wetland is facing increasingly severe challenges, and the protection and monitoring of the wetland have become an important topic in ecological research; the hyperspectral remote sensing technology can accurately classify different ground object types by providing continuous spectral band information, and provides strong support for the ecological protection and resource management of the wetland; the airborne hyperspectral remote sensing equipment carried by the unmanned aerial vehicle can realize observation with high spatial resolution and high spectral resolution, and still obtain remote sensing images of a large range after effective image splicing.

[0003] The existing hyperspectral image classification technology has the following significant defects in complex environment application: first, the existing hyperspectral image classification method usually adopts single-scale convolution operation or uses some basic attention mechanisms (such as spatial attention), and in the face of complex hyperspectral images, the single scale cannot fully capture the rich local details and global structure information in the image; second, important features in the image are not uniformly distributed, and direct feature extraction through a conventional convolution layer may miss some local and global important information; in addition, in a deep neural network, especially when a deep convolutional network is used, with the increase of the number of layers, information may be lost, and the spatial information of the feature map may be gradually compressed, so that the details are lost, and the classification performance is affected.

[0004] The invention patent with the application number CN201810191507.8 provides a multi-scale space-spectrum collaborative classification method for hyperspectral images, step one: feature extraction is performed on the original hyperspectral image H to obtain a spectral information set Hspec composed of a band subset; step two: multi-scale spatial information is extracted from the spectral information set Hspec to obtain a plurality of multi-scale spatial information data sets Hspet; step three: the plurality of multi-scale spatial information data sets Hspet and the spectral information set Hspec are fused and preliminarily classified to obtain a preliminary classification result graph Qinit; step four: the preliminary classification result graph Qinit is decomposed into k probability result graphs P; step five: the k probability result graphs P are post-processed to obtain a final classification result graph Ofin, thereby completing the space-spectrum collaborative classification of the hyperspectral image.

[0005] The prior art reduces the complexity of the model by extracting multi-scale spatial information, although the training time and the consumption of computing resources are reduced, the classification efficiency is improved, but there is no adaptive weight adjustment mechanism, it is difficult to highlight the features of key ground objects in complex terrain, and the classification result will be affected when applied to the hyperspectral image classification of the Yellow River wetland, SUMMARY

[0006] The application aims to provide a hyperspectral image classification method for complex environments, which can realize efficient classification of hyperspectral images in large-scale scenes and complex environments through multi-scale feature extraction and information redistribution technology.

[0007] The application adopts the following technical solutions:

[0008] A hyperspectral image classification method for complex environments comprises:

[0009] S1: obtaining and preprocessing unmanned aerial vehicle hyperspectral remote sensing image data;

[0010] S2: labeling different ground object categories in the preprocessed image data, and dividing the training set, the validation set and the test set;

[0011] S3: constructing a convolutional neural network model, comprising:

[0012] A data preprocessing module is configured to extract features and perform nonlinear transformation on the input unmanned aerial vehicle hyperspectral image in the training set;

[0013] An information redistribution module is configured to normalize the features output by the data preprocessing module, and redistribute the high-entropy part features and the low-entropy part features of the features using scaling parameters and offset parameters;

[0014] A local attention module is configured to extract local spatial information from the features output by the information redistribution module, generate spatial local attention weights, and perform weighted adjustment on the input features to obtain locally weighted features;

[0015] A global attention module is configured to obtain globally weighted features by performing global convolution, global average pooling and full connection layer on the features output by the local attention module;

[0016] A feature fusion module is configured to calculate residual features, and fuse the locally weighted features, the globally weighted features and the residual features to obtain a fused feature map;

[0017] A classification module is configured to obtain pooled features by performing global pooling on the fused feature map, and output class prediction results by performing the first full connection layer on the pooled features;

[0018] S4: verifying the convolutional neural network model by using the unmanned aerial vehicle hyperspectral image verification set;

[0019] S5: inputting a test set of unmanned aerial vehicle hyperspectral images to be classified to obtain a classification result of the unmanned aerial vehicle hyperspectral images.

[0020] Preferably, in step S1, the image data is acquired by using an unmanned aerial vehicle onboard hyperspectral camera and a visible light camera, and the preprocessing includes grayscale and filtering and denoising processing, and image stitching based on feature point matching.

[0021] Preferably, in step S2, the pixel points are divided into a training set, a test set and a verification set in a ratio of 1:2:2.

[0022] Preferably, in step S3, the data preprocessing module outputs a feature map from the input unmanned aerial vehicle hyperspectral image through convolution operation, performs batch normalization on the output feature map, and applies a ReLU activation function to obtain features with specific information.

[0023] Preferably, in step S3, the information redistribution module processes the features output by the data preprocessing module through batch normalization BN to obtain normalized features; the information redistribution module continuously adjusts the scaling parameter and the offset parameter through training and learning, dynamically adjusts the normalized features, generates high-entropy partial features; the high-entropy partial features are subtracted from the normalized features to obtain low-entropy partial features; the high-entropy partial features and the low-entropy partial features are added and output to the local attention module.

[0024] Preferably, in step S3, the local attention module obtains a local feature map by performing convolution operation on the features output by the information redistribution module, performs batch normalization and Sigmoid activation function on the local feature map to generate spatial local attention weights, and multiplies the spatial local attention weights with the features output by the information redistribution module to obtain features weighted locally.

[0025] Preferably, in step S3, the global attention module obtains a global feature map by performing convolution operation on the features weighted locally, performs global pooling on the global feature map to generate a global feature vector, and processes the global feature vector through a first full connection layer and a second full connection layer of a full connection layer to obtain spatial global attention weights, multiplies the spatial global attention weights with the features weighted locally to perform global weighting, and obtains features weighted globally.

[0026] Preferably, in step S3, the feature fusion module adjusts the number of channels of the input unmanned aerial vehicle hyperspectral image to be consistent with the number of channels of the features output by the global attention module through convolution operation to obtain residual features.

[0027] Preferably, in step S3, the classification module obtains the pooled features by performing global pooling on the feature maps output by the feature fusion module; and then maps the pooled features to the class space through the first fully connected layer to obtain the class prediction result.

[0028] Preferably, in step S5, the to-be-classified image data test set is sequentially input into the trained convolutional neural network model to obtain the class prediction result of each pixel point, and all the class prediction results are spliced according to the order of the original image to obtain the classification result of the unmanned aerial vehicle hyperspectral image.

[0029] The present application has the advantages that an information redistribution module is designed, the scaling parameters and the offset parameters are continuously adjusted through training and learning, the features of the high-entropy part are dynamically adjusted and optimized, and the influence of the low-entropy part is reduced; the local and global attention mechanisms are combined, different sizes of convolution kernels are used to extract detailed and global features, the multi-scale attention mechanism can adaptively adjust the focus of the model according to the needs of different regions, thereby significantly improving the recognition ability of the hyperspectral image details and global patterns, and being crucial for the classification of ground objects in complex scenes; in the classification of hyperspectral images, a large amount of band and detail information needs to be processed, the residual connection module is used to ensure that each layer can effectively transmit information, thereby improving the training effect of the deep network, and the residual connection provides an effective solution, especially in the deep network. BRIEF DESCRIPTION OF DRAWINGS

[0030] Fig. 1 The flowchart of the hyperspectral image classification method for complex environments according to the present application;

[0031] Fig. 2 The schematic diagram of the multi-scale attention network in the present application. DETAILED DESCRIPTION

[0032] The present application will be described in detail below in combination with the drawings and embodiments:

[0033] As shown in the accompanying Figs. 1-2 The flowchart of the hyperspectral image classification method for complex environments according to the present application is as follows,

[0034] S1: Obtain and pre-process the unmanned aerial vehicle hyperspectral remote sensing image data;

[0035] S2: Label different ground object classes in the pre-processed image data, and divide the training set, the validation set and the test set;

[0036] S3: Construct a convolutional neural network model, which includes a data preprocessing module, an information redistribution module, a local attention module, a global attention module, a feature fusion module and a classification module; wherein,

[0037] a data preprocessing module configured to perform feature extraction and nonlinear transformation on the input unmanned aerial vehicle (UAV) hyperspectral image in the training set;

[0038] an information redistribution module configured to normalize the features output by the data preprocessing module and to perform feature redistribution on the high-entropy and low-entropy features by using the scaling and offset parameters that are constantly adjusted through training learning;

[0039] a local attention module configured to extract local spatial information from the features output by the information redistribution module, generate spatial local attention weights, and perform weighted adjustment on the input features to enhance the importance of local regions;

[0040] a global attention module configured to obtain globally weighted features by performing global convolution, global average pooling, and a fully connected layer on the features output by the local attention module;

[0041] a feature fusion module configured to calculate residual features and fuse the locally weighted features, the globally weighted features, and the residual features to obtain a fused feature map;

[0042] a classification module configured to obtain pooled features by performing global pooling on the fused feature map, and to obtain and output a class prediction result by performing a first fully connected layer on the pooled features;

[0043] S4: verifying the convolutional neural network model by using the obtained UAV hyperspectral image verification set;

[0044] S5: inputting a UAV hyperspectral image test set to be classified to obtain a classification result of the UAV hyperspectral image.

[0045] In this embodiment, the image data with high spatial and spectral resolution can be obtained by using a UAV-borne hyperspectral camera and a visible light camera in step S1, the visible light image is used for identification and confirmation, and each class in the hyperspectral data is counted.

[0046] In the present application, the preprocessing operation in step S1 specifically includes the following steps,

[0047] S101: performing grayscale and filtering and denoising processing on the input image; first, adjusting the spectral reflectance based on a standard spectral curve to reduce the influence of environmental factors; then, removing random noise in the hyperspectral data by using an SG smoothing filter to improve data quality; finally, eliminating the non-uniformity of the sensor response by using multiplicative scattering correction to make the data more accurately reflect the spectral characteristics of the ground objects;

[0048] S102: Image stitching is performed on the two images based on feature point matching; that is, by detecting key feature points in the images, matching point pairs of the two images are found, a transformation relationship between the two images is calculated, and the images are seamlessly stitched together.

[0049] In the present application, in step S2, different ground object categories in the image are labeled, and each labeled pixel point is randomly divided; the pixel points are divided into a training set, a test set and a verification set in a ratio of 1:2:2, at least 400 sample points of each category are selected for labeling, the more the number of labeled samples and the higher the quality, the more accurate the model will perform in subsequent classification tasks.

[0050] In the present embodiment, this division method ensures the representativeness of the training samples and the test samples, and lays a solid foundation for the training and performance evaluation of the subsequent model.

[0051] In the present application, the data preprocessing module in step S3 first extracts low-level features by a convolution kernel of 3x3 for the unmanned aerial vehicle hyperspectral image x input into the convolutional neural network model, outputs a feature map, performs batch normalization on the output feature map to speed up the training process and stabilize the dynamic range of the training, reduces the internal covariance shift, and then applies a ReLU activation function to obtain a feature x with specific information. stem ; The formula is as follows,

[0052] x stem =ReLU(BN(Conv2D(x,64,3,1,1))) (1)

[0053] Where x stem represents the feature with specific information output by the data preprocessing module; x represents the input unmanned aerial vehicle hyperspectral image; Conv2D(x,64,3,1,1) represents applying a 3x3 convolution operation with 64 channels output; BN() represents a batch normalization function; ReLU() is an activation function.

[0054] In the present embodiment, the hyperspectral image is usually composed of hundreds of bands, each band representing different spectral information, and this process helps to reduce the computational complexity and ensure the computational efficiency of the model; the above operation can speed up the training process, stabilize the dynamic range of the training, reduce the internal covariance shift, ensure the nonlinearity of the output, and enhance the ability of the model to express complex features.

[0055] The information redistribution module in step S3 performs batch normalization BN processing on the feature x stem output by the data preprocessing module, to obtain a normalized feature x norm ; The formula is as follows,

[0056] xnorm = BN(x stem ) (2)

[0057] The information redistribution module continuously adjusts the scaling parameter scale and the offset parameter shift through training learning, for dynamically adjusting the normalized feature x norm to enhance the expression of high-entropy part features and suppress the influence of low-entropy part features, and the generated high-entropy part feature high_entropy is:

[0058] high_entropy = scale x x norm + shift (3)

[0059] By subtracting the high-entropy part feature high_entropy from the normalized feature x norm , the low-entropy part feature low-entropy is obtained; the formula is as follows,

[0060] low-entropy = x norm -high-entropy (4)

[0061] Adding the high-entropy part feature high_entropy and the low-entropy part feature low-entropy obtains the feature x irm after information redistribution, at this time the high-entropy part is enhanced and the low-entropy part is suppressed; the formula is as follows,

[0062] x irm = high-entropy + low-entropy (5)

[0063] In the embodiment, the information redistribution module aims to strengthen the high-entropy part features and reduce the influence of the low-entropy part features, so that the model can better utilize the features with higher information entropy in the image to optimize the expression ability of the features, and can make the network more flexibly adjust the attention degree to different features; the high-entropy part features often contain more information (such as complex background or boundary), while the low-entropy part features may represent some less useful information, by dynamically adjusting the feature weight, the model can highlight more important information and improve the classification precision, in addition, using batch normalization helps to stabilize network training and improve the convergence speed of the model.

[0064] The local attention module in the step S3 performs a convolution operation with a 5x5 convolution kernel on the feature x irm output by the information redistribution module, to obtain a local feature map, performs batch normalization on the local feature map to stabilize the data distribution, and generates a spatial local attention weight AttentionMap local through a Sigmoid activation function,

[0065] AttentionMap local = σ(BN(Conv2D local (x irm ))) (6)

[0066] where AttentionMap local represents the spatial local attention weight, and the value size represents the importance degree of each local area; AttentionMap local ∈ [0, 1], the closer to 1, the more important the position feature is, and vice versa; σ() represents the Sigmoid activation function; Conv2D local () represents a convolution operation with a convolution kernel of 5x5; BN() represents a batch normalization function;

[0067] The calculated spatial local attention weight AttentionMap local is multiplied by the feature x irm output by the information redistribution module to obtain a locally weighted feature x local .

[0068] x local = x irm x AttentionMap local (7)

[0069] In this embodiment, this step can perform weighting processing on the key area in the hyperspectral image, so that important spatial and spectral features are highlighted, and the sensitivity of the model to subtle differences in complex scenes is enhanced; the local attention mechanism focuses on small areas in the image, highlighting those important detail features for classification; for hyperspectral images, local details such as texture, boundaries, etc. often contain important class information, and the local attention mechanism can accurately locate these key areas, improving the sensitivity of the model to subtle differences in complex scenes.

[0070] The global attention module in step S3 calculates a global feature map AttentionMap local from the locally weighted feature x global through a convolution operation with a convolution kernel of 11x11.

[0071] AttentionMap global = Conv2D global (x local ) (8)

[0072] where AttentionMap global represents the global feature map; Conv2Dglocal () represents a convolution operation with a convolution kernel of 11x11;

[0073] Then, the global feature map AttentionMap global is globally pooled to obtain a global feature vector Pooled, which represents the context information of the entire image,

[0074] Pooled = GlobalPool(AttentionMap global ) (9)

[0075] wherein Pooled represents the global feature vector; GlobalPool() represents a global pooling function;

[0076] The global feature vector after global pooling is processed through fully connected layers fc1 and fc2, fc1 is a first fully connected layer, which converts the global feature vector into a lower-dimensional feature space for abstracting features; fc2 is a second fully connected layer, which further processes the features output by fc1 to obtain a spatial global attention weight fc2_out,

[0077] fc1-out = ReLU(fc1(Pooled)) (10)

[0078] fc2_out = σ(fc2(fc1_out)) (11)

[0079] wherein Pooled represents the global feature vector; fc1() represents a first fully connected layer function; ReLU() is an activation function; fc1-out represents the features output from the first fully connected layer fc1; fc2() represents a second fully connected layer function; σ() represents a Sigmoid activation function; fc2-out represents a spatial global attention weight;

[0080] The spatial global attention weight fc2_out is multiplied by the feature x local that has been locally weighted to obtain a globally weighted feature x global , which further improves the expression of global information;

[0081] x global = x local x fc2_out (12)

[0082] In this embodiment, the global attention mechanism is used to capture the overall structural information of the hyperspectral image, so that the model can not only identify local details, but also obtain an understanding of the overall image structure; this multi-scale feature extraction method can improve the recognition ability of the model for class boundaries and global patterns, thereby effectively reducing classification errors.

[0083] The feature fusion module in step S3 adjusts the channel number of the unmanned aerial vehicle hyperspectral image x input into the convolutional neural network model through a convolution operation with a convolution kernel of 1*1, so that the channel number is consistent with the locally weighted feature x local and the globally weighted feature x global , to obtain residual feature x residual .

[0084] x residual = Conv2D residual (x, 1*1) (13)

[0085] wherein x represents the unmanned aerial vehicle hyperspectral image input into the convolutional neural network model; x residual represents the residual feature; Conv2D residual (x, 1*1) represents a convolution operation with a convolution kernel of 1*1 applied to the unmanned aerial vehicle hyperspectral image input into the convolutional neural network model.

[0086] The locally weighted feature x local , the globally weighted feature x global and the residual feature x residual are added to form a fusion feature map x final ; the formula is as follows,

[0087] x final = x local + x global + x residual (14)

[0088] In the embodiment, the addition operation helps the network to retain the original information of the input while effectively fusing the processed features; the residual connection module ensures that important information will not be lost in the deep network, effectively solving the problems of gradient disappearance and information loss in the deep network; in this way, the network can be more stably trained and avoid performance degradation.

[0089] The classification module in step S3 performs global pooling on the fusion feature map x final , compresses the feature map into a fixed-size vector, and obtains a pooled feature x pooled ; the formula is as follows,

[0090] x pooled = GlobalPool(x final ) (15)

[0091] wherein GlobalPool() represents a global pooling function.

[0092] The pooled feature xpooled mapping to a category space, finally outputting a category prediction result;

[0093] x class = fc1(x pooled ) (16)

[0094] wherein fc1() represents a first full connection layer function; x class represents a category prediction result.

[0095] In the embodiment, step S4 verifies the convolutional neural network model in step S3 by using the unmanned aerial vehicle hyperspectral image verification set obtained in step S2, ensures to retain the best model parameters, evaluates the constructed network model through the verification set every round, and if the accuracy of the current model on the verification set exceeds the previous best record, saves the weight of the current model to a file to ensure to retain the best model parameters.

[0096] In step S5, the to-be-classified image data test set is sequentially input into the trained convolutional neural network model, the category prediction result of each pixel point is obtained, all category prediction results are spliced according to the order of the original image, and then the category to which the hyperspectral image belongs is predicted and classified.

Claims

1. A hyperspectral image classification method for complex environments, characterized by: The method comprises the following steps, S1: acquiring and preprocessing unmanned aerial vehicle hyperspectral remote sensing image data; S2: labeling different ground object categories in the preprocessed image data and dividing the training set, the verification set and the test set; S3: constructing a convolutional neural network model, comprising: a data preprocessing module for feature extraction and nonlinear transformation of the unmanned aerial vehicle hyperspectral image in the input training set; an information redistribution module for normalizing the features output by the data preprocessing module and redistributing the high-entropy part features and the low-entropy part features of the features by using scaling parameters and offset parameters; a local attention module for extracting local spatial information from the features output by the information redistribution module, generating spatial local attention weights, and weighting and adjusting the input features to obtain locally weighted features; a global attention module for obtaining globally weighted features by performing global convolution, global average pooling and full connection layer on the features output by the local attention module; a feature fusion module for calculating residual features and fusing the locally weighted features, the globally weighted features and the residual features to obtain a fused feature map; a classification module for obtaining pooled features by performing global pooling on the fused feature map, and obtaining class prediction results by performing full connection layer on the pooled features and outputting the class prediction results; S4: verifying the convolutional neural network model by using the unmanned aerial vehicle hyperspectral image verification set; S5: inputting the unmanned aerial vehicle hyperspectral image test set to be classified to obtain the classification result of the unmanned aerial vehicle hyperspectral image.

2. The hyperspectral image classification method for complex environment according to claim 1, characterized in that: In step S1, the image data is acquired by the unmanned aerial vehicle onboard hyperspectral camera and visible light camera, and the preprocessing includes grayscale and filtering and noise reduction processing, and image stitching based on feature point matching.

3. The hyperspectral image classification method for complex environment according to claim 1, characterized in that: In step S2, the marked pixel points are divided into a training set, a test set and a validation set in a ratio of ​ 4. The hyperspectral image classification method for complex environment of claim 1, wherein: In step S3, the data preprocessing module performs convolution operation on the input unmanned aerial vehicle hyperspectral image to output a feature map, performs batch normalization on the output feature map, and then applies The activation function obtains features with specific information.

5. The hyperspectral image classification method for complex environment according to claim 4, characterized in that: In step S3, the information reassignment module performs batch normalization on the features output by the data preprocessing module to obtain normalized features. processing, to obtain normalized features. The information redistribution module continuously adjusts the scaling parameters and offset parameters through training and learning, dynamically adjusts the normalized features, generates high-entropy part features, subtracts the high-entropy part features from the normalized features to obtain low-entropy part features, and outputs the high-entropy part features and the low-entropy part features to the local attention module.

6. The hyperspectral image classification method for complex environment according to claim 5, characterized in that: In step S3, the local attention module obtains a local feature map by performing convolution operation on the features output by the information redistribution module, and performs batch normalization and Sigmoid The activation function generates spatial local attention weights, and the spatial local attention weights are multiplied with the features output by the information redistribution module to obtain locally weighted features.

7. The hyperspectral image classification method for complex environments according to claim 6, wherein: In step S3, the global attention module obtains a global feature map by performing convolution operation on the locally weighted features, performs global pooling on the global feature map to generate a global feature vector, processes the global feature vector through a first full connection layer and a second full connection layer of the global connection layer to obtain spatial global attention weights, multiplies the spatial global attention weights with the locally weighted features to perform global weighting, and obtains the globally weighted features.

8. The hyperspectral image classification method for complex environment according to claim 7, characterized in that: In step S3, the feature fusion module adjusts the channel number of the input unmanned aerial vehicle hyperspectral image to be consistent with the channel number of the feature output by the global attention module by convolution operation, and obtains the residual features.

9. The hyperspectral image classification method for complex environments according to claim 8, wherein: In step S3, the classification module obtains pooled features by performing global pooling on the feature map output by the feature fusion module; and maps the pooled features to a class space by a full connection layer to obtain a class prediction result.

10. The hyperspectral image classification method for complex environments of claim 1, wherein: In step S5, the image data test set to be classified is input into the trained convolutional neural network model in sequence, the class prediction result of each pixel point is obtained, all class prediction results are spliced according to the order of the original image, and the classification result of the unmanned aerial vehicle hyperspectral image is obtained.

Images