A method for calibrating hyperspectral features using global-local channel attention module

Abstract

The present invention discloses a method for calibrating hyperspectral features using a global-local channel attention module, comprising: extracting deep semantic features of a hyperspectral original image through a backbone network to obtain a local hyperspectral feature map; using a global-local channel attention module, through five steps of compression, replication, connection, excitation and calibration, to achieve recalibration of the local hyperspectral feature map. The present invention proposes a global-local channel attention module that works in a plug-and-play manner, flexibly and accurately recalibrates channel characteristic responses in a pixel manner, enhances useful information, suppresses useless information, recalibrates feature mapping, reduces redundancy and noise, and effectively improves the performance of a hyperspectral image classification method based on a deep learning network, and can be applied to any existing deep hyperspectral image classification without requiring additional computational costs.

Description

Technical Field

[0001] The invention relates to the technical field of image classification, and in particular to a method for calibrating hyperspectral features by using a global-local channel attention module. Background Art

[0002] Hyperspectral images are composed of dozens or hundreds of continuous narrow-band images, which can simultaneously capture the spatial and spectral dimensions of the target scene, and are called "data cubes". Hyperspectral image classification is an important step in remote sensing applications ranging from agricultural production, environmental monitoring to geological exploration. Early research focused on machine learning solutions, such as support vector machines, but their classification accuracy was not ideal. In recent years, due to the strong robustness and performance of deep hyperspectral image classification methods, they have received more and more attention, and stacked autoencoders and fully convolutional networks have emerged.

[0003] However, these deep learning algorithms use all feature channel information, ignoring the different importance of different channels, and these methods are prone to introduce noise. Learning how to select useful channels and suppress other channels is an important solution. The pixel spectra at different spatial locations are the same, but the information channel response in the generated convolution features will change with the position. How to select channels based on the characteristics of each pixel is an important issue and has great significance. Summary of the invention

[0004] In view of this, the present invention provides a method for calibrating hyperspectral features using a global-local channel attention module, which flexibly and accurately recalibrates the channel characteristic response in a pixel manner, enhances useful information, suppresses useless information, recalibrates the feature mapping, reduces redundancy and noise, and effectively improves the performance of a hyperspectral image classification method based on a deep learning network.

[0005] The present invention discloses a method for calibrating hyperspectral features using a global-local channel attention module, comprising:

[0006] The deep semantic features of the original hyperspectral image are extracted through the backbone network to obtain the local hyperspectral feature map;

[0007] A global-local channel attention module is used to recalibrate the local hyperspectral feature map through five steps of compression, replication, connection, excitation and calibration.

[0008] Preferably, the global-local channel attention module is used to recalibrate the local hyperspectral feature map through five steps of compression, replication, connection, excitation and calibration, including:

[0009] Compressing the local hyperspectral feature map to obtain a first global feature map;

[0010] Performing a copy operation on the first global feature map to obtain a second global feature map;

[0011] Connecting the local hyperspectral feature map and the second global feature map to obtain a connected image;

[0012] Performing excitation processing on the connected image to obtain a three-dimensional modulation weight matrix;

[0013] The three-dimensional modulation weight matrix and the local hyperspectral feature map are calibrated to achieve recalibration of the local hyperspectral feature map.

[0014] Preferably, the number of categories of the classification of the first global feature map is the same as the number of categories of the classification of the local hyperspectral feature map.

[0015] Preferably, the dimension of the second global feature map is the same as the dimension of the local hyperspectral feature map.

[0016] Preferably, the excitation processing is performed on the connected images to obtain a three-dimensional modulation weight matrix:

[0017] The global and local information between the channels of the connected image is modeled to generate pixel-by-pixel and channel-by-channel modulation weight results, that is, a three-dimensional modulation weight matrix is ​​obtained.

[0018] Preferably, the three-dimensional modulation weight matrix has the same spatial size and number of channels as the local hyperspectral feature map, so that each pixel of the local hyperspectral feature map has its corresponding weight to stimulate or inhibit channel characteristic response.

[0019] Preferably, the calibrating the three-dimensional modulation weight matrix and the local hyperspectral feature map to achieve recalibration of the local hyperspectral feature map includes:

[0020] The three-dimensional modulation weight matrix and the local hyperspectral feature map are multiplied at the pixel level, and the characteristic response of each pixel of the local hyperspectral feature map is recalibrated through its corresponding channel weight to highlight important features while suppressing useless or noise features.

[0021] Due to the adoption of the above technical solution, the present invention has the following advantages: the present invention provides a global-local channel attention module that works in a plug-and-play manner, flexibly and accurately recalibrates the channel characteristic response in a pixel-by-pixel manner, enhances useful information, suppresses useless information, recalibrates feature mapping, reduces redundancy and noise, and effectively improves the performance of the hyperspectral image classification method based on deep learning networks. It can be applied to any existing deep hyperspectral image classification without additional computational costs. Sufficient experiments have proved the effectiveness of our algorithm, and we have achieved the best performance on multiple hyperspectral image classification datasets. By integrating global information and local spectral information, channel selection is performed in a pixel-by-pixel manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0023] Figure 1 A schematic diagram of a process of calibrating hyperspectral features using a global-local channel attention module according to an embodiment of the present invention;

[0024] Figure 2 This is a structural diagram of an embodiment of the present invention in which a global-local channel attention module is inserted into a non-local fully convolutional network. DETAILED DESCRIPTION

[0025] The present invention is further described in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0026] Embodiment 1:

[0027] An embodiment of the present invention provides an embodiment of a method for calibrating hyperspectral features using a global-local channel attention module, the method comprising:

[0028] The deep semantic features of the original hyperspectral image are extracted through the backbone network to obtain the local hyperspectral feature map;

[0029] A global-local channel attention module is used to recalibrate the local hyperspectral feature map through five steps: compression, replication, concatenation, excitation, and calibration.

[0030] In this embodiment, a global-local channel attention module is used to recalibrate the local hyperspectral feature map through five steps of compression, replication, connection, excitation and calibration, including:

[0031] Compressing the local hyperspectral feature map to obtain a first global feature map;

[0032] Performing a copy operation on the first global feature map to obtain a second global feature map;

[0033] Connecting the local hyperspectral feature map and the second global feature map to obtain a connected image;

[0034] Performing excitation processing on the connected image to obtain a three-dimensional modulation weight matrix;

[0035] The three-dimensional modulation weight matrix and the local hyperspectral feature map are calibrated to achieve recalibration of the local hyperspectral feature map.

[0036] In this embodiment, the number of categories of the first global feature map is the same as the number of categories of the local hyperspectral feature map.

[0037] In this embodiment, the dimension of the second global feature map is the same as the dimension of the local hyperspectral feature map.

[0038] In this embodiment, the connected image is subjected to excitation processing to obtain a three-dimensional modulation weight matrix:

[0039] The global and local information between the channels of the connected image is modeled to generate pixel-by-pixel and channel-by-channel modulation weight results, that is, a three-dimensional modulation weight matrix is ​​obtained.

[0040] In this embodiment, the three-dimensional modulation weight matrix has the same spatial size and number of channels as the local hyperspectral feature map, so that each pixel of the local hyperspectral feature map has its corresponding weight to stimulate or inhibit the channel characteristic response.

[0041] In this embodiment, the three-dimensional modulation weight matrix and the local hyperspectral feature map are calibrated to achieve recalibration of the local hyperspectral feature map, including:

[0042] The three-dimensional modulation weight matrix is ​​multiplied with the local hyperspectral feature map at the pixel level, and the characteristic response of each pixel in the local hyperspectral feature map is recalibrated by its corresponding channel weight to highlight important features while suppressing useless or noise features.

[0043] Embodiment 2:

[0044] See also Figure 1 and Figure 2, the present invention also provides a more specific embodiment: the global-local channel attention module combines the global spatial information and the local spectral feature information through the channel weights generated by learning, and excites or inhibits the channel response of a given feature map. The global-local channel attention module first merges the global information and the local information, and then compresses the entire feature map to gather the global distribution of spatial features, and then merges the merged features with the original input to comprehensively enhance the global information and the specific spectral features of each position. Finally, a series of 1×1 kernel convolutions and excitations are performed on the results, and new weights are mapped to the input features through pixel-level multiplication to generate pixel-level channel weights.

[0045] Given a hyperspectral image X∈R HxWxB , extracting deep semantic features through the backbone network and finally outputting the segmentation result map to represent each pixel category output segmentation result map Represents the predicted category of each pixel, where H, W, B, and C represent the image height, image width, number of spectra, and number of categories, respectively. The proposed global-local channel attention module can be flexibly inserted into the backbone network of the global-local channel attention module for deep hyperspectral image classification, thereby enhancing the representation learning function based on pixel-level channel attention and recalibrating the hyperspectral feature map using five steps: compression, replication, connection, excitation, and calibration.

[0046] 1. Compression

[0047] The global-local channel attention module has an intermediate function result map U∈R HxWxC As input U = J (X) = [u1, u2, ...., u c ], where J(·) represents the mapping of the input image X to the feature map learned by the backbone network, and u c ∈R HxW represents the cth channel of the feature map. The compression step can achieve a global distribution in the perceived channel response through global average pooling. The result is M = [m1,m2,....,m c ]∈R C The statistics of the overall image can be represented as a collection of local detailed information.

[0048] 2. Copy

[0049] This step repeats M in space, obtains the global feature map and makes its dimension the same as the original one, and outputs the copied feature map = [p1, p2, ...., p c ]∈R HxWxC , where p i ∈R HxW can be calculated as, Therefore, the size of the copied global features is the same as the size of the input features U.

[0050] 3. Connect

[0051] Given the local spectral feature U and the global spatial information P respectively, this step connects the two features together. T = [U, P], and the result T∈R H×W×2C Finally, the local structure information and the global shape information are combined.

[0052] 4. Motivation

[0053] The purpose of this step is to model the global and local information between the channels of the result T of the previous step, and generate pixel-by-pixel and channel-by-channel modulation weight results. To achieve this goal, this step needs to follow two criteria: 1) This step has the ability to establish a nonlinear model that can reflect the relationship between channels. 2) The generated weight result should have the same spatial size and number of channels as the input feature map U. So that each pixel has its own weight to stimulate or inhibit the channel feature response. In order to meet these standards, the formula of this step is as follows: E = f sig (f con2 (δ(f con1 (T)))), where E∈R HxWxC ,f coni (·),i∈[1,2], represents the convolution f coni (T) = W i ×T+B i ,i∈[1,2], we use C 1×1 convolution kernels to process T at the pixel level to ensure that the spatial size of the output result remains unchanged. δ represents the ReLU function, which is used between two convolutional layers to give the model nonlinear representation capabilities. Represents the Sigmoid function It can make the output weight range from 0 to 1.

[0054] 5. Calibration

[0055] This step applies the generated pixel-wise, channel-wise modulation weights E to the feature map U to generate the output of the global-local channel attention module. Where * represents a pixel-level multiplication operation. As a result, the feature response of each pixel is recalibrated by its corresponding channel weight, highlighting important features while suppressing useless or noisy features. The recalibrated output can be directly input into the subsequent layers of the network.

[0056] The embodiment of the present invention proposes a plug-and-play global-local channel attention module. The global-local channel attention module assigns different weights to information in different regions of the feature map in the channel dimension by learning global spatial information and local spectral features. The recalibrated feature map enhances useful information, suppresses useless information, reduces redundancy and noise, and effectively improves the performance of the hyperspectral image classification method based on the deep learning network.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for calibrating hyperspectral features using a global-local channel attention module, characterized in that: include: The deep semantic features of the original hyperspectral image are extracted through the backbone network to obtain the local hyperspectral feature map; A global-local channel attention module is used to recalibrate the local hyperspectral feature map through five steps of compression, replication, connection, excitation and calibration; Compression: The intermediate function result graph of the global-local channel attention module As input ,in, represents the mapping of the input image X to the feature map learned by the backbone network, Represents the cth channel of the feature map; global distribution of perceptual channel responses is achieved through global average pooling; ,result is the statistical information of the whole image, It is represented as a collection of local detailed information, where H, W, and C represent the image height, image width, and the number of classification categories, respectively; Copy: By repeating M in space, obtain the global feature map and make its dimension the same as the original, and output the copied feature map ,in, , , the size of the copied global feature is the same as the size of the input feature U; Connection: Given the local spectral feature U and the global spatial information P respectively, connect the two features together. ,as a result Combine local structural information with global shape information; Excitation: Model the global and local information between the channels of T and generate pixel-by-pixel and channel-by-channel modulation weight results; ,in , , which represents convolution Use C The convolution kernel processes T at the pixel level. represents the Relu function, Represents the Sigmoid function ; Calibration: Apply the generated pixel-by-pixel, channel-by-channel modulation weights E to the feature map U to generate the output of the global-local channel attention module =E U, where Represents a pixel-level multiplication operation.

2. The method according to claim 1, characterized in that The global-local channel attention module is used to recalibrate the local hyperspectral feature map through five steps of compression, replication, connection, excitation and calibration, including: Compressing the local hyperspectral feature map to obtain a first global feature map; Performing a copy operation on the first global feature map to obtain a second global feature map; Connecting the local hyperspectral feature map and the second global feature map to obtain a connected image; Performing excitation processing on the connected image to obtain a three-dimensional modulation weight matrix; The three-dimensional modulation weight matrix and the local hyperspectral feature map are calibrated to achieve recalibration of the local hyperspectral feature map.

3. The method according to claim 2, characterized in that The number of categories of the classification of the first global feature map is the same as the number of categories of the classification of the local hyperspectral feature map.

4. The method according to claim 2, characterized in that: The dimension of the second global feature map is the same as the dimension of the local hyperspectral feature map.

5. The method according to claim 2, characterized in that: The three-dimensional modulation weight matrix obtained by performing excitation processing on the connected image is specifically: The global and local information between the channels of the connected image is modeled to generate pixel-by-pixel and channel-by-channel modulation weight results, that is, a three-dimensional modulation weight matrix is ​​obtained.

6. The method according to claim 5, characterized in that The three-dimensional modulation weight matrix has the same spatial size and number of channels as the local hyperspectral feature map, so that each pixel of the local hyperspectral feature map has its corresponding weight to stimulate or inhibit channel characteristic response.

7. The method according to claim 2, characterized in that The calibrating process of the three-dimensional modulation weight matrix and the local hyperspectral feature map to achieve recalibration of the local hyperspectral feature map includes: The three-dimensional modulation weight matrix and the local hyperspectral feature map are multiplied at the pixel level, and the characteristic response of each pixel of the local hyperspectral feature map is recalibrated through its corresponding channel weight to highlight important features while suppressing useless or noise features.

Images