Image classification method of Gaussian weighted network based on adaptive feature enhancement
By introducing adaptive feature enhancement module and Gaussian weighted feature fusion module in hyperspectral image classification, combined with the multi-head collaborative attention mechanism, the problems of insufficient interaction of the Transformer attention head information and limited receptive field of the convolution kernel are solved, which significantly improves the Accuracy and robustness of hyperspectral image classification.
Patent Information
- Application Number
- CN202411015695.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
AI Technical Summary
In the existing hyperspectral image classification methods, there is a lack of effective information interaction between the attention heads of the Transformer calculation method, which leads to the scoring matrix that can only use limited information during the calculation process, affecting the classification accuracy, and the convolution kernel receptive field is limited. The problem has not been fully alleviated.
It proposes a Gaussian-weighted network based on adaptive feature enhancement, including an adaptive feature enhancement module, a Gaussian-weighted feature fusion module and a multi-head collaborative attention mechanism. Through adaptive adjustment of features, local and global information are fused, and multi-head collaborative attention is introduced in Transformer. Force mechanism to enhance information interaction.
The accuracy and robustness of hyperspectral image classification are improved, and the classification performance is significantly improved by effectively fusion of feature information and enhancing the feature extraction ability of the model.
Smart Images

Figure CN118968154A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an image classification method. Background Art
[0002] Hyperspectral images (HSIs) are cube data that simultaneously obtain the spatial information and spectral information of the target object. It combines imaging technology with spectral technology to form continuous, narrow-band spectral image data, so that the characteristics of the objects can be revealed more comprehensively and meticulously. Hyperspectral images are also widely used in medical, agricultural, and mineral exploration fields due to their unique image-spectrum integration characteristics. Hyperspectral image classification is the process of assigning corresponding category labels to each pixel in the image by analyzing the information in the image. The early popular hyperspectral image classification technology mainly uses the unique spectral information of different objects for classification. Representative methods include: K nearest neighbor classifier, support vector machine (SVM) and sparse representation classification. Although these methods are simple and easy to implement, they only use spectral information and do not take into account the continuity of spatial object distribution, resulting in incomplete feature extraction. In addition, traditional classification methods rely too much on expert experience, making it difficult to ensure the robustness and generalization of the algorithm.
[0003] In recent years, deep learning (DL) has shown great application potential in the field of hyperspectral image classification. Models such as convolution neural network (CNN), deep belief network (DBN), and stacked autoencoder (SAE) have achieved good research results in the study of hyperspectral image classification. However, both DBN and SAE require the input data to be a one-dimensional vector, which leads to great limitations in the application of the model. CNN can extract deeper data features by virtue of its local perception and parameter sharing characteristics, while avoiding a sharp increase in model parameters, and has therefore attracted widespread attention. A spectral-spatial residual network (SSRN) based on spectral-spatial was proposed. This method uses a three-dimensional convolution kernel to jointly extract the spectral-spatial features of hyperspectral images and introduces the residual structure in ResNet. This residual structure helps to alleviate the gradient diffusion problem caused by too many neural network layers, thereby improving the accuracy of feature extraction and the classification performance of the network. However, the simple use of a three-dimensional convolution kernel will also lead to a sharp increase in computational complexity. To alleviate this problem, a hybrid spectral CNN (HybridSN) was proposed. Compared with the method of simply using three-dimensional convolution kernels, this method combines two-dimensional convolution kernels on the basis of three-dimensional convolution kernels, which not only reduces the complexity of the model, but also can learn more abstract spatial representations. In addition, a method based on step activation quantization (step activation network with binary weight, SAWB) was proposed, which can replace floating-point operations with integers, thereby speeding up the network's reasoning process and reducing computational costs.
[0004] The attention mechanism originates from the study of human vision. In the field of cognitive science, due to the limitations of information processing, humans inevitably selectively focus on part of the information and ignore other equally visible information. By simulating this feature of the human visual and cognitive system, the attention mechanism was introduced in the field of deep learning. This mechanism aims to enable neural networks to focus on relevant information like humans when processing input data, thereby improving the efficiency and accuracy of information processing. By introducing the attention mechanism, neural networks can automatically learn and selectively focus on important information in the input data, thereby improving the performance and generalization ability of the model. A hierarchical residual network (HResNetAM) with an attention mechanism was proposed. This method uses the attention mechanism to set adaptive weights for spectral features of different scales, further improving the discriminative ability of the extracted features. A cross-attention spectral-spatial network (CASSN) was proposed. This method uses cross-spectral and cross-spatial attention components to generate band weights and spectral-spatial features to alleviate the problem that previous methods are not robust to HSI rotation. A cross-channel dense connection and multi-scale dual aggregated attention network (CDC_MDAA) is proposed to alleviate the problem of difficult labeled data in hyperspectral image classification. A dual global-local attention network (DGLANet) is proposed to utilize both global and local spatial-spectral information at a lower computational cost.
[0005] Although convolutional neural networks (CNNs) have achieved many remarkable results in the field of hyperspectral image classification, the problem of limited receptive field has been a long-standing challenge. This problem makes it relatively difficult for the network to learn long-range spatially distributed dependencies. Although this problem can be alleviated by using convolution kernels of different sizes, this will also increase the complexity of the model and may cause overfitting of network training.
[0006] Transformer is a new neural network structure proposed by Vaswani et al. in 2017. It was initially widely used in the field of natural language processing (NLP) and achieved remarkable results. However, in the application of computer vision (CV), Transformer has certain limitations. The emergence of research results such as ViT (vision transformer, ViT) has greatly expanded the application scope of Transformer in the field of computer vision. Based on ViT, a backbone network (spectralformer, SF) for hyperspectral image classification was proposed. This method can not only learn local spectral representations, but also transfer similar components from shallow layers to deep layers. Similarly, a group-aware hierarchical transformer (GAHT) was proposed. This method extracts more detailed local spatial-spectral relationships through a group embedding module. A spectral-spatial transformer network (SSTN) was proposed. This method consists of spatial attention and spectral association modules to overcome the constraint of limited receptive field of convolution kernels. An efficient lightweight spectral-spatial transformer (ELS2T) is proposed, which can model the long-range global dependencies of hyperspectral data, thereby alleviating the problem of limited receptive field of CNN.
[0007] However, since the Transformer uses a global attention mechanism, it calculates the association between all elements in the input sequence and cannot focus on local features. To alleviate this problem, some researchers have tried to combine the Transformer with CNN to give full play to the advantages of the Transformer in processing long sequences and the CNN in processing local image features. A network mixed with CNN and Transformer (convolution transformer mixer, CTMixer) was proposed. This method uses CNN to improve the multi-head attention mechanism, thereby realizing the combination of CNN and Transformer. A bottleneck spatial-spectral transformer (BS2T) was proposed. This method combines CNN and Transformer to achieve feature extraction and capture of long-distance dependencies. Specifically, CNN is used for feature compression and expansion, while Transformer enhances feature representation by capturing long-distance dependencies. A spectral-spatial feature tokenization transformer (SSFTT) was proposed. This method extracts spectral-spatial features through CNN and uses feature tokenizers to transform features to obtain high-level semantic features.
[0008] Although these methods have alleviated the problems of CNN and Transformer, some challenges still exist.
[0009] include:
[0010] 1) After generating the attention map, the common attention module directly performs weighted fusion with the feature map. This processing method ignores the further processing of the main and secondary features in the data, which leads to the suppression of features that can improve network performance, and ultimately affects the overall training effect.
[0011] 2) Previous methods to alleviate the problem of limited receptive field of convolution kernels mainly adopted the strategy of simply combining CNN with Transformer, either in parallel or serial mode, without making more in-depth improvements to CNN itself.
[0012] 3) In the multi-head attention mechanism, the core component of Transformer, the calculation of each score matrix is usually independent of each other. This non-interference calculation method leads to a lack of effective information interaction between attention heads, so that each score matrix can only use very limited information during the calculation process. Summary of the invention
[0013] The purpose of the present invention is to solve the problem that the existing methods cannot be adaptively adjusted according to the characteristics of the data, and there is a lack of effective information interaction between the attention heads of the Transformer calculation method, so that each score matrix can only use very limited information in the calculation process, resulting in low classification accuracy of the hyperspectral image classification method. An image classification method based on a Gaussian weighted network with adaptive feature enhancement is proposed.
[0014] The specific process of the image classification method based on Gaussian weighted network with adaptive feature enhancement is as follows:
[0015] Step 1: Establish a Gaussian weighted network AFEGW based on adaptive feature enhancement;
[0016] The Gaussian weighted network AFEGW based on adaptive feature enhancement consists of three parts: feature enhancement, feature extraction and feature fusion, and global dependency extraction and classification;
[0017] The feature enhancement part includes the adaptive feature enhancement module AFEM;
[0018] The feature extraction and feature fusion part includes Gaussian weighted feature fusion module GWF2;
[0019] The Gaussian weighted feature fusion module GWF2 includes three-dimensional convolution with a void rate of 1, BN, ReLU, Dropout, spectral feature fusion, ReLU, Dropout, Sigmoid function, two-dimensional convolution, BN, ReLU, Dropout, spatial feature fusion, ReLU, Dropout, spectral feature fusion, and spatial feature fusion;
[0020] The global dependency extraction and classification part includes Transformer encoder and linear layer;
[0021] The Transformer encoder includes a layer normalization layer, a multi-head collaborative attention MHCA, a layer normalization, and a multi-layer perceptron;
[0022] The hyperspectral image training set X is input into the Gaussian weighted network AFEGW based on adaptive feature enhancement for training until convergence, and the trained Gaussian weighted network AFEGW based on adaptive feature enhancement is obtained;
[0023] Step 2: Input the hyperspectral image to be tested into the trained Gaussian weighted network AFEGW based on adaptive feature enhancement to complete the classification of the hyperspectral image to be tested.
[0024] The beneficial effects of the present invention are:
[0025] An adaptive feature enhanced gaussian weighted network (AFEGW) is proposed. First, an adaptive feature enhancement module (AFEM) is designed. AFEM can accurately evaluate the impact of different pixel features on the network classification performance and enhance the features that are more conducive to improving the network classification performance. This mechanism enhances the adaptability of the network. Secondly, in order to alleviate the problem of limited convolution receptive field, a gaussian weighted feature fusion module (GWF2) is proposed to fuse local and global feature information. Specifically, local spectral-spatial information is extracted through CNN, and the extracted local information is further mapped to the global information by using mapping operation, so as to achieve effective fusion of local and global information. Finally, in order to alleviate the problem of insufficient information interaction between attention heads, a multi-head coordination attention mechanism (MHCA) is constructed. MHCA compensates for the lack of information interaction between attention heads in a collaborative way, so that the key features in the input sequence can be captured more comprehensively and the feature extraction ability of the model can be enhanced. In order to verify the effectiveness of the proposed method, comprehensive experiments were conducted on three datasets: Indian Pines, Pavia University, and Houston 2013. The experimental results show that the proposed method has superior classification performance compared with some current advanced methods.
[0026] The main contributions of the present invention can be summarized as follows:
[0027] An adaptive feature enhancement module (AFEM) is proposed, which can accurately evaluate the impact of different pixel features on network classification performance and dynamically enhance those pixel features that most significantly improve network classification performance.
[0028] A Gaussian weighted feature fusion module (GWF2) is proposed, which can effectively integrate local and global feature information and alleviate the problem of limited convolution receptive field.
[0029] A multi-head collaborative attention mechanism (MHCA) is designed. MHCA realizes complementary information exchange between multiple attention heads by working together, so as to more comprehensively capture the key features of the input sequence.
[0030] In the present invention, a Gaussian weighted network based on adaptive feature enhancement is proposed for hyperspectral image classification. An adaptive feature enhancement module (AFEM) is proposed, which can accurately judge the influence of different sample features on the network classification performance, and then dynamically adjust the learnable parameters between different partitions to enable the network to learn feature representations that are more conducive to classification performance. Secondly, in order to alleviate the problem of limited receptive field of traditional convolution kernels, a Gaussian weighted feature fusion module (GWF2) is designed. GWF2, which sets the initial weights through Kaiming normal distribution, can not only extract local detail information well, but also effectively fuse information from different channels, so that the network can understand feature information more accurately. Finally, a multi-head collaborative attention mechanism (MHCA) is constructed. MHCA combines squeeze-incentive attention and multi-head self-attention, so that the network can focus more on the extraction and processing of classification information. A large number of experimental results show that compared with other CNN and Transformer-based methods, AFEGW can provide more excellent classification performance with limited samples and has strong robustness. In future work, the removal of redundant spectral information is an important research direction, and we will continue to work in this direction. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 The overall network structure diagram of AFEGW proposed in the present invention;
[0032] Figure 2 is a structural diagram of the AFEM of the present invention;
[0033] Figure 3 It is a structural diagram of GWF2 of the present invention;
[0034] Figure 4 is a structural diagram of MHCA of the present invention;
[0035] Figure 5 Schematic diagram of GroundTruth. (a)-(c) are Indian Pines, Pavia University, and Houston 2013.
[0036] Figure 6 The effect of different window sizes and learning rates on classification performance. The datasets (a)-(c) are Indian Pines, Pavia University, and Houston 2013 respectively;
[0037] Figure 7 This is a graph showing the impact of different similarity thresholds on overall accuracy;
[0038] Figure 8 It is the growth trend diagram of the learnable parameter α in different data sets;
[0039] Fig. 9 It is the growth trend diagram of the learnable parameter β on different data sets;
[0040] Fig.10 This is a comparative experimental diagram of different sample ratios on the IP dataset;
[0041] Fig.11 This is a comparative experimental diagram of different sample ratios on the UP dataset;
[0042] Fig.12 This is a comparative experimental diagram of different sample ratios on the HT dataset;
[0043] Fig.13 Classification diagrams obtained by different methods on the IP dataset, (a)-(j) are: GroundTruth, HybridSN, HResNetAM, GAHT, SpectralFormer, SSTN, CTMixer, BS2T, SSFTT, Proposed
[0044] Fig.14 Classification diagrams obtained by different methods on the UP dataset, (a)-(j) are: Ground Truth, HybridSN, HResNetAM, GAHT, SpectralFormer, SSTN, CTMixer, BS2T, SSFTT, Proposed
[0045] Fig.15 Classification diagrams obtained by different methods on the HT dataset, (a)-(j) are: Ground Truth, HybridSN, HResNetAM, GAHT, SpectralFormer, SSTN, CTMixer, BS2T, SSFTT, Proposed
[0046] Fig.16 t-SNE visualization results obtained by different methods on the IP dataset, (a)-(d): HResNetAM, SSTN, SSFTT, AFEGW;
[0047] Fig.17 t-SNE visualization results obtained by different methods on the UP dataset, (a)-(d): HResNetAM, SSTN, SSFTT, AFEGW;
[0048] Fig.18 t-SNE visualization results obtained by different methods on the HT dataset, (a)-(d): HResNetAM, SSTN, SSFTT, AFEGW. DETAILED DESCRIPTION
[0049] Specific implementation method 1: Combination Figure 1 This embodiment is described. The specific process of the image classification method based on the Gaussian weighted network with adaptive feature enhancement is as follows:
[0050] Step 1: Establish an adaptive feature enhanced Gaussian weighted network AFEGW (adaptive feature enhanced gaussian weighted network, AFEGW);
[0051] The Gaussian weighted network AFEGW based on adaptive feature enhancement consists of three parts: feature enhancement, feature extraction and feature fusion, and global dependency extraction and classification;
[0052] The feature enhancement part includes the adaptive feature enhancement module AFEM;
[0053] The feature extraction and feature fusion part includes Gaussian weighted feature fusion module GWF2;
[0054] The Gaussian weighted feature fusion module GWF2 includes three-dimensional convolution with a void rate of 1, BN, ReLU, Dropout, spectral feature fusion, ReLU, Dropout, Sigmoid function, two-dimensional convolution, BN, ReLU, Dropout, spatial feature fusion, ReLU, Dropout, spectral feature fusion, and spatial feature fusion;
[0055] The global dependency extraction and classification part includes Transformer encoder and linear layer;
[0056] The Transformer encoder includes a layer normalization layer, a multi-head collaborative attention MHCA, a layer normalization, and a multi-layer perceptron;
[0057] The hyperspectral image training set X is input into the Gaussian weighted network AFEGW based on adaptive feature enhancement for training until convergence, and the trained Gaussian weighted network AFEGW based on adaptive feature enhancement is obtained;
[0058] Step 2: Input the hyperspectral image to be tested into the trained Gaussian weighted network AFEGW based on adaptive feature enhancement to complete the classification of the hyperspectral image to be tested.
[0059] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the process of obtaining the hyperspectral image training set X in step 1 is:
[0060] Randomly sampled hyperspectral image training set Tagged with
[0061] Among them, h, w, and l represent the height, width, and number of spectral bands of the hyperspectral image HSI respectively. Represents a real number.
[0062] The other steps and parameters are the same as those in the first embodiment.
[0063] Specific implementation method three: This implementation method is different from specific implementation method one or two in that in step one, the hyperspectral image X is input into the Gaussian weighted network AFEGW based on adaptive feature enhancement for training until convergence, and a trained Gaussian weighted network AFEGW based on adaptive feature enhancement is obtained;
[0064] The specific process is:
[0065] Step 1: Use PCA (Principal Component Analysis, PCA) to analyze the hyperspectral image training set Perform dimensionality reduction to obtain the data after PCA dimensionality reduction b represents the number of spectral bands;
[0066] In this way, the redundant spectral information in the hyperspectral image can be removed while extracting the main spectral information, thus reducing the amount of calculation;
[0067] Step 1 and 2: Reduce the dimension of the data X pca Input Adaptive Feature Enhancement Module AFEM, Adaptive Feature Enhancement Module AFEM output features
[0068] Where s represents the height and width of the window, and b represents the number of spectral bands;
[0069] Step 13: Output features of the adaptive feature enhancement module AFEM Input the Gaussian weighted feature fusion module GWF2, and the Gaussian weighted feature fusion module GWF2 outputs features;
[0070] Step 14: The Gaussian weighted feature fusion module GWF2 outputs features and inputs the global dependency extraction and classification part, and the global dependency extraction and classification part outputs the classification result;
[0071] Step 15: Repeat step 11, step 12, step 13, and step 14 until convergence to obtain a trained Gaussian weighted network AFEGW based on adaptive feature enhancement.
[0072] The other steps and parameters are the same as those in the first or second embodiment.
[0073] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that the dimension-reduced data X is pca Input Adaptive Feature Enhancement Module AFEM, Adaptive Feature Enhancement Module AFEM output features The specific process is:
[0074] Step 121: Data X after PCA dimensionality reduction pca Perform 3D cube data extraction to obtain data
[0075] Step 122: Data Input Adaptive Feature Enhancement Module AFEM, Adaptive Feature Enhancement Module AFEM output features
[0076] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0077] Specific implementation method 5: This implementation method is different from specific implementation methods 1 to 4 in that the data is Input an adaptive feature enhancement module (AFEM), and the adaptive feature enhancement module (AFEM) outputs features;
[0078] The specific process is:
[0079] Step 1221. From data x p Extract the center pixel vector from
[0080] Step 1222: Calculate the center pixel vector x using cosine similarity c With surrounding pixel data x p The first-order attention map M1 is expressed as:
[0081]
[0082] Among them, x i represents the i-th pixel vector in the surrounding pixel data, cosine similarity(·) represents the cosine similarity function, s represents the height and width of the window; ||·|| represents the modulus length of the corresponding vector;
[0083] Cosine similarity is a commonly used similarity metric to measure the similarity between two vectors. Hyperspectral images have high dimensions and usually contain hundreds of bands, and cosine similarity can handle high-dimensional data well.
[0084] Step 1223: perform a dot product between the mean of the attention map M1 (average of the elements in M1, M1 is a matrix) and the preset threshold parameter λ to generate a threshold matrix with the same shape as the first-order attention map M1;
[0085] Step 1224: Perform matrix subtraction on M1 and the threshold matrix generated in step 1223, and then divide M1 into two attention maps with different score intervals to obtain the second-order high-score attention map M h And the second-order low-score attention map M l ;
[0086]
[0087] in, represents matrix multiplication, represents matrix subtraction, mean(·) represents the mean function, and split(·) represents the split function;
[0088] Step 1225: multiply the attention maps of the two different score intervals obtained in step 1224 by the learnable parameters corresponding to each interval;
[0089] The multiplied high-score attention map M h and low score attention map M l Fusion generates a third-order attention map M f ;
[0090]
[0091] in, represents matrix addition; α represents the learnable parameter between high partitions, and β represents the learnable parameter between low partitions;
[0092] By introducing learnable parameters, the weights of the main features and secondary features can be dynamically adjusted according to the back-propagation process to better adapt to different task requirements. This mechanism makes AFEM more adaptable and flexible.
[0093] Step 1226: The third-order attention map M f The data x obtained in step 121 p Multiply them together to get the output weighted by the attention mechanism;
[0094] Combine the weighted output of the attention mechanism with the data x obtained in step 121 p Add them together to get feature A;
[0095] Use the Sigmoid function to normalize feature A and finally get feature
[0096] The process is:
[0097]
[0098] Among them, σ s (·) represents the Sigmoid function, which is a squeezing function that can compress a wide range of input values to between 0 and 1, making it more suitable as an activation function of a neural network, thereby effectively controlling the distribution range of data.
[0099] In order to retain some of the original features, a skip connection is added to the output result to ensure that the output contains the integrity of the original features. In order to avoid the problem of gradient explosion or gradient diffusion, the output result is normalized using the Sigmoid function, and finally obtained
[0100] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0101] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that in step 13, the output feature of the adaptive feature enhancement module AFEM is input into the Gaussian weighted feature fusion module GWF2, and the Gaussian weighted feature fusion module GWF2 outputs the feature; the specific process is:
[0102] Step 131. Adaptive Feature Enhancement Module AFEM output features Input three-dimensional convolution with a void rate of 1, BN, ReLU, and Dropout in sequence, and Dropout outputs data
[0103] Where c represents the number of channels;
[0104] By jointly extracting spectral and spatial information, GWF2 is able to capture the relationship between primary spectral and spatial features, which will be further fused in subsequent operations to form higher-level feature representations;
[0105] The expression is:
[0106]
[0107] Among them, φ(·) represents the Dropout operation, and the probability is set to 0.1; σ r represents the ReLU activation function; ψ 3d (·) represents a 3D batch normalization layer; H i , W i and D i Represent the height, width and depth of the three-dimensional convolution kernel respectively; h, w and d represent the current height position, the current width position and the current depth position respectively; represents the output of the nth feature map of the mth layer at (i, j, k), represents the output of the cth feature map of the m-1th layer at the position (i+h,j+w,k+d), and c represents the number of channels of the feature map connected between the mth layer and the m-1th layer; represents the weight parameter of the nth convolution kernel in the mth layer at (h, w, d), b m,n represents the bias, · represents the multiplication sign;
[0108] Step 132: Output data for Dropout Reshape the spectral dimension data of different channels to obtain the reshaped data
[0109] Through the linear layer, each component in the spectral dimension of the reshaped data x is mapped so that each component can obtain the weight from other components, thereby realizing spectral feature fusion and obtaining spectral dimension fusion data;
[0110] The spectral dimension fusion data is sequentially input into ReLU and Dropout, and Dropout outputs the data;
[0111] Dropout output data and feature enhancement module output features Add them up, and use the Sigmoid function to normalize the added features to finally get the feature x spe ;
[0112] The expression is:
[0113]
[0114] Among them, σ r represents the ReLU activation function, represents the fusion of spectral dimensions, reshape(·) represents the deformation operation; σ s (·) represents the Sigmoid function, φ(·) represents the Dropout operation;
[0115] Dropout and ReLU operations are performed on the features to enhance the generalization ability of the model and introduce nonlinearity. Finally, skip connections are used to retain some of the original features, and all data are normalized using the Sigmoid function;
[0116] After the data is fused by spectral features, it is fused in spatial dimension;
[0117] Step 133: Set feature x spe Input two-dimensional convolution, BN, ReLU, Dropout in sequence, and Dropout outputs data
[0118] Where p and q represent the channel dimension and spatial dimension after two-dimensional convolution transformation respectively;
[0119] The expression is:
[0120]
[0121] Among them, ψ 2D (·) represents a 2D batch normalization layer; H i , W i Represent the height and width of the two-dimensional convolution kernel respectively; h and w represent the current height position and width position respectively; represents the output of the nth feature map of the mth layer at (i, j), represents the output of the cth feature map of the m-1th layer at the position (i+h,j+w), and c represents the number of feature maps connected between the mth layer and the m-1th layer; represents the weight parameter of the nth convolution kernel in the mth layer at (h, w), b m,n represents bias; σ r Represents the ReLU activation function;
[0122] Step 1, 3, 4: Use the linear layer to transform the data x 2d Mapping operation is performed on each component in the spatial dimension, so that each component can obtain the weight from other components, thereby realizing spatial feature fusion and obtaining spatial feature fusion data;
[0123] The spatial feature fusion data is input into ReLU and Dropout in sequence, and Dropout outputs the data;
[0124] Step 1, 3, 5. Use the linear layer to transform the data x spe Mapping operation is performed on each component in the spectral dimension, so that each component can obtain weighting from other components, thereby realizing spectral feature fusion and obtaining spectral feature fusion data;
[0125] Through the linear layer, each component in the spatial dimension of the spectral feature fusion data is mapped so that each component can obtain the weight from other components, thereby realizing spatial feature fusion and obtaining spatial feature fusion data. x re2d It is the same data shape as the Dropout output data in steps 1, 3 and 4;
[0126] It is expressed as:
[0127]
[0128] in, It represents the fusion of spatial dimensions;
[0129] Step 136: Fusion data x with the spatial features obtained in step 135 re2d The data obtained in steps 1, 3 and 4 are summed to obtain the output feature x of the Gaussian weighted feature fusion module GWF2 spa .
[0130] The expression is:
[0131]
[0132] in, represents the fusion of spatial dimensions, x re2d Represents the reshaped data x 2d ; σ r Represents the ReLU activation function;
[0133] In convolutional neural networks, the size of the convolution kernel is fixed, which limits the adaptability of the network to receptive fields of different sizes. However, receptive fields of different sizes are essential for feature extraction of different types and scales. Therefore, using a fixed-size convolution kernel limits the network in processing inputs with different spatial scales. Although convolution kernels of different sizes can be used to alleviate the above problem, increasing the number of convolution kernels will also increase the complexity of the model and may lead to overfitting problems.
[0134] In order to alleviate the above problems, a Gaussian weighted feature fusion module (GWF2) is designed. Its detailed structure is as follows: Figure 3 As shown in the figure. GWF2 can fuse local and global spectral-spatial feature information, so that the network's ability to extract features is enhanced. Specifically, GWF2 contains two residual-like structures. At the starting point of each residual-like structure, a convolutional block is used to extract shallow feature relationships. Subsequently, the local feature information of different channels is fused through a Gaussian-weighted linear layer to form a deeper feature representation. Finally, the network can extract spectral-spatial features under the global receptive field.
[0135] Deep networks require that the variance of activation values in each layer is similar to ensure effective information transmission, but traditional initialization methods such as uniform initialization or Xavier initialization are difficult to adapt to this requirement. In order to ensure the optimization of module performance, both residual structures are assigned initial weights according to the Kaiming normal distribution. Kaiming normal distribution is a weight initialization method proposed by He Kaiming et al. in 2015. It is based on the standard normal distribution and can maintain the stability of the variance of the activation value, thereby avoiding the problem of gradient explosion or disappearance.
[0136] After passing through GWF2, the output data x spa It not only includes the spectral-spatial dependencies at the global scale, but also conducts an in-depth exploration of the spectral-spatial features at the global scale, which greatly facilitates the subsequent operation of Transformer. In summary, GWF2 can better capture the complex features in the data, allowing the model to understand and process the input data more accurately, thereby improving the performance and generalization ability of the model.
[0137] The other steps and parameters are the same as those in Specific Implementations 1 to 5.
[0138] Specific implementation method seven: This implementation method is different from one of the specific implementation methods one to six in that in step one four, the Gaussian weighted feature fusion module GWF2 outputs the feature input global dependency extraction and classification part, and the global dependency extraction and classification part outputs the classification result; the specific process is:
[0139] Next, we enter the Transformer encoder part, whose structure consists of layer normalization (LN), multi-head coordination attention (MHCA) and multi-layer perceptron (MLP).
[0140] Step 141: Before data input, the learnable classification label x cls Cascaded in Gaussian weighted feature fusion module GWF2 output features In front, and add the position code x pos , get the input feature x in ; The expression is:
[0141] x in =concat(x cls ,x spa )+x pos (11)
[0142] Among them, concat means cascade;
[0143] Step 142: Feature x in Input Transformer encoder, Transformer encoder output features;
[0144] Step 143: Transformer encoder output features are input into the linear layer, and the linear layer output features are input into the softmax classifier for classification.
[0145] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0146] Specific implementation eight: This implementation differs from specific implementations one to seven in that in step one to four, feature x is in Input Transformer encoder, Transformer encoder output features;
[0147] The specific process is:
[0148] Step 1421: Set feature x in Input layer normalization layer, layer normalization layer output feature input multi-head collaborative attention, multi-head collaborative attention output feature and feature x in Performing sum processing to obtain the features after sum processing;
[0149] Step 1422. Input the features processed by step 1421 and add them into the normalization layer. Input the output features of the normalization layer into the multilayer perceptron MLP. Add the output features of the multilayer perceptron MLP and the features processed by step 1421 and add them to obtain the added features as the output features of the Transformer encoder.
[0150] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0151] Specific implementation method 9: This implementation method is different from specific implementation methods 1 to 8 in that in step 1421, feature x is in Input layer normalization layer, layer normalization layer output feature input multi-head collaborative attention, multi-head collaborative attention output feature and feature x in Perform sum processing to obtain the features after sum processing; the specific process is:
[0152] Step 14211: Set feature x in Input layer normalization layer, layer normalization layer output feature x i ' n ;
[0153] Step 14212: Input multi-head collaborative attention, and multi-head collaborative attention output features; the specific process is:
[0154] Normalize the layer output features x based on the layer i ' n Get the Q, K and V matrices; the expression is:
[0155]
[0156] Among them, linear means linearization processing; * means multiplication sign;
[0157] Key and Value split their own multi-head branches from the original features through mapping operations, so that there is a certain correlation between them. Query directly uses the multi-head information mapped from the original information to participate in the calculation process of the score matrix. In this way, each head in the Key branch can get the original feature input to varying degrees, thereby achieving more accurate matching calculations. In this way, each score matrix can establish a dependency relationship between itself and the original input, thereby better capturing the features in the input sequence.
[0158] Calculate multi-head attention based on Q, K and V matrices; the expression is:
[0159]
[0160] Among them, x SA represents the result of an attention head, attention(Q,K,V) represents scaled dot product attention, softmax(·) represents the normalization function, represents the square root of the dimension of K, · represents dot product multiplication, and the superscript T represents transposition;
[0161]
[0162] Among them, MHSA(·) represents multi-head attention, concat(·) represents cascade operation, represents the result of the first attention head, represents the result of the second attention head, represents the result of the h-th attention head;
[0163] Combine SE and MHSA to output feature x out , as the multi-head collaborative attention output feature, can further improve the model's understanding ability; the expression is:
[0164]
[0165] SE(x i ' n )=σ s (linear(σ m (linear(avgpool(x i ' n )+maxpool(x i ' n ))))) (16)
[0166] Among them, SE(·) represents the squeeze-excitation attention function, linear(·) represents linear mapping, avgpool(·) represents average pooling, and maxpool(·) represents maximum pooling; σ m represents the Mish activation function; σ s (·) represents the Sigmoid function;
[0167] The input features are processed by the squeeze and excitation attention mechanism (SE) to enhance the representation ability of the features. In order to further improve the performance of the model, a maximum pooling branch is added on the basis of SE. This improvement enables the model to focus more on the peaks in the input features, thereby more accurately capturing the key information in the input sequence.
[0168] Afterwards, the modified SE attention is fused with multi-head attention. On the one hand, SE can enable the model to better focus on the important information in the input features, thereby improving the interpretability of the model and reducing the impact of noise interference. On the other hand, multi-head attention can calculate the score matrix between multiple heads in parallel, and each head can independently calculate the attention weights and generate output vectors. By synergistically combining SE and MHSA, the model's understanding ability can be further improved.
[0169] Step 14213: Multi-head collaborative attention output feature x out With feature x in Perform sum processing to obtain the features after sum processing.
[0170] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0171] Simulation experiment:
[0172] In order to verify the effectiveness of the proposed AFEGW, experiments were conducted on three public classic datasets. In order to avoid the accidental nature of the experiments, all experiments were repeated 10 times, and the average results of the 10 repeated experiments were used as the final results.
[0173] A. HSI Dataset
[0174] In order to verify the performance of the proposed method, this paper conducted extensive experiments on three public datasets, namely Indian Pines, Pavia University, and Houston 2013.
[0175] The Indian Pines dataset was imaged in Indiana, USA, in June 1992, and was taken by the Airborne Visible Infrared Imaging Spectrometer (AVIRIS). The size of the dataset is 145×145, the spatial resolution is about 20 meters, the wavelength range is 0.4-2.5nm, and it contains 16 types of objects and 224 continuous bands. After removing 20 water-absorbing bands (104-108, 150-163, 220), there are actually 200 bands that can be used for training.
[0176] The PaviaUniversity dataset is an image acquired by the ROSIS sensor while flying over Pavia in northern Italy. The size of the dataset is 610×340, the spatial resolution is 1.3 meters, and it contains 9 types of objects and 103 bands.
[0177] The Houston 2013 dataset was imaged in Houston, Texas, and its surrounding rural areas, and was taken by the ITERS CASI-1500 sensor. The dataset has a size of 349×1905, a spatial resolution of 2.5 meters, and contains 15 types of objects and 144 bands.
[0178] Figure 5 The ground-truth graphs of the three datasets are shown, and Tables I to III list the land cover class names, the number of training samples, and the number of test samples of the three datasets. Each dataset is divided into a training set and a test set, where the number of training samples of the Indian Pines and Houston 2013 datasets accounts for 5% of the total number of samples, and the number of training samples of the Pavia University dataset accounts for 1% of the total number of samples.
[0179] Table I
[0180] Land cover categories of the IndianPines dataset, and the number of training and test samples for each category
[0181]
[0182]
[0183] Table II
[0184] Land cover classes of the Pavia University dataset, and the number of training and test samples for each class
[0185]
[0186] Table III
[0187] Land cover categories of the Houston 2013 dataset, and the number of training and test samples for each category
[0188]
[0189]
[0190] B. Experimental Setup
[0191] a. Hardware Configuration
[0192] On the Pytorch 1.10.1 platform, the proposed method AFEGW is implemented in the Python 3.7.0 environment, and an Intel Core i9-9900K processor with 128G memory and an NVIDIAGeForce RTX3090 graphics card with 24G video memory is configured for verification experiments.
[0193] b. Parameter analysis
[0194] Table IV
[0195] Different batch sizes
[0196]
[0197] For a fair comparison, all the compared methods followed the optimal parameter configurations in their respective papers when evaluating the proposed AFEGW method. During the evaluation process, the effects of parameters such as window size, learning rate, batch size, etc. on the classification performance were deeply analyzed. For different datasets, the effects of different parameters on the classification performance are shown in the following figure. Figure 6 As shown in Table IV. Therefore, for the Indian Pines and Pavia University datasets, the window size is set to 13×13, and the learning rate is set to 1e-3. For the Houston 2013 dataset, the learning rate is set to 1e-2, and the window size is 11×11. In addition, a learning rate adjustment strategy with fixed step decay (StepLR) is adopted, that is, the learning rate is multiplied by a factor of 0.9 for each tenth of the total training rounds. Finally, the Adam optimizer is used to train the network, the training rounds are set to 100, and the batch size is set to 64, as shown in Table IV. These parameter settings enable the proposed AFEGW to achieve the best performance on different datasets.
[0198] C. Experimental Description
[0199] a. First-order attention map segmentation threshold analysis
[0200] In the AFEM module, there is a pre-set threshold, which is used in the segmentation process of the first-order attention map. Specifically, this threshold is set to: In order to ensure that the attention map can be effectively segmented accurately, this section experimentally studies the impact of different λ values on the classification performance of the proposed method. The range of λ is set from 1.00 to 1.20, and the classification performance is tested in detail at an interval of 0.05. The experimental results are shown in Figure 7 As shown in the figure. The optimal parameter settings are different for different datasets. Specifically, on the Houston 2013 dataset, when the parameter λ is set to 1.00, the classification performance of the network reaches the best state. Similarly, for the Indian Pines and Pavia University datasets, we also found their respective optimal settings, which are 1.10 and 1.15 respectively.
[0201] b. The influence of learnable parameters α and β in AFEM
[0202] In the AFEM module, the differentiated scaling of different features is achieved through the predefined learnable parameters α and β. This section will verify how the AFEM module achieves the above process through the learnable parameters α and β. The experimental results are shown in Figures 8 to 9 As shown in Table Ⅴ, the growth curves of these two parameters on different datasets are significantly different. Specifically, on the Indian Pines and Pavia University datasets, the parameter α corresponding to the pixel features between high partitions shows an upward trend, and the parameter β corresponding to the pixel features between low partitions shows a downward trend. This change is consistent with the results of most studies. However, on the Houston 2013 dataset, the change trend is different. It can be clearly found that the parameter β shows an upward trend, while the parameter α basically remains near the initial value, which indicates that the pixel features between low partitions are enhanced on the Houston 2013 dataset. Case 1 and Case 2 in Table Ⅴ confirm this. These experimental results prove that the pixel features between low partitions can contribute to the improvement of the classification performance of the network in some scenarios; on the other hand, they also show the adaptability of AFEM, which can judge the impact of different pixel features on the classification performance of the network according to the specific scenario. This flexibility enables AFEM to perform well on different hyperspectral datasets.
[0203] c. Ablation experiment
[0204] Table V
[0205] Ablation experiment
[0206]
[0207] The proposed AFEGW consists of three modules, namely, adaptive feature enhancement module (AFEM), Gaussian weighted feature fusion module (GWF2), and multi-head collaborative attention (MHCA). This section will verify the effectiveness of the three modules proposed, and conduct some ablation experiments on three datasets, using OA as the evaluation indicator, and the results are shown in Table V.
[0208] First, without using any of the proposed modules, the network only contains the basic convolutional layer and the Transformer encoder part. Observing the experimental results in Table V, it can be seen that the classification accuracy of the network is the lowest at this time.
[0209] Secondly, after introducing the adaptive feature enhancement module (AFEM) into the network, the classification accuracy has been improved to a certain extent. This shows that the AFEM module can effectively enhance the features that are conducive to improving the classification performance of the network.
[0210] Then, a Gaussian weighted feature fusion module (GWF2) is added to the network. As can be seen from Table V, the classification accuracy of the network is also significantly improved. Especially on the Indian Pines dataset, OA is improved by 1.92% and 0.92% respectively compared with the first two cases. This shows that GWF2 can effectively integrate local and global feature information, thereby improving classification accuracy.
[0211] Next, after introducing the multi-head collaborative attention (MHCA) module, the improvement in classification accuracy is similar to that of introducing AFEM. This shows that the MHCA module can effectively increase the network's attention to important information, thereby enhancing classification performance.
[0212] Starting from adding two or more modules, the classification accuracy of the network still maintains a certain degree of improvement, which shows that the proposed multiple modules can further enhance the performance of the network through mutual cooperation.
[0213] Finally, when all the proposed modules are added, the classification accuracy of the network reaches the best. Compared with the first case, OA improves by 2.47%, 0.57% and 1.07% on the three datasets respectively. This fully proves the effectiveness of the main modules in the AFEGW network.
[0214] d. Comparative test
[0215] This section compares the proposed AFEGW with eight excellent hyperspectral image classification methods (HSIC), including HybridSN, HResNetAM, BS2T, GAHT, SSTN, CTMixer, SpectralFormer, and SSFTT. Among them, HybridSN and HResNetAM are two HSIC methods based on CNN, GAHT, SpectralFormer, and SSTN are three HSIC methods based on Transformer, and BS2T, CTMixer, and SSFTT are three HSIC methods that combine CNN and Transformer.
[0216] a) Quantitative evaluation
[0217] Tables VI to VIII show the performance of various methods in hyperspectral image classification tasks on different datasets. These methods include CNN-based methods, Transformer-based methods, and methods that combine CNN and Transformer.
[0218] First, in the CNN-based method, HybridSN combines 3D convolution kernels and 2D convolution kernels to explore spectral-spatial features. However, due to the limited receptive field of traditional convolution kernels, it did not achieve good classification accuracy on the three datasets. HResNetAM introduces an attention mechanism based on convolution, which enables the network to pay attention to more important information, so its classification accuracy is improved compared to HybridSN. However, since it can only pay attention to part of the information, it limits the further improvement of its classification accuracy.
[0219] Secondly, among the Transformer-based methods, GAHT and SpectralFormer fully exploit the spectral information in HSI, providing new ideas for hyperspectral image classification. However, due to the lack of extraction of spatial information, its classification accuracy is not ideal. SSTN is the latest Transformer-based HSIC method. It better explores spectral-spatial information through the framework of structured search and also obtains good classification performance.
[0220] Then, in the method combining CNN and Transformer, it can be observed that its classification performance is better than that of the method using only CNN or only Transformer. CTMixer combines CNN and Transformer in parallel, BS2T introduces Transformer on the basis of dense network, and SSFTT extracts shallow information through CNN and then uses feature labeling to give full play to the ability of Transformer to extract high-level semantic features. It can be found that these three methods all take advantage of the advantages of CNN and Transformer at the same time, so they all achieve good classification performance.
[0221] Finally, the proposed method AFEGW integrates CNN and Transformer, and achieves the highest OA, AA and Kappa on the three datasets. The number of categories with the highest classification accuracy also exceeds half of the total number of categories, which is sufficient to prove the advantages of AFEGW over other methods in classification performance. Specifically, in the CNN-based method, compared with the HResNetAM method with the highest classification accuracy, the OA of the proposed AFEGW method on the three datasets is 0.97%, 0.66%, and 2.72% higher respectively; in the Transformer-based method, compared with the SSTN with the highest classification accuracy, the OA of the proposed AFEGW method on the three datasets is 1.61%, 1.26%, and 3.68 higher respectively; in the method combining CNN and Transformer, compared with the SSFTT with the highest classification accuracy, the OA of the proposed AFEGW method on the three datasets is 1.10%, 0.61%, and 0.92% higher respectively. This fully proves the advantages of AFEGW over other methods in classification performance.
[0222] Table VI
[0223] Classification performance obtained by different methods on the IndianPines dataset (the best result is bolded)
[0224]
[0225] Table VII
[0226] Classification performance obtained by different methods on the Pavia University dataset (the best result is bolded)
[0227]
[0228]
[0229] Table VIII
[0230] Classification performance obtained by different methods on the Houston 2013 dataset (the best result is bolded)
[0231]
[0232] b) Comparison of different sample proportions
[0233] In order to explore the robustness of the network in terms of the number of training samples, this section conducts a series of comparative experiments with different sample ratios. The experimental results are as follows: Figure 10 to Figure 12 As shown. It can be clearly observed that, regardless of the sample ratio training scenario, the proposed AFEGW method consistently shows the highest classification accuracy on the three datasets. For the four training ratios of the IndianPines dataset, the OA obtained by AFEGW is 0.66%, 1.10%, 0.46% and 0.25% higher than the best comparison method, respectively. For the five training ratios of the PaviaUniversity dataset, the leading advantages are 0.61%, 0.08%, 0.02%, 0.22% and 0.16%, respectively. For the four training ratios of the Houston 2013 dataset, AFEGW also performs well, leading the best comparison method by 0.94%, 0.92%, 0.24% and 0.16%, respectively. These experimental results fully demonstrate the excellent classification performance of the AFEGW method and its strong robustness.
[0234] c) Visual assessment
[0235] Figures 13 to 15 The classification graphs obtained by all methods on the three datasets of Indian Pines, Pavia University and Houston 2013 are shown. It can be seen that the classification effect of the proposed AFEGW is closer to the real image. This is because the proposed AFEM module can more flexibly weigh the importance of different sample features in the current scene, thereby providing more ideal enhanced features. At the same time, after extracting local features, GWF2 integrates local and global features through Gaussian weighted mapping, so that the network can learn richer feature relationships. Finally, MHCA complements the feature relationships learned by the two attention branches in a collaborative way. These operations make the classification results of AFEGW more accurate.
[0236] d) t-SNE visualization
[0237] In order to visualize the distribution of features more clearly, Figures 16 to 18The t-SNE visualization results of four methods with excellent classification performance on three datasets are shown, including CNN-based HResNetAM, Transformer-based SSTN, SSFTT combining CNN and Transformer, and the proposed AFEGW. It can be observed that the visualization effect of AFEGW is better than other methods, especially on the Indian Pines dataset. Compared with other methods, on the IP dataset, AFEGW achieves larger inter-class distance and smaller intra-class distance, while other classification methods show more serious confusion between different categories. These visualization results fully prove that the proposed method can extract more discriminative features, thereby improving the classification performance of hyperspectral images.
[0238] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. An image classification method based on a Gaussian weighted network with adaptive feature enhancement, characterized in that: The specific process of the method is: Step 1: Establish a Gaussian weighted network AFEGW based on adaptive feature enhancement; The Gaussian weighted network AFEGW based on adaptive feature enhancement consists of three parts: feature enhancement, feature extraction and feature fusion, and global dependency extraction and classification; The feature enhancement part includes the adaptive feature enhancement module AFEM; The feature extraction and feature fusion part includes Gaussian weighted feature fusion module GWF2; The Gaussian weighted feature fusion module GWF2 includes three-dimensional convolution with a void rate of 1, BN, ReLU, Dropout, spectral feature fusion, ReLU, Dropout, Sigmoid function, two-dimensional convolution, BN, ReLU, Dropout, spatial feature fusion, ReLU, Dropout, spectral feature fusion, and spatial feature fusion; The global dependency extraction and classification part includes Transformer encoder and linear layer; The Transformer encoder includes a layer normalization layer, a multi-head collaborative attention MHCA, a layer normalization, and a multi-layer perceptron; The hyperspectral image training set X is input into the Gaussian weighted network AFEGW based on adaptive feature enhancement for training until convergence, and the trained Gaussian weighted network AFEGW based on adaptive feature enhancement is obtained; Step 2: Input the hyperspectral image to be tested into the trained Gaussian weighted network AFEGW based on adaptive feature enhancement to complete the classification of the hyperspectral image to be tested.
2. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 1, characterized in that: The process of obtaining the hyperspectral image training set X in step 1 is as follows: Randomly sampled hyperspectral image training set Tagged with Among them, h, w, and l represent the height, width, and number of spectral bands of the hyperspectral image HSI respectively. Represents a real number.
3. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 2, characterized in that: In the step 1, the hyperspectral image X is input into the Gaussian weighted network AFEGW based on adaptive feature enhancement for training until convergence, and a trained Gaussian weighted network AFEGW based on adaptive feature enhancement is obtained; The specific process is: Step 1: Use PCA to train the hyperspectral image set Perform dimensionality reduction to obtain the data after PCA dimensionality reduction b represents the number of spectral bands; Step 1 and 2: Reduce the dimension of the data X pca Input Adaptive Feature Enhancement Module AFEM, Adaptive Feature Enhancement Module AFEM output features Where s represents the height and width of the window, and b represents the number of spectral bands; Step 13: Output features of the adaptive feature enhancement module AFEM Input the Gaussian weighted feature fusion module GWF2, and the Gaussian weighted feature fusion module GWF2 outputs features; Step 14: The Gaussian weighted feature fusion module GWF2 outputs features and inputs the global dependency extraction and classification part, and the global dependency extraction and classification part outputs the classification result; Step 15: Repeat step 11, step 12, step 13, and step 14 until convergence to obtain a trained Gaussian weighted network AFEGW based on adaptive feature enhancement.
4. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 3 is characterized in that: In the steps 1 and 2, the dimension-reduced data X pca Input Adaptive Feature Enhancement Module AFEM, Adaptive Feature Enhancement Module AFEM output features The specific process is: Step 121: Data X after PCA dimensionality reduction pca Perform 3D cube data extraction to obtain data Step 122: Data Input Adaptive Feature Enhancement Module AFEM, Adaptive Feature Enhancement Module AFEM output features 5. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 4 is characterized in that: In step 122, the data Input the adaptive feature enhancement module AFEM, the adaptive feature enhancement module AFEM outputs features; The specific process is: Step 1221. From data x p Extract the center pixel vector from Step 1222: Calculate the center pixel vector x using cosine similarity c With surrounding pixel data x p The first-order attention map M1 is expressed as: Among them, x i represents the i-th pixel vector in the surrounding pixel data, cosine similarity(·) represents the cosine similarity function, s represents the height and width of the window; ||·|| represents the modulus length of the corresponding vector; Step 1, 2, 2, 3, dot product the mean of the attention map M1 with the preset threshold parameter λ to generate a threshold matrix with the same shape as the first-order attention map M1; Step 1224: Perform matrix subtraction on M1 and the threshold matrix generated in step 1223, and then divide M1 into two attention maps with different score intervals to obtain the second-order high-score attention map M h And the second-order low-score attention map M l ; in, represents matrix multiplication, represents matrix subtraction, mean(·) represents the mean function, and split(·) represents the split function; Step 1225: multiply the attention maps of the two different score intervals obtained in step 1224 by the learnable parameters corresponding to each interval; The multiplied high-score attention map M h and low score attention map M l Fusion generates a third-order attention map M f ; in, represents matrix addition; α represents the learnable parameter between high partitions, and β represents the learnable parameter between low partitions; Step 1226: The third-order attention map M f The data x obtained in step 121 p Multiply them together to get the output weighted by the attention mechanism; Combine the weighted output of the attention mechanism with the data x obtained in step 121 p Add them together to get feature A; Use the Sigmoid function to normalize feature A and finally get feature 6. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 5, characterized in that: In the step 1-3, the output feature of the adaptive feature enhancement module AFEM is input into the Gaussian weighted feature fusion module GWF2, and the Gaussian weighted feature fusion module GWF2 outputs the feature; The specific process is: Step 131. Adaptive Feature Enhancement Module AFEM output features Input three-dimensional convolution with a void rate of 1, BN, ReLU, and Dropout in sequence, and Dropout outputs data Where c represents the number of channels; Step 132: Output data for Dropout Reshape the spectral dimension data of different channels to obtain the reshaped data Map each component in the spectral dimension of the reshaped data x through a linear layer to obtain spectral dimension fused data; The spectral dimension fusion data is sequentially input into ReLU and Dropout, and Dropout outputs the data; Dropout output data and feature enhancement module output features Add them up, and use the Sigmoid function to normalize the added features to finally get the feature x spe ; Step 133: Set feature x spe Input two-dimensional convolution, BN, ReLU, Dropout in sequence, and Dropout outputs data Where p and q represent the channel dimension and spatial dimension after two-dimensional convolution transformation respectively; Step 1, 3, 4: Use the linear layer to transform the data x 2d Mapping operation is performed on each component in the spatial dimension to obtain spatial feature fusion data; The spatial feature fusion data is input into ReLU and Dropout in sequence, and Dropout outputs the data; Step 1, 3, 5. Use the linear layer to transform the data x spe Mapping operation is performed on each component in the spectral dimension to obtain spectral feature fusion data; Through the linear layer, each component in the spatial dimension of the spectral feature fusion data is mapped to obtain the spatial feature fusion data. Step 136: Fusion data x with the spatial features obtained in step 135 re2d The data obtained in steps 1, 3 and 4 are summed to obtain the output feature x of the Gaussian weighted feature fusion module GWF2 spa .
7. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 6, characterized in that: The Gaussian weighted feature fusion module GWF2 in step 1-4 outputs features into the global dependency extraction and classification part, and the global dependency extraction and classification part outputs the classification result; The specific process is: Step 141: Label the learnable classification x cls Cascaded in Gaussian weighted feature fusion module GWF2 output features In front, and add the position code x pos , get the input feature x in ; The expression is: x in =concat(x cls ,x spa )+x pos (11) Among them, concat means cascade; Step 142: Feature x in Input Transformer encoder, Transformer encoder output features; Step 143: Transformer encoder output features are input into the linear layer, and the linear layer output features are input into the softmax classifier for classification.
8. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 7, characterized in that: In step 142, feature x in Input Transformer encoder, Transformer encoder output features; The specific process is: Step 1421: Set feature x in Input layer normalization layer, layer normalization layer output feature input multi-head collaborative attention, multi-head collaborative attention output feature and feature x in Performing sum processing to obtain the features after sum processing; Step 1422. Input the features processed by step 1421 and add them into the normalization layer. Input the output features of the normalization layer into the multilayer perceptron MLP. Add the output features of the multilayer perceptron MLP and the features processed by step 1421 and add them to obtain the added features as the output features of the Transformer encoder.
9. The image classification method based on Gaussian weighted network with adaptive feature enhancement according to claim 8, characterized in that: In the step 1421, feature x in Input layer normalization layer, layer normalization layer output feature input multi-head collaborative attention, multi-head collaborative attention output feature and feature x in Performing sum processing to obtain the features after sum processing; The specific process is: Step 14211: Set feature x in Input layer normalization layer, layer normalization layer output feature x i ' n ; Step 14212: Input multi-head collaborative attention, and output features of multi-head collaborative attention; The specific process is: Normalize the layer output features x based on the layer i ' n Get the Q, K and V matrices; the expression is: Among them, linear means linearization processing; * means multiplication sign; Calculate multi-head attention based on Q, K and V matrices; the expression is: Among them, x SA represents the result of an attention head, attention(Q,K,V) represents scaled dot product attention, softmax(·) represents the normalization function, represents the square root of the dimension of K, · represents dot product multiplication, and the superscript T represents transposition; Among them, MHSA(·) represents multi-head attention, concat(·) represents cascade operation, represents the result of the first attention head, represents the result of the second attention head, represents the result of the h-th attention head; Combine SE and MHSA to output feature x out , as the multi-head collaborative attention output feature; the expression is: SE(x i ′ n )=σ s (linear(σ m (linear(avgpool(x i ′ n )+maxpool(x i ′ n ))))) (16) Where SE(·) represents the squeeze-excitation attention function, linear(·) represents linear mapping, avgpool(·) represents average pooling, and maxpool(·) represents maximum pooling; σ m represents the Mish activation function; σ s (·) represents the Sigmoid function; Step 14213: Multi-head collaborative attention output feature x out With feature x in Perform sum processing to obtain the features after sum processing.
Citation Information
Cited By
Fire coal working condition identification method and system based on flame video multi-scale attention
CN119559561A
