Hyperspectral image classification method based on spectral-spatial self-attention and Transformer network
By using the spectral-space self-attention and Transformer network methods in hyperspectral image classification, combined with dimensionality reduction and multiple attention modules, the problem of inconsistent contributions between bands and pixels in hyperspectral images is solved, and the classification effect with high precision and low computational volume is achieved.
Patent Information
- Application Number
- CN202311369853.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2043-10-23
AI Technical Summary
The spectral bands in hyperspectral images are very redundant, and the contributions of each band and each pixel to the classification are different, resulting in increased model complexity and difficulty in training.
Adaptive selection and feature extraction of channels are achieved through principal component analysis and dimensionality reduction through CBAM module, spectral attention module, spatial attention module and multi-scale convolution module.
It improves the classification accuracy of hyperspectral images, reduces the computational amount, realizes lightweight and efficient feature extraction and channel selection, and enhances the spatial extraction capability and model performance.
Smart Images
Figure CN117315481B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a hyperspectral image classification method based on spectral-spatial self-attention and Transformer network. Background Art
[0002] Each pixel in a hyperspectral remote sensing image has hundreds of narrow spectral bands from ultraviolet to infrared, which contain the spectral information and spatial information of the imaging target. Due to its large spectral dimension, the spectral information in hyperspectral images is much richer than that in multispectral and panchromatic remote sensing images. The spectral and spatial information can be used to classify imaging targets, and has been widely used in urban construction, marine observation, precision agriculture, disaster prevention and mitigation, resource exploration and other fields. Among the remote sensing hyperspectral image analysis tasks, the hyperspectral image classification task is a relatively important and basic image analysis task.
[0003] In the early days, hyperspectral image classification was mainly based on machine learning. The main steps of classification methods based on machine learning are feature extraction and classifier training. Feature extraction methods include random minimum spanning forest and local covariance matrix representation. Classifier training methods include linear discriminant method and support vector machine. However, these hyperspectral image classification methods based on machine learning are insufficient in feature extraction of spatial information.
[0004] In recent years, deep learning technology has shown strong discrimination and learning capabilities in hyperspectral image classification tasks. Among them, convolutional neural network (CNN) has the characteristics of local perception and parameter sharing. At the same time, it can retain the original structure when learning features and greatly reduce the number of network parameters. However, one-dimensional CNN can only use spectral information without considering spatial information, and two-dimensional CNN lacks the use of spectral information in the feature extraction process. Therefore, it is difficult to further improve the classification performance. In order to simultaneously utilize the spatial and spectral information of the target, three-dimensional CNN is proposed to learn the spatial and spectral joint features of the target. However, hyperspectral images have many bands and high redundancy. The importance of each band is different, and the importance of each spatial feature is also different. Therefore, it is necessary to introduce an attention mechanism so that we can selectively focus on the significant parts instead of treating each part equally, so as to improve the discrimination of spectral and spatial features. At the same time, the convolution operation is mainly aimed at the extraction of neighborhood features, and it is difficult to obtain the long-distance dependency relationship of pixels. At this time, the Transformer network is needed.
[0005] In summary, compared with traditional machine learning methods, hyperspectral images based on deep learning have made significant progress, with better performance and stronger generalization ability. However, hyperspectral image data is highly redundant, and the contributions to classification between bands and pixels are different, which will increase the complexity of the model and make training more difficult. Under a series of challenges, efficient extraction of hyperspectral image features and accurate classification still deserve further research. Summary of the invention
[0006] In order to solve the problem that spectral bands have large redundancy and each band and each pixel have different contributions to classification, the purpose of the present invention is to provide a hyperspectral image classification method based on spectral-spatial self-attention and Transformer network, which can adaptively select channels, mine the correlation between central feature vectors and neighborhood feature vectors in spatial features, and has high classification accuracy and low computational complexity.
[0007] To achieve the above object, the present invention adopts the following technical solution: a hyperspectral image classification method based on spectral-spatial self-attention and Transformer network, the method comprising the following steps in order:
[0008] (1) Input hyperspectral image H∈R h×w×l , where h, w, and l represent the length, width, and number of bands of the hyperspectral image, respectively;
[0009] (2) For the input hyperspectral image H∈R h×w×l Use principal component analysis to reduce the dimension and obtain the hyperspectral image H after dimension reduction. PCA ∈R h×w×b , reduce the number of bands from l to b;
[0010] (3) Hyperspectral image H after dimensionality reduction PCA The CBAM module is used to perform preliminary spectral-spatial feature learning to obtain the first output result;
[0011] (4) The first output result is passed through a batch normalization layer, a two-dimensional convolution layer, and a Relu activation function in turn for preliminary feature learning; then the spectral features are obtained through a spectral attention module, and finally the spectral features are further mined through a batch normalization layer, a two-dimensional convolution layer, and a Relu activation function to obtain a second output result P i，j ;
[0012] (5) The second output result P i，j Input the spatial attention module and get the third output result [P1, P2, ..., P ω ];
[0013] (6) The third output result [P1, P2, ..., P ω] Input the Transformer module to learn the global features and obtain the hyperspectral image after global feature learning;
[0014] (7) The hyperspectral image that has undergone global feature learning is finally subjected to global pooling, flattening, batch normalization, and linear layers to obtain the final classification result.
[0015] The step (3) specifically refers to: the hyperspectral image H after dimensionality reduction PCA The spectral attention is obtained by the channel attention module of the CBAM module, and the spectral attention is combined with the dimensionally reduced hyperspectral image H PCA Multiply the spatial attention as the input of the spatial attention module of the CBAM module to obtain the spatial attention, and multiply the spatial attention by the input of the spatial attention module to obtain the first output result.
[0016] The step (4) specifically comprises the following steps:
[0017] (4a) The first output result is passed through a batch normalization layer, a 3×3 two-dimensional convolution, and a ReLU activation function for preliminary feature learning;
[0018] (4b) Then, the spectral features are obtained through the spectral attention module, wherein the spectral attention module is composed of two groups of depthwise separable convolution blocks, namely, a first group of depthwise separable convolution blocks and a second group of depthwise separable convolution blocks, wherein the first group of depthwise separable convolution blocks comprises a batch normalization layer, a 1×1 point convolution, an activation function LeakyReLU, an SE attention module, a 3×3 channel-by-channel convolution, and an activation function ReLU; and the second group of depthwise separable convolution blocks comprises a batch normalization layer, a 1×1 point convolution, an activation function LeakyReLU, an SE attention module, a 5×5 channel-by-channel convolution, and an activation function ReLU;
[0019] (4c) Then, the spectral feature representation is enhanced through the batch normalization layer, 3×3 two-dimensional convolution and ReLU activation function to obtain the second output result P i，j .
[0020] The step (5) specifically refers to:
[0021] The spatial attention module includes a Manhattan similarity self-attention module for enhancing spatial feature representation and improving feature discriminability, and a multi-scale convolution module for extracting features from different scales and further extracting spatial features;
[0022] The input of the Manhattan similarity self-attention module is P i，j ∈R s×s×c , c is the number of channels, s is the spatial dimension; first, a 1×1 convolution is used to generate three Q, K, V∈R s×s×cFeature map space, feature vector x i，j ∈R 1×1×c is the central feature vector of the feature mapping space Q. Using the Manhattan distance similarity metric, we get the self-similar representation of the central feature vector:
[0023] M i，j =MDSim(x i，j ,K) (1)
[0024] Among them, MDSim is calculated by calculating the eigenvector x i，j The Manhattan self-similarity relationship with K gives the self-similarity matrix M i，j ∈R s×s :
[0025]
[0026] Among them, m i′，j′ is the Manhattan self-similarity value, m i′，j′ It is M i，j The value at position (i′, j′), i′=1,…,s,, j′=1,…,s,m i′，j′ By calculating MDSimV, that is, calculating x i，j and x′ i′，j′ The Manhattan similarity is worth getting;
[0027] After further normalization by the Softmax function, the self-similar attention map MA based on Manhattan distance is obtained. i，j ∈R s×s :
[0028]
[0029] Finally, the recalibrated spatial spectral feature map P′ is obtained. i，j ∈R s×s×c :
[0030]
[0031] in, Represents element-by-element multiplication, and the feature map space V passes through MA i，j Re-weighted, and then similar to the Manhattan-like self-attention module input P i，j Add element by element;
[0032] The multi-scale convolution module includes a first convolution branch, a second convolution branch and a third convolution branch. The multi-scale convolution module further extracts spatial features through three convolution branches. The first convolution branch includes a 5×5 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function. The second convolution branch includes a 3×3 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function. The third convolution branch includes a 1×1 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function. Then, the different features of the three convolution branches are fused together through element-by-element addition operations. Finally, a batch normalization layer and a Relu activation function are used to obtain a third output result [P1, P2, ..., P ω ].
[0033] The step (6) specifically comprises the following steps:
[0034] (6a) The third output result [P1, P2, ..., P ω ] and a learnable label P for classification cls Connect together and add position code PE pos :
[0035] T in =[P cls , P1, P2, …, P ω ]+PE pos (5)
[0036] After that, it is normalized and then passed through a multi-head attention MSA. The multi-head attention MSA consists of multiple independent self-attention SAs. Each self-attention SA learns different relationships, allowing the Transformer network to learn different feature representations at different attention points. The self-attention SA is obtained by three matrices Q1, K1, and V1:
[0037]
[0038] Among them, d k is the dimension of K1;
[0039] Multi-head attention MSA integrates multiple self-attention SAs together and then projects:
[0040] MSA=Concat(SA1,SA2,…,SA h )W (7)
[0041] Where h is the number of SAs, is a learnable parameter matrix;
[0042] (6b) After that, the residual operation is performed, and then the normalization layer and the multi-layer perceptron are passed. The batch normalization layer is used to reduce the internal covariate shift, while the multi-layer perceptron performs further nonlinear transformation and feature combination.
[0043] It can be seen from the above technical scheme that the beneficial effects of the present invention are: first, the present invention uses two-dimensional convolution instead of three-dimensional convolution to take hyperspectral images as the research object, which can reduce the amount of calculation and save costs while maintaining calculation efficiency; second, the spectral attention module proposed in the present invention embeds SE into the deep separable convolution, so as to achieve lightweight and efficient feature extraction and channel selection; third, the spatial attention mechanism proposed in the present invention explores the similarity between the central feature vector and the surrounding feature vector, improves the accuracy of the central feature vector recognition, and enhances the spatial extraction capability; fourth, the multi-scale convolution fusion module proposed in the present invention realizes the complementarity and comprehensive utilization of scale information, further optimizes the processing of spatial information and feature extraction, thereby improving performance; fourth, the present invention uses Transformer to capture long-distance dependencies and nonlinear associations in the data to extract global features. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 Schematic diagram of the method of the present invention
[0045] Figure 2 is a schematic diagram of the spectral attention module in the present invention;
[0046] Figure 3 Schematic diagram of the Manhattan similarity self-attention module in the present invention;
[0047] Figure 4 Schematic diagram of the multi-scale convolution fusion module in the present invention. DETAILED DESCRIPTION
[0048] like Figure 1 As shown, a hyperspectral image classification method based on spectral-spatial self-attention and Transformer network includes the following steps in order:
[0049] (1) Input hyperspectral image H∈R h×w×l , where h, w, and l represent the length, width, and number of bands of the hyperspectral image, respectively;
[0050] (2) For the input hyperspectral image H∈R h×w×l Use principal component analysis to reduce the dimension and obtain the hyperspectral image H after dimension reduction. PCA ∈R h×w×b , reduce the number of bands from l to b;
[0051] (3) Hyperspectral image H after dimensionality reductionPCA The CBAM module is used to perform preliminary spectral-spatial feature learning to obtain the first output result;
[0052] (4) The first output result is passed through a batch normalization layer, a two-dimensional convolution layer, and a Relu activation function in turn for preliminary feature learning; then the spectral features are obtained through a spectral attention module, and finally the spectral features are further mined through a batch normalization layer, a two-dimensional convolution layer, and a Relu activation function to obtain a second output result P i，j ;
[0053] (5) The second output result P i，j Input the spatial attention module and get the third output result [P1, P2, ..., P ω ];
[0054] (6) The third output result [P1, P2, ..., P ω ] Input the Transformer module to learn the global features and obtain the hyperspectral image after global feature learning;
[0055] (7) The hyperspectral image that has undergone global feature learning is finally subjected to global pooling, flattening, batch normalization, and linear layers to obtain the final classification result.
[0056] The step (3) specifically refers to: the hyperspectral image H after dimensionality reduction PCA The spectral attention is obtained by the channel attention module of the CBAM module, and the spectral attention is combined with the dimensionally reduced hyperspectral image H PCA The multiplication is used as the input of the spatial attention module of the CBAM module to obtain the spatial attention, and the spatial attention is multiplied with the input of the spatial attention module to obtain the first output result. The channel attention module uses global maximum pooling and global average pooling in the spatial dimension, and obtains the output result by adding the shared multi-layer perceptron weights; the spatial attention module uses global maximum pooling and global average pooling along the spectral dimension, and then obtains the output result through convolution and activation function.
[0057] The step (4) specifically comprises the following steps:
[0058] (4a) The first output result is passed through a batch normalization layer, a 3×3 two-dimensional convolution, and a ReLU activation function for preliminary feature learning;
[0059] (4b) Then the spectral features are obtained through the spectral attention module, such as Figure 2As shown, the spectral attention module is composed of two groups of depth-separable convolution blocks, namely, a first group of depth-separable convolution blocks and a second group of depth-separable convolution blocks, the first group of depth-separable convolution blocks includes a batch normalization layer, a 1×1 point convolution, an activation function LeakyReLU, an SE attention module, a 3×3 channel-by-channel convolution, and an activation function ReLU; the second group of depth-separable convolution blocks includes a batch normalization layer, a 1×1 point convolution, an activation function LeakyReLU, an SE attention module, a 5×5 channel-by-channel convolution, and an activation function ReLU;
[0060] The SE attention module captures the relationship between channels through compression and excitation operations, emphasizing channels that are useful for classification tasks, thereby improving the discriminability of features. Compression refers to global average pooling in the spatial dimension, and excitation refers to performing two fully connected layer operations and two activation functions to finally obtain the channel weight value;
[0061] (4c) The spectral feature representation is then enhanced through a batch normalization layer, a 3×3 two-dimensional convolution, and a Relu activation function to obtain the second output result Pi,j.
[0062] The step (5) specifically refers to:
[0063] The spatial attention module includes a Manhattan similarity self-attention module for enhancing spatial feature representation and improving feature discriminability, and a multi-scale convolution module for extracting features from different scales and further extracting spatial features;
[0064] like Figure 3 As shown, the input of the Manhattan similarity self-attention module is P i，j ∈R s×s× c, c is the number of channels, s is the spatial dimension; first, a 1×1 convolution is used to generate three Q, K, V∈R s×s×c Feature map space, feature vector x i，j ∈R 1×1×c is the central feature vector of the feature mapping space Q. Using the Manhattan distance similarity metric, we get the self-similar representation of the central feature vector:
[0065] M i，j =MDSim(x i，j ,K) (1)
[0066] Among them, MDSim is calculated by calculating the eigenvector x i，j The Manhattan self-similarity relationship with K gives the self-similarity matrix M i，j ∈R s×s :
[0067]
[0068] Among them, m i′，j′ is the Manhattan self-similarity value, m i′，j′ It is M i，j The value at position (i′, j′), i′=1,…,s,, j′=1,…,s,m i′，j′ By calculating MDSimV, that is, calculating x i，j and x′ i′，j′ The Manhattan similarity is worth getting;
[0069] After further normalization by the Softmax function, the self-similar attention map MA based on Manhattan distance is obtained. i，j ∈R s×s :
[0070]
[0071] Finally, the recalibrated spatial spectral feature map P′ is obtained. i，j ∈R s×s×c :
[0072]
[0073] in, Represents element-by-element multiplication, and the feature map space V passes through MA i，j Re-weighted, and then similar to the Manhattan-like self-attention module input P i，j Add element by element;
[0074] like Figure 4 As shown in the figure, the multi-scale convolution module includes a first convolution branch, a second convolution branch and a third convolution branch. The multi-scale convolution module further extracts spatial features through three convolution branches. The first convolution branch includes a 5×5 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function, the second convolution branch includes a 3×3 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function, and the third convolution branch includes a 1×1 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function; then, through element-by-element addition operation, the different features of the three convolution branches are fused together, thereby realizing the complementarity and comprehensive utilization of scale information, which is helpful to further optimize the processing and feature extraction of spatial information, thereby enhancing the performance and representation ability of the model; finally, through a batch normalization layer and a Relu activation function, the third output result [P1, P2, ..., P ω ].
[0075] The step (6) specifically comprises the following steps:
[0076] (6a) The third output result [P1, P2, ..., P ω] and a learnable label P for classification cls Connect together and add position code PE pos :
[0077] T in =[P cls , P1, P2, …, P ω ]+PE pos (5)
[0078] After that, it is normalized and then passed through a multi-head attention MSA. The multi-head attention MSA consists of multiple independent self-attention SAs. Each self-attention SA learns different relationships, allowing the Transformer network to learn different feature representations at different attention points. The self-attention SA is obtained by three matrices Q1, K1, and V1:
[0079]
[0080] Among them, d k is the dimension of K1;
[0081] Multi-head attention MSA integrates multiple self-attention SAs together and then projects:
[0082] MSA=Concat(SA1,SA2,…,SA h )W (7)
[0083] Where h is the number of SAs, is a learnable parameter matrix;
[0084] (6b) After that, the residual operation is performed, and then the normalization layer and the multi-layer perceptron are passed. The batch normalization layer is used to reduce the internal covariate shift, while the multi-layer perceptron performs further nonlinear transformation and feature combination.
[0085] In summary, the present invention uses two-dimensional convolution instead of three-dimensional convolution to study hyperspectral images, which can reduce the amount of calculation and save costs while maintaining computational efficiency; the spectral attention module proposed in the present invention embeds SE into deep separable convolution to achieve lightweight and efficient feature extraction and channel selection; the spatial attention mechanism proposed in the present invention explores the similarity between the central feature vector and the surrounding feature vectors, improves the accuracy of central feature vector recognition, and enhances spatial extraction capabilities; the multi-scale convolution fusion module proposed in the present invention realizes the complementarity and comprehensive utilization of scale information, further optimizes the processing of spatial information and feature extraction, thereby improving performance; fourth, the present invention uses Transformer to capture long-distance dependencies and nonlinear associations in data to extract global features.
Claims
1. A hyperspectral image classification method based on spectral-spatial self-attention and Transformer network, characterized by: The method comprises the following steps in order: (1) Input hyperspectral image H∈R h×w×l , where h, w, and l represent the length, width, and number of bands of the hyperspectral image, respectively; (2) For the input hyperspectral image H∈R h×w×l Use principal component analysis to reduce the dimension and obtain the hyperspectral image H after dimension reduction. PCA ∈R h×w×b , reduce the number of bands from l to b; (3) Hyperspectral image H after dimensionality reduction PCA The CBAM module is used to perform preliminary spectral-spatial feature learning to obtain the first output result; (4) The first output result is sequentially passed through a batch normalization layer, a two-dimensional convolutional layer, and a ReLU activation function for preliminary feature learning; Then the spectral features are obtained through the spectral attention module, and finally the spectral features are further mined through the batch normalization layer, the two-dimensional convolution layer and the Relu activation function to obtain the second output result P i，j ; (5) The second output result P i，j Input the spatial attention module and get the third output result [P1, P2, ..., P ω ]; (6) The third output result [P1, P2, ..., P ω ] Input the Transformer module to learn the global features and obtain the hyperspectral image after global feature learning; (7) The hyperspectral image after global feature learning is finally subjected to global pooling, flattening, batch normalization and linear layers to obtain the final classification result; The step (4) specifically comprises the following steps: (4a) The first output result is passed through a batch normalization layer, a 3×3 two-dimensional convolution, and a ReLU activation function for preliminary feature learning; (4b) Then, the spectral features are obtained through the spectral attention module, wherein the spectral attention module is composed of two groups of depthwise separable convolution blocks, namely, a first group of depthwise separable convolution blocks and a second group of depthwise separable convolution blocks, wherein the first group of depthwise separable convolution blocks comprises a batch normalization layer, a 1×1 point convolution, an activation function LeakyReLU, an SE attention module, a 3×3 channel-by-channel convolution, and an activation function ReLU; and the second group of depthwise separable convolution blocks comprises a batch normalization layer, a 1×1 point convolution, an activation function LeakyReLU, an SE attention module, a 5×5 channel-by-channel convolution, and an activation function ReLU; (4c) Then, the spectral feature representation is enhanced through the batch normalization layer, 3×3 two-dimensional convolution and ReLU activation function to obtain the second output result P i，j .
2. The hyperspectral image classification method based on spectral-spatial self-attention and Transformer network according to claim 1, characterized in that: The step (3) specifically refers to: the hyperspectral image H after dimensionality reduction PCA The spectral attention is obtained by the channel attention module of the CBAM module, and the spectral attention is combined with the dimensionally reduced hyperspectral image H PCA Multiply the spatial attention as the input of the spatial attention module of the CBAM module to obtain the spatial attention, and multiply the spatial attention by the input of the spatial attention module to obtain the first output result.
3. The hyperspectral image classification method based on spectral-spatial self-attention and Transformer network according to claim 1, characterized in that: The step (5) specifically refers to: The spatial attention module includes a Manhattan similarity self-attention module for enhancing spatial feature representation and improving feature discriminability, and a multi-scale convolution module for extracting features from different scales and further extracting spatial features; The input of the Manhattan similarity self-attention module is P i，j ∈R s×s×c , c is the number of channels, s is the spatial dimension; first, a 1×1 convolution is used to generate three Q, K, V∈R s×s×c Feature map space, feature vector x i，j ∈R 1×1×c is the central feature vector of the feature mapping space Q. Using the Manhattan distance similarity metric, we get the self-similar representation of the central feature vector: M i，j =MDSim(x i，j ,K) (1) Among them, MDSim is calculated by calculating the eigenvector x i，j The Manhattan self-similarity relationship with K gives the self-similarity matrix M i，j ∈R s ×s : Among them, m i′，j′ is the Manhattan self-similarity value, m i′，j′ It is M i，j The value at position (i′, j′), i′=1,…,s,, j′=1,…,s,m i′，j′ By calculating MDSimV, that is, calculating x i，j and x′ i′，j′ The Manhattan similarity is worth getting; After further normalization by the Softmax function, the self-similar attention map MA based on Manhattan distance is obtained. i，j ∈R s ×s : Finally, the recalibrated spatial spectral feature map P′ is obtained. i，j ∈R s×s×c : in, Represents element-by-element multiplication, and the feature map space V passes through MA i，j Re-weighted, and then similar to the Manhattan-like self-attention module input P i，j Add element by element; The multi-scale convolution module includes a first convolution branch, a second convolution branch and a third convolution branch. The multi-scale convolution module further extracts spatial features through three convolution branches. The first convolution branch includes a 5×5 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function. The second convolution branch includes a 3×3 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function. The third convolution branch includes a 1×1 two-dimensional convolution, a batch normalization layer and a LeakyRelu activation function. Then, the different features of the three convolution branches are fused together through element-by-element addition operations. Finally, a batch normalization layer and a Relu activation function are used to obtain a third output result [P1, P2, ..., P ω ].
4. The hyperspectral image classification method based on spectral-spatial self-attention and Transformer network according to claim 1, characterized in that: The step (6) specifically comprises the following steps: (6a) The third output result [P1, P2, ..., P ω ] and a learnable label P for classification cls Connect together and add position code PE pos : T in =[P cls ,P1,P2,...,P ω ]+OR pos (5) After that, it is normalized and then passed through a multi-head attention MSA. The multi-head attention MSA consists of multiple independent self-attention SAs. Each self-attention SA learns different relationships, allowing the Transformer network to learn different feature representations at different attention points. The self-attention SA is obtained by three matrices Q1, K1, and V1: Among them, d k is the dimension of K1; Multi-head attention MSA integrates multiple self-attention SAs together and then projects: MSA = Concat(SA1, SA2,…, SA h )W (7) Where h is the number of SAs, is a learnable parameter matrix; (6b) After that, the residual operation is performed, and then the normalization layer and the multi-layer perceptron are passed. The batch normalization layer is used to reduce the internal covariate shift, while the multi-layer perceptron performs further nonlinear transformation and feature combination.