Visual Transform classification method based on shunt decomposition attention
By adopting a shunt decomposition attention mechanism in visual Transformer, combining position-related local attention and sparse global grid attention, the problem of computing efficiency and feature details retention when processing small goals is solved, and the image classification is significantly improved accuracy.
Patent Information
- Application Number
- CN202410900869.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-26
AI Technical Summary
The visual Transformer model is difficult to maintain computational efficiency and retain local feature details of small targets in image classification, especially when dealing with small target objects, resulting in the impact of recognition performance.
Using the visual Transformer classification method based on shunt decomposition attention, a more efficient shunt decomposition attention mechanism (SDSA) is designed to be used to connect position-related local attention and sparse global grid attention. Pay attention to receptive fields of different sizes in the same layer to achieve modeling of local details and global dependencies.
On the Mini-ImageNet and CIFAR-10 datasets, the Top-1 accuracy is 6.34% and 3.93% higher than the baseline model PVT, respectively, proving that this method effectively retains local feature details of small targets while maintaining computational efficiency while maintaining the localization of small targets. ability.
Smart Images

Figure CN119027709A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a visual Transformer classification method based on split-flow decomposition attention. Background Art
[0002] In the field of deep learning visual models, the Transformer architecture has attracted much attention in recent years for its excellent performance, especially in tasks such as image classification and object detection. The rise of this architecture has not only promoted the innovation of model architecture design, but also brought broad application prospects to the field of deep learning. However, the visual Transformer model faces a major challenge in practical applications, namely its huge amount of computation. In order to reduce the amount of computation, the commonly used method is to divide the image into image blocks and stretch it into a one-dimensional vector as the input of the self-attention mechanism. The size of the image block can be regarded as the sampling granularity of the feature map. The larger the divided image block, the coarser the sampling, and the smaller the amount of computation; the smaller the divided image block, the finer the sampling, and the greater the amount of computation. In order to reduce the amount of computation, ViT divides the image blocks into 16×16 blocks, while PVT and Shunted Transformer downsample the K and V feature maps. Although this method can significantly reduce the computational burden of the model, it often leads to the loss of local feature details due to the coarse-grained division of the image blocks, especially when dealing with small target objects. This problem is particularly prominent, affecting the recognition performance of the model.
[0003] To meet this challenge, researchers have proposed a method based on local attention. This method effectively reduces the amount of computation by modeling the relationship within the local window while maintaining the ability to model local features. However, similar to the convolution method, local attention also has limitations, that is, it cannot directly model global dependencies. In order to obtain global dependencies, it is often necessary to superimpose multiple layers of attention mechanisms, which is contrary to the original intention of using the attention mechanism. Therefore, in the study of the visual Transformer model, how to effectively retain the local feature details of small targets while maintaining the computational efficiency of the model is still a problem that needs to be solved. Summary of the invention
[0004] In view of the above technical problems raised by the background technology, the technical problem to be solved by the present invention is: how to effectively retain the local feature details of small targets while maintaining the calculation efficiency of the model.
[0005] To solve the above technical problems, the present invention adopts the following technical solution: a visual Transformer classification method based on split-flow decomposition attention, comprising the following steps:
[0006] S1: Get several publicly labeled images as a training set, one of which is used as a training sample.
[0007] S2: Build and train an improved Transformer classification model. The improved Transformer classification model is an improvement on the existing Transformer classification model by using the attention mechanism. The spatial reduction attention is replaced by the more efficient shunt decomposition attention SDSA. The SDSA is composed of the position-dependent local attention PWSA and the sparse global grid attention GridSA in series, and a multi-branch structure is designed in PWSA and GridSA.
[0008] Input the training samples in the training set into the improved Transformer classification model, and pass through the image block embedding layer, the shunt decomposition attention module, and the feedforward neural network respectively, continuously update the feature map, and finally output the final predicted classification result through the linear layer, calculate the loss between the real label and the predicted label of the training sample, and use back propagation to calculate the gradient of each layer parameter of the model, and adjust the model parameters according to the calculated gradient to minimize the loss function. Continuously iterate the above steps until the loss function is minimized, and then obtain the trained Transformer classification model;
[0009] S3: Input an image to be classified into the trained Transformer classification model. The trained Transformer classification model calculates the probability value of the image to be classified belonging to each category, and outputs the class corresponding to the maximum probability value as the predicted class of the image to be classified.
[0010] Furthermore, in S2, the features output by the image block embedding layer are sent to SDSA, and the process of calculating the feature map corresponding to each training sample is as follows:
[0011]
[0012] X1=X+Concat(head1,…,head h )W O
[0013]
[0014] X2=X1+Concat(head1,…,head h )W O
[0015] Output=X2+MSLE_FFN(LN(X2))
[0016] Among them, the input features First, after position-dependent local attention PWSA, k1 and k2 represent the window sizes used by the two groups of attention heads, h represents the number of attention heads, and W O It represents the linear projection layer that finally merges the multi-head attention. X1 represents the aggregation of the local attention calculation results of multiple heads and the residual connection. X1 is used as the input of the sparse global grid attention GridSA. The grid sizes of the two branches in the grid attention are G1 and G2. X2 represents the aggregation of the global attention results of multiple heads and the residual connection. Then X2 is input into the multi-scale local enhanced feedforward neural network MSLE_FFN. Output is the feature of the final output of the attention module.
[0017] Furthermore, the processing process of the PWSA is as follows:
[0018]
[0019] Q i ,K i ,V i =Reshape(Q i ,Win k (K i ),Win k (V i ))
[0020]
[0021] PWSA(X)=Concat(head0,…,head h )W O
[0022] Among them, X represents the input feature, W i Q , W i K , W i V are the initialization matrices corresponding to Q, K, and V of the i-th head, Q i , K i 、V i =W(·) indicates that the input feature is multiplied by the corresponding matrix to obtain the feature map. Wink(·) indicates that the features within the window size of k×k are aggregated in the channel dimension. Q is updated through the Reshape operation. i , K i 、V i Features, head i represents the attention calculation result obtained by each attention head, Softmax is the activation function, d h is the hidden layer dimension, PWSA represents position-dependent local attention, W ORepresents the final linear projection layer that incorporates multi-head attention.
[0023] Furthermore, the processing process of the GridSA is as follows:
[0024] X'=Split G (X)
[0025]
[0026] GridSA(X,G)=Concat(head0,…,head h )W O
[0027] Among them, Split G It means dividing the Q, K, V feature maps into G×G image blocks, and GridSA means performing grid attention.
[0028] Compared with the prior art, the present invention has at least the following beneficial effects:
[0029] Aiming at the problem that the visual Transformer cannot model local details and global dependencies at the same time, the method of the present invention improves the PVT image classification algorithm, designs a more efficient shunting decomposition attention (SDSA) on the basis of the sparse global grid attention (GridSA), and proposes to decompose attention into position-dependent local attention (PWSA) and sparse global grid attention (GridSA). By adding position information to the local attention, each position can obtain the information interaction with the strongest dependence on itself, which is more conducive to the modeling of local information. PWSA and GridSA are connected in series, and a multi-branch structure is added at the same time, which can pay attention to receptive fields of different sizes in the same layer, which is more conducive to the recognition of multi-scale objects. It has only linear complexity and can model local details and global dependencies at the same time. Finally, this paper names the model that combines image block processing technology and shunting decomposition attention mechanism as shunting decomposition transformer (SDT), and verifies the effectiveness of the proposed model on the Mini-ImageNet and CIFAR-10 datasets. Compared with the baseline model PVT, the Top-1 accuracy on the two datasets is improved by 6.34% and 3.93% respectively. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a position-dependent local attention mechanism.
[0031] Figure 2 Schematic diagram of grid attention.
[0032] Figure 3 Schematic diagram of decomposing attention.
[0033] Figure 4 Schematic diagram of attention decomposition for splitting
[0034] Figure 5 Heat map visualization results for four categories of objects. DETAILED DESCRIPTION
[0035] The present invention proposes a method of decomposing attention into local attention and sparse global attention, and at the same time setting multiple branches on the local and global attention modules to focus on receptive fields of different scales, thereby improving the recognition performance of objects of different sizes. It is a more efficient attention mechanism.
[0036] 1. Position-dependent local attention mechanism
[0037] In the local attention proposed by Swin, a k×k local window is divided and the self-attention mechanism is performed within the window. However, this non-overlapping way of dividing the window makes the features at the edge of the window perform attention calculations on the features on one side, and lacks symmetrical features. At the same time, the pixel point at the center of the window can obtain more comprehensive local information than the other positions. For example, in Figure 1 In the left image, the pixel marked by the five-pointed star calculates local attention with all the pixels to its upper left (in the black box), but lacks feature interaction with the right and bottom sides, which destroys the symmetry and locality. According to the visualization of ViT and DETR, queries in different semantic areas involve completely different key-value pairs. Forcing all queries to process the same set of keys and values may be suboptimal. Generally speaking, the visual dependence of images on close areas is stronger than that on farther areas. Therefore, based on the original local attention, the present invention proposes a position-related local attention mechanism (Position-related window self-attention, PWSA), which divides the window with the position (i, j) of the pixel as the center. Each position can obtain the information interaction with the strongest dependence on itself. At the same time, this overlapping division method can retain the locality of features, such as Figure 1 As shown in the right figure, a local window centered on the pixel itself is set.
[0038] Specifically, the input features are first multiplied by different weight matrices to obtain Q, K, and V feature maps. For the K and V feature maps, the feature values in the k×k region centered at the current position are gathered at each position, so that each pixel on the Q feature map can correspond to the K and V features related to its own position, thereby realizing the position-related local attention mechanism. The overall calculation process is shown in formula (1).
[0039]
[0040] Among them, X represents the input feature, Wi Q , W i K , W i V are the initialization matrices corresponding to Q, K, and V of the i-th head, Q i , K i 、V i =W(·) indicates that the input feature is multiplied by the corresponding matrix to obtain the feature map. Wink(·) indicates that the features within the window size of k×k are aggregated in the channel dimension. Q is updated through the Reshape operation. i , K i 、V i Features, head i represents the attention calculation result obtained by each attention head, Softmax is the activation function, d h is the hidden layer dimension, PWSA represents position-dependent local attention, W O represents the linear projection layer that merges the multi-head attention at the end, Wink(·) represents the features within the window size of k×k in the channel dimension, so the size of the K and V feature maps is
[0041] Q i , K i 、V i The transformations are as follows:
[0042]
[0043] Where d = C / h, h represents the number of attention heads. H, W, C represent the height, width and number of channels of the input features respectively, and k is the size of the local window.
[0044] 2. Sparse Global Grid Attention Mechanism
[0045] Grid attention uses an adaptive image block size method to divide the input image into a fixed number of G×G. The size of the image block is H / G×W / G. Figure 2 As shown in the figure, the image is divided into 3×3 blocks, the pixels at the same position in each image block are grouped, and the pixels of the same color calculate attention to each other, which is similar to the void convolution, so it is a sparse global attention mechanism.
[0046] The calculation process of the grid attention diagram is shown in formula (3).
[0047]
[0048] Among them, Split GIndicates that the Q, K, V feature maps are divided into G×G image blocks, Q i , K i 、V i Indicates that the input feature is multiplied by the corresponding matrix to obtain the feature map, head i represents the attention calculation result obtained by each attention head, Softmax is the activation function, d h is the hidden layer dimension, GridSA means performing grid attention, W O Represents the final linear projection layer that incorporates multi-head attention.
[0049] Split G The transformation of (X) is as follows:
[0050]
[0051] Among them, d = C / h, h represents the number of attention heads.
[0052] 3. Split-Decompose Attention Mechanism
[0053] This paper proposes a decomposing attention mechanism for fusing local and global information, named DSA (Decomposing Self-Attention). Figure 3 As shown, the position-related local attention proposed by the present invention is connected in series with the sparse global grid attention, which can simultaneously obtain local and global receptive fields in the same layer, realize the interaction of local information and global information, and have the ability to model global dependencies while retaining locality. Compared with global self-attention, it can retain the local information of features, and compared with local self-attention, it has the ability to model global dependencies. At the same time, the performance and computational complexity of the model can be balanced by setting the size of the local attention image block and the number of grid attention.
[0054] Due to the multi-scale characteristics of objects, existing attention has poor recognition effects on objects of different scales in natural scenes. This is attributed to the fact that each image block in the attention layer has a uniform information granularity and the same size of receptive field, so it is impossible to capture features of different scales at the same time. In order to effectively model objects of different scales in the same layer, DSA continues to incorporate the "Shuted" idea in the Shunted Transformer, adding multiple branches to the position-related local attention and sparse global grid attention, and obtaining receptive fields of different sizes in one layer. This is named the Shunted Decomposing Self-Attention (SDSA), as shown in Figure 4 shown.
[0055] The calculation process of split-flow decomposition attention is as follows:
[0056]
[0057] X1=X+Concat(head1,…,head h )W O
[0058]
[0059] X2=X1+Concat(head1,…,head h )W O
[0060] Output=X2+MSLE_FFN(LN(X2)) (5)
[0061] Among them, the input features First, after position-dependent local attention PWSA, the local window sizes of the two branches are k1 and k2 respectively, the local attention calculation results of multiple heads are aggregated, and residual connection is performed. The output is used as the input of sparse global grid attention. The grid sizes of the two branches in the grid attention are G1 and G2. The global attention results of multiple heads are aggregated as output, residual connection is performed and input into the multi-scale local enhanced feedforward neural network MsLe_FFN.
[0062] 4. Analysis of the amount of attention calculation by splitting and decomposing
[0063] In the attention calculation, two matrix multiplications are used, a matrix of size m×n is multiplied by a matrix of size n×r. Each position of the output (shown in the black box) will undergo n multiplications, so there are a total of mnr operations. In the attention, the size of K and V is the same, so the computational cost of multiplying the three matrices Q, K, and V is 2mnr.
[0064] For conventional multi-head self-attention MSA, for the input Its computational complexity is shown in formula (6), which mainly consists of the computational complexity of obtaining the Q, K, and V matrices through linear transformation, the computational complexity of multiplying the Q, K, and V matrices in the self-attention mechanism, and the computational complexity of multiplying the weight matrix during the final multi-head attention splicing.
[0065]
[0066] Among them, h represents the number of attention heads.
[0067] The computational complexity of local attention W-MSA is shown in formula (7):
[0068]
[0069] Wherein, M is the window size of local attention, which is set to 7 in the present invention. Compared with formula (6), it can be found that the complexity of W-MSA is proportional to the image size and is reduced to linear complexity.
[0070] The computational complexity of the position-dependent local attention PWSA proposed in the present invention is shown in formula (8):
[0071]
[0072] Among them, k is the size of the local window. Compared with W-MSA, the position-related local attention does not increase the computational complexity of the model, and can obtain feature representations with higher position dependence.
[0073] The computational cost of sparse global grid attention is shown in formula (9):
[0074]
[0075] Where G is the size of the grid. Grid attention has a computational cost that is linearly related to the image size and is a sparse global attention mechanism.
[0076] The split decomposition attention mechanism proposed in the present invention is obtained by connecting the position-improved local attention and the grid attention in series, ignoring the constant terms k and G. The split decomposition attention is the addition of the computational complexity of the local and global attentions, and also has linear complexity. Compared with W-MSA, it can have a global receptive field with only an increase in linear computational complexity, and at the same time has multi-scale characteristics and better recognition performance.
[0077] Introduction of MS-PVT:
[0078] The PVT image classification method is improved. On this basis, multi-scale feature fusion is performed on the image block. A multi-scale feature fusion image block embedding layer (MSFF_PE) is proposed. The lightweight multi-scale feature fusion module is designed to enrich the multi-scale feature information in the image block and enhance the feature representation ability of the image block. In view of the lack of local information modeling in the encoder structure, a multi-scale local enhanced feedforward neural network layer (MSLE_FF) is proposed to restore the one-dimensional sequence output by the attention mechanism to a two-dimensional image for local information interaction, and introduce distance importance to obtain the weighted information of local features. Finally, the above improvements are added to the baseline model and named MS-PVT.
[0079] MSFF_PE is specifically:
[0080] y1 = conv7(input)
[0081] y2=MSFF(y1)
[0082] y3=conv2(y2)
[0083] patch1,…,patch n = flatten(y3)×E
[0084] in, Represents the image of the input MSFF_PE, conv k represents a convolution with a kernel size of k, MSFF represents the multi-scale feature fusion module imp_Inception(a), the step size of conv7 and conv2 convolutions are both set to 2, and the MSFF module does not change the size of the input feature map, so the output is embedded in the image block layer. Flatten means stretching the spatial dimensions of the feature, i.e., length H and width W, so there are n = H / 4 × W / 4 image blocks. is a fully connected layer, so the output As the input of the attention layer. H, W, C represent the height, width and channel dimensions of the feature map of the input image, respectively. D represents the dimension of the hidden layer setting.
[0085] MSLE_FF is as follows:
[0086] x1=FC(input”;θ1)
[0087] y'1'=DWconv3(x1)
[0088] y'2'=DWDilated3conv3(x1)
[0089] y'3'=GELU(x1+y'1'+y'2' / d)
[0090] output"=FC(y'3';θ2)
[0091] Among them, "input" represents the feature map of the input MSLE_FF, "output" represents the output of MSLE_FF, θ1 and θ2 represent the parameters for setting dropout, FC represents the fully connected layer, and y i " represents the output of the i-th layer, DWDilated m conv n Represents a dilated convolution, with a dilation rate of d=m and a convolution kernel of k=n.
[0092] Introduction of LightPVT:
[0093] The visual Transformer classification method based on lightweight spatial reduced attention is referred to as LightPVT. It is based on an improved Transformer classification model. The improved Transformer classification model replaces the spatial reduction layer in the spatial reduced attention of the existing Transformer classification model with a soft segmentation downsampling module. The calculation process of the soft segmentation downsampling module is as follows:
[0094] Q i =XW i Q ,K i =XW i K ,V i =XW i V
[0095] K i =SSD(K i ,R i ),V i =SSD(K i ,R i )
[0096] V i =V i +LE(V i )
[0097]
[0098] SSD-SRA(X)=Concat(head0,…,head h )W O
[0099] Where SSD-SRA represents the soft segmentation downsampling module, SSD represents spatial reduced attention, LE represents the local enhancement layer, X represents the input feature, and W i Q , W i K , W i V are the initialization matrices corresponding to Q, K, and V of the i-th head, Q i , K i 、V i Indicates that the input feature is multiplied by the corresponding matrix to obtain the feature map, R i Represents the downsampling coefficient mentioned above, K i 、V i After the soft segmentation downsampling module is used to reduce the feature map, head i represents the attention calculation result obtained by each attention head, Softmax is the activation function, dh is the hidden layer dimension, SEAttn represents SE attention, W O Represents the final linear projection layer that incorporates multi-head attention.
[0100] In the soft segmentation downsampling module, depthwise separable convolution is used instead of ordinary convolution, and a channel fusion module is added after the spliced attention. The calculation process is as follows:
[0101] Q i =XW i Q ,K i =XW i K ,V i =XW i V
[0102] K i =LightSSD(K i ,R i ),V i =LightSSD(K i ,R i )
[0103] V i =V i +LE(V i )
[0104]
[0105] LightSRA(X)=SEAttn(Concat(head0,…,head h )W O )
[0106] Among them, LightSSD represents the soft segmentation downsampling module using deep separable convolution, LightSRA represents the improved lightweight spatial reduced attention, and SEAttn represents SE attention.
[0107] The processing process of LightSSD is to divide the input features into two branches in the channel dimension, one of which is connected in series. convolutional layer, and the other branch has a stride of R i The maximum pooling of .
[0108] 5. Experimental results and analysis
[0109] The MS-PVT and the proposed split-flow decomposition attention were experimentally verified on the Mini-ImageNet and CIFAR-10 datasets. The experiment used an A100 graphics card, the deep learning framework used in the experiment was pytorch1.7.1, and the python version was 3.7. We used the common training strategy of visual Transformer and the same data enhancement and regularization as DeiT to prevent overfitting, such as Mixup, RandAugment, and CutMix. The model was trained for 300 epochs using the AdamW optimizer, with a weight decay of 0.05 and a learning rate of 5×10 -4 , the batch size is set to 128 for the Mini-ImageNet dataset and 64 for CIFAR-10.
[0110] Mini-ImageNet is a dataset created by the Google DeepMind team by extracting 100 categories of images from the ImageNet dataset. Each category has 600 images, for a total of 60,000 images. Since this dataset is used in the field of small samples, its training set contains 64 categories, the validation set contains 16 categories, and the test set contains 20 categories, so the dataset also needs to be preprocessed. By collecting the corresponding images according to the category and dividing them into training set and test set in a ratio of 8:2, a dataset that can be used for image classification is obtained. Mini-ImageNet has a total of 100 categories, covering a variety of categories from various animals, portraits, food, scenery, etc., with 48,000 images in the training set and 12,000 images in the test set.
[0111] (1) Evaluation of the results of the split-flow decomposition attention mechanism
[0112] In order to verify the effectiveness of the proposed method, the MS-PVT model is used as the baseline model, and the local attention in Swin is combined with the grid attention of MaxViT as the comparison model of the decomposed and split attention model proposed in this invention, referred to as W-GridSA, and the decomposed attention DSA and split decomposed attention SDSA are verified. The Top-1 accuracy, Top-5 accuracy, precision, recall, F1 score and model parameter quantity of each model are shown in Tables 1 and 2.
[0113] Table 1 Performance indicators of the split-flow decomposition attention model on the Mini-ImageNet dataset
[0114]
[0115] Table 2 Performance indicators of the split-flow decomposition attention model on the CIFAR-10 dataset
[0116]
[0117] As can be seen from Table 1 and Table 2, the performance of W-GridSA, DSA, and SDSA is better than the original attention. This also proves the effectiveness of decomposing global attention into local attention and sparse global grid attention, which can reduce the complexity of attention to a linear relationship with the image size while ensuring accuracy. By comparing W-GridSA and DSA attention, it can be found that under the same number of parameters, by introducing position-related local attention, the pixels can focus on the local features centered on themselves, and the performance is better. Compared with MS-PVT, the Top-1 accuracy of DSA on Mini-ImageNet and CIFAR-10 datasets is improved by 0.62% and 0.85% respectively. The performance of the SDSA model with multiple branches is only improved by 0.12% and 0.16% compared with DSA. This may be because the decomposed attention can already learn locality well, so the effectiveness of more fine-grained local attention and grid attention is limited.
[0118] We conducted a weight heat map visualization experiment on the visual Transformer model SDT using SDSA attention, such as Figure 5 As shown in the figure, four types of objects, namely dog, ship, jellyfish and bird, are selected, and the output of the last attention layer in the first stage is visualized. The four rows in the figure represent dog, ship, jellyfish and bird respectively, and the three columns in the figure represent the visualization results of the original image, MS-PVT and SDT models respectively, where the redder the color, the greater the weight, corresponding to the part that the model focuses on, and the lighter the color, the smaller the weight, corresponding to the part that the model roughly pays attention to. It can be found from the figure that the SDT model can pay better attention to the target than the baseline model, and at the same time, the recognition performance of small targets has also been improved. For example, the whole dog in the first row, the middle of the ship in the second row, and the body of the jellyfish. In the fourth row, the model we proposed also assigns weights to the two birds in the picture. Although there are still other weights assigned to the branches, the bird is redder, which also means that the network pays more attention to the target itself. It can be proved that the proposed method of decomposing global attention into local attention and sparse global grid attention has better modeling ability for small targets, and can retain the local detail features of small targets to a large extent.
[0119] (2) Performance comparison of mainstream methods
[0120] We use two datasets, Mini-ImageNet and CIFAR10, to compare with the mainstream methods in the field of image splitting. All models do not use pre-trained weights and are trained from scratch until convergence. All experimental settings are the same as in (1). The Top-1 accuracy, Top-5 accuracy, precision, recall, F1 score and model parameter count of each model are shown in Tables 3 and 4.
[0121] Table 3 Performance comparison of mainstream methods on the Mini-ImageNet dataset
[0122]
[0123] Table 4 Performance comparison of mainstream methods on CIFAR-10 dataset
[0124]
[0125] From the experimental data in Table 3 and Table 4, we can see that the Top-1 accuracy of ViT on the two datasets is 60.02% and 72.39% respectively, which is quite different from the performance on large-scale datasets. It also reflects that the visual Transformer cannot obtain the two-dimensional properties of the image, resulting in poor performance on small and medium-sized datasets. Other visual Transformer models similar to ViT include DeiT, TNT, and PVT. The models BiFormer-S, Shunted-S, and our proposed MS-PVT, which use overlapping convolutions to divide image blocks, can still achieve good performance on small and medium-sized datasets. It can be seen that after using the proposed image block processing technology, the performance of PVT has been effectively improved. The LightPVT method based on the lightweight spatial reduction attention mechanism has the least parameters among many models, and its performance is better than most visual Transformer models, proving the effectiveness of the proposed lightweight spatial reduction attention. The SDT method based on the split-flow decomposition attention mechanism improves the Top-1 accuracy of the baseline model by 6.34% and 3.93% on the two datasets respectively. It can improve the detection performance of the model while reducing the number of parameters. It also achieves better accuracy than other visual Transformer and convolutional neural network models.
Claims
1. A visual Transformer classification method based on split-flow decomposition attention, characterized in that: The steps include: S1: Obtain several publicly labeled images as a training set, one of which is used as a training sample; S2: Build and train an improved Transformer classification model. The improved Transformer classification model is an improvement on the existing Transformer classification model by making an attention mechanism. The spatial reduction attention is changed to the diversion decomposition attention SDSA. The SDSA is composed of the position-related local attention PWSA and the sparse global grid attention GridSA in series, and a multi-branch structure is designed in PWSA and GridSA. The training samples are input into the improved Transformer classification model, and they are respectively passed through the image block embedding layer, the shunting decomposition attention module, and the feedforward neural network, and the feature map is continuously updated. Finally, the final predicted classification result is output through the linear layer, and the loss between the real label and the predicted label of the training sample is calculated. The gradient of each layer parameter of the model is calculated by back propagation, and the model parameters are adjusted according to the calculated gradient to minimize the loss function. The above steps are continuously iterated until the loss function is minimized, and the trained Transformer classification model is obtained at this time; S3: Input an image to be classified into the trained Transformer classification model. The trained Transformer classification model calculates the probability value of the image to be classified belonging to each category, and outputs the class corresponding to the maximum probability value as the predicted class of the image to be classified.
2. The visual Transformer classification method based on split-flow decomposition attention as claimed in claim 1, characterized in that: In S2, the features output by the image block embedding layer are sent to SDSA, and the process of calculating the feature map corresponding to each training sample is as follows: X1=X+Concat(head1,…,head h )W O X2=X1+Concat(head1,…,head h )W O Output=X2+MSLE_FFN(LN(X2)) Among them, the input features First, after position-dependent local attention PWSA, k1 and k2 represent the window sizes used by the two groups of attention heads, h represents the number of attention heads, and W O It represents the linear projection layer that finally merges the multi-head attention. X1 represents the aggregation of the local attention calculation results of multiple heads and the residual connection. X1 is used as the input of the sparse global grid attention GridSA. The grid sizes of the two branches in the grid attention are G1 and G2. X2 represents the aggregation of the global attention results of multiple heads and the residual connection. Then X2 is input into the multi-scale local enhanced feedforward neural network MSLE_FFN. Output is the feature of the final output of the attention module.
3. The visual Transformer classification method based on split-flow decomposition attention as claimed in claim 2, characterized in that: The processing process of the PWSA is as follows: Q i =XW i Q ,K i =XW i K ,V i =XW i V Q i ,K i ,V i =Reshape(Q i ,Win k (K i ),Win k (V i )) PWSA(X)=Concat(head0,…,head h )W O Among them, X represents the input feature, W i Q , W i K , W i V are the initialization matrices corresponding to Q, K, and V of the i-th head, Q i , K i 、V i =W(·) indicates that the input feature is multiplied by the corresponding matrix to obtain the feature map. Wink(·) indicates that the features within the window size of k×k are aggregated in the channel dimension. Q is updated through the Reshape operation. i , K i 、V i Features, head i represents the attention calculation result obtained by each attention head, Softmax is the activation function, d h is the hidden layer dimension, PWSA represents position-dependent local attention, W O Represents the final linear projection layer that incorporates multi-head attention.
4. The visual Transformer classification method based on split-flow decomposition attention as claimed in claim 3, characterized in that: The processing process of the GridSA is: X'=Split G (X) Q i =X'W i Q ,K i =X'W i K ,V i =X'W i V GridSA(X,G)=Concat(head0,…,head h )W O Among them, Split G It means dividing the Q, K, V feature maps into G×G image blocks, and GridSA means performing grid attention.
Citation Information
Patent Citations
Pollen image classification method based on cross attention distillation Transformer
CN113887610A
Fine-grained image classification optimization method and system
CN116543216A
Novel CNN (Convolutional Neural Network) and Transform hybrid neural network structure method and system for visual identification
CN117808044A
Joint modeling method and apparatus for enhancing local features of pedestrians
US11810366B1
