An image classification method based on linear self-attention Transformer

Abstract

The present invention relates to the field of computer vision technology, and in particular to an image classification method based on a linear self-attention Transformer. The method comprises the following steps: S1: sending a picture to an overlapping convolutional coding module of the first stage, and encoding the picture into a picture token by using a data operation; S2: sending the picture token to the Transformer module in the stage, and extracting a picture feature vector; S3: sending the extracted picture feature vector to an overlapping convolutional coding module of the next stage, reducing the number of feature vectors while increasing the feature vector dimension; S4: repeating S2 and S3, from the Transformer module of the last stage to the final output vector; S5: converting the final output vector into a probability representation through a classifier unit, and completing image classification. The present invention can not only effectively extract picture features from pictures, but also significantly reduce the computational complexity of the Transformer module, and improve the ability of the model to extract picture features through overlapping convolutional coding modules and convolutional feedforward neural network modules.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image classification method based on linear self-attention Transformer. Background Art

[0002] Since AlexNet achieved revolutionary performance in the ImageNet classification challenge, CNN network architecture has developed rapidly. Since then, people have proposed deeper and more effective convolutional network structures, such as VGG, GoogleNet, Resnet, DenseNet, HRNet and EfficientNet, which further promoted the wave of deep learning in the field of computer vision. CNN and its variants have become the main backbone architecture for computer vision applications.

[0003] Transformer was first proposed by the Google team in 2017 for translation tasks in the field of natural language processing (NLP). It is able to model long-range dependencies by using a multi-head attention mechanism and can perform parallel computing to improve computational efficiency. Therefore, the Transformer model quickly replaced RNN as the preferred architecture in the field of NLP. However, in the field of computer vision (CV), the convolutional neural network architecture still dominates. Inspired by the great success of Transformer in the field of NLP, researchers began to try to apply Transformer to computer vision tasks. In 2020, Carionet et al. proposed DETR, which treats object detection as a direct set prediction problem and solves it using the Transformer Encoder-Decoder architecture. On the COCO object detection dataset, DETR performs better than Faster-RCNN on large object detection, but worse on small objects. In 2020, Dosovitskiy et al. first tried to directly apply the standard Transformer model to image classification tasks with minimal modifications, calling it Vision Transformer (ViT). ViT first divides the image into non-overlapping patches of fixed size and marks these patches as a series of visual tokens. These visual tokens are then directly fed into the Encoder module in the standard Transformer. These visual tokens are processed in the same way as tokens in NLP, and the image is classified based on the output of the Encoder module. After pre-training on the large dataset JFT-300M, ViT can achieve close to or exceed the state-of-the-art on multiple image recognition task sets. In 2021, Hugo Touvron et al. proposed a training data augmentation method and a distillation strategy based on distilled tokens to efficiently train the ViT model, called DeiT. DeiT achieved an accuracy of 83.1% on the ImageNet dataset using only 86M parameters through a set of excellent hyperparameters and distillation training strategies. Subsequently, Li Yuan et al. believed that ViT's method of tokenizing image patches was too simple and could not fully model local information such as edges and lines in the image. Therefore, a novel progressive tokenization method was proposed to aggregate the information of adjacent tags into one tag, called T2T. When the resolution of the pre-training images is 384×384, T2T achieves 83.3% accuracy on the ImageNet dataset with only 21.5M parameters.

[0004] After pre-training on large datasets, the ViT model and its variants can achieve good results on many image classification task sets. However, the ViT model also has many shortcomings. Since the self-attention mechanism is a key component of the Transformer architecture, and the computational complexity of the self-attention mechanism is quadratic with the number of input tokens. Therefore, when the input image resolution is too high, the amount of computation required for image classification using ViT will be very large. In addition, ViT lacks inductive biases such as translation invariance and locality compared to convolutional neural networks, so the amount of data required for ViT model training is greater than the amount of data required for convolutional neural networks. Summary of the invention

[0005] The purpose of the present invention is to solve the above problems and provide an image classification method based on linear self-attention Transformer. It uses two linear self-attention mechanisms to construct alternating Transformer layers, which can greatly reduce the computational complexity of the Transformer module. The overlapping convolutional encoding modules and convolutional feedforward neural networks can bring locality to the model and reduce the model's dependence on large data sets.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An image classification method based on linear self-attention Transformer includes the following steps:

[0008] S1: Send the image to the overlapping convolutional coding module of the first stage, and use convolution operations to encode the image into image tokens;

[0009] S2: Send the image token to the Transformer module in this stage to extract the image feature vector;

[0010] S3: Send the extracted image feature vector to the overlapping convolutional coding module of the next stage to reduce the number of feature vectors while increasing the feature vector dimension;

[0011] S4: Repeat S2 and S3 to get the final output vector from the Transformer module of the last stage;

[0012] S5: The final output vector is converted into a probability representation through the classifier unit to complete the image classification.

[0013] As a preferred technical solution of the present invention: the overlapping convolutional coding modules in S1 and S3 use overlapping patches to mark images.

[0014] As a preferred technical solution of the present invention: the Transformer module in S2 includes a local linear self-attention module, a global linear self-attention module and a convolutional feedforward neural network; the local linear self-attention module and the convolutional feedforward neural network constitute a Transformer layer for capturing local features; the global linear self-attention module and the convolutional feedforward network constitute another Transformer layer for capturing global features; the Transformer layer for capturing local features and the Transformer layer for capturing global features are alternately arranged, first capturing local self-attention and then capturing global attention.

[0015] As a preferred technical solution of the present invention: the local linear self-attention module is based on the multi-head self-attention mechanism, first inputting the two-dimensional feature vector x ∈ R N×d , transformed into a three-dimensional feature map x ∈ R (H×W×d) , and then re-divide the three-dimensional feature map into feature maps x containing M sub-windows p ∈R (M×p×p×d) , where N represents the number of feature vectors, d is the pass number, H, W are the length and width of the two-dimensional feature map, that is, N = H × W; M represents the number of sub-windows, p represents the length and width of the sub-window, that is, M = (H / p×W / p).

[0016] As the preferred technical solution of the present invention: the global linear self-attention module is based on self-attention, and each input is first linearly projected into three sets of trainable query Q, key K, and value V; before calculating the attention score map, the key K and value V are pooled into a feature tensor with a fixed spatial size through an average pooling operation, which can greatly reduce the amount of calculation for subsequent calculations; then the pooled key K is dot-producted with all tokens Q, and divided by the scaling factor, and the attention weight map is obtained through the Softmax activation function; finally, the obtained attention weight map and the pooled value V are matrix multiplied to obtain a new token vector.

[0017] As a preferred technical solution of the present invention: the convolutional feedforward neural network includes two linear layers, a depthwise separable convolutional layer and a GELU activation function; the two linear layers and the GELU activation function can fuse vectors, transform each position, and project them to the required dimension; the depthwise separable convolutional layer is a convolutional layer in which the number of convolution kernels is the same as the number of channels, one convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel; the depthwise separable convolutional layer can bring a locality mechanism to the Transformer and improve network performance.

[0018] As a preferred technical solution of the present invention: S3 is specifically: first input the two-dimensional feature vector x ∈ R N×d , transformed into a three-dimensional feature map x ∈ R (H×W×d) , where N represents the number of feature vectors, d is the pass number, H, W are the length and width of the three-dimensional feature map, that is, N = H × W; then use overlapping patches to mark the feature map, that is, enlarge the patch window so that adjacent windows overlap half of the area, and fill 0 around the feature map to maintain the resolution; finally, reduce the number of feature vectors by half and double the dimension of the feature vector.

[0019] As a preferred technical solution of the present invention: the feature vector in S4 will pass through the convolutional coding module and the Transformer module of each stage in turn; the size of the feature vector in each stage is different, that is, multi-scale feature vectors can be obtained.

[0020] As a preferred technical solution of the present invention: in S5, the feature vector obtained in the last stage is average pooled to obtain a one-dimensional vector; the one-dimensional vector is sent to the classifier to obtain the classification result, which is compared with the actual result, the model is optimized, and this process is repeated to train the model; after the training is completed, the model can output the classification result of the corresponding input image through the classifier.

[0021] The image classification method based on linear self-attention Transformer described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0022] (1) The present invention utilizes two linear attention mechanisms to construct two Transformer layers. The local linear self-attention Transformer is used to capture local information, and the global linear self-attention Transformer is used to capture global information. The two Transformer layers are alternately arranged. Since the computational complexity of the two attention mechanisms is linearly related to the number of input tokens, the computational complexity of the Transformer module can be greatly reduced.

[0023] (2) In order to fully model local information, the overlapping convolutional encoding module uses overlapping patches to mark feature maps and adds a deep separable convolutional layer to the standard feedforward neural network. This can bring a locality mechanism to the Transformer, improve network performance, and reduce dependence on data.

[0024] (3) The overall architecture of the model of the present invention is composed of multiple stages. The number and dimension of feature vectors in each stage are different. Therefore, the model of the present invention can generate multi-scale feature representations and can serve as the backbone network for many downstream visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall architecture of the network model in the method of the present invention;

[0026] Figure 2 A schematic diagram of an architecture in which two Transformer layers are alternately arranged in the method of the present invention;

[0027] Figure 3 Schematic diagram of an overlapped convolutional coding module in the method of the present invention;

[0028] Figure 4 Schematic diagram of local linear self-attention in the present invention;

[0029] Figure 5 Schematic diagram of global linear self-attention in the present invention;

[0030] Figure 6 Schematic diagram of the convolutional feedforward neural network of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can have a deeper understanding of the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0032] like Figure 1 As shown, the present invention proposes an image classification method based on linear self-attention Transformer, and its network model is mainly composed of four different stages, each stage is composed of overlapping convolutional coding modules and Transformer modules. The computational complexity of the attention mechanism of the Transformer module is linearly related to the number of input tokens. Therefore, compared with the ViT model and some of its variants, the computational complexity of the network model of the present invention is significantly reduced, and in order to fully model local information, the present invention uses overlapping convolution modules to mark feature maps. Finally, in order to bring a locality mechanism to the model of the present invention, the present invention introduces deep separable convolutions in the standard feedforward neural network; at the same time, the number and dimension of feature vectors in each stage are different, so the present invention can obtain multi-scale features, providing more possibilities for the subsequent application of the model of the present invention.

[0033] An image classification method based on linear self-attention Transformer includes the following steps: S1: sending an image to an overlapping convolutional coding module of the first stage, and encoding the image into an image token by using a convolution operation; S2: sending the image token to the Transformer module in the stage to extract an image feature vector; S3: sending the extracted image feature vector to an overlapping convolutional coding module of the next stage, reducing the number of feature vectors while increasing the feature vector dimension; S4: repeating S2 and S3, from the Transformer module of the last stage to the final output vector; S5: converting the final output vector into a probability representation through a classifier unit to complete image classification.

[0034] The overlapping convolutional coding modules in S1 and S3 use overlapping patches to mark images. Figure 3 As shown, the present invention uses overlapping patches to mark feature maps, that is, the patch window is enlarged so that adjacent windows overlap half of the area, and zeros are filled around the feature map to maintain resolution; this is achieved using 0-filled convolution. Specifically, given an H×W×C input, it is input into a convolution layer with a stride of S, a convolution kernel size of 2S-1, a padding size of S-1, and a convolution kernel number of 2C, and the output size is (H / S×H / S×2C).

[0035] like Figure 2 As shown, the Transformer module in S2 includes a local linear self-attention module, a global linear self-attention module and a convolutional feedforward neural network; the local linear self-attention module and the convolutional feedforward neural network constitute a Transformer layer for capturing local features; the global linear self-attention module and the convolutional feedforward network constitute another Transformer layer for capturing global features; the Transformer layer for capturing local features and the Transformer layer for capturing global features are alternately arranged, first capturing local self-attention and then capturing global attention.

[0036] like Figure 4 As shown in the figure, the local linear self-attention module is based on the multi-head self-attention mechanism. First, the input two-dimensional feature vector x ∈ R N×d , transformed into a three-dimensional feature map x ∈ R (H×W×d) , and then re-divide the three-dimensional feature map into feature maps x containing M sub-windows p ∈R (M×p×p×d), where N represents the number of feature vectors, d is the pass number, H, W are the length and width of the two-dimensional feature map, that is, N = H × W; M represents the number of sub-windows, p represents the length and width of the sub-window, that is, M = (H / p×W / p).

[0037] This process can be achieved through the following calculation:

[0038] x = x.reshape(B,H,W,C)

[0039] x=x.view(B,H / / window_size,window_size,W / / window_size,window_size,C)

[0040] windows=x.permute(0,1,3,2,4,5).contiguous().view(-1,window_size,window_size,C)

[0041] Among them, B represents the number of batch_size, H and W represent the length and width of the feature map, window_size represents the size of the sub-window, and d represents the number of channels. Multi-head self-attention is calculated in each sub-window.

[0042] like Figure 5 As shown in the figure, the global linear self-attention module is built on the basis of self-attention. First, each input is linearly projected into three sets of trainable query Q, key K, and value V. Before calculating the attention score map, the key K and value V are pooled into a feature tensor with a fixed spatial size through an average pooling operation, which can greatly reduce the amount of computation for subsequent calculations. Then, the dot product operation is performed on the pooled key K through all tokens Q, and divided by the scaling factor, and the attention weight map is obtained through the Softmax activation function. Finally, the obtained attention weight map and the pooled value V are matrix multiplied to obtain a new token vector.

[0043] like Figure 6 As shown, the convolutional feedforward neural network includes two linear layers, one depthwise separable convolutional layer and a GELU activation function; the two linear layers and the GELU activation function can fuse vectors, transform each position, and project them to the required dimension; the depthwise separable convolutional layer is a convolutional layer in which the number of convolution kernels is the same as the number of channels, one convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel; the depthwise separable convolutional layer can bring a locality mechanism to the Transformer and improve network performance.

[0044] S3 is as follows: first, the input two-dimensional feature vector x ∈ R N×d, transformed into a three-dimensional feature map x ∈ R (H×W×d) , where N represents the number of feature vectors, d is the pass number, H, W are the length and width of the three-dimensional feature map, that is, N = H × W; then use overlapping patches to mark the feature map, that is, enlarge the patch window so that adjacent windows overlap half of the area, and fill 0 around the feature map to maintain the resolution; finally, reduce the number of feature vectors by half and double the dimension of the feature vector.

[0045] The feature vector in S4 will pass through the convolutional coding module and the Transformer module of each stage in turn; the size of the feature vector in each stage is different, that is, multi-scale feature vectors can be obtained, which is convenient for the future expansion of the model of the present invention.

[0046] In S5, the feature vector obtained in the last stage is average pooled to obtain a one-dimensional vector; the one-dimensional vector is sent to the classifier to obtain the classification result, which is compared with the actual result to optimize the model, and this process is repeated for model training; cross entropy loss is used for model training here; after the training, the model obtains a classification vector through the classifier, and the one-dimensional vector with the largest value is the category of the image.

[0047] The present invention proposes an image classification method based on a linear self-attention Transformer, which can not only effectively extract image features from images but also significantly reduce the computational complexity of the Transformer module, and improve the ability of the model to extract image features through overlapping convolutional coding modules and convolutional feedforward neural network modules. Based on the extracted image feature vectors, the model of the present invention can effectively complete the classification of images. The model of the present invention can also provide multi-scale image features and can serve as the backbone network for many downstream visual tasks.

[0048] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. An image classification method based on linear self-attention Transformer, characterized in that: The following steps are involved: S1: Send the image to the overlapping convolutional coding module of the first stage, and use convolution operations to encode the image into image tokens; S2: Send the image token to the Transformer module in this stage to extract the image feature vector; The Transformer module in S2 includes a local linear self-attention module, a global linear self-attention module and a convolutional feedforward neural network; The local linear self-attention module and the convolutional feedforward neural network constitute a Transformer layer for capturing local features; the global linear self-attention module and the convolutional feedforward network constitute another Transformer layer for capturing global features; the Transformer layer for capturing local features and the Transformer layer for capturing global features are alternately arranged, first capturing local self-attention and then capturing global attention; The local linear self-attention module is based on the multi-head self-attention mechanism. First, the input two-dimensional feature vector x∈R N×d , transformed into a three-dimensional feature map x∈R (H×W×d) , and then re-divide the three-dimensional feature map into feature maps x containing M sub-windows p ∈R (M×p×p×d) , where N represents the number of feature vectors, d is the number of passes, H, W are the length and width of the two-dimensional feature map, that is, N=H×W; M represents the number of sub-windows, p represents the length and width of the sub-window, that is, M=(H / p × W / p); S3: Send the extracted image feature vector to the overlapping convolutional coding module of the next stage to reduce the number of feature vectors while increasing the feature vector dimension; The specific step S3 is: first input the two-dimensional feature vector x∈R N×d , transformed into a three-dimensional feature map x∈R (H×W×d) , where N represents the number of feature vectors, d is the pass number, H, W are the length and width of the three-dimensional feature map, that is, N=H×W; then use overlapping patches to mark the feature map, that is, enlarge the patch window so that adjacent windows overlap half of the area, and fill 0 around the feature map to maintain the resolution; finally, reduce the number of feature vectors by half and double the dimension of the feature vector; S4: Repeat S2 and S3 to get the final output vector from the Transformer module of the last stage; The feature vector in S4 will pass through the convolutional coding module and the Transformer module of each stage in turn; the size of the feature vector in each stage is different, that is, multi-scale feature vectors can be obtained; S5: The final output vector is converted into a probability representation through the classifier unit to complete the image classification.

2. The image classification method based on linear self-attention Transformer according to claim 1, characterized in that: The overlapping convolutional coding modules in S1 and S3 use overlapping patches to mark images.

3. The image classification method based on linear self-attention Transformer according to claim 1, characterized in that: The global linear self-attention module is built on the basis of self-attention. Each input is first linearly projected into three sets of trainable query Q, key K, and value V. Before calculating the attention score map, the key K and value V are pooled into a feature tensor with a fixed spatial size through an average pooling operation, which can greatly reduce the amount of computation for subsequent calculations. Then, the dot product operation is performed on the pooled key K through all tokens Q, and divided by the scaling factor, and the attention weight map is obtained through the Softmax activation function. Finally, the obtained attention weight map and the pooled value V are matrix multiplied to obtain a new token vector.

4. The image classification method based on linear self-attention Transformer according to claim 3, characterized in that: The convolutional feedforward neural network includes two linear layers, one depth-separable convolution layer and a GELU activation function; the two linear layers and the GELU activation function can fuse vectors, transform each position, and project them to the required dimension; the depth-separable convolution layer is a convolution layer in which the number of convolution kernels is the same as the number of channels, one convolution kernel is responsible for one channel, and one channel is convolved by only one convolution kernel; the depth-separable convolution layer can bring a locality mechanism to the Transformer and improve network performance.

5. The image classification method based on linear self-attention Transformer according to claim 4, characterized in that: In S5, the feature vector obtained in the last stage is average pooled to obtain a one-dimensional vector; the one-dimensional vector is sent to the classifier to obtain the classification result, which is compared with the actual result to optimize the model, and this process is repeated to perform model training; After training, the model can output the classification results of the corresponding input images through the classifier.

Images