Fine-grained image recognition method based on DLA +-ViT +

Abstract

The invention provides a fine-grained image recognition method based on DLA +-ViT +, and the method comprises the steps: designing a deep aggregation network image feature enhancement module DLA + which is improved and designed on a deep aggregation network, different-scale semantic and spatial information of an original image is fused, a mask template with the same size as the original image is generated to obtain a feature enhanced image, the feature enhanced image is input into a subsequent network, and the problem of rough ViT model feature extraction granularity is effectively solved. Designing ViT +, redesigning a multi-head attention mechanism in a Transform Encoder module of a ViT model, and introducing three control variables alpha, beta and gamma to change S1 matrix calculation so as to improve an aggregation mode of an attention layer, so that the model pays attention to global features and low-level visual features at the same time; and a convolution module is introduced between an improved multi-head attention mechanism and an MLP module to fuse channel domain features, so that the network flexibility and the nonlinear expression capability are enhanced, and the final recognition precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of fine-grained image recognition, and in particular to a method based on DLA + -ViT + A fine-grained image recognition method. Background Art

[0002] Traditionally, image recognition tasks can be divided into two main categories: semantic-level image recognition and instance-level image recognition. Fine-grained image (FGI) recognition is between the two. It is a more detailed sub-class division of the same type of image, such as identifying bird species. Its characteristics are large intra-class variance and small inter-class variance. Compared with semantic-level image recognition tasks, the difficulty of this recognition task is that category differences usually only exist in very small local areas. Compared with instance-level image recognition tasks, the difficulty is that there is recognition confusion caused by category subdivision, and the problem of interference information in the image affecting the recognition results. At present, the research on fine-grained image recognition tasks at home and abroad mainly includes two recognition methods: artificial feature-based and deep learning-based.

[0003] Recognition methods based on artificial features usually extract artificial features from the image first, and use Fisher Vector or VLAD (Vector of Locally Aggregated Descriptors) algorithms to encode the features, thereby obtaining a vector expression of the image features. Since this type of method relies on manual labor, the artificially extracted features have weak expression capabilities and insufficient discrimination, and have great limitations in practice. They are mainly suitable for small-scale samples and users with a certain field background.

[0004] The recognition method based on deep learning has gone through methods based on local detection, bilinear features and attention mechanism. With the continuous development of natural language processing in recent years, many algorithms used in this field have been proven to be applicable in the field of computer vision, including the Transformer model. This model was introduced into the field of computer vision and is called ViT (Vison Transformer). Its application is becoming more and more extensive and has also been applied to the problem of fine-grained image recognition. Compared with the previous deep learning convolutional neural network model, the research on image recognition based on the ViT model has better recognition effect, but it also has certain shortcomings. For example, the way that the ViT architecture divides the input image into fixed-size image blocks sacrifices the granularity of the features of the significant discriminant area, resulting in the inability to effectively extract the small-size detail features. Compared with the traditional convolution, which can adjust the receptive field and feature extraction method by changing the convolution kernel size and continuously deepening the network, the feature extraction granularity of ViT is too coarse. The deep feature collapse in the ViT architecture due to the self-attention mechanism leads to the disappearance of low-level visual features. The ViT architecture uses the self-attention mechanism to fuse spatial domain features, and then fuses spatial domain and channel domain features through MLP (Multilayer Perceptron). Compared with the channel operation that uses convolution kernels to fuse channel features in CNN convolution operations, the network flexibility and nonlinear ability are weaker.

[0005] The above problems of the ViT model will affect the final image recognition accuracy. The above problems need to be further solved. To this end, the present invention provides a DLA-based + -ViT + A fine-grained image recognition method. Summary of the invention

[0006] The present invention provides a DLA-based + -ViT + A fine-grained image recognition method is proposed to solve the problems of coarse granularity of regional feature extraction, collapse of deep features, disappearance of low-level visual features, and weak network flexibility and nonlinearity in the existing ViT model, which leads to unsatisfactory recognition effect.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The present invention provides a DLA-based + -ViT + The fine-grained image recognition method comprises the following specific steps:

[0009] Step 1: Data preprocessing: Use the CUB200-2011 fine-grained dataset to preprocess the images in the dataset, and use the preprocessed images as training and testing data;

[0010] Step 2: Training: Designing the DLA + -ViT + model and apply the preprocessed image to DLA + -ViT + The model is trained and the model weight with the highest recognition accuracy during the training process is saved;

[0011] Step 3: Test: DLA + -ViT + The model is loaded based on the trained weight file obtained in step 2, and the test set is tested to obtain the test set accuracy, thereby verifying DLA + -ViT + The recognition performance of the model.

[0012] Furthermore, in step 1: data preprocessing, the experiment uses the public fine-grained image benchmark dataset CUB200-2011, and in order to improve the generalization ability and robustness of the network model, the unified data enhancement method applied to fine-grained image recognition tasks is referred to, and data enhancement operations such as scaling, flipping, and cropping are performed on the dataset images. Finally, the dataset images are enhanced to 70,680, and the dataset is divided into a training set and a test set according to an 8:2 ratio. Then, random cropping is used to crop the dataset images into images of a size of 224×224, and then the images are preprocessed by horizontal flipping and normalization operations in sequence.

[0013] Furthermore, in step 2, it specifically includes:

[0014] Step 2.1: Design DLA + -ViT + DLA + Part of the structure, a deep aggregation network image feature enhancement module DLA is proposed based on the improved design of the deep aggregation network (DLA) + ;

[0015] Step 2.2: Design DLA + -ViT + ViT + Partial structure: redesign the multi-head attention mechanism in the Transformer Encoder module of the ViT model, and introduce convolution into the Transformer Encoder module of the ViT model;

[0016] Step 2.3: Complete the DLA + -ViT + Model training, save the model weight with the highest recognition accuracy during the training process.

[0017] Furthermore, in step 2.1, the deep layer aggregation network (DLA) is improved and designed + Module, DLA + The downsampling multiple is increased to 64 times on the original DLA, and then the downsampling results of different multiples are fused from bottom to top in a residual connection manner. After the fusion, three convolution modules and one deconvolution module are added to generate a template of the same size as the original image. The sigmoid activation function is then added to generate a mask, and a dot multiplication operation is added to multiply the generated mask with the original image to obtain the final feature-enhanced image, which is then input into the subsequent network to solve the problem of coarse granularity of feature extraction in the ViT model.

[0018] Furthermore, in step 2.2, DLA is designed + -ViT + ViT + Part of the structure, redesign the multi-head attention mechanism in the Transformer Encoder module of the ViT (Vision Transformer) model, and introduce convolution into the Transformer Encoder module of the ViT model.

[0019] (1) Redesign the multi-head attention mechanism in the Transformer Encoder module of the ViT model. Redesign the S l The matrix calculation formula introduces three control variables α, β, and γ to improve the aggregation method of the attention layer, so that the model pays attention to both global features and low-level visual features. l The matrix is ​​the inner product of each row vector of matrix Q and K, where l is the number of multi-head attention layers.

[0020] The values ​​of the introduced control variables α, β, and γ are between 0 and 1. The optimal values ​​are determined through experiments. Finally, the values ​​of the introduced control variables α, β, and γ are determined to be 0.75, 0.6, and 0.3. The Transformer Encoder part is composed of 12 stacked blocks, which means there are 12 multi-head attention mechanism layers. The first layer is based on the original formula. Calculate S l , (where l is the number of multi-head attention layers, Q l is the query vector of the lth layer, K l is the key vector of the lth layer, T represents transposition, d l is the dimension of the word vector of the first layer), and the second layer calculates S according to the improved formula introducing α l , the second multi-head attention mechanism layer is weighted summed with the first layer, and the remaining layers are calculated according to the improved formula introducing β and γ. l, the weighted sum of the first two multi-head attention mechanism layers and the current layer is realized to obtain the global feature content during feature extraction while also extracting important low-level visual feature expressions, enhancing the feature diversity of the entire network layer, solving the problem of feature collapse, and improving the final recognition progress. The improved S l The matrix calculation formula is:

[0021]

[0022] In the above formula: l represents the number of multi-head attention mechanism layers, Q l is the query vector of the lth layer, K l is the key vector of the lth layer, V l is the value vector of the lth layer, T is the transpose, d l is the dimension of the word vector of the lth layer, and α, β, and γ are control variables.

[0023] (2) The designed convolution module (including two 1×1 convolution kernels and GELU activation function) is introduced into the Transformer Encoder module of the ViT model between the multi-head attention mechanism and the MLP module to fuse channel domain features and enhance the network flexibility and nonlinear expression ability.

[0024] Beneficial effects of the present invention:

[0025] (1) Aiming at the problem that the ViT architecture divides the input image into fixed-size image blocks, which sacrifices the granularity of the salient discriminative region features and makes it impossible to effectively extract the smaller-sized detail features in the image, a deep aggregation network image feature enhancement module (DLA) is proposed based on the deep layer aggregation network (DLA). + ), the semantic and spatial information of different scales of the original image are fused and a mask template with the same size as the original image is generated to obtain a feature-enhanced image, which is input into the subsequent network to effectively solve the problem of coarse granularity of feature extraction of the ViT model;

[0026] (2) To address the problem of deep feature collapse in the ViT architecture due to the self-attention mechanism, which leads to the disappearance of low-level visual features, we redesigned the multi-head attention mechanism in the Transformer Encoder module of the ViT model and redesigned the S l Matrix calculation formula, introducing three control variables α, β, γ to change S l Matrix calculations to improve the aggregation method of the attention layer, so that the model pays attention to both global features and low-level visual features;

[0027] (3) The ViT architecture uses the self-attention mechanism to fuse spatial domain features, and then fuses spatial domain and channel domain features through MLP. Compared with the channel operation using convolution kernel to fuse channel features in CNN convolution operation, the network flexibility and nonlinear ability are weaker. The present invention introduces convolution into the Transformer Encoder module of the ViT model to fuse channel domain features, thereby enhancing the network flexibility and nonlinear expression ability.

[0028] The recognition accuracy is used as the evaluation index. The recognition accuracy mainly refers to the percentage of samples with correct category recognition in the total number of images in the test set. Experiments show that on the CUB200-2011 fine-grained image dataset, compared with the traditional ViT model, the accuracy of the present invention is increased by 1.8%, compared with the StackedLSTM network, the accuracy of the present invention is increased by 1.3%, and compared with the TransFG which is also improved based on ViT, the recognition accuracy is increased by 0.5%. This shows that the DLA obtained by improving ViT is + -ViT + , which improves the recognition ability of fine-grained images. Therefore, within an acceptable time and computing power consumption, by comprehensively using multiple methods to extract, fuse and recognize the feature information of different levels of fine-grained images, the image recognition accuracy can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0030] Figure 1 The DLA of the present invention + -ViT + Flowchart of fine-grained image recognition method;

[0031] Figure 2 The DLA of the present invention + -ViT + The overall structure diagram;

[0032] Figure 3 The DLA of the present invention + Detailed structure diagram of the module;

[0033] Figure 4 ViT of the present invention + Detailed structural diagram of the part;

[0034] Figure 5 For the present invention ViT +A detailed process diagram of the self-attention mechanism of the multi-head attention mechanism that introduces control variables α, β, and γ in the Transformer Encoder module;

[0035] Figure 6 For the present invention ViT + Detailed diagram of some of the convolutional modules added to the Transformer Encoder module. DETAILED DESCRIPTION

[0036] The exemplary embodiments disclosed in the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to be able to fully convey the scope of the present application to those skilled in the art. If not specifically indicated, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0037] Combination Figures 1 to 6 As shown, the present invention provides a method based on DLA + -ViT + The fine-grained image recognition method of the embodiment has the following specific steps:

[0038] Step 1: Data preprocessing: Use the CUB200-2011 fine-grained dataset to preprocess the images in the dataset, and use the preprocessed images as training and testing data;

[0039] Step 2: Training: Designing the DLA + -ViT + Model, and use the preprocessed images to train the model, saving the model weight with the highest recognition accuracy during the training process;

[0040] Step 2.1: Design DLA + -ViT + DLA + Part of the structure, a deep aggregation network image feature enhancement module DLA is proposed based on the improved design of the deep aggregation network (DLA) + ;

[0041] Step 2.2: Design DLA + -ViT + ViT + Partial structure: redesign the multi-head attention mechanism in the Transformer Encoder module of the ViT model, and introduce convolution into the Transformer Encoder module of the ViT model;

[0042] Step 2.3: Complete the DLA + -ViT + Model training, save the model weight with the highest recognition accuracy during the training process.

[0043] Step 3: Test: DLA + -ViT + The model is loaded based on the trained weight file obtained in step 2, and the test set is tested to obtain the test set accuracy and verify the DLA + -ViT + The recognition performance of the model.

[0044] Further, step 1: adopt the public fine-grained image benchmark dataset CUB200-2011, which is a public fine-grained benchmark dataset. The dataset has 200 categories of birds, 16,185 images, 5,994 training set images, and 5,794 test set images. In order to improve the generalization ability and robustness of the network model, the unified data enhancement method applied to the fine-grained image recognition task is used to perform data enhancement operations such as scaling, flipping, and cropping on the dataset images. After enhancement, there are a total of 70,680 images, and the dataset is divided into a training set and a test set according to an 8:2 ratio. The images are cropped to a size of 224×224 using random cropping, and then the images are preprocessed by horizontal flipping and normalization operations in sequence.

[0045] Further, step 2: DLA of the present invention + -ViT + The model is improved on the ViT model (the basic network is ViT-B_16), combined with Figure 2 Compared with the original ViT model, the present invention solves the problems of coarse regional feature extraction, deep feature collapse, disappearance of low-level visual features, and weak network flexibility and nonlinear ability in the ViT model, which result in unsatisfactory recognition effect. + module, and proposed ViT to improve the original structure of ViT + , and get the improved model DLA + -ViT + . The preprocessed dataset images are first input into DLA + The module generates a feature-enhanced image, and then inputs the feature-enhanced image into ViT + Perform image recognition tasks in the model and obtain category results (this process is a round of model training). Apply the preprocessed dataset images according to the set pretrained weight files and hyperparameters to complete DLA + -ViT +Model training, after multiple rounds of model training, save the model weight with the highest recognition accuracy during the training process.

[0046] Furthermore, step 2.1: compared with the original ViT model, in order to solve the problem that the ViT architecture divides the input image into fixed-size image blocks, which sacrifices the granularity of the significant discriminant region features, resulting in the inability to effectively extract the small-size detail features, thereby affecting the final recognition accuracy, the present invention proposes a deep aggregation network image feature enhancement module DLA based on a deep layer aggregation network (DLA) + , combined with Figure 3 , all improvements are marked with dashed boxes. The left part, where the dashed box outside the digital block is an improvement based on the original DLA network, that is, the Hierarchical Deep Aggregation (HDA) module increases the downsampling multiple to 64 times, and then the different downsampling results are fused bottom-up using residual connections. The dashed box on the right is the three convolutional modules (i.e. Figure 3 The dotted box on the right side of the figure (①) and a deconvolution module (i.e. Figure 3 The dotted box on the right (part ②) is used to generate a template of the same size as the original image, and then the sigmoid activation function is added to generate a mask, and the dot product operation is added to multiply the generated mask with the original image to obtain the final feature-enhanced image, which is then input into the subsequent network to solve the problem of coarse granularity of feature extraction in the ViT model.

[0047] Combination Figure 3 , where the upsampling multiples are all 2, the number in the downsampling multiple module is the downsampling multiple, and the number in the aggregation node is the sampling multiple of the current aggregated feature map.

[0048] Combination Figure 3 , DLA +The specific operation of the module structure is as follows: First, the network input original image is downsampled by 4 times, 8 times, 16 times, 32 times, and 64 times in turn through the hierarchical deep aggregation module (HDA), and then the sampling results of different stages are fused from bottom to top in a residual connection manner. That is, first, the information rich in semantics but low resolution is fused with the sampling results of the previous layer. The 64-fold downsampled feature map is fused with the 32-fold downsampled feature map in the previous level after upsampling. This process is repeated until the resolution after fusion is the same as the scale of the 4-fold downsampled original image. The result is then operated by three convolution blocks. Each convolution block contains two 3×3 convolutions (Conv) and a Relu activation function. The convolution operation does not change the size of the feature map, but further extracts image features and reduces the dimension of the feature map channel. The original fused feature map before the input convolution block and the feature map after each convolution block are merged in the channel dimension through the Concat splicing method to deepen the fusion of image semantic information, and then it is input to the next convolution block. Repeat this operation for a total of three convolution blocks. Finally, the result is input into the deconvolution module, which contains two deconvolutions (TransposeConv) and a Relu activation function to generate a template x with the same size as the original image. t , and then generate the auxiliary ViT mask x by performing sigmoid activation on the template mask , the mask is dot-multiplied with the original image to obtain the final feature-enhanced image x en , the process formula is as follows:

[0049]

[0050] x en = x·x mask

[0051] Further, step 2.2: Compared with the original ViT model, the present invention proposes a ViT improved structure to solve the problems of deep feature collapse, disappearance of low-level visual features, and weak network flexibility and nonlinear ability, which result in unsatisfactory recognition effect. + . Combined Figure 4 , Figure 4 ViT on the left + structure, Figure 4 The right side of the figure shows the detailed structure of each encoder block in the improved Transformer Encoder, and the dotted part is the improved part. + The specific operation of the structure is as follows:

[0052] (1) First, pass through DLA + The feature-enhanced image x obtained by the moduleen Image segmentation is performed using a convolutional layer with a convolution kernel size of 16×16 and a step size of 16. The 224×224 three-channel image of the dataset preprocessed in step 1 is divided into 196 16×16 patches, and each patch is encoded as an embedding vector. ( Figure 4 The example in this example is to divide the image into 9 patches.)

[0053] (2) Then the 196 patch embedding vectors obtained in the previous step are input into the linear projection layer (LinearProjection of Flattened Patches), which is flattened into a 196×768 token sequence, namely PatchEmbedding ( Figure 4 The blank ellipse without a number symbol represents the obtained Patch Embedding. Figure 4 In this example, only the Patch Embedding corresponding to 9 patches is used. In addition, add the special character [class] token (i.e. Figure 4 The ellipse with the symbol * in the figure is randomly initialized and continuously updated as the network is trained. Its main function is to encode the statistical characteristics of the entire data set and aggregate the information on all other tokens (global feature aggregation). That is, the [class] token learned by the model integrates the information of all patches and is global. It is more reasonable to use it to predict the label of the entire image, providing more flexibility and expression ability, so that the model has better performance in classification tasks. Its dimension is 1×768. It is concatenated to Patch Embedding using the concat method, and the resulting dimension is 197×768. Then, position encoding Position Embedding (i.e. Figure 4 The ellipse with numbers 0-9 in the figure corresponds to the Position Embedding of each of the 9 patches. Its true value is not a number, but a vector table with a vector dimension of 768. Only numbers are used as examples in the figure. The dimension is 197×768. All Position Embeddings and PatchEmbeddings are added to get the final token sequence with a dimension of 197×768, which is input into the improved TransformerEncoder.

[0054] (3) The improved Transformer Encoder is composed of L (L=12) Encoder Blocks stacked together. Each Block consists of a normalization layer (LayerNorm layer), a multi-head attention mechanism (Multi-HeadAttention) that introduces control variables α, β, and γ, a convolution (Conv) module, a multi-layer perceptron (Multilayer Perceptron, MLP) module, and a residual connection. Figure 4 The right side of the figure shows the detailed structure of each Encoder Block in the improved Transformer Encoder, and the dotted part is the improved part. The dotted box on the lower right side of the figure is the improved multi-head attention mechanism in the original ViT model Transformer Encoder, and the dotted box on the upper right side is the convolution module added to the original ViT model Transformer Encoder. Before adding the convolution module, the result of the previous module needs to be normalized, so a normalization layer is added before the convolution module. The specific operation of each Block is as follows:

[0055] aFirst, input the token sequence obtained in the previous step into the normalization layer to perform layer normalization on the input, standardize the input distribution, alleviate internal covariate shift, improve model generalization ability, and reduce model complexity.

[0056] b Then the layer-normalized sequence obtained in the previous step is input into the multi-head attention mechanism layer. Compared with the original ViT model, in order to solve the problem that the deep features in the ViT architecture are collapsed due to the multi-head attention mechanism, resulting in the disappearance of low-level visual features, thereby affecting the final recognition accuracy, the present invention redesigns the multi-head attention mechanism layer (a multi-head attention mechanism has 12 heads, each head is a self-attention mechanism, that is, there are 12 self-attention mechanisms in total), see Figure 5 The dotted line frame part (where Figure 5 The left part of the solid line is the calculation process diagram of each self-attention mechanism of the second-layer multi-head attention mechanism that introduces the control variable α. Figure 5 The right side of the solid line is the calculation process diagram of each self-attention mechanism of the third to twelfth layer of the multi-head attention mechanism with the control variables β and γ introduced, and the dashed box parts are the improved parts). The calculation process of each self-attention mechanism of the multi-head attention mechanism of the original ViT model is redesigned. lThe matrix calculation method introduces three control variables α, β, and γ, and aggregates the query vector Q (Query) and key vector K (Key) of the first two multi-head attention layers with the current layer (the first layer is not aggregated if there is no other layer before it, and the second layer is only aggregated with the previous layer), so that the model pays attention to global features while also paying attention to low-level visual features. l The matrix is ​​the inner product of each row vector of matrix Q and K, where l is the number of multi-head attention layers.

[0057] Specific process combination Figure 5 The control variables α, β, and γ are between 0 and 1. The optimal values ​​are determined through experiments. Finally, the control variables α, β, and γ are determined to be 0.75, 0.6, and 0.3, which are used to balance the aggregation of Q and K of the first two multi-head attention mechanism layers with the current layer (the first layer will not be aggregated if there is no other layer before it, and the second layer will only aggregate with the previous layer). The Transformer Encoder part is composed of 12 stacked blocks, so there are 12 multi-head attention mechanism layers in total. The first layer is based on the original formula. Calculate S l , (where l is the number of multi-head attention layers, l value is 0-11, Q l is the query vector of the lth layer, K l is the key vector of the lth layer, T is the transpose, d l is the dimension of the word vector of the first layer) The second layer calculates S according to the improved formula introducing α l , realize the weighted summation of the second multi-head attention mechanism layer and the first layer (see Figure 5 The remaining layers are calculated using the improved formula that introduces β and γ. l , realize the weighted summation of the first two multi-head attention mechanism layers and the current layer (see Figure 5 The improved S l The calculation formula for each self-attention mechanism in the matrix and multi-head attention mechanism is:

[0058]

[0059] A l =softmax(S l )

[0060] X l =A l ·V l

[0061] In the above formula: l represents the number of multi-head attention mechanism layers, Ql is the query vector of the lth layer, K l is the key vector of the lth layer, V l is the value vector of the lth layer, T is the transpose, d l is the dimension of the word vector of the first layer, α, β, γ are control variables. The softmax function represents the normalization process and is used to calculate the normalized weight coefficient. l S is the lth layer l The matrix is ​​normalized, X l It is the result value of the self-attention mechanism of a single head of the multi-head attention mechanism in the l-th layer.

[0062] The multi-head attention mechanism layer is used to extract features, allowing features at different positions to interact fully. There are 12 heads in total, each of which is a self-attention mechanism (that is, there are 12 attention mechanisms in total). First, the corresponding result of each self-attention mechanism is calculated, and then a fully connected layer is used to concatenate the attention results obtained by each head to obtain the final value.

[0063] c combination Figure 4 On the right side, the residual of the value obtained by the multi-head attention mechanism layer in the previous step is added to the original token sequence initially input into the EncoderBlock part, and the resulting value is used as the input of the next module layer.

[0064] dThen the value after adding the residuals in the previous step is input into the newly introduced normalization layer to perform layer normalization on the input, standardize the input distribution, alleviate internal covariate shift, improve the model generalization ability, and reduce model complexity.

[0065] eThen the normalized value of the layer obtained in the previous step is input into the convolution module. Compared with the original ViT model, the convolution module is newly introduced to solve the problem of weak network flexibility and nonlinear ability. Figure 6 , the convolution module consists of a 1×1 convolution kernel, a GELU activation function, and a 1×1 convolution kernel. The token sequence input to the Encoder Block remains unchanged after the multi-head attention mechanism, but the sequence dimension needs to be transposed before the convolution operation. The specific formula for the convolution operation is as follows:

[0066]

[0067] Among them, Z i represents the output of the token sequence after the convolution block, X iis the input, W is the weight matrix of the convolution layer (W1 is the weight matrix of the first convolution kernel, W2 is the weight matrix of the second convolution kernel), σ() is the Gelu (Gaussian error linear units) activation function, N() is the layer normalization function, i is the i-th token in all token sequences, n is the length of the token sequence, and T represents transposition. This convolution module is added between the multi-head attention mechanism and the MLP module in the Transformer Encoder to increase the network flexibility and nonlinear expression ability, and improve the final recognition progress.

[0068] f adds the residual of the value obtained by the convolution module in the previous step to the value obtained in step c, and the resulting value is used as the input of the next module layer.

[0069] gThen the normalized value of the layer obtained in the previous step is input into the MLP module for feature extraction. Its structure is Linear, GELU, Dropout, Linear, Dropout. The first Linear layer enlarges the input value sequence dimension by 4 times to 197×3072; the second Linear layer reduces the sequence dimension and projects it back to 197×768 as the output value.

[0070] h adds the residual of the value obtained by the MLP module in the previous step to the value obtained by step f, and the result value is used as the input of the next module layer. In the entire improved Encoder Block, the multi-head attention mechanism is first used to fuse the spatial domain features, and then the channel domain features are fused through two 1×1 convolution kernels, and finally the channel domain and spatial domain features are integrated through the multi-layer perceptron (MLP) module.

[0071] Repeat the above steps 1) to 8) for L times (12 times) to obtain the final output sequence.

[0072] (4) Combination Figure 4 On the left, the value obtained by the improved Transformer Encoder is subjected to LayerNorm operation. Then the corresponding value of [class] token is extracted by list slicing, with a dimension of 1×768, and the result is input into the subsequent fully connected feedforward neural network (MLP Head) for recognition. Its structure consists of a linear layer (Linear), which is used for the final recognition and output of the image category.

[0073] Further, step 2.3: complete the DLA + -ViT + Model training, saving the model weight with the highest recognition accuracy during the training process;

[0074] The hardware platform used in this experiment is the NVIDIA RTX 3090 server, whose configuration system is Linux. The network model uses the PyTorch framework and python 3.8. The pre-trained model weight file uses the ViT network parameters pre-trained on the ImageNet21K dataset, divides the image slices according to patch size = 16, sets the training parameter batch size to 64, sets the number of training epochs to 500, uses the stochastic gradient descent SGD optimizer, and sets the momentum to 0.9, and the weight decay weight_decay to 5×10 -5 , the initial learning rate lr is set to 0.001, the learning rate decay strategy is cosine annealing, and the cross entropy CrossEntropyLoss loss function is used. + -ViT + The model is trained in a loop according to the set pre-trained weight file and hyperparameters, and the images of the dataset pre-processed in step 1 are trained according to steps 2.1 and 2.2, and the model weight with the highest recognition accuracy during the training process is saved.

[0075] Step 3: Testing: Using DLA + -ViT + Model loading is based on the trained weights obtained in step 2, loading DLA + -ViT + The model is tested on the test set to obtain the test set accuracy and verify the recognition performance of the model.

[0076] In order to verify the effectiveness and rationality of the algorithm proposed in the present invention, an ablation experiment was conducted under the same experimental environment and conditions. The ablation experiment results are shown in Table 1.

[0077] Table 1 Ablation experiment results

[0078]

[0079] The model method of the present invention is based on weakly supervised fine-grained image recognition research. Therefore, in order to verify the effectiveness of the overall improvement, the network of the present invention is compared with the traditional ViT network and other mainstream weakly supervised image recognition methods in recent years. The comparison results are shown in Table 2.

[0080] According to the experimental results in Table 2, the proposed method has a higher recognition accuracy than other methods on the CUB200-2011 dataset. Compared with the original ViT, the proposed model has a recognition accuracy improvement of 1.8%; compared with the currently higher recognition accuracy StackedLSTM, the recognition accuracy has been improved by 1.3%; compared with the TransFG which is also improved based on ViT, the recognition accuracy has been improved by 0.5%. This shows that the DLA obtained by a series of improvements to the ViT model is + -ViT + , which improves the recognition ability of fine-grained images.

[0081] Table 2 DLA + -ViT + Comparison of recognition accuracy with other methods

[0082]

[0083] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A DLA-based + -ViT + A fine-grained image recognition method is characterized in that: The specific steps of this method include: Step 1: Data preprocessing: Use the CUB200-2011 fine-grained dataset to preprocess the images in the dataset, and use the preprocessed images as training and testing data; Step 2: Training: Designing the DLA + -ViT + Model, and use the preprocessed images to train the model, saving the model weight with the highest recognition accuracy during the training process; Step 3: Test: DLA + -ViT + The model is loaded based on the trained weight file obtained in step 2, and the test set is tested to obtain the test set accuracy, thereby verifying DLA + -ViT + The recognition performance of the model.

2. The DLA-based method according to claim 1 + -ViT + A fine-grained image recognition method is characterized in that: In step 1: In data preprocessing, the experiment uses a public fine-grained benchmark dataset CUB200-2011, and in order to improve the generalization ability and robustness of the network model, data enhancement operations such as scaling, flipping, and cropping are performed on the dataset images. Finally, the dataset images are enhanced and divided into training and test sets according to the ratio of 8:2; then random cropping is used to crop the dataset images into images of size 224×224, and then the images are preprocessed by horizontal flipping and normalization operations.

3. The DLA-based method according to claim 1 + -ViT + A fine-grained image recognition method is characterized in that: In step 2, specifically including: Step 2.1: Design DLA + -ViT + DLA + Part of the structure, a deep aggregation network image feature enhancement module DLA is proposed based on the deep layer aggregation network Deep LayerAggregation. + ; Step 2.2: Design DLA + -ViT + ViT + Partial structure: redesign the multi-head attention mechanism in the Transformer Encoder module of the Vision Transformer model, and introduce convolution into the Transformer Encoder module of the ViT model; Step 2.3: Complete the DLA + -ViT + Model training, save the model weight with the highest recognition accuracy during the training process.

4. The DLA-based method according to claim 3 + -ViT + A fine-grained image recognition method is characterized in that: In step 2.1, an improved DLA is designed on the deep aggregation network DLA + Module, DLA + The module increases the downsampling multiple to 64 on the original DLA network, and then fuses the downsampling results of different multiples from bottom to top using residual connections. After the fusion, three convolution modules and one deconvolution module are added to generate a template of the same size as the original image. The sigmoid activation function is then added to generate a mask, and a dot multiplication operation is added to multiply the generated mask with the original image to obtain the final feature-enhanced image, which is then input into the subsequent network to solve the coarse granularity of feature extraction in the ViT model.

5. The DLA-based method according to claim 3 + -ViT + A fine-grained image recognition method is characterized in that: In step 2.2, design the DLA + -ViT + ViT + Part of the structure, improve the multi-head attention mechanism in the TransformerEncoder module of the Vision Transformer model, and introduce convolution into the Transformer Encoder module of the ViT model.

6. The DLA-based method according to claim 5 + -ViT + A fine-grained image recognition method is characterized in that: The multi-head attention mechanism in the Transformer Encoder module of the improved Vision Transformer model includes: Redesign the multi-head attention mechanism in the Transformer Encoder module of the ViT model; redesign the S l The matrix calculation formula introduces three variables α, β, and γ to improve the aggregation method of the attention layer, so that the model pays attention to both global features and low-level visual features; S l The matrix is ​​the inner product of each row vector of matrix Q and K, where l is the number of multi-head attention layers; The values ​​of the introduced control variables α, β, and γ are between 0 and 1. The optimal values ​​are determined through experiments. Finally, the introduced control variables α, β, and γ are determined to be 0.75, 0.6, and 0.

3. The Transformer Encoder part is composed of 12 stacked blocks, which means there are 12 multi-head attention mechanism layers. The first layer is based on the original formula Calculate S l , where l is the number of multi-head attention layers, Q l is the query vector of the lth layer, K l is the key vector of the lth layer, T represents transposition, d l is the dimension of the word vector at the lth layer; The second layer calculates S according to the improved formula introducing α l , the second multi-head attention mechanism layer is weighted summed with the first layer, and the remaining layers are calculated according to the improved formula introducing β and γ. l , realize the weighted summation of the first two multi-head attention mechanism layers and the current layer, so as to obtain the global feature content during feature extraction while also extracting important low-level visual feature expressions, enhance the feature diversity of the entire network layer, and improve the final recognition accuracy; Improved S l The matrix calculation formula is: In the above formula: l represents the number of multi-head attention mechanism layers, Q l is the query vector of the lth layer, K l is the key vector of the lth layer, V l is the value vector of the lth layer, T is the transpose, d l is the dimension of the word vector of the lth layer, and α, β, and γ are control variables.

7. The DLA-based method according to claim 5. + -ViT + A fine-grained image recognition method is characterized in that: The introduction of convolution into the Transformer Encoder module of the ViT model includes: The designed convolution module is introduced between the multi-head attention mechanism and the MLP module in the Transformer Encoder module of the ViT model to fuse channel domain features and enhance network flexibility and nonlinear expression capabilities. The convolution module contains two 1×1 convolution kernels and a GELU activation function.

Images