Fine-grained visual classification method based on multi-level progressive refinement network

Abstract

The invention relates to a fine-grained visual classification method based on a multi-level progressive refinement network. The method comprises the following steps: acquiring an image data set and preprocessing the image data set; the method comprises the following steps: constructing a ViT-based multi-level progressive refinement network MHPR-Net which comprises a global coding module, a local positioning module and a multi-level token refinement module; training the MHPR-Net by using the preprocessed data set; and inputting the inference image to be classified into the MHPR-Net to obtain a classification result of the inference image. According to the method, through a multi-level processing mode, input images are processed from coarse-grained and fine-grained levels, the coarse-grained levels rapidly extract global features under a low-resolution scale, and the fine-grained levels mine fine discriminative local features under a high-resolution scale. According to the method, the calculation cost can be effectively reduced, and the discriminative local features which are beneficial to identifying tiny differences between different subclasses are mined, so that the classification precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a fine-grained visual classification method based on a multi-level progressive refinement network. Background Art

[0002] Fine-grained visual classification aims to identify different subclasses within a given superclass, such as identifying different types of birds (such as "seagulls", "swifts", and "egrets", etc.), or identifying different types of insect pests (such as rice pests "rice leaf roller", "striped stem borer", and "rice water weevil", etc.). This technology plays an important role in many real-world application scenarios, such as biodiversity conservation, smart agriculture, commodity retail, smart agriculture and other fields. Compared with traditional image classification, fine-grained visual classification faces two major challenges: one is the small inter-class differences, and the other is the large intra-class differences. Taking insect pests as an example, the appearance of different species may be very similar (i.e., small inter-class differences), while the same species may show significant morphological differences in different stages such as eggs, larvae, pupae, and adults (i.e., large intra-class differences). These two challenges greatly increase the difficulty of fine-grained visual classification.

[0003] In recent years, the rapid development of deep learning technology has significantly promoted progress in the field of fine-grained visual classification. Deep learning-based fine-grained visual classification models typically use convolutional neural networks (CNNs) such as ResNet and DenseNet as their backbone for feature extraction, demonstrating their high efficiency and classification accuracy. Inspired by the success of transformer models in natural language processing, the Vision Transformer (ViT) has been introduced as an emerging architecture in computer vision. By segmenting the input image into non-overlapping local tokens and leveraging a multi-head attention mechanism to capture global features, ViT has demonstrated superior performance compared to traditional convolutional neural networks in tasks such as image classification and object detection. However, in the specific field of fine-grained visual classification, ViT faces several challenges. Due to its architectural design's emphasis on integrating global information, ViT lacks the ability to capture the subtle local features crucial for fine-grained visual classification, resulting in unsatisfactory performance on these tasks. Furthermore, ViT's computational cost is a significant issue. ViT has a very high computational overhead, especially when the image is divided into more patches to enhance global feature extraction. This computational burden increases rapidly, which greatly limits the widespread application of ViT in fine-grained visual classification. Therefore, finding effective solutions to achieve a good balance between performance and resource efficiency has become an important research direction. Summary of the Invention

[0004] In order to solve the above problems in the prior art, the present invention proposes a fine-grained visual classification method based on a multi-level progressive refinement network to maximize the classification accuracy and reduce the required computing resources.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a fine-grained visual classification method based on a multi-level progressive refinement network. The steps are as follows:

[0006] Step S1: Obtain an original image dataset and preprocess the image dataset.

[0007] Step S2: Construct a Multi-Hierarchical Progressive Refinement Network (MHPR-Net). MHPR-Net uses the encoder in ViT as a feature extractor to extract image features. MHPR-Net consists of three main components: a global encoding module, a localization module, and a multi-level token refinement module.

[0008] Step S3: Global encoding module: In the first stage, i.e., the coarse-grained classification stage, the input image is first converted into a low-resolution scale. The encoder in ViT is used as a feature extractor to quickly and comprehensively capture the global features of the input image at the coarse-grained level. The image is then input into the classifier for classification to obtain a global classification vector at the coarse-grained level.

[0009] Step S4: Discriminative local tokens, such as bird beaks and claws, are rich in detailed information crucial for fine-grained visual classification. To accurately locate and identify these key regions, a local localization module is designed based on the multi-head self-attention scores calculated by the encoder during the coarse-grained classification stage. This module calculates the attention weights between each local token in the image and the classification token, which is then used to select discriminative local tokens with high attention weights. Local tokens with high attention weights indicate that the region in the image where the token resides contains richer detailed information and contributes more to the classification task.

[0010] Step S5: Multi-level Token Refinement Module: In the second stage, the fine-grained classification stage, the multi-level token refinement module is used to capture detailed and discriminative features. Based on the attention weights output from step S4, the image is differentiated to obtain image tokens at different scales: regions corresponding to high-attention-weight tokens are converted to a high-resolution scale to accurately capture detailed information; regions corresponding to low-attention-weight tokens are maintained at a low resolution to ensure the integrity of the image content and reduce computational resources. The processed image tokens are input into the encoder for feature extraction, obtaining fine-grained local features. These are then input into the classifier for classification, obtaining fine-grained local classification vectors.

[0011] Step S6: Using the preprocessed image dataset to train the proposed MHPR-Net network to obtain a trained MHPR-Net network.

[0012] Step S7: Input the inference image to be classified into the trained MHPR-Net to obtain the coarse-grained global classification vector and fine-grained local classification vector of the inference image, and further process them to output the classification result of the inference image.

[0013] The pre-processing of step S1 specifically includes the following steps:

[0014] S11: The image dataset is divided into a training set and a test set according to a specific ratio. The training set is used to train the MHPR-Net network, while the test set is used to evaluate the classification performance of the network.

[0015] S12: To enrich the dataset and improve the generalization ability of the model, data augmentation is performed on the images in the training set. Specific operations include image scaling, random cropping, horizontal flipping, brightness adjustment, and normalization. For the test set, preprocessing operations such as image scaling, center cropping, and normalization are mainly performed to ensure the consistency of the test data.

[0016] Step S2 uses ViT as a feature extractor and includes the following steps:

[0017] S21: The preprocessed input image Divide into a series of non-overlapping image-partial tokens H×W and P 2 Represents image x and image token x p Resolution, C represents the number of channels, N = (H / P) × (W / P) represents the number of local tokens. p Linear projection to d-dimensional embedding space, adding additional classification tokens x cls Used to represent global image information. All tokens are embedded with learnable positions. Corresponding to the input token sequence of the ViT model for:

[0018]

[0019] in represents a linear projection, Represents positional embedding.

[0020] S22: Input Z0 into an encoder consisting of L transformer layers. Each transformer layer is composed of a multi-head self-attention (MSA) layer and a multi-layer perceptron (MLP) layer stacked together. The output Z of the lth layer is l for:

[0021] Z′ l =MSA(LN(Z l-1 ))+Z l-1 ,l∈1,2,···,L (2)

[0022] Z l =MLP(LN(Z′ l ))+Z′ l ,l∈1,2,···,L (3)

[0023] Where LN(·) represents the layer normalization operation, is the image representation output by the encoder.

[0024] The global encoding module in step S3 includes the following steps:

[0025] S31: Downsample the input image x by two times to obtain a low-resolution image after sampling The size of is h×w×C, where h=H / 2 and w=W / 2. Processing, at this time the coarse-grained local token sequence at low resolution

[0026]

[0027] Where n = (h / P) × (w / P) = N / 4 represents the number of local tokens, represents non-overlapping image-local tokens, represents a linear projection, Represents positional embedding.

[0028] S32: According to step S22, Input into the encoder, the output of the l-th layer encoder is

[0029] S33: Output of the last encoder layer Denote as the global feature of the coarse-grained stage, Classification tokens in Input classifier Get the global classification vector of the coarse-grained stage Where S represents the number of categories,

[0030] The local positioning module in step S4 includes the following steps:

[0031] S41: In the coarse-grained classification stage, the calculation process of the multi-head self-attention mechanism is as follows: Given an input of the transformer layer (For simplicity, the subscript of this layer is ignored here.) Perform linear projection to obtain the query vector Key Vector

[0032]

[0033] in and are weight matrices, d k The dimension of is usually set to d / M. The attention score of a single head The calculation is as follows

[0034]

[0035] S42: The attention score of the lth layer is M represents the number of heads. Calculate the average attention score of all heads, add a unit matrix I, and take the average to get the attention score of each layer.

[0036]

[0037] S43: Attention scores of all layers Recursively multiply to perform attention aggregation to obtain the final attention weight between image tokens and classification tokens

[0038]

[0039] at this time Each element in Indicates the degree of attention of the i-th token to the j-th token. The attention weight on the classification token can represent the importance of the image token to the final classification. Attention weights corresponding to classification tokens in Used in the subsequent multi-level token refinement module, local tokens with high attention weights contain detailed information that is helpful for classification.

[0040] The multi-level token refinement module in step S5 includes the following steps:

[0041] S51: The vector A calculated at a low resolution scale contains the attention weights of all local tokens and can be flattened into a two-bit matrix A′ with n rows h =h / P (number of tokens in the height direction), column n w = w / P (number of tokens in the width direction to be changed). For any token at position (j, k) in the matrix A′, where 0≤j≤n h -1,0≤k≤nw -1, its position in the one-dimensional vector A can be expressed by j×n w +k is calculated. Given that local tokens with high attention weights are rich in discriminative information, the attention weights are selected from A to be ranked first. The local tokens are designed to focus on the most discriminative local areas in the image. Where α represents the ratio of the selected discriminative local tokens, Indicates rounding up.

[0042] S52: For each selected discriminative local token, convert the corresponding image region into a high-resolution scale. As described in S31, the original input image x is relative to That is, the high-resolution scale, the original input image is processed according to the method described in S21 to obtain the token sequence Z0 of formula (1). For each local token to be refined selected at the low-resolution scale, assume that its index position is r i , the corresponding two-dimensional spatial position is (j, k). In order to find the four corresponding local tokens from the high-resolution scale, the position relationship of A′ is first doubled and upsampled by the nearest neighbor, and then the one-dimensional vector is reshaped Note the index position r i The four index positions corresponding to U are The corresponding calculations of these index positions are: the upper left corner is The upper right corner is The lower left corner is lower right corner Select the local tokens at four index positions from Z0 As located in r i The refined local token corresponding to the discriminative local token.

[0043] S53: For local tokens with low attention weight, that is, the attention weight in A is ranked at the end The local tokens contain weakly discriminative object parts and most of the background information. In order to ensure the integrity of the image content and optimize the use of computing resources, tokens are selected directly from the low-resolution scale. Specifically, for each selected local token, assume that its index position is r m , from S31 Extract the corresponding local token

[0044] S54: Concatenate the refined local tokens selected in S52 and the coarse-grained local tokens selected in S53, and add the fine-grained classification tokens at the front Get the fine-grained local token sequence Z0′.

[0045]

[0046] in, N fine represents the number of local tokens selected after refinement, calculated as follows:

[0047]

[0048] The discriminative local tokens spliced in this way can effectively capture local details thanks to more detailed segmentation; at the same time, the areas where other local tokens are located maintain a low-resolution scale at a coarse-grained level, which can ensure the integrity of the image content and effectively reduce the number of local tokens.

[0049] S55: Input Z0′ into the encoder containing L Transformer layers for feature extraction, and output the local feature Z at the fine-grained level L ′. L Classification token in ′ Input classifier Get the local classification vector of the fine-grained stage Where S represents the number of categories,

[0050] In step S6: when the pre-processed image dataset is used to train the proposed MHPR-Net network, in order to ensure the global classification vector y in the coarse-grained stage coarse and the local classification vector y in the fine-grained stage fine The classification accuracy of the two, using cross entropy loss to supervise the training process of MHPR-Net:

[0051]

[0052] in, represents the cross entropy loss and y represents the true value label.

[0053] In step S7, the inference image to be classified is input into the trained MHPR-Net to obtain the coarse-grained global classification vector y of the inference image. coarse and the fine-grained local classification vector y fine , the final classification result It is y coarse and y fine The combined output

[0054]

[0055] Calculate classification results The probability of belonging to each category is calculated, and the category with the highest probability is selected as the prediction result. Finally, the prediction result is compared with the true value label, the prediction accuracy is calculated, and the classification result of the inference image is output.

[0056] Beneficial effects

[0057] This paper uses a multi-level processing approach to process input images at both coarse-grained and fine-grained levels. The former quickly extracts global features at a low-resolution scale, while the latter mines fine, discriminative local features at a high-resolution scale. This approach effectively reduces computational costs and mines discriminative local features that help identify subtle differences between different subclasses, thereby improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 1 is a flow chart of a fine-grained visual classification method based on a multi-level progressive refinement network according to the present invention;

[0059] Figure 2 Schematic diagram of the multi-level progressive refinement network MHPR-Net framework of the present invention;

[0060] Figure 3 This is a schematic diagram of the framework of a transformer layer in the encoder and the multi-head self-attention layer in the layer;

[0061] Figure 4 It is a schematic diagram of the framework of the multi-level token refinement module of the present invention;

[0062] Figure 5 This is an ablation experiment diagram of the hyperparameter α in the multi-level progressive refinement network MHPR-Net of the present invention;

[0063] Figure 6 This is a visualization diagram of the multi-level progressive refinement network MHPR-Net of the present invention under different α values. DETAILED DESCRIPTION

[0064] The present invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the following detailed description is for illustrative purposes only and is intended to further illustrate the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs.

[0065] Example 1:

[0066] like Figure 1 As shown, a fine-grained visual classification method based on a multi-level progressive refinement network includes the following steps:

[0067] Step S1: Obtain an original image dataset and preprocess the image dataset.

[0068] In this example, a public dataset was obtained and divided into a training set and a test set according to the official partitioning information. To enrich the dataset and improve the generalization ability of the model, data augmentation was performed on the images in the training set. Specifically, the following operations were performed: scaling the images to 600×600, randomly cropping them to 448×448, horizontally flipping the cropped images, randomly changing the brightness, contrast, and saturation of the images, and normalizing each color channel of the images. For the images in the test set, the following operations were performed: scaling the images to 600×600, center-cropping them to 448×448, and normalizing each color channel of the images.

[0069] Step S2: Construct a Multi-Hierarchical ProgressiveRefinement Network (MHPR-Net). MHPR-Net uses the encoder in ViT as a feature extractor to extract image features. MHPR-Net consists of three main components: a global encoding module, a local positioning module, and a multi-level token refinement module. Figure 2 As shown, parameters are shared between the encoders on the left and right parts of the figure.

[0070] Step S3: Use the global encoding module to extract the global features of the coarse-grained level And calculate the global classification vector y coarse The global encoding module corresponds to Figure 2 In the left part, bilinear interpolation is first used to downsample the input image x to a quarter of the original image size to obtain a low-resolution scale image. Low-resolution image at this time The resolution is 224 × 224. The low-resolution image is divided into n local tokens of equal size and non-overlapping Here n is 196 and the size of local tokens is 16×16. Project to d = 768 dimensional linear space and add classification tokens at the front To extract global information, all tokens are embedded with learnable positions Get a coarse-grained local token sequence at low resolution

[0071] The coarse-grained local token sequence Input is fed into an encoder with L transformer layers. The architecture of the transformer layer is as follows Figure 3 As shown in the left half of , the framework of the multi-head self-attention mechanism is as follows Figure 3 The right half of the encoder is shown. Recorded as the global feature of the coarse-grained stage. Next, Select the classification token Feed it into the classifier Get the global classification vector of the coarse-grained stage Where S represents the number of categories,

[0072] Step S4: Design a local positioning module, such as Figure 2 As shown in the upper middle part, this module calculates the attention weights between each local token and the classification token based on the multi-head self-attention scores calculated by each transformer layer of the encoder in the coarse-grained classification stage, which are used to subsequently select discriminative local tokens with high attention weights.

[0073] The multi-head self-attention scores of each layer are given by Figure 3 The multi-head self-attention mechanism calculation shown on the right, specifically, for the input of the transformer layer Linearly project it to get the query vector and key vector Attention score of a single head It can be calculated by the following formula:

[0074]

[0075] The attention score of the lth layer is recorded as M represents the number of heads. Calculate the average attention score of all heads, add a unit matrix I, and take the average to get the attention score of each layer.

[0076]

[0077] The attention scores of all layers Recursively multiply to perform attention aggregation to obtain the final attention weight between image tokens and classification tokens

[0078]

[0079] at this time Each element in Indicates the degree of attention of the i-th token to the j-th token. The attention weight on the classification token can represent the importance of the image token to the final classification. Attention weights corresponding to classification tokens in Used in the subsequent multi-level token refinement module, local tokens with high attention weights contain detailed information that is helpful for classification.

[0080] Step S5: Use the multi-level token refinement module to extract fine-grained local features Z L ′ and calculate the global classification vector y fine , the module is as follows Figure 4 As shown. The A calculated above contains the attention weights of all local tokens on the final classification. The attention weights ranked first in A are selected. The local tokens are designed to focus on the most discriminative local areas in the image. Where α represents the ratio of the selected discriminative local tokens, Rearrange the attention weight matrix A and flatten it into a two-dimensional matrix A′ with n rows. h =h / P, listed as n w =w / P, the positional relationship of each local token in A′ remains consistent with the original input image.

[0081] For each discriminative local token selected in A, assume its index position is r i , the corresponding two-dimensional spatial position in A′ is (j, k). In order to convert the corresponding image area into a high-resolution scale, the specific approach is to select the corresponding local token from the high-resolution scale as the refinement of the discriminative local token. This first requires finding the corresponding position of the discriminative local token in the high-resolution scale.

[0082] For a high-resolution image x, whose size is 448×448, divide x into N local tokens x of size 16×16 p , at this time N = 784, and N = 4n, which means that the local tokens obtained at high resolution are divided more finely than those obtained at low resolution, and more image tokens are generated. p , use a learnable linear projection to map it to a d = 768 dimensional linear space and add a classification token x in front of it cls To extract global information, all tokens are added with learnable position embedding E pos , and obtain the fine-grained token sequence Z0 at high resolution.

[0083] Next, the position relationship of A′ is doubled and upsampled to the nearest neighbor, and then reshaped into a one-dimensional vector U of length N. Note the index position r i The four index positions corresponding to U are The corresponding calculations of these index positions are: the upper left corner is The upper right corner is The lower left corner is lower right corner Select the local tokens at four index positions from Z0 As located in r iThe refined local token corresponding to the discriminative local token.

[0084] For local tokens with low attention weights, i.e., the attention weights in A are ranked at the end, The local tokens contain weakly discriminative object parts and most of the background information. In order to ensure the integrity of the image content and optimize the use of computing resources, tokens are selected directly from the low-resolution scale. Specifically, for each selected local token, assume that its index position is r m , from the coarse-grained token sequence Extract the corresponding coarse-grained local token

[0085] Concatenate the selected refined local tokens with the coarse-grained local tokens and add the fine-grained classification tokens at the front Get the fine-grained local token sequence Z0′.

[0086]

[0087] Z0′ is input into the encoder for feature extraction (corresponding to Figure 2 The right part of the output is the local feature Z at the fine-grained level. L ′. L Classification token Z in ′ L ' cls Input classifier Get the local classification vector of the fine-grained stage Where S represents the number of categories,

[0088] Step S6: Use the preprocessed image dataset to train the proposed MHPR-Net network to ensure the global classification vector y in the coarse-grained stage coarse and the local classification vector y in the fine-grained stage fine The classification accuracy of the two, using cross entropy loss to supervise the training process of MHPR-Net:

[0089]

[0090] in, represents the cross entropy loss and y represents the true value label.

[0091] Step S7: Input the inference image to be classified into the trained MHPR-Net to obtain the coarse-grained global classification vector y of the inference image coarse and the fine-grained local classification vector y fine , the final classification result It is y coarse and y fine The combined output is:

[0092]

[0093] Example 2: Testing the effect of biodiversity conservation on the CUB-200-2011, Stanford Dogs, and NABirds datasets

[0094] In this example, three public datasets were obtained: the bird dataset CUB-200-2011, the canine dataset Stanford Dogs, and the bird dataset NABirds. According to the official division information, the datasets were divided into training sets and test sets. Table 1 shows the number of categories in each dataset, the number of images in the training set and test set, and the total number of samples.

[0095] Table 1 Dataset partitioning table

[0096] Dataset Number of categories Number of training set images Number of test set images Total number of images CUB-200-2011 200 5994 5794 11788 Stanford Dogs 120 12000 8580 20580 NABirds 555 23929 24633 48562

[0097] To enrich the dataset and improve the generalization of the model, data augmentation was performed on the images in the training set. This included scaling the images to 600×600, randomly cropping them to 448×448, horizontally flipping the crops, randomly adjusting the brightness, contrast, and saturation, and normalizing each color channel. For the images in the test set, this included scaling the images to 600×600, center-cropping them to 448×448, and normalizing each color channel. For all three datasets, all input images were first uniformly resized to 448×448 to accommodate the fine-grained stage. In the coarse-grained stage, the images were further resized to 224×224. The ViT-B / 16 model, pre-trained on the ImageNet22K dataset, was chosen as the feature extractor. During training, the batch size was set to 6, and the stochastic gradient descent (SGD) optimizer with momentum of 0.9 was used. The learning rate was set to 0.03 on the CUB-200-2011 and NABirds datasets, and to 0.003 on the Stanford Dogs dataset. The number of training steps on the CUB-200-2011, Stanford Dogs, and NABirds datasets was set to 10,000, 20,000, and 6,000, respectively.

[0098] In the global encoding module, each input image is divided into 16×16 image tokens, and the total number of tokens is (224 / 16)×(224 / 16)=14×14=196. In the multi-level token refinement module, the ratio α of the selected discriminative local tokens is set to 0.4. At this time, the number of discriminative local tokens selected is 79, the number of refined local tokens is 316, the number of coarse-grained tokens selected is 117, and the total number of local tokens is 433.

[0099] The MHPR-Net method of the embodiment of the present invention is compared with existing methods such as ViT, TransFG, and AA-Trans. The experimental results are shown in Table 2, which shows the comparison of MHPR-Net and the comparison methods in terms of parameter quantity, computational complexity, and classification accuracy under three data sets. The computational complexity in the table refers to the number of floating-point operations.

[0100] Table 2 Parameters, computational complexity, and classification accuracy of different methods based on ViT-B / 16

[0101]

[0102]

[0103] From the perspective of computational complexity, MHPR-Net demonstrates significant advantages, with a computational complexity of only 54.1G, a figure significantly lower than all the comparison methods. Compared with the baseline model ViT, the number of parameters of MHPR-Net remains unchanged, but the computational complexity is reduced by 13.4G. On the three datasets of CUB-200-2011, Stanford Dogs, and NABirds, the classification accuracy of MHPR-Net increased by 0.9%, 1.5%, and 0.7%, respectively. This proves that the MHPR-Net proposed in this patent can effectively reduce the high computational cost caused by the interaction of a large number of background tokens and classification tokens in ViT (significantly reduced computational complexity), and better mine discriminative local features that help identify subtle differences between different categories (improved classification accuracy).

[0104] Compared to AA-Trans, MHPR-Net reduces computational overhead by 13.4 GB and improves classification accuracy by 1.3% and 0.1% on the Stanford Dogs and NABirds datasets, respectively. TransFG's computational overhead is as high as 108.4 GB. This is because TransFG divides the image into 1,369 overlapping local tokens, requiring a large number of interaction calculations between these local tokens, resulting in a significant increase in computational overhead. Although TransFG achieves slightly higher classification accuracy on the CUB-200-2011 and NABirds datasets than the MHPR-Net proposed in this patent, its high computational overhead makes it unsuitable for practical applications. Furthermore, the MHPR-Net proposed in this patent achieves the best classification result of 93.1% on the Stanford Dogs dataset while significantly reducing computational overhead.

[0105] Example 3: Testing the effectiveness of MHPR-Net in smart agriculture on the IP102 insect pest dataset

[0106] In the field of smart agriculture, fine-grained visual recognition technology helps accurately identify insect pests on crops such as rice, grapes, and apples, distinguishing between different pest species and growth stages, thereby providing more accurate pest and disease information. This technology enables farmers to promptly detect signs of pests and diseases, effectively preventing their spread and progression, and significantly reducing crop losses. Furthermore, this technology can help farmers develop targeted control measures for different pest types, such as implementing precision pesticide application, which can reduce pesticide waste and pollution.

[0107] The IP102 dataset is a challenging, large-scale pest recognition benchmark designed specifically for crop pest identification. The dataset covers eight crops: rice, grapes, citrus, mango, corn, wheat, alfalfa, and sugar beets, and contains images of 102 different pest species. These images capture all key stages of the pest life cycle, from egg, larva, pupal, to adult, comprehensively demonstrating the pests' morphological changes. In this dataset, 45,095 images are used for training and 22,619 images are used for testing.

[0108] To enrich the dataset and improve the generalization of the model, data augmentation was performed on the images in the training set. This included rescaling the images to 600×600, randomly cropping them to 448×448, horizontally flipping the crops, randomly adjusting the brightness, contrast, and saturation, and normalizing each color channel. For the images in the test set, this process involved rescaling them to 600×600, center-cropping them to 448×448, and normalizing each color channel. For MHPR-Net, all input images in the IP102 dataset were first uniformly resized to 448×448 to accommodate the fine-grained stage. In the coarse-grained stage, the images were further resized to 224×224. The ViT-B / 16 model, pre-trained on the ImageNet22K dataset, was chosen as the feature extractor. During training, the batch size is set to 6, the stochastic gradient descent (SGD) optimizer with momentum of 0.9 is used for training, the learning rate is set to 0.03, and the number of training steps is set to 100,000.

[0109] In the global encoding module, each input image is divided into 16×16 image tokens, and the total number of tokens is (224 / 16)×(224 / 16)=14×14=196. In the multi-level token refinement module, the ratio α of the selected discriminative local tokens is set to 0.4. At this time, the number of discriminative local tokens selected is 79, the number of refined local tokens is 316, the number of coarse-grained tokens selected is 117, and the total number of local tokens is 433. The method MHPR-Net of the embodiment of the present invention is compared with existing methods such as ViT, TransFG and AA-Trans on the IP102 dataset. The experimental results are shown in Table 3. The table shows the comparison of MHPR-Net and the comparison methods in terms of parameter quantity, computational complexity and classification accuracy under three datasets. The computational complexity in the table is the number of floating-point operations.

[0110] Table 3 Parameters, computational complexity, and classification accuracy of different methods based on ViT-B / 16

[0111] method Parameter quantity / M Computational Amount / G IP102 ViT 86.5 67.5 75.2 TransFG 86.5 108.4 76.1 AA-Trans 86.5 65.9 75.6 MHPR-Net 86.5 54.1 76.7

[0112] As can be seen from the table, the parameters of the four methods remain consistent. From the perspective of computational complexity, MHPR-Net demonstrates a significant advantage, with the lowest computational complexity of only 54.1G. On the IP102 dataset, MHPR-Net improves classification accuracy by 1.5%, 0.6%, and 1.1% compared to ViT, TransFG, and AA-Trans, respectively. The comparative results demonstrate that the MHPR-Net proposed in this patent can effectively reduce computational complexity and better mine discriminative local features that help identify subtle differences between different categories, thereby improving classification accuracy.

[0113] Example 4: Ablation experiments and visualization analysis on CUB-200-2011, Stanford Dogs, NABirds, and IP102 datasets

[0114] (1) Ablation experiments on four datasets

[0115] In the MHPR-Net network, α represents the ratio of the selected discriminative local tokens. In order to study the effect of the hyperparameter α on the classification accuracy of the MHPR-Net network, α is set to 0.1, 0.2, 0.3, 0.4, 0.5 to 0.6, and ablation experiments are conducted on four datasets: CUB-200-2011, Stanford Dogs and NABirds, and IP102. The experimental results are shown in Figure 2. Figure 5 As shown in Figure 2, it contains 4 sub-graphs, which correspond to the ablation experiment results on the above four datasets. Figure 5 (a) is the ablation on the CUB-200-2011 dataset, Figure 5 (b) is the ablation on the Stanford Dogs dataset, Figure 5 (c) is the ablation on the NABirds dataset, Figure 5 (d) shows the ablation results on the IP102 dataset. Each sub-figure shows not only the classification accuracy (the percentage of correctly classified images in the test set to the total number of images in the test set, unit: %) for different α values, but also the model computational overhead (measured by the number of floating-point operations, unit: GB) for different α values.

[0116] from Figure 5It can be found that on each dataset, the classification accuracy of the MHPR-Net network first improves with the increase of α. This improvement is due to the fact that the fine-grained layer can refine more discriminative local tokens, thereby more comprehensively capturing the detailed information in the image, which is crucial for distinguishing subtle differences between different subclasses, thereby improving the classification effect. However, after the classification accuracy reaches its peak, as α increases, the classification accuracy of the MHPR-Net network begins to show a downward or stable trend. This is because background tokens begin to gradually mix into the refined discriminative local tokens. The introduction of these non-target areas has a negative impact on the classification performance, resulting in a decrease in accuracy or no further improvement. In addition, as α increases, the computational complexity of the MHPR-Net model shows a continuous growth trend. This phenomenon is attributed to the fact that the increase in the α value causes the number of selected local tokens in the fine-grained layer to increase linearly. When using the ViT network to extract discriminative local features, this increase will lead to a continuous increase in the computational complexity. Therefore, it is necessary to balance the relationship between the classification performance of the model and the computational complexity. In Examples 2 and 3, the α value of MHPR-Net on the four datasets is set to 0.4.

[0117] (2) Visual analysis on four datasets

[0118] Figure 6 The visualization of the discriminative local areas of the MHPR-Net network under different α values is presented. The first two rows of images are visualization results of two test images selected from the CUB-200-2011 dataset, the third and fourth rows are visualization results of two test images selected from the StanfordDogs dataset, the fifth and sixth rows are visualization results of two test images selected from the NABirds dataset, and the seventh and eighth rows are visualization results of two test images selected from the IP102 dataset. Figure 6 In the last six columns of visualization results, the gray boxes mark the areas where non-discriminative tokens are located, while the parts not blocked by the gray boxes represent the discriminative local token areas that are selected for further refinement.

[0119] By observation Figure 6, it can be clearly seen that the selected local tokens are highly correlated with the target object in the image. As the α value gradually increases, more discriminative local tokens are selected and refined, which helps the model capture the details in the image more comprehensively and plays a vital role in accurately distinguishing the subtle differences between different subclasses. However, when the α value reaches a certain critical point, background tokens begin to gradually mix in during the refinement process. This means that as α increases further, not all refined local tokens correspond to the discriminative areas of the target object, but begin to contain some background-related tokens, which may have an adverse effect on the classification performance of the model.

[0120] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A fine-grained visual classification method based on a multi-level progressive refinement network, characterized in that: The following steps are involved: Step S1: obtaining an original image dataset and preprocessing the original image dataset; Step S2: Construct a multi-level progressive refinement network MHPR-Net, which uses the encoder in the visual Transformer as a feature extractor to extract image features. MHPR-Net includes a global encoding module, a local positioning module, and a multi-level token refinement module; Step S3: Global Encoding Module Processing: First, the input image is converted to a low-resolution scale. The encoder in the visual Transformer is used as a feature extractor to capture the global features of the input image at a coarse-grained level. The global features are then input into the classifier for classification to obtain a coarse-grained global classification vector. Step S4: Local localization module processing: Based on the multi-head self-attention scores calculated by the encoder in the coarse-grained classification stage, a local localization module is designed to calculate the attention weights between each local token in the image and the classification token, which is used to subsequently select discriminative local tokens with high attention weights; Step S5: Multi-level token refinement module processing: The multi-level token refinement module is used to capture fine and discriminative features. Based on the attention weights output in step S4, the image is differentiated to obtain image tokens of different scales: the areas corresponding to high attention weight tokens are converted to high-resolution scales; The areas corresponding to low attention weight tokens are kept at low resolution. The processed image tokens are input into the encoder for feature extraction to obtain fine-grained local features, which are then input into the classifier for classification to obtain fine-grained local classification vectors. Step S6: using the preprocessed image dataset to train the MHPR-Net to obtain a trained MHPR-Net; Step S7: Input the inference image to be classified into the trained MHPR-Net to obtain the coarse-grained global classification vector and fine-grained local classification vector of the inference image.

2. A fine-grained visual classification method based on a multi-level progressive refinement network according to claim 1, characterized in that: The step S1 is specifically as follows: S11: The image dataset is divided into a training set and a test set in proportion. The training set is used to train the MHPR-Net network, and the test set is used to evaluate the classification performance of the network. S12: Perform data enhancement processing on the images in the image dataset, where the processing method includes at least one of image scaling, random cropping, horizontal flipping, brightness adjustment, normalization, and center cropping.

3. The fine-grained visual classification method based on a multi-level progressive refinement network according to claim 1 is characterized in that: In step S2, a visual transformer is used as a feature extractor, and the specific steps are as follows: S21: The preprocessed input image Divide into a series of non-overlapping image-partial tokens H×W and P 2 Represents image x and image token x p Resolution, C represents the number of channels, N = (H / P) × (W / P) represents the number of local tokens; the local token x p Linear projection to d-dimensional embedding space, adding additional classification tokens x cls Used to represent global image information; All tokens are added with learnable position embeddings corresponding to the input token sequence of the visual Transformer model for: in represents a linear projection, Represents positional embedding; S22: Input Z0 into an encoder consisting of L transformer layers. Each transformer layer is composed of a multi-head self-attention layer and a multi-layer perceptron layer stacked together. The output Z of the lth layer is l for: Z l ′=MSA(LN(Z l-1 ))+Z l-1 ,l∈1,2,···,L (2) WITH l =MLP(LN(Z l ′))+Z l ′,l∈1,2,···,L (3) Where LN(·) represents the layer normalization operation, is the image representation output by the encoder.

4. The fine-grained visual classification method based on a multi-level progressive refinement network according to claim 3 is characterized in that: The step S3 is specifically as follows: S31: Downsample the input image x by two times to obtain a low-resolution image after sampling The size of is h×w×C, where h=H / 2, w=W / 2; according to step S21 Processing, at this time the coarse-grained local token sequence at low resolution Where n = (h / P) × (w / P) = N / 4 represents the number of local tokens, represents non-overlapping image-local tokens, represents a linear projection, Represents positional embedding; S32: According to step S22, Input into the encoder, the output of the l-th layer encoder is S33: Output of the last encoder layer Denote as the global feature of the coarse-grained stage, Classification tokens in Input classifier Get the global classification vector of the coarse-grained stage Where S represents the number of categories, 5. The fine-grained visual classification method based on a multi-level progressive refinement network according to claim 4 is characterized in that: The local positioning module in step S4 is specifically: S41: In the coarse-grained classification stage, the calculation process of the multi-head self-attention mechanism: given an input of the transformer layer Will Perform linear projection to obtain the query vector Key Vector in and are weight matrices, d k The dimension is set to d / M, where M represents the number of heads; the attention score of a single head Calculation formula: S42: The attention score of the lth layer is Calculate the average attention score of all heads, add a unit matrix I, and take the average to get the attention score of each layer S43: Attention scores of all layers Recursively multiply to perform attention aggregation to obtain the final attention weight between image tokens and classification tokens at this time Each element in Indicates the degree of attention of the i-th token to the j-th token, select Attention weights corresponding to classification tokens in Used in the subsequent multi-level token refinement module, local tokens with high attention weights contain detailed information that is helpful for classification.

6. The fine-grained visual classification method based on a multi-level progressive refinement network according to claim 5, characterized in that: The multi-level token refinement module in step S5 is specifically: S51: The vector A calculated at a low resolution scale contains the attention weights of all local tokens and can be flattened into a two-bit matrix A′ with n rows h =h / P, listed as n w = w / P, for any token at position (j,k) in the matrix A′, where 0≤j≤n h -1,0≤k≤n w -1, its position in the one-dimensional vector A can be expressed by j×n w +k is calculated; given that local tokens with high attention weights are rich in discriminative information, the attention weights are selected from A to be ranked first The local tokens are used to focus on the most discriminative local areas in the image, where α represents the ratio of the selected discriminative local tokens, Indicates rounding up; S52: For each selected discriminative local token, the corresponding image region is converted into a high-resolution scale. As described in S31, the original input image x is relative to That is, the high-resolution scale, the original input image is processed according to the method described in S21 to obtain the token sequence Z0 of formula (1). For each local token to be refined selected at the low-resolution scale, assume that its index position is r i , corresponding to the two-dimensional spatial position (j, k), first perform a two-fold nearest neighbor upsampling of the position relationship of A′, and then reshape the one-dimensional vector Note the index position r i The four index positions corresponding to U are Corresponding calculation of index position: the upper left corner is The upper right corner is The lower left corner is lower right corner Select the local tokens at four index positions from Z0 As located in r i The refined local token corresponding to the discriminative local token; S53: For local tokens with low attention weight, that is, the attention weight in A is ranked at the end The local tokens contain weakly discriminative object parts and most of the background information, and the tokens are selected directly from the low-resolution scale: for each selected local token, assume that its index position is r m , from S31 Extract the corresponding local token S54: Concatenate the refined local tokens selected in S52 and the coarse-grained local tokens selected in S53, and add the fine-grained classification tokens at the front Get the fine-grained local token sequence Z0′: in, N fine represents the number of local tokens selected after refinement, calculated as follows: S55: Input Z0′ into the encoder containing L Transformer layers for feature extraction, and output the local feature Z at the fine-grained level L ′, Z L Classification token in ′ Input classifier Get the local classification vector of the fine-grained stage Where S represents the number of categories, 7. The fine-grained visual classification method based on a multi-level progressive refinement network according to claim 1, characterized in that: The step S6 is specifically as follows: When the preprocessed image dataset is used to train the proposed MHPR-Net, in order to ensure the global classification vector y coarse and the local classification vector y in the fine-grained stage fine The classification accuracy of the two, using cross entropy loss to supervise the training process of MHPR-Net: in, represents the cross entropy loss and y represents the true value label.

8. The fine-grained visual classification method based on a multi-level progressive refinement network according to claim 7, characterized in that: The step S7 is specifically as follows: Input the inference image to be classified into the trained MHPR-Net to obtain the coarse-grained global classification vector y of the inference image coarse and the fine-grained local classification vector y fine , the final classification result It is y coarse and y fine The combined output is: Calculate classification results The probability of belonging to each category is calculated, the category with the highest probability is selected as the prediction result, the prediction result is compared with the true value label, the prediction accuracy is calculated, and the classification result of the inference image is output.

Images