Fine-grained image classification method and system based on two-stage attention positioning framework

Abstract

The application discloses a fine-grained image classification method and system based on a two-stage attention positioning framework, and belongs to the technical field of image processing. The method comprises the following steps: processing an image by using two different division modes to obtain a non-overlapping patch sequence and an overlapping patch sequence, and merging a class token for each sequence respectively; in the first stage, the non-overlapping patch sequence is input into a feature extractor based on a Transformer encoder to generate an attention score and a class token; in the second stage, the attention scores of all layers are integrated, a key patch position is identified through a sorting operation, and an overlapping patch sequence at the position is extracted as an input of a reuse feature extractor, local information of adjacent areas is reserved, and a more fine class token and image feature are obtained; finally, the class tokens of the two stages and the final image feature are generated as a component of a training loss. The method is convenient for improving the fine-grained image classification precision.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a fine-grained image classification method and system based on a two-stage attention localization framework. Background Technology

[0002] Fine-grained image classification tasks further classify subclasses within a basic category, and have wide-ranging applications, such as food selection and pest prediction, thus attracting increasing attention. Compared to traditional classification tasks, its core challenge lies in distinguishing between class similarities and within-class differences. Accurately differentiating these subclasses under varying lighting, poses, and shooting angles becomes even more difficult.

[0003] To effectively identify subtle differences between subcategories, early work typically used data with localization annotations. However, manual annotation is costly and prone to errors. To alleviate this problem, weakly supervised model frameworks emerged, requiring only a single category label to achieve good performance. Common weakly supervised models often use local localization methods, primarily identifying subtle differences in the input image by recognizing the bounding boxes of the target region. However, this increases network complexity, making the network difficult to train. Some methods have attempted to apply the Vision Transformer (ViT) to fine-grained classification, achieving good results. However, current ViT-based methods mainly focus on integrating feature dependencies at high levels, making the model susceptible to interference from low-level background information.

[0004] Therefore, how to amplify the feature information of fine-grained images and reduce the interference of background information unrelated to the target in the image is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, in order to solve the problem of how to amplify the feature information of fine-grained images and reduce the interference of background information unrelated to the target in the image, the present invention provides a fine-grained image classification method and system based on a two-stage attention localization framework, which can effectively remove background information unrelated to the target in the image, amplify the inter-class differences of fine-grained images, and more accurately determine the specific category of the subclass under the base category.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, embodiments of the present invention provide a fine-grained image classification method based on a two-stage attention localization framework, the method comprising the following steps:

[0008] S1. Construct a two-stage fine-grained image classification model; the constructed fine-grained image classification model includes a data processing module, a first backbone feature extraction network, an attention selection module, a second backbone feature extraction network, and a loss function calculation module connected in sequence;

[0009] S2. The data processing module uses two partitioning modes to partition the fine-grained image to obtain non-overlapping patch sequences and overlapping patch sequences, and merges a class token for each sequence.

[0010] S3. In the first stage, the non-overlapping patch sequence after division is input into the first backbone feature extraction network to generate an attention score and a class token.

[0011] S4. In the second stage, the attention selection module is used to integrate the obtained multi-layer attention scores, and overlapping patch sequences are selected from the corresponding positions through position transformation.

[0012] S5. Input the selected overlapping patch sequence into the second backbone feature extraction network of the reuse parameters to obtain more refined class tokens and image features.

[0013] S6. The loss function calculation module uses the class tokens of the two stages and the final image features as components of the training loss, and obtains fine-grained image classification results based on the class tokens of the second stage.

[0014] In an optional implementation, in step S1, both the first backbone feature extraction network and the second backbone feature extraction network of the fine-grained image classification model are feature extractors based on Transformer encoders; the feature extractor contains 12 layers, each layer containing a normalization module, a multi-head attention module, and a multilayer perceptron module; the multilayer perceptron module connects a linear layer and a GELU activation function.

[0015] In an optional implementation, step S2 specifically includes:

[0016] The data processing module uses convolution operations to segment and embed the image, and obtains patch sequences of two segmentation modes by setting different parameters. When the convolution kernel and stride are both set to P, the input dimension is the RGB dimension of the original image, and the output dimension is the embedding dimension D, a non-overlapping patch sequence is obtained. When the stride is set to S, such that S is less than P, an overlapping patch sequence is obtained. After the segmentation and embedding are completed, a class token is merged between the overlapping and non-overlapping patch sequences respectively. A parameterized position embedding is then added to each of these two sequences to obtain the final overlapping patch sequence input and the non-overlapping patch sequence input.

[0017] In an optional implementation, the specific operation of adding a parameterized position embedding to each of the two sequences is as follows:

[0018]

[0019] Where z0 represents the final processed sequence, x class This represents a learnable class token used for classification. E represents the Nth patch of the partitioned embedding. pos This represents a learnable positional encoding parameter.

[0020] In an optional implementation, step S4 specifically includes:

[0021] S41. Average the attention scores of each layer output by the first backbone feature extraction network in the first stage, using the following formula:

[0022]

[0023] Among them, a l represents the average attention score of the l-th layer, and k represents the index of the attention head.

[0024] S42. Starting from the attention score of the preset layer, integrate the attention scores using an exponential moving average, with the following formula:

[0025]

[0026] Where α represents the patch selection ratio. This represents the average attention score of the first l layers of integration;

[0027] S43. In the score matrix, retain only the attention score of the first class token; sort the scores, convert the patch positions with scores higher than the preset value into the corresponding positions of the overlapping patch sequence, and reselect the overlapping patch sequence for input.

[0028] In an optional implementation, step S6, the operation of the loss function calculation module includes:

[0029] The first loss is obtained by calculating the cross-entropy between the tokens output from the second stage and the ground real labels;

[0030] The image features output from the second stage are compared and lost in batches to obtain the second loss.

[0031] The class tokens output by the second stage are used to supervise the class tokens output by the first stage, and the third loss is calculated by Kullback-Leibler divergence.

[0032] The three losses are summed to obtain the final loss for training supervision; during testing, fine-grained image classification results are obtained based on the class tokens from the second stage.

[0033] Secondly, embodiments of the present invention also provide a fine-grained image classification system based on a two-stage attention localization framework. This system applies the aforementioned fine-grained image classification method based on a two-stage attention localization framework to perform fine-grained image classification. The system includes:

[0034] The model building module is used to build a two-stage fine-grained image classification model; the built fine-grained image classification model includes a data processing module, a first backbone feature extraction network, an attention selection module, a second backbone feature extraction network, and a loss function calculation module connected in sequence.

[0035] The image segmentation module is used to segment a fine-grained image using two segmentation modes through the data processing module, to obtain non-overlapping patch sequences and overlapping patch sequences, and to merge a class token for each sequence.

[0036] The feature extraction module is used to input the non-overlapping patch sequences after division into the first backbone feature extraction network in the first stage to generate an attention score and a class token; and in the second stage, the attention selection module is used to integrate the obtained multi-layer attention scores, select overlapping patch sequences from the corresponding positions through position transformation; and input the selected overlapping patch sequences into the second backbone feature extraction network with reuse parameters to obtain more refined class tokens and image features.

[0037] The image classification module is used to utilize the loss function calculation module based on the class tokens of the two stages and the final image features as components of the training loss, and to obtain fine-grained image classification results based on the class tokens of the second stage.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] This invention provides a fine-grained image classification method and system based on a two-stage attention localization framework. This invention fully leverages the advantages of deep learning by constructing a two-stage fine-grained image classification model. Utilizing the data processing module, backbone feature extraction network, attention selection module, and loss function calculation module within the model, it is easy to effectively locate important target regions in an image. By extracting more refined low-level features through position transformation, it provides more comprehensive information, reduces interference from low-level background information, and thus helps improve the accuracy of fine-grained image classification.

[0040] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0044] Figure 1 This is a flowchart illustrating a fine-grained image classification method based on a two-stage attention localization framework, as provided in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the basic architecture and working principle of the model provided in the embodiments of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0047] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0048] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0049] See Figure 1 As shown, this invention provides a fine-grained image classification method based on a two-stage attention localization framework, which mainly includes the following steps:

[0050] S1. Construct a two-stage fine-grained image classification model; the constructed fine-grained image classification model includes a data processing module, a first backbone feature extraction network, an attention selection module, a second backbone feature extraction network, and a loss function calculation module connected in sequence;

[0051] S2. The fine-grained image is divided using two partitioning modes through the data processing module to obtain non-overlapping patch sequences and overlapping patch sequences, and a class token is merged for each sequence.

[0052] S3. In the first stage, the non-overlapping patch sequence after division is input into the first backbone feature extraction network to generate an attention score and a class token.

[0053] S4. In the second stage, the attention selection module is used to integrate the obtained multi-layer attention scores, and overlapping patch sequences are selected from the corresponding positions through position transformation.

[0054] S5. Input the selected overlapping patch sequence into the second backbone feature extraction network of the reuse parameters to obtain more refined class tokens and image features.

[0055] S6. The loss function calculation module uses the class tokens of the two stages and the final image features as components of the training loss, and obtains fine-grained image classification results based on the class tokens of the second stage.

[0056] The specific implementation of the method of the present invention will be described in detail below:

[0057] In this embodiment, in the constructed two-stage fine-grained image classification model, both the first backbone feature extraction network and the second backbone feature extraction network are feature extractors based on Transformer encoders; the preferred feature extractor contains 12 layers, each layer containing a normalization module, a multi-head attention module and a multilayer perceptron module; wherein the multilayer perceptron module connects the linear layer and the GELU activation function.

[0058] In one specific embodiment, step S2 includes the following:

[0059] Convolutional operations are used to partition and embed an original image with an initial dimension of H×W×3, and patch sequences with two partitioning modes are obtained by setting different parameters. When the convolution kernel and stride are both set to P, the input dimension is the RGB dimension of the original image (3), and the output dimension is the embedding dimension D, the sequence length is (H×W) / P.2 The non-overlapping sequences. When the step size is set to S, such that S is less than P, the resulting overlapping patch sequences are:

[0060]

[0061] Where p represents padding.

[0062] To further implement the above technical solution, after the embedding is divided, a class token is merged between the overlapping patch sequence and the non-overlapping patch sequence. In this embodiment, to better preserve position information, a parameterized position embedding is preferably added to each of the two sequences, as follows:

[0063]

[0064] Where z0 represents the final processed sequence, x class This represents a learnable class token used for classification. E represents the Nth patch of the partitioned embedding. pos This represents a learnable positional encoding parameter. After the above processing, the final overlapping patch sequence input and non-overlapping patch sequence input are obtained.

[0065] In one specific embodiment, step S3 includes the following:

[0066] The processed non-overlapping patch sequence is input into the first backbone feature extraction network, and the attention score of each layer in the backbone network is saved:

[0067]

[0068] Among them, A l This represents the attention score of the l-th layer. This represents the attention score of the k-th head in layer l; and it retains the class token in the final output.

[0069] In one specific embodiment, step S4 includes the following:

[0070] S41. Average the attention scores of each layer output by the first backbone feature extraction network in the first stage: The formula is:

[0071]

[0072] Among them, a l This represents the average attention score of the l-th layer;

[0073] S42. Starting from the attention score of the fourth layer, integrate the attention scores using an exponential moving average, as follows:

[0074]

[0075] Where α represents the patch selection ratio. The average attention score of the first l layers is represented; after the exponential moving average operation, the final integration score is obtained in the Lth layer.

[0076] S43. In the score matrix, only the attention score of the first class token needs to be retained; sort the scores, transform the high-scoring patch positions into the corresponding positions of the overlapping patch sequence, and reselect the overlapping patch sequence for input.

[0077] In one specific embodiment, step S6 includes the following:

[0078] S61. Calculate the cross-entropy between the tokens output from the second stage and the ground real labels to obtain the first loss:

[0079] L cross =CE(y,y2)

[0080] Where y represents the ground real label, and y2 represents the token-like output of the second stage;

[0081] S62. Perform batch-wise comparison loss calculation on the image features output from the second stage to obtain the second loss:

[0082]

[0083] Where B represents the batch size, and z i and z j Represents image features, β is the boundary introduced by the contrast loss, which takes effect when the value exceeds this range, and y gi and y gj Let i and j represent the ground truth labels for the i-th and j-th images, respectively.

[0084] S63. Supervise the class tokens output from the second stage with the class tokens output from the first stage, and calculate the third loss using Kullback-Leibler divergence:

[0085] L kull =KL(y1,y2)

[0086] S64. Add the above three losses to obtain the final loss and then use it for training supervision:

[0087] L total =L cross +L con +L kull

[0088] S65. During testing, fine-grained image classification results are obtained based on the class tokens from the second stage.

[0089] In this embodiment, see Figure 2 The diagram illustrates the basic architecture and working principle of the network model in this invention. First, the image is processed using two different partitioning modes to obtain non-overlapping patch sequences and overlapping patch sequences, and a class token is merged for each sequence. In the first stage, the non-overlapping patch sequences are processed by a Transformer-based encoder feature extractor, generating attention scores and a class token. In the second stage, the attention scores from all layers are integrated, and the locations of key patches are identified through a sorting operation. Then, overlapping patch sequences are extracted from these locations and used as input to a reused feature extractor to obtain finer-grained class tokens and image features. Finally, this architecture generates the class tokens from both stages and the final image features, which are used as components of the training loss.

[0090] As described in the above embodiments, those skilled in the art will understand that, in order to address the problem of how to amplify the feature information of fine-grained images and reduce the interference of background information unrelated to the target in the image, this invention proposes a fine-grained image classification method based on a two-stage attention localization framework. This method fully leverages the advantages of deep learning, constructing a two-stage fine-grained image classification model. Utilizing the data processing module, backbone feature extraction network, attention selection module, and loss function calculation module within the model, it effectively locates important target regions in the image, extracts more refined low-level features through position transformation, provides more comprehensive information, reduces interference from low-level background information, and thus improves the accuracy of fine-grained image classification. This method not only reduces the interference of background information unrelated to the fine-grained target but also captures subtle differences between different categories by learning more discriminative features, amplifying the inter-class differences in fine-grained images and helping to more accurately determine the specific category of a subcategory under the base category.

[0091] Furthermore, the present invention also provides a fine-grained image classification system based on a two-stage attention localization framework, applied to a fine-grained image classification method based on a two-stage attention localization framework in the above embodiments, to improve the accuracy of fine-grained image classification. The system includes:

[0092] The model building module is used to build a two-stage fine-grained image classification model. The built fine-grained image classification model includes a data processing module, a first backbone feature extraction network, an attention selection module, a second backbone feature extraction network, and a loss function calculation module connected in sequence.

[0093] The image segmentation module is used to segment fine-grained images using two segmentation modes through the data processing module, resulting in non-overlapping patch sequences and overlapping patch sequences, and merging a class token for each sequence respectively;

[0094] The feature extraction module is used to input the non-overlapping patch sequences after segmentation into the first backbone feature extraction network in the first stage to generate an attention score and a class token; and in the second stage, the attention selection module integrates the obtained multi-layer attention scores, selects overlapping patch sequences from the corresponding positions through position transformation, and inputs the selected overlapping patch sequences into the second backbone feature extraction network with reused parameters to obtain more refined class tokens and image features.

[0095] The image classification module utilizes the loss function calculation module based on the class tokens of the two stages and the final image features as components of the training loss, and obtains fine-grained image classification results based on the class tokens of the second stage.

[0096] The system provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0097] In addition, embodiments of the present invention also provide a storage medium storing one or more programs readable by a computing device. The one or more programs include instructions that, when executed by the computing device, cause the computing device to perform a fine-grained image classification method based on a two-stage attention localization framework as described in the above embodiments.

[0098] In embodiments of the present invention, the storage medium may be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, and any suitable combination thereof.

[0099] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0100] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fine-grained image classification method based on a two-stage attention localization framework, characterized in that, The method includes the following steps: S1. Construct a two-stage fine-grained image classification model; the constructed fine-grained image classification model includes a data processing module, a first backbone feature extraction network, an attention selection module, a second backbone feature extraction network, and a loss function calculation module connected in sequence; S2. The data processing module uses two partitioning modes to partition the fine-grained image to obtain non-overlapping patch sequences and overlapping patch sequences, and merges a class token for each sequence. S3. In the first stage, the non-overlapping patch sequence after division is input into the first backbone feature extraction network to generate an attention score and a class token. S4. In the second stage, the attention selection module is used to integrate the obtained multi-layered attention scores, and overlapping patch sequences are selected from the corresponding positions through position transformation; specifically including: S41. Average the attention scores of each layer output by the first backbone feature extraction network in the first stage, using the following formula: in, Indicates the first The average attention score of the layer, Indicates the sequence number of the attention head; S42. Starting from the attention score of the preset layer, integrate the attention scores using an exponential moving average, with the following formula: in, Indicates the patch selection ratio. Indicates the previous integration The average attention score of the layer; S43. In the score matrix, retain only the attention score of the first class token; sort the scores, convert the patch positions with scores higher than the preset value into the corresponding positions of the overlapping patch sequence, and reselect the overlapping patch sequence for input. S5. Input the selected overlapping patch sequence into the second backbone feature extraction network of the reuse parameters to obtain more refined class tokens and image features. S6. The loss function calculation module uses the class tokens of the two stages and the final image features as components of the training loss, and obtains fine-grained image classification results based on the class tokens of the second stage.

2. The fine-grained image classification method based on a two-stage attention localization framework according to claim 1, characterized in that, In step S1, the first backbone feature extraction network and the second backbone feature extraction network of the fine-grained image classification model are both feature extractors based on Transformer encoder; the feature extractor contains 12 layers, each layer contains a normalization module, a multi-head attention module and a multilayer perceptron module; the multilayer perceptron module connects a linear layer and the GELU activation function.

3. The fine-grained image classification method based on a two-stage attention localization framework according to claim 1, characterized in that, Step S2 specifically includes: The data processing module uses convolution operations to segment and embed the image, and obtains patch sequences of two segmentation modes by setting different parameters. When the convolution kernel and stride are both set to P, the input dimension is the RGB dimension of the original image, and the output dimension is the embedding dimension, a non-overlapping patch sequence is obtained. When the stride is set to S, such that S is less than P, an overlapping patch sequence is obtained. After the segmentation and embedding are completed, a class token is merged between the overlapping patch sequence and the non-overlapping patch sequence. A parameterized position embedding is then added to each of the two sequences to obtain the final overlapping patch sequence input and the non-overlapping patch sequence input.

4. The fine-grained image classification method based on a two-stage attention localization framework according to claim 3, characterized in that, The specific operation of adding a parameterized position embedding to each of the two sequences is as follows: in, This represents the final sequence after processing. This represents a learnable class token used for classification. Represents the first part of the embedding. One patch, This represents the position encoding parameter.

5. A fine-grained image classification method based on a two-stage attention localization framework according to claim 1, characterized in that, In step S6, the working process of the loss function calculation module includes: The first loss is obtained by calculating the cross-entropy between the tokens output from the second stage and the ground real labels; The image features output from the second stage are compared and lost in batches to obtain the second loss. The class tokens output by the second stage are used to supervise the class tokens output by the first stage, and the third loss is calculated by Kullback-Leibler divergence. The three losses are summed to obtain the final loss for training supervision; during testing, fine-grained image classification results are obtained based on the class tokens from the second stage.

6. A fine-grained image classification system based on a two-stage attention localization framework, characterized in that, A fine-grained image classification method based on a two-stage attention localization framework as described in any one of claims 1-5 is used to perform fine-grained image classification. The system includes: The model building module is used to build a two-stage fine-grained image classification model; the built fine-grained image classification model includes a data processing module, a first backbone feature extraction network, an attention selection module, a second backbone feature extraction network, and a loss function calculation module connected in sequence. The image segmentation module is used to segment a fine-grained image using two segmentation modes through the data processing module, to obtain non-overlapping patch sequences and overlapping patch sequences, and to merge a class token for each sequence. The feature extraction module is used to input the non-overlapping patch sequences after division into the first backbone feature extraction network in the first stage to generate an attention score and a class token; and in the second stage, the attention selection module is used to integrate the obtained multi-layer attention scores, select overlapping patch sequences from the corresponding positions through position transformation; and input the selected overlapping patch sequences into the second backbone feature extraction network with reuse parameters to obtain more refined class tokens and image features. The image classification module is used to utilize the loss function calculation module based on the class tokens of the two stages and the final image features as components of the training loss, and to obtain fine-grained image classification results based on the class tokens of the second stage.

7. A fine-grained image classification system based on a two-stage attention localization framework according to claim 6, characterized in that, In the constructed fine-grained image classification model, both the first backbone feature extraction network and the second backbone feature extraction network are feature extractors based on Transformer encoders; the feature extractor contains 12 layers, each layer containing a normalization module, a multi-head attention module, and a multilayer perceptron module; the multilayer perceptron module connects linear layers and the GELU activation function.

Images