Low-resolution fine-grained image classification method based on component attention correction and detail enhancement

Abstract

The invention discloses a low-resolution fine-grained image classification method based on component attention correction and detail enhancement, and belongs to the field of low-resolution fine-grained image recognition. The method comprises the following steps: firstly, mining diversified and discriminative local representations from a low-resolution image; using learnable category embedding vectors to encode global fine-grained feature representation embedded in the low-resolution image blocks; and designing an attention correction loss for adjusting the attention activation intensity of each component query vector embedded in each image block in the middle coding layer. Inputting the extracted local fine-grained feature representation and global fine-grained feature representation into a feature fusion layer for information interaction and fusion by forcibly mapping the high-level semantic features of the low-resolution image blocks into the bottom-layer texture details of the corresponding high-resolution image blocks; and inputting the fused feature representation into a classification layer to carry out category prediction. According to the invention, the local fine-grained features in the target component can be accurately perceived and associated, and the classification precision of the low-resolution fine-grained image is improved.

Description

Technical Field

[0001] The present invention belongs to the field of low-resolution fine-grained image recognition, and relates to a low-resolution fine-grained image classification method based on component attention correction and detail enhancement. Background Art

[0002] Fine-grained image classification aims to distinguish different subcategories under the same coarse-grained category. These subcategories usually have highly similar appearance features and very small fine-grained differences. Traditional fine-grained image classification methods mainly study the location of key local areas from high-resolution input images to extract target-rich fine-grained features. Thanks to the powerful feature encoding ability of Transformer, some research works select image patches containing key fine-grained features by designing image patch sampling mechanisms to filter out image patch features that are not discriminative. These methods have been experimentally verified on multiple high-resolution fine-grained datasets, achieving high recognition accuracy and greatly promoting the development of fine-grained image recognition.

[0003] However, in practical application scenarios such as remote monitoring and automatic retail, high-resolution image data is not easy to collect due to the large shooting distance resulting in the small overall image area occupied by the target. Therefore, it is inevitable to face the problem of low-resolution fine-grained image classification. Due to the large difference in feature distribution between high-resolution data and low-resolution data, the fine-grained recognition model trained on high-resolution images often produces unreliable prediction results when directly applied to low-resolution images. Although the problem of inconsistent feature distribution between high-resolution training images and low-resolution test images can be alleviated by reducing the resolution of training images, it is still difficult to overcome the negative impact of resolution reduction and sparse image features on classification performance by training models only with low-resolution data. To solve this problem, an intuitive solution is to use image super-resolution methods to reconstruct the texture details of low-resolution images. Some methods integrate super-resolution models into recognition models to construct an end-to-end network structure, and the downstream recognition task guides the task of restoring fine-grained details.

[0004] In addition to image super-resolution-based methods, another paradigm for solving low-resolution image classification tasks is to use the features of high-resolution data to guide and constrain the feature learning process of low-resolution data. Some methods use the feature information of different levels in high-resolution data to design various knowledge distillation strategies and transfer them to the feature learning process of low-resolution data. In addition, the encoder-decoder structure-based method constrains the encoder to extract resolution-invariant features from low-resolution images by forcing the high-level semantic features of low-resolution images to reconstruct the texture details of the corresponding high-resolution images.

[0005] Although the above methods have improved the ability to distinguish low-resolution fine-grained objects to some extent, most of them tend to recognize low-resolution image objects by learning global feature representations consistent with high-resolution images, while the ability to learn local discriminative features of low-resolution images is still insufficient. Summary of the invention

[0006] In view of the above problems existing in the prior art, the present invention provides a low-resolution fine-grained image classification method based on component attention correction and detail enhancement, wherein a component attention correction module is used to mine diverse and discriminative component features; an attention correction loss is designed to adjust the attention activation strength of each component query vector in the intermediate coding layer to the features of each image block, reduce the attention of the component query vector to the non-discriminative area in the low-resolution image, and enhance the sensitivity of the component query vector to fine-grained features; by applying a dynamic reconstruction loss to the low-resolution image block, the high-level semantic features of the low-resolution image block are forced to be mapped to the underlying texture details of the corresponding high-resolution image block, thereby further enhancing the local fine-grained feature representation capability, thereby improving the low-resolution fine-grained image classification accuracy.

[0007] The technical solutions adopted to achieve the above objectives are:

[0008] A low-resolution fine-grained image classification method based on component attention correction and detail enhancement, the low-resolution fine-grained image classification method comprising the following steps:

[0009] Step 1: Divide the low-resolution image and the high-resolution image into non-overlapping image block sequences, and perform linear transformation on the two sets of image block sequences and map them into high-dimensional image block features; specifically:

[0010] For a given low-resolution image I LR , which is divided into N 16×16 image block sequences, which are recorded as It is then input into the linear transformation layer and mapped into a D-dimensional low-resolution image block feature, denoted as {v 1 , v 2 , ..., v N}, where D is the dimension of the low-resolution image patch features.

[0011] For the high-resolution image I in the training phase HR , perform the above operation and divide it into N image block sequences of 16×16 size, which are recorded as It is then input into the linear transformation layer and mapped into a D-dimensional high-resolution image block feature, denoted as {u 1 ,u 2 , ..., u N}, where D is the dimension of the high-resolution image patch features.

[0012] Step 2: Construct a component attention correction module, set a set of learnable component query vectors and a learnable category embedding, and use them together with the low-resolution image block features and high-resolution image block features generated in step 1 as the input of the component attention correction module, which is used to perceive and associate the local and global fine-grained features of the targets in the low-resolution image and the high-resolution image, and output the encoded low-resolution image component features, the global fine-grained feature representation of the low-resolution image, the high-resolution image component features, and the global fine-grained feature representation of the high-resolution image; specifically:

[0013] The feature extractor of the component attention correction module adopts ViT-B_16, which includes L encoding layers, and the parameters of the low-resolution image block feature and high-resolution image block feature extractors are shared. The input of the feature extractor is Q learnable component query vectors A category embedding v cls , N low-resolution image patch features and N high-resolution image features.

[0014] In the feature extractor, the component query vector performs feature perception and interaction with the low-resolution image patch features, and calculates the attention score of each component query vector on all low-resolution image patch features. (,in represents the attention score of the i-th component query vector on the j-th image block in the l-th encoding layer of the feature extractor, i∈{1,...,Q},j∈{1,...,N},l∈{1,...,L}); the low-resolution image block features are re-weighted and summed using the attention score to aggregate the fine-grained features of different parts of the image target and output Q low-resolution image component features and N low-resolution image patch features {v (L)，1 , v (L)，2 , ..., v (L)，N In the feature extractor, the category embedding is obtained by computing the attention map E between it and all low-resolution image patch features. l , aggregate all low-resolution image patch features, extract and output the global fine-grained feature representation of the low-resolution image

[0015] At the same time, the component query vector encodes the fine-grained features of different parts of the high-resolution image target by performing feature perception and interaction with all high-resolution image block features, and outputs Q high-resolution image component features and N high-resolution image patch features {u (L)，1 ,u (L)，2 , ..., u (L)，NIn the feature extractor, the category embedding extracts and outputs the global fine-grained feature representation of the high-resolution image by aggregating all high-resolution image patch features.

[0016] Step 3: Use the classification loss function to classify the low-resolution image component features output in step 2 and high-resolution image component features The discriminability is supervised. The orthogonal loss function is used to supervise the query vectors of different components in step 2 to focus on different regions of the target. The contrast loss function is used to further increase the distance between the global fine-grained feature representations of low-resolution images of different categories in step 2, and to reduce the distance between the global fine-grained feature representations of low-resolution images of the same category; specifically:

[0017] The classification loss function The calculation formula is as follows:

[0018]

[0019] Where C[·] represents the classification head; y i Indicates that the image belongs to the i-th category in the dataset, and C is the total number of categories; represents the low-resolution image component features output from step 2, represents the high-resolution image component feature output by step 2; j represents the index of the component query vector, and Q represents the number of learnable component query vectors.

[0020] The orthogonal loss function L o The calculation formula is as follows:

[0021]

[0022] in, and They represent the i-th and j-th low-resolution image component query vectors output by the last encoding layer respectively. and They represent the i-th and j-th high-resolution image component query vectors output by the last encoding layer, respectively. ||·||2 represents the L2 norm.

[0023] The contrast loss function The calculation is as follows:

[0024]

[0025] in, Represents the global fine-grained feature representation of a low-resolution image with class label yi; Indicates that the category label is y jThe global fine-grained feature representation of the low-resolution image; Sim(·) represents the function of calculating the similarity of two sample features; α is used to control the distance between features of different categories.

[0026] Step 4: Based on the attention score of each component query vector on the image patch feature in step 2 Extract the index and attention score of the first K low-resolution image block features corresponding to each component query vector (here the attention score of the first K image block features corresponding to the i-th component query vector is recorded as ), construct an attention correction loss function to correct the attention activation score of each component query vector to the low-resolution image patch feature;

[0027] The attention correction loss function L ac The calculation is as follows:

[0028]

[0029] stA[d]={A j [d]|A i [d]≥A j [d], d∈D i，j},

[0030] Among them, A[d] represents the attention score that needs to be optimized on the d-th image block of the part query vector, A i [d] represents the attention score of the i-th component query vector on the d-th image patch; A j [d] represents the attention score of the j-th component query vector on the d-th image patch; d represents the image patch index; D i，j represents the index set of repeated image blocks in the first K image blocks selected by the i-th and j-th component query vectors, and is calculated as follows:

[0031] D i，j =H(P i , P j ),

[0032] in, represents the set of the first K image block features that the i-th component query vector focuses on, H(P i , P j ) is used to calculate the index of the i-th and j-th component query vectors focusing on the repeated image block features.

[0033] During the training process, the number of repeated image block features searched by different component query vectors will gradually approach 0, so |D i，j The value of | varies between [0, K].

[0034] Step 5: Superimpose the attention scores of each component query vector on the first K low-resolution image block features in step 4 to obtain the key area attention map of all component query vectors on the image block features. Then extract the attention maps of all encoding layers of the feature extractor in the component attention correction module described in step 2 and fuse them, and fuse the fused attention map with the key area attention map to obtain a dynamic attention map; specifically:

[0035] The dynamic attention map A mod The calculation formula is as follows:

[0036]

[0037] Among them, M i Represents the binary mask of the i-th component query vector, and the K image patch features with the highest attention scores on the image patch features are in M i The mask value of the corresponding position in is 1, and the values ​​of the other positions are 0. β is a threshold used to further adjust the attention score of each component query vector on the image block feature. c represents the attention map E in all encoding layers output by step 2 l The attention map is obtained by multiplication. ⊙ represents the Hadamard product.

[0038] The first term on the right side of the equal sign in the above formula is used to extract and adjust the attention scores of the image block features that all component query vectors pay attention to, and the second term is used to extract the activation scores of the discriminative areas paid attention to in the cumulative attention map.

[0039] Step 6: Introduce a detail enhancement module, which takes the low-resolution image block features as input, maps them to the texture details of the high-resolution image blocks, and uses the dynamic attention map A obtained in step 5 mod A dynamic reconstruction loss function of a low-resolution image is constructed to constrain the fine-grained feature representation of a low-resolution image block to approach the fine-grained feature representation of a high-resolution image block during training; specifically:

[0040] The detail enhancement module is composed of L′ decoding layers stacked together, and its input is the N low-resolution image block features {v (L)，1 , v (L)，2 , ..., v (L)，N}, the output is the reconstructed high-resolution image patch

[0041] The dynamic reconstruction loss function of the low-resolution image The calculation is as follows:

[0042]

[0043] in, represents the jth high-resolution image block; Represents the jth high-resolution image block reconstructed from the jth low-resolution image block features.

[0044] In order to further improve the reconstruction capability of the detail enhancement module for the texture details of the key area, the present invention uses the high-resolution image block features {u (L)，1， u (L)，2 , ..., u (L)，N}As the input of the detail enhancement module, the corresponding high-resolution image block is reconstructed The dynamic reconstruction loss function of high-resolution images is calculated as follows:

[0045]

[0046] in, represents the high-resolution image patch reconstructed from the high-level semantic features of the high-resolution image patch. The total loss of dynamic reconstruction is calculated as follows:

[0047]

[0048] It should be noted that during the testing phase, the detail enhancement module was removed and the low-resolution images did not need to pass through this module.

[0049] Step 7: Introduce a feature fusion layer in the component attention correction module and reset a learnable category embedding v′ cls , and combine it with the Q low-resolution image component features output from step 2 The images are connected in series and input into the feature fusion layer for information interaction and fusion, and the fused feature representation is output; a classification layer is introduced to predict the category of the low-resolution image. The input of the classifier is the fused feature representation, and the output is the predicted category. Specifically:

[0050] The Q part features extracted in step 2 A learnable category embedding v′ with the new setting cls As the input of the feature fusion layer, information interaction and fusion are performed in the feature fusion layer, and the fused feature representation is output It is input into the classification layer and the classification supervision loss function is used for category prediction.

[0051] The classification supervision loss function L c The calculation is as follows:

[0052]

[0053] Among them, C represents the number of categories in the data set, yi represents the category label of the image; C[·] represents the classification layer, Represents the probability that the classification layer predicts the input as the i-th class.

[0054] Step 8: Input the low-resolution fine-grained test image into the component attention correction module trained in steps 1 to 7, and output the predicted category of the low-resolution fine-grained image.

[0055] It should be noted that in the testing phase, the model only requires low-resolution images as input, which are predicted by the component attention correction module, while high-resolution images and detail enhancement modules are only used in the training phase.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) The present invention designs a component attention correction module with attention correction loss, which accurately perceives and associates the local fine-grained features in the target component by correcting the attention activation score of each component query vector in the intermediate coding layer to the features of each image block.

[0058] (2) On the other hand, the present invention introduces a detail enhancement module with dynamic reconstruction loss into the detail enhancement module, thereby mapping the high-level semantic features of the low-resolution image blocks to the corresponding high-resolution texture details, so that the fine-grained feature representations of the low-resolution image blocks and the high-resolution image blocks remain consistent.

[0059] (3) The present invention can fully exploit the local fine-grained visual differences of targets in low-resolution images and improve the classification accuracy of low-resolution fine-grained images. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of a low-resolution fine-grained image classification method based on component attention correction and detail enhancement of the present invention.

[0061] Figure 2 It is a flow chart of a low-resolution fine-grained image classification method based on component attention correction and detail enhancement of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] like Figure 1 As shown, the present invention provides a low-resolution fine-grained image classification method based on component attention correction and detail enhancement, comprising the following steps:

[0065] Step 1: Divide the low-resolution image and the high-resolution image into non-overlapping image block sequences, and perform linear transformation on the two sets of image block sequences and map them into high-dimensional image block features; specifically:

[0066] For a given low-resolution image I LR , which is divided into N 16×16 image block sequences, which are recorded as It is then input into the linear transformation layer and mapped into a D-dimensional low-resolution image block feature, denoted as {v 1 , v 2 , ..., v N}, where D is the dimension of the low-resolution image patch features.

[0067] For the high-resolution image I in the training phase HR , perform the above operation and divide it into N image block sequences of 16×16 size, which are recorded as It is then input into the linear transformation layer and mapped into a D-dimensional high-resolution image block feature, denoted as {u 1 ,u 2 , ..., u N}, where D is the dimension of the high-resolution image patch features.

[0068] Step 2: Construct a component attention correction module, set a set of learnable component query vectors and a learnable category embedding, and use them together with the low-resolution image block features and high-resolution image block features generated in step 1 as the input of the component attention correction module, which is used to perceive and associate the local and global fine-grained features of the targets in the low-resolution image and the high-resolution image, and output the encoded low-resolution image component features, the global fine-grained feature representation of the low-resolution image, the high-resolution image component features, and the global fine-grained feature representation of the high-resolution image; specifically:

[0069] The feature extractor of the component attention correction module adopts ViT-B_16, which includes L coding layers, and the parameters of the low-resolution image block feature and high-resolution image block feature extractors are shared. Figure 1 As shown in (a), the input of the feature extractor is Q learnable component query vectors. A category embedding v cls , N low-resolution image block features and N high-resolution image block features.

[0070] In the feature extractor, the component query vector performs feature perception and interaction with the low-resolution image patch features, and calculates the attention score of each component query vector on all low-resolution image patch features. (,in represents the attention score of the i-th component query vector on the j-th image block in the l-th encoding layer of the feature extractor, i∈{1,...,Q},j∈{1,...,N},l∈{1,...,L}); the low-resolution image block features are re-weighted and summed using the attention score to aggregate the fine-grained features of different parts of the image target and output Q low-resolution image component features and N low-resolution image patch features {v (L)，1 , v (L)，2 , ..., v (L)，N In the feature extractor, the category embedding is obtained by computing the attention map E between it and all low-resolution image patch features. l , aggregate all low-resolution image patch features, extract and output the global fine-grained feature representation of the low-resolution image

[0071] At the same time, the component query vector encodes the fine-grained features of different parts of the image target by performing feature perception and interaction with all high-resolution image block features, and outputs Q high-resolution image component features and N high-resolution image patch features {u (L)，1 ,u (L)，2 , ..., u (L)，NIn the feature extractor, the category embedding extracts and outputs the global fine-grained feature representation of the high-resolution image by aggregating all high-resolution image patch features.

[0072] Step 3: Use the classification loss function to classify the low-resolution image component features output in step 2 and high-resolution image component features The discriminability is supervised. The orthogonal loss function is used to supervise the query vectors of different components in step 2 to focus on different regions of the target. The contrast loss function is used to further increase the distance between the global fine-grained feature representations of low-resolution images of different categories in step 2, and to reduce the distance between the global fine-grained feature representations of low-resolution images of the same category; specifically:

[0073] The classification loss function The calculation formula is as follows:

[0074]

[0075] Where C[·] represents the classification head; y i Indicates that the image belongs to the i-th category in the dataset, and C is the total number of categories; represents the low-resolution image component features output from step 2, represents the high-resolution image component feature output by step 2; j represents the index of the component query vector, and Q represents the number of learnable component query vectors.

[0076] The orthogonal loss function L o The calculation formula is as follows:

[0077]

[0078] in, and They represent the i-th and j-th low-resolution image component query vectors output by the last encoding layer respectively. and They represent the i-th and j-th high-resolution image component query vectors output by the last encoding layer, respectively. ||·||2 represents the L2 norm.

[0079] The contrast loss function The calculation is as follows:

[0080]

[0081] in, Represents the global fine-grained feature representation of a low-resolution image with class label yi; Indicates that the category label is y jThe global fine-grained feature representation of the low-resolution image; Sim(·) represents the function of calculating the similarity of two sample features; α is used to control the distance between features of different categories.

[0082] Step 4: Based on the attention score of each component query vector on the image patch feature in step 2 Extract the index and attention score of the first K low-resolution image block features corresponding to each component query vector (here the attention score of the first K image block features corresponding to the i-th component query vector is recorded as ), construct an attention correction loss function to correct the attention activation score of each component query vector to the low-resolution image patch feature;

[0083] The attention correction loss function L ac The calculation is as follows:

[0084]

[0085] stA[d]={A j [d]|A i [d]≥A j [d], d∈D i，j},

[0086] Among them, A[d] represents the attention score that needs to be optimized on the d-th image block of the part query vector, A i [d] represents the attention score of the i-th component query vector on the d-th image patch; A j [d] represents the attention score of the j-th component query vector on the d-th image patch; d represents the image patch index; D i，j represents the index set of repeated image blocks in the first K image blocks selected by the i-th and j-th component query vectors, and is calculated as follows:

[0087] D i，j =H(P i , P j ),

[0088] in, represents the set of the first K image block features that the i-th component query vector focuses on, H(P i , P j ) is used to calculate the index of the repeated image block features that the i-th and j-th component query vectors focus on. During the training process, the number of repeated image block features searched by different component query vectors will gradually approach 0, so |D i，j The value of | varies between [0, K].

[0089] Step 5: Superimpose the attention scores of each component query vector on the first K low-resolution image block features in step 4 to obtain the key area attention map of all component query vectors on the image block features. Then extract the attention maps of all encoding layers of the feature extractor in the component attention correction module described in step 2 and fuse them, and fuse the fused attention map with the key area attention map to obtain a dynamic attention map; specifically:

[0090] like Figure 1 As shown in (b), the dynamic attention map A mod The calculation formula is as follows:

[0091]

[0092] Among them, M i Represents the binary mask of the i-th component query vector, and the K image patch features with the highest attention scores on the image patch features are in M i The mask value of the corresponding position in is 1, and the values ​​of the other positions are O. β is a threshold used to further adjust the attention score of each component query vector on the image block feature. c represents the attention map E in all encoding layers output by step 2 l The attention map is obtained by multiplication. ⊙ represents the Hadamard product.

[0093] The first term on the right side of the equal sign in the above formula is used to extract and adjust the attention scores of the image block features that all component query vectors pay attention to, and the second term is used to extract the activation scores of the discriminative areas paid attention to in the cumulative attention map.

[0094] Step 6: Introduce a detail enhancement module, which takes the low-resolution image block features as input, maps them to the texture details of the high-resolution image blocks, and uses the dynamic attention map A obtained in step 5 mod A dynamic reconstruction loss function of a low-resolution image is constructed to constrain the fine-grained feature representation of a low-resolution image block to approach the fine-grained feature representation of a high-resolution image block during training; specifically:

[0095] The detail enhancement module is composed of L′ decoding layers stacked together, and its input is the N low-resolution image block features {v (L)，1 , v (L)，2 , ..., v (L)，N}, the output is the reconstructed high-resolution image patch

[0096] The dynamic reconstruction loss function of the low-resolution image The calculation is as follows:

[0097]

[0098] in, represents the jth high-resolution image block; Represents the jth high-resolution image block reconstructed from the jth low-resolution image block features.

[0099] In order to further improve the reconstruction capability of the detail enhancement module for the texture details of the key area, the present invention uses the high-resolution image block features {u (L)，1 ,u (L ) ，2 , ..., u (L)，N}As the input of the detail enhancement module, the corresponding high-resolution image block is reconstructed The dynamic reconstruction loss function of high-resolution images is calculated as follows:

[0100]

[0101] in, represents the high-resolution image patch reconstructed from the high-level semantic features of the high-resolution image patch. The total loss of dynamic reconstruction is calculated as follows:

[0102]

[0103] It should be noted that during the testing phase, the detail enhancement module was removed and the low-resolution images did not need to pass through this module.

[0104] Step 7: Introduce a feature fusion layer in the component attention correction module and reset a learnable category embedding v′ cls , and combine it with the Q low-resolution image component features output from step 2 The images are connected in series and input into the feature fusion layer for information interaction and fusion, and the fused feature representation is output; a classification layer is introduced to predict the category of the low-resolution image. The input of the classifier is the fused feature representation, and the output is the predicted category. Specifically:

[0105] The Q part features extracted in step 2 A learnable category embedding v′ with the new setting cls As the input of the feature fusion layer, information interaction and fusion are performed in the feature fusion layer, and the fused feature representation is output It is input into the classification layer and the classification supervision loss function is used for category prediction.

[0106] The classification supervision loss function L c The calculation is as follows:

[0107]

[0108] Among them, C represents the number of categories in the data set, y i represents the category label of the image; C[·] represents the classification layer, Represents the probability that the classification layer predicts the input as the i-th class.

[0109] Step 8: Input the low-resolution fine-grained test image into the component attention correction module trained in steps 1 to 7, and output the predicted category of the low-resolution fine-grained image.

[0110] It should be noted that in the testing phase, the model only requires low-resolution images as input, which are predicted by the component attention correction module, while high-resolution images and detail enhancement modules are only used in the training phase.

[0111] According to the above steps, the present invention trains and tests the proposed low-resolution fine-grained image classification method based on component attention correction and detail enhancement. The low-resolution fine-grained dataset used is generated by bilinear downsampling of the high-resolution fine-grained benchmark datasets CUB-200-2011, Nabirds, Standford-Dogs, Standford-Cars and FGVC-Aircraft, and is labeled as CUB-S, Nabird-S, Dog-S, Car-S and Aircraft-S respectively. The sizes of the low-resolution and high-resolution images generated here are 28×28 and 224×224 respectively. In the testing phase, this method only uses low-resolution images as model input.

[0112] The evaluation index used in this method is accuracy, that is, the ratio of the number of samples of the correctly predicted category in all test data to the number of samples of all test data. The higher the accuracy, the better the recognition result. The following experimental results show that the method proposed in the present invention has better classification accuracy. The present invention guides the component query vector to discover more discriminative fine-grained clues by correcting the attention activation score of the component query vector to each image block feature, and enhances the local fine-grained feature representation capability by introducing the dynamic reconstruction loss function constraint, thereby improving the classification accuracy of low-resolution fine-grained images.

[0113] Table 1 Comparison results between the present invention and the existing traditional method

[0114] method CUB-S Nabird-S Dog-S Cars-S Aircraft-S Baseline Network 76.26 68.13 74.27 69.08 73.17 TransFG 76.75 68.91 73.62 70.50 68.91 SIM-Trans 76.61 68.92 72.69 71.69 69.92 The present invention 80.47 72.54 76.70 75.15 74.52

[0115] The above-described embodiments merely express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A low-resolution fine-grained image classification method based on component attention correction and detail enhancement, characterized in that: The low-resolution fine-grained image classification method comprises the following steps: Step 1: Divide the low-resolution image and the high-resolution image into non-overlapping image block sequences, and perform linear transformation on the two sets of image block sequences and map them into high-dimensional image block features; Step 2: Build a component attention correction module, set a set of learnable component query vectors and a learnable category embedding, and use them together with the low-resolution image block features and high-resolution image block features generated in step 1 as the input of the component attention correction module, which is used to perceive and associate the local and global fine-grained features of the targets in the low-resolution image and the high-resolution image, and output the encoded low-resolution image component features, the global fine-grained feature representation of the low-resolution image, the high-resolution image component features, and the global fine-grained feature representation of the high-resolution image; Step 3: Use the classification loss function to classify the low-resolution image component features output in step 2 and high-resolution image component features The discriminability of the image is supervised; the orthogonal loss function is used to supervise the query vectors of different components in step 2 to focus on different areas of the target; the contrast loss function is used to expand the distance of the global fine-grained feature representation of low-resolution images of different categories in step 2, and to reduce the distance of the global fine-grained feature representation of low-resolution images of the same category; Step 4: Based on the attention score of each component query vector on the image patch feature in step 2 Extract the index and attention score of the first K low-resolution image block features corresponding to each component query vector, and record the attention score of the first K image block features corresponding to the i-th component query vector as Construct an attention correction loss function to correct the attention activation score of each component query vector to the low-resolution image patch feature; Step 5: Superimpose the attention scores of each component query vector on the first K low-resolution image block features in step 4 to obtain the key area attention map of all component query vectors on the image block features; then extract the attention maps of all encoding layers of the feature extractor in the component attention correction module in step 2 and fuse them, and fuse the fused attention map with the key area attention map to obtain the dynamic attention map A. mod ; Step 6: Introduce a detail enhancement module, which takes the low-resolution image block features as input, maps them to the texture details of the high-resolution image blocks, and uses the dynamic attention map A obtained in step 5 mod A dynamic reconstruction loss function of a low-resolution image is constructed to constrain the fine-grained feature representation of a low-resolution image block to approach the fine-grained feature representation of a high-resolution image block during training; Step 7: Introduce a feature fusion layer in the component attention correction module and reset a learnable category embedding v′ cls , and combine it with the Q low-resolution image component features output from step 2 The images are connected in series and input into the feature fusion layer for information interaction and fusion, and the fused feature representation is output; a classification layer is introduced to predict the category of the low-resolution image. The input of the classifier is the fused feature representation, and the output is the predicted category; Step 8: Input the low-resolution fine-grained test image into the component attention correction module trained in steps 1 to 7, and output the predicted category of the low-resolution fine-grained image.

2. According to claim 1, a low-resolution fine-grained image classification method based on component attention correction and detail enhancement is characterized in that: The step 1 is specifically as follows: For a given low-resolution image I LR , divide it into N image block sequences of the same size, which are recorded as It is then input into the linear transformation layer and mapped into a D-dimensional low-resolution image block feature, denoted as {v 1 ,v 2 ,...,v N }, where D is the dimension of the low-resolution image patch feature; For the high-resolution image I in the training phase HR , perform the above operation and divide it into N image block sequences of the same size, which are recorded as It is then input into the linear transformation layer and mapped into a D-dimensional high-resolution image block feature, denoted as {u 1 ,u 2 ,...,u N }, where D is the dimension of the high-resolution image patch features.

3. The low-resolution fine-grained image classification method based on component attention correction and detail enhancement according to claim 2 is characterized in that: The step 2 is specifically as follows: The feature extractor of the component attention correction module adopts ViT-B_16, which includes L encoding layers, and the parameters of the low-resolution image block feature and the high-resolution image block feature extractor are shared; the input of the feature extractor is Q learnable component query vectors A category embedding v cls , N low-resolution image block features and N high-resolution image features; In the feature extractor, the component query vector performs feature perception and interaction with the low-resolution image patch features, and calculates the attention score of each component query vector on all low-resolution image patch features. in represents the attention score of the i-th component query vector on the j-th image block in the l-th encoding layer of the feature extractor, i∈{1,…,Q}, j∈{1,…,N}, l∈{1,…,L}); the attention score is used to re-weight and sum the low-resolution image block features, aggregate the fine-grained features of different parts of the image target, and output Q low-resolution image component features and N low-resolution image patch features {v (L),1 ,v (L),2 ,...,v (L),N }; In the feature extractor, the category embedding is obtained by computing the attention map E between it and all low-resolution image patch features l , aggregate all low-resolution image patch features, extract and output the global fine-grained feature representation of the low-resolution image At the same time, the component query vector encodes the fine-grained features of different parts of the high-resolution image target by performing feature perception and interaction with all high-resolution image block features, and outputs Q high-resolution image component features and N high-resolution image patch features {u (L),1 ,u (L),2 ,...,u (L),N In the feature extractor, the category embedding extracts and outputs the global fine-grained feature representation of the high-resolution image by aggregating all high-resolution image patch features.

4. The low-resolution fine-grained image classification method based on component attention correction and detail enhancement according to claim 3 is characterized in that: The step 3 is specifically as follows: The classification loss function The calculation formula is as follows: Where C[·] represents the classification head; y i Indicates that the image belongs to the i-th category in the dataset, and C is the total number of categories; represents the low-resolution image component features output from step 2, represents the high-resolution image component feature outputted in step 2; j represents the index of the component query vector, and Q represents the number of learnable component query vectors; The orthogonal loss function L o The calculation formula is as follows: in, and Respectively represent the i-th and j-th low-resolution image component query vectors output by the last encoding layer; and They represent the i-th and j-th high-resolution image component query vectors output by the last encoding layer respectively; ∥·∥2 represents the L2 norm; The contrast loss function The calculation is as follows: in, Indicates that the category label is y i Global fine-grained feature representation of low-resolution images; Indicates that the category label is y j The global fine-grained feature representation of the low-resolution image; Sim(·) represents the function of calculating the similarity of two sample features; α is used to control the distance between features of different categories.

5. The low-resolution fine-grained image classification method based on component attention correction and detail enhancement according to claim 4 is characterized in that: The step 4 is specifically as follows: The attention correction loss function L ac The calculation is as follows: s.t.A[d]={A j [d]∣A i [d]≥A j [d],d∈D i,j }, Among them, A[d] represents the attention score that needs to be optimized on the d-th image block of the part query vector, A i [d] represents the attention score of the i-th component query vector on the d-th image patch; A j [d] represents the attention score of the j-th component query vector on the d-th image patch; d represents the image patch index; D i,j represents the index set of repeated image blocks in the first K image blocks selected by the i-th and j-th component query vectors, and is calculated as follows: D i,j =H(P i ,P j ), in, represents the set of the first K image block features that the i-th component query vector focuses on, H(P i ,P j ) is used to calculate the index of the i-th and j-th component query vectors focusing on the repeated image block features; During the training process, the number of repeated image block features searched by different component query vectors will gradually approach 0, so |D i,j The value of | varies between [0,K].

6. The low-resolution fine-grained image classification method based on component attention correction and detail enhancement according to claim 5, characterized in that: The step 5 is specifically as follows: The dynamic attention map A mod The calculation formula is as follows: Among them, M i Represents the binary mask of the i-th component query vector, and the K image patch features with the highest attention scores on the image patch features are in M i The mask value of the corresponding position in is 1, and the values ​​of the other positions are 0. β is a threshold used to further adjust the attention score of each component query vector on the image block feature. c represents the attention map E in all encoding layers output by step 2 l The attention map obtained by cumulative multiplication; ⊙ represents the Hadamard product.

7. The low-resolution fine-grained image classification method based on component attention correction and detail enhancement according to claim 6 is characterized in that: The step 6 is specifically as follows: The detail enhancement module is composed of L' decoding layers stacked together, and its input is the N low-resolution image block features {v (L),1 ,v (L),2 ,...,v (L),N }, the output is the reconstructed high-resolution image patch The dynamic reconstruction loss function of the low-resolution image The calculation is as follows: in, represents the jth high-resolution image block; represents the jth high-resolution image block reconstructed from the jth low-resolution image block features; Using the high-resolution image block features {u (L),1 ,u (L),2 ,...,u (L),N }As the input of the detail enhancement module, the corresponding high-resolution image block is reconstructed The dynamic reconstruction loss function of high-resolution images is calculated as follows: in, represents the high-resolution image patch reconstructed by the high-level semantic features of the high-resolution image patch; the total loss of dynamic reconstruction is calculated as follows:

8. The low-resolution fine-grained image classification method based on component attention correction and detail enhancement according to claim 6, characterized in that: The step 7 is specifically as follows: The Q part features extracted in step 2 A learnable category embedding v′ with the new setting cls As the input of the feature fusion layer, information interaction and fusion are performed in the feature fusion layer, and the fused feature representation is output Input it into the classification layer and use the classification supervision loss function to make category prediction; The classification supervision loss function L c The calculation is as follows: Among them, C represents the number of categories in the data set, y i represents the category label of the image; C[·] represents the classification layer, Represents the probability that the classification layer predicts the input as the i-th class.

Images