Low-resolution image fine-grained recognition method based on confusion-prone feature mining and decoupling

Abstract

The application proposes a low-resolution image fine-grained recognition method based on easily confused feature mining and decoupling. First, low-resolution and high-resolution images are input into a feature extractor respectively to obtain corresponding global class representations. Cosine similarity scores of the global class representations of low-resolution and high-resolution images and learnable class prototype vectors are calculated respectively, and a loss function is constructed to guide the learning of class prototypes and feature extractor parameters. The response of each class prototype vector on the feature map is calculated, and the response features of each class on the feature map are aggregated respectively to obtain corresponding grouping features. According to the similarity scores of the global representation of the low-resolution image and the class prototype vector, the K easily confused grouping features with the highest scores are selected. The feature representation of the feature map based on the class prototype is calculated to decouple the K easily confused grouping features, so as to decompose the fine-grained class characteristics. Finally, the decoupled features and the global class features are fused, and the target class is predicted.

Description

Technical Field

[0001] This invention relates to the fields of computer vision technology and image processing, specifically to a method for fine-grained recognition of low-resolution images based on the mining and decoupling of easily confused features. Background Technology

[0002] Traditional image recognition categorizes targets in an image into coarse-grained categories, which differ significantly in appearance features, such as bicycles, cars, and flowers. In contrast, fine-grained image recognition divides images into more refined subcategories within these coarse-grained categories, such as Arctic terns, Foster terns, and Lesser white-fronted terns. Targets within these subcategories typically share similar appearance features, with minimal inter-category differences, but are susceptible to variations in shooting angle and growth cycle, leading to greater intra-category differences. Furthermore, in some real-world applications, low-resolution images due to shooting distance and environmental factors result in missing target content, making it difficult for existing fine-grained recognition models to extract sufficient and accurate fine-grained features.

[0003] Existing super-resolution methods aim for good pixel-level approximation or high visual quality, but are not specifically designed for high-level semantic tasks. Fine-grained image recognition methods, on the other hand, aim to learn high-level semantic features, creating a task gap between the two. To address this, some researchers have constructed multi-task joint frameworks by combining image super-resolution and recognition tasks. These frameworks utilize downstream recognition tasks to constrain the super-resolution model to reconstruct the features required for recognition. A classic example of this approach is the 2022 paper by Yan et al., "Discriminative Feature Mining and Enhancement Network for Low-resolution Fine-grained Image Recognition," published in IEEE TCSVT. This algorithm guides the model to reconstruct and extract key discriminative features from both macroscopic and microscopic perspectives. Macroscopically, it trains the network's ability to recover and extract global discriminative features. Microscopically, it designs an information component mining module that, guided by difference and discriminative loss functions, mines a set of the most informative target components. These target components are then used to enhance the network's ability to recover and extract local discriminative fine-grained features.

[0004] Compared to joint multi-task frameworks, another type of approach extracts target semantic features from high-resolution images as additional information, constructs a constraint loss, and guides the recognition model to extract key robust discriminative features to improve the recognition accuracy of low-resolution data. A typical example of this type of approach is the paper "Dualdirected capsule network for very low resolution image recognition" published by Singh et al. at the ICCV conference in 2019. This algorithm uses capsule networks to learn features in low-resolution images that do not change with resolution, and constrains these features to reconstruct the original high-resolution image. At the same time, it designs a high-resolution anchor loss function to constrain the low-resolution image features to approximate the high-resolution image features, thereby improving the model's ability to extract key features.

[0005] However, subtle differences between fine-grained classes are usually distributed in the regions of semantic responses of similar classes. When the model extracts fine-grained features, it is more susceptible to interference from easily confused features of similar classes, which limits the recognition performance of low-resolution images. Summary of the Invention

[0006] In response to the technical problems mentioned in the background section, this invention provides a method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling. Addressing the aforementioned problems in existing technologies, this invention provides a method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling. It uses a set of learnable class prototypes as a medium for information interaction. By constraining the similarity scores between the global category features of low-resolution and high-resolution images and their corresponding class prototype vectors to maintain consistency, it guides the feature representation of low-resolution images to approximate the feature representation of high-resolution images. Furthermore, by mining features of easily confused categories in the feature map and decoupling these features, it extracts key fine-grained features hidden within the easily confused features, enhancing the target feature representation capability and improving the fine-grained recognition performance of low-resolution images.

[0007] The technical means employed in this invention are as follows: A fine-grained recognition method for low-resolution images based on easily confused feature mining and decoupling includes the following steps: Step 1: Input the low-resolution image and the high-resolution image into the feature extractor respectively to perform preliminary feature extraction and obtain the global category features of the low-resolution image and the high-resolution image respectively; Step 2: Set up a set of learnable class prototype vectors, calculate the cosine similarity scores between the global category features and class prototype vectors of the low-resolution and high-resolution images learned in Step 1, and construct a loss function to guide the parameter learning of the class prototype and feature extractor using the image category label as supervision information. Step 3: Calculate the response of each class prototype vector in Step 2 on the feature map, and aggregate the response features of each class on the feature map to obtain the corresponding grouped features; Step 4: Based on the similarity score between the global representation of the low-resolution image and the class prototype vector, select the K group features with the highest scores, which are the easily confused category features; Step 5: Using feature maps in the feature representation based on class prototypes, decouple the K sets of easily confused features obtained in Step 4, decompose fine-grained category characteristics from the easily confused features, and enhance the fine-grained category representation of the target. Step 6: The final category representation of the low-resolution image is the category representation output by the feature fusion module, which is composed of global category features and decoupled grouped features.

[0008] Compared with the prior art, the present invention has the following advantages: This invention models from the perspective of feature mining and decoupling, and combines a set of learnable class prototype vectors to complete the transmission of fine-grained discriminative information, thereby enhancing the model's ability to extract information from fine-grained features of low-resolution images and improving the accuracy of fine-grained recognition of low-resolution images.

[0009] This invention constructs a feature constraint loss function using a set of learnable class prototype vectors to achieve information transfer between high-resolution and low-resolution image features, enabling the global category features of low-resolution images to approximate the global category features of high-resolution images. On the other hand, by mining easily confused category features in the response of the feature map and performing feature decoupling, the fine-grained category characteristics hidden in the response features are decomposed, thereby enhancing the expressive power of fine-grained category features of low-resolution images. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a low-resolution image fine-grained recognition method based on easily confused feature mining and decoupling according to the present invention.

[0012] Figure 2 This is a schematic diagram of the easily confused feature mining module described in an embodiment of the present invention.

[0013] Figure 3 This is a schematic diagram of the easily confused feature decoupling module described in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] like Figure 1-3 As shown, this invention provides a method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling, comprising the following steps: Step 1: Input the low-resolution and high-resolution images into the feature extractor respectively for preliminary feature extraction, obtaining the global category features of the low-resolution and high-resolution images respectively; use ResNet50 as the feature extractor, and the parameters of the feature extractors for the high-resolution and low-resolution images are not shared; use the features of the last pooling layer of the feature extractor as the global category features of the high-resolution and low-resolution images respectively, denoted as... and .

[0017] Step 2: Set a set of learnable class prototype vectors, calculate the cosine similarity scores between the global category features and class prototype vectors of the low-resolution and high-resolution images learned in Step 1, and construct a loss function to guide the parameter learning of the class prototypes and feature extractors, using the image category labels as supervision information; in Step 2, set a set of learnable class prototypes ( ) feature vector dimension DThe dimension is the same as the global category feature dimension of the image; the learnable class prototype vector is used as the feature repository and information transmission medium. By constraining the similarity scores between the global category features of low-resolution and high-resolution images and their corresponding class prototype vectors, the feature representation of low-resolution images is guided to approximate the feature representation of high-resolution images; the corresponding loss function is calculated as follows: ; in Category labels indicating low-resolution and high-resolution images. This indicates the number of categories, and T represents the temperature coefficient. Represents an exponential function. Represents a logarithmic function; and They represent the first i The similarity score between the prototype feature vectors of each class and the high-resolution and low-resolution class representations is calculated using the following formula: ; .

[0018] Step 3: Calculate the response of each class prototype vector in Step 2 on the feature map, and aggregate the response features of each class on the feature map to obtain the corresponding grouped features; in Step 3, calculate the feature map of the last convolutional block of the feature extractor for each class prototype vector. The response is then aggregated and calculated by category to obtain the grouped features of each category on the feature map; the calculation formula for the grouped features is as follows: .

[0019] Step 4: Based on the similarity score between the global representation of the low-resolution image and the class prototype vector, select the one with the highest score. Each group feature is a easily confused category feature; in step 4, based on the similarity score... Select the highest The grouping features corresponding to each score are the most easily confused ones. The features of each category are denoted as... .

[0020] Step 5: Calculate the feature representation of the feature map based on the class prototype, and then use the feature map obtained in Step 4. The easily confused features are decoupled to extract fine-grained category characteristics, thereby enhancing the fine-grained category representation of the target; in step 5, by... The easily confused features are orthogonalized with the features of the corresponding categories to decompose the information of the interfering categories, explicitly model fine-grained category characteristics, and strengthen the representation of subtle differences between classes; the decoupling calculation formula for easily confused features is as follows: ; in, , Represents the weight matrix. This represents a category based on a class prototype. The calculation formula is as follows: ; ; Since the first grouping feature is composed of similarity scores The highest-level class prototype vector, obtained by aggregating it with the feature map, contains rich, fine-grained discriminative information, therefore the aforementioned feature decoupling operation is not performed.

[0021] Step 6: The final category representation of the low-resolution image is the category representation output by the feature fusion module, which combines the global category features and the decoupled grouped features. By setting a learnable category embedding, and utilizing the inherent correlation between the global category features and the decoupled grouped features of the low-quality image, feature fusion is performed to obtain the final category representation of the low-resolution image.

[0022] Example 1 This invention provides a method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling. A flowchart is shown below. Figure 1 Specifically, it includes the following steps: Step 1: Input the low-resolution image and the high-resolution image into the feature extractor respectively to perform preliminary feature extraction and obtain the global category features of both.

[0023] It should be noted that this invention uses ResNet50 as the feature extractor. The feature extractors for high-resolution and low-resolution images are independent and do not share parameters. The features from the last pooling layer of the feature extractor are used as the global category features for the high-resolution and low-resolution images, respectively denoted as... and The feature dimension is .

[0024] Step 2: Set up a set of learnable class prototype vectors, calculate the cosine similarity scores between the global class features and class prototype vectors of the low-resolution and high-resolution images learned in Step 1, and construct a loss function to guide the parameter learning of the class prototypes and feature extractors, using the image class labels as supervision information.

[0025] It should be noted that this invention sets up a learnable class prototype ( ) feature vector dimension DThe dimension is the same as the global category feature dimension of the image; using learnable class prototype vectors as the feature repository and information transmission medium, the similarity scores between the global category features of low-resolution and high-resolution images and their corresponding class prototype vectors are constrained to maintain consistency, guiding the feature representation of low-resolution images to approximate the feature representation of high-resolution images. The corresponding loss function is calculated as follows: , in, and Let represent the similarity scores between the i-th class prototype feature vector and the high-resolution and low-resolution class representations, respectively, calculated using the following formula: , . Step 3: Input the low-resolution image feature map extracted in Step 1 and the learnable class prototype vectors in Step 2 into the easily confused feature mining module. Calculate the response of each class prototype vector in Step 2 on the feature map, and aggregate the response features of each class on the feature map to obtain the corresponding grouped features.

[0026] It should be noted that, as Figure 2 As shown, in the easily confused feature mining module, the response of each category vector in the class prototype to the feature map of the last convolutional block of the feature extractor is calculated. Then, the response features are aggregated by category to obtain the grouped features of each category on the feature map. The calculation formula for the grouped features is as follows: .

[0027] Step 4: Based on the similarity scores between the global representation of the low-resolution image and the class prototype vector, select the grouping features of the K most similar categories, which are the easily confused category features.

[0028] It should be noted that, in order to select feature vectors that potentially contain target discriminative information from grouped features, this invention uses similarity scores... Arrange from highest to lowest, and select the top... The grouping features corresponding to each score are the most easily confused ones. The features of each category are denoted as... .

[0029] Step 5: Use the low-resolution image feature map and the learnable class prototype vector as input to the easily confused feature decoupling module. Calculate the feature representation of the low-resolution image feature map based on the class prototype, and use the vector obtained in step (4) as input. The system decouples easily confused features and extracts fine-grained category characteristics from them, thereby enhancing the fine-grained category representation of the target.

[0030] It should be noted that, as Figure 3As shown, in the easily confused feature decoupling module, by... The easily confused features are orthogonalized with the features of their corresponding categories to decompose the information of interfering categories, explicitly model fine-grained category characteristics, and strengthen the representation of subtle differences between classes. The decoupling calculation formula for easily confused features is as follows: , in, , It is a weight matrix. It is a category representation based on the class prototype. The calculation formula is as follows: , , Since the first grouping feature is the one most relevant to the image category, the feature decoupling operation described above is not performed.

[0031] Step 6: The final category representation of the low-resolution image is the category representation output by the feature fusion module, which is composed of global category features and decoupled grouped features.

[0032] It should be noted that this invention achieves feature fusion by setting a learnable category embedding and utilizing the inherent correlation between the global category features of the low-resolution image and the decoupled grouped features to obtain the final category representation of the low-resolution image. Specifically, the learnable category embedding, the global category features of the low-quality image, and the decoupled grouped features are input into a Transformer Layer. After feature encoding and interaction, the output category embedding features serve as the final category representation of the image, and are then input into a classification layer for category prediction and supervision.

[0033] Based on the above steps, this invention trains and tests the proposed fine-grained recognition method for low-resolution images based on easily confused feature mining and decoupling. The public datasets CUB-200-2011, Stanford Cars, and FGVC-Aircraft are used. Pairs of high-resolution and low-resolution data are constructed by downsampling the high-resolution data. Here, the sizes of the low-resolution and high-resolution images are respectively... and The evaluation metric used is accuracy, which is the ratio of the number of samples correctly predicted to the total number of samples in all test data. Higher accuracy indicates better recognition results. The experimental results below demonstrate that the method proposed in this invention achieves better recognition accuracy. Existing methods are easily affected by easily confused category features in the feature representation of low-resolution fine-grained images, resulting in lower recognition accuracy than this invention. This invention improves the ability to represent the characteristics of low-resolution fine-grained images by mining and decoupling easily confused features of similar categories of image targets, thus improving the final recognition accuracy.

[0034] Table 1 Comparison results with existing advanced methods

[0035] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling, characterized in that, Includes the following steps: Step 1: Input the low-resolution image and the high-resolution image into the feature extractor respectively to perform preliminary feature extraction and obtain the global category features of the low-resolution image and the high-resolution image respectively; Step 2: Set up a set of learnable class prototype vectors, calculate the cosine similarity scores between the global category features and class prototype vectors of the low-resolution and high-resolution images learned in Step 1, and construct a loss function to guide the parameter learning of the class prototype and feature extractor using the image category label as supervision information. Step 3: Calculate the response of each class prototype vector in Step 2 on the feature map, and aggregate the response features of each class on the feature map to obtain the corresponding grouped features; Step 4: Based on the similarity score between the global representation of the low-resolution image and the class prototype vector, select the K group features with the highest scores, which are the easily confused category features; Step 5: Calculate the feature representation of the feature map based on the class prototype, decouple the K sets of easily confused features obtained in Step 4, and decompose fine-grained category characteristics from the easily confused features to enhance the fine-grained category representation of the target; in Step 5, by... The easily confused features are orthogonalized with the features of the corresponding categories to decompose the information of the interfering categories, explicitly model fine-grained category characteristics, and strengthen the representation of subtle differences between classes; the decoupling calculation formula for easily confused features is as follows: ； in, , Represents the weight matrix. This represents the category representation based on the class prototype; since the first grouping feature is based on the similarity score. The highest-level class prototype vector, obtained by aggregating with the feature map, contains rich fine-grained discriminative information, therefore the above feature decoupling operation is not performed; Step 6: The final category representation of the low-resolution image is the category representation output by the feature fusion module, which is composed of global category features and decoupled grouped features.

2. The method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling as described in claim 1, characterized in that, In step 1, ResNet50 is used as the feature extractor, and the parameters of the feature extractors for high-resolution and low-resolution images are not shared. The features of the last pooling layer of the feature extractor are used as the global category features for the high-resolution and low-resolution images, respectively denoted as... and .

3. The method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling as described in claim 1, characterized in that, In step 2, the dimension of the learnable class prototype vector is set. D The dimension is the same as the global category feature dimension of the image; the learnable class prototype vector is used as the feature repository and information transmission medium. By constraining the similarity scores between the global category features of low-resolution and high-resolution images and their corresponding class prototype vectors, the feature representation of low-resolution images is guided to approximate the feature representation of high-resolution images; the corresponding loss function is calculated as follows: ； in Category labels indicating low-resolution and high-resolution images. Indicates the number of categories. Indicates the temperature coefficient; Represents an exponential function. Represents a logarithmic function; and They represent the first i Similarity scores between the prototype feature vectors of each class and the high-resolution and low-resolution class representations.

4. The method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling according to claim 1, characterized in that, In step 3, the feature map of each category vector in the class prototype is calculated in the last convolutional block of the feature extractor. The response is then aggregated and calculated by category to obtain the grouped features of each category on the feature map.

5. The method for fine-grained recognition of low-resolution images based on easily confused feature mining and decoupling according to claim 1, characterized in that, In step 4, based on the similarity score Select the highest The grouping features corresponding to each score are the most easily confused ones. The features of each category are denoted as... .

6. The method according to claim 1, characterized in that, In step 6, a learnable category embedding is set up, and the inherent correlation between the global category features of the low-resolution image and the decoupled grouped features is used to perform feature fusion to obtain the final category representation of the low-resolution image.

Images