Image classification method and system based on multi-granularity mixed fine-grained prototype network

Abstract

The application provides an image classification method and system based on a multi-granularity mixed fine-grained prototype network, wherein different granularities of bottom layer fine-grained features are extracted from different layers of a Swin-Transformer backbone network of a visual model by using an MIE module, and are used as a source of corresponding category prototype learning; a prototype distillation strategy is used to refine the prototypes in the prototype library, so that the prototypes can more accurately express the bottom layer features of objects; in order to enable the visual model to learn more robust fine-grained representations, a prototype exchange strategy is used, that is, the prototypes of different categories are exchanged, so that the model learns the differences between the representations of different categories; finally, the prototype vectors are matched with the prototype representations corresponding to the prototypes of each category in the prototype library, and the prediction results of the classification head of the visual model are reliably fused, so as to obtain the final predicted probability distribution, thereby improving the classification accuracy and efficiency.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, and specifically relates to an image classification method and system based on multi-granularity hybrid fine-grained prototype networks. Background Technology

[0002] Fine-grained visual classification (FGVC) aims to further refine the classification of different subcategories within the same broad category, such as birds, dogs, cars, and airplanes, and has wide-ranging applications. However, due to the high visual similarity between different subcategories in terms of shape, color, and texture, and the influence of factors such as pose, angle, lighting, and background, images within the same subcategory can exhibit significant differences, making it difficult to extract accurate fine-grained feature representations.

[0003] With the development of artificial intelligence, models are no longer solely focused on accuracy; the interpretability and reliability of decision-making and reasoning are becoming increasingly important, especially in the medical field. Prototype networks have gained significant attention in recent years due to their excellent interpretability. Recent work on FGVC uses prototypes to search and calculate similarity on feature maps to find the existence of prototypes for the corresponding categories, and then obtains the final category prediction based on the score of each prototype. However, current mainstream prototype network works often use single-feature representations, extracting only single-granularity features to represent the corresponding category, leaving a large portion of low-level discriminative features unrepresented in the prototypes. Furthermore, directly matching prototypes on feature maps increases computational resource consumption. Current work aims to obtain fine-grained representations for each category to guide classification. However, in FGVC, many samples have similar appearances, leading to similar feature representations, making it difficult to directly use the average of sample features to obtain fine-grained prototypes for the corresponding subcategories and learn the differences between different categories. Summary of the Invention

[0004] Based on this, the present invention provides an image classification method and system based on a multi-granularity hybrid fine-grained prototype network, which aims to solve the problems of low classification accuracy due to insufficient detail discrimination and low efficiency due to high computational resource overhead when using traditional fine-grained visual classification for image classification in the prior art.

[0005] A first aspect of this invention provides an image classification method based on a multi-granularity hybrid fine-grained prototype network, implemented through a visual model, the method comprising:

[0006] The image is acquired and input into the Swin-Transformer backbone network of the visual model. It outputs feature maps of different granularities and inputs the feature maps of the last three layers into the MIE module to extract the fine-grained features of different granularities. The fine-grained features of the bottom layer are concatenated to obtain the prototype vector.

[0007] The prototype distillation strategy is used to refine the prototypes in the prototype library, wherein the prototype vectors are fused with the prototypes of the real category labels in the prototype library, and the prototypes in the prototype library are classified in this way to optimize the prototypes in the prototype library.

[0008] The prototype exchange strategy forces the visual model to learn the differences between different prototypes. The prototype vector is mixed and exchanged with prototypes with non-real category labels in the prototype library in a certain proportion and used for classification. It is constrained and updated with a preset loss function. Finally, a robust prototype representation of multi-granular fine-grained low-level objects is obtained and stored in the prototype library.

[0009] The prototype vector is matched with the prototype representations corresponding to each type of prototype in the prototype library, and then reliably fused with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution.

[0010] Furthermore, the MIE module includes a discriminative feature downsampling submodule, a partial sampling compression submodule, and a cross-attention fusion submodule arranged sequentially. The discriminative feature downsampling submodule is used to downsample to the size of the last layer feature map through 2D convolution. The partial sampling compression submodule is used to learn fine-grained features of the object that have key features and are not limited to rectangular regions. The cross-attention fusion submodule is used to perform cross-attention fusion processing on the output of the partial sampling compression submodule and the feature map of the deepest layer of the Swin-Transformer backbone network.

[0011] Furthermore, in the discriminative feature downsampling submodule, the feature map is first downsampled to the size of the last layer feature map using 2D convolution, then the sampled feature map is squeezed and excited, and finally the obtained feature map is normalized using Layernorm.

[0012] Furthermore, the expression for the cross-attention fusion processing is:

[0013] ;

[0014] ;

[0015] Where softmax(•) is the softmax function, q(•) is the query generator function, k(•) is the key generator function, and X... iLet F4 be the feature map of the i-th layer, F4 be the feature map of the deepest layer of the Swin-Transformer backbone network, A be the fusion weight, and p be the fused feature.

[0016] Furthermore, in the step of fusing the prototype vector with the prototypes of the real category labels in the prototype library for classification to optimize the prototypes in the prototype library, the prototype vector and the prototypes of the real category labels in the prototype library are fused based on similarity, and classification prediction is performed on the distilled and fused features. Cross-entropy loss is used, and the prototypes in the prototype library are constrained and updated through backpropagation. The expression for the prototype distillation strategy is:

[0017] ;

[0018] in, p represents the same type of prototype obtained from the prototype library, q(•) represents the fused feature, k(•) represents the query generator function, and k(•) represents the key generator function.

[0019] Furthermore, the expression for the prototype exchange strategy is:

[0020] ;

[0021] in, The prototype after the exchange, IP represents purity. For the prototype of the current type in the prototype library, This refers to the different types of prototypes currently in the prototype library.

[0022] Furthermore, the expression for the preset loss function is:

[0023] ;

[0024] Where L is the preset loss function, y1 and y2 are the labels of different categories, P1 is the adaptive weight of category y1 during training, and P2 is the adaptive weight of category y2 during training. Let y1 be the cross-entropy loss function. Let y2 be the cross-entropy loss function.

[0025] A second aspect of this invention provides an image classification system based on a multi-granularity hybrid fine-grained prototypical network, used to implement the image classification method based on a multi-granularity hybrid fine-grained prototypical network described in the first aspect, the system comprising:

[0026] The acquisition module is used to acquire images. It inputs the images into the Swin-Transformer backbone network of the visual model and outputs feature maps of different granularities. The feature maps of the last three layers are then input into the MIE module to extract the fine-grained features of different granularities. The fine-grained features of the bottom layer are then concatenated to obtain the prototype vector.

[0027] The refinement module is used to refine the prototypes in the prototype library using a prototype distillation strategy, wherein the prototype vectors are fused with the prototypes with real category labels in the prototype library, and the prototypes in the prototype library are classified in this way to optimize the prototypes in the prototype library.

[0028] The exchange module is used to force the visual model to learn the differences between different prototypes using a prototype exchange strategy. The prototype vector is mixed and exchanged with prototypes with non-real category labels in the prototype library in a certain proportion and used for classification. It is constrained and updated with a preset loss function and finally obtains a robust prototype representation of multi-granular fine-grained low-level objects, which is stored in the prototype library.

[0029] The matching module is used to perform prototype matching between the prototype vector and the prototype representations corresponding to each type of prototype in the prototype library, and to perform reliable fusion with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution.

[0030] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image classification method based on a multi-granularity hybrid fine-grained prototype network provided in the first aspect.

[0031] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image classification method based on a multi-granularity hybrid fine-grained prototype network provided in the first aspect.

[0032] The image classification method and system based on multi-granularity hybrid fine-grained prototype networks provided in this invention extracts low-level fine-grained features of different granularities from different layers of the Swin-Transformer backbone network of the visual model using the MIE module, and uses these features as the source for learning corresponding class prototypes. A prototype distillation strategy is used to refine the prototypes in the prototype library, allowing the prototypes to more accurately represent the low-level features of the object. To enable the visual model to learn more robust fine-grained representations, a prototype exchange strategy is used, that is, exchanging prototypes of different types, allowing the model to learn the differences in representation between different types. Finally, prototype vectors are matched with the prototype representations corresponding to each type of prototype in the prototype library, and reliably fused with the prediction results of the visual model's classification head to obtain the final predicted probability distribution, thereby improving classification accuracy and efficiency. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the visual model's structure.

[0034] Figure 2 This is a flowchart illustrating the implementation of an image classification method based on a multi-granularity hybrid fine-granularity prototype network, as provided in Embodiment 1 of the present invention.

[0035] Figure 3 This is a schematic diagram of the extrusion excitation module;

[0036] Figure 4 This is a structural block diagram of an image classification system based on a multi-granularity hybrid fine-granularity prototype network provided in Embodiment 2 of the present invention;

[0037] Figure 5 This is a structural block diagram of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0038] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0039] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0041] Example 1

[0042] According to an embodiment of the present invention, an image classification method based on a multi-granularity hybrid fine-grained prototype network is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] This first embodiment provides an image classification method based on a multi-granularity hybrid fine-grained prototype network, implemented through a visual model, which can be used in electronic devices, such as computers. Please refer to [link to relevant documentation]. Figure 1 This is a schematic diagram of the structure of a visual model. Specifically, before the visual model training begins, a prototype library is initialized. The prototype library contains all kinds of prototype features, and the prototypes will be updated as training progresses.

[0044] Training begins with the Swin-Transformer used as the backbone network for feature extraction. The feature maps from the last three layers of the Swin-Transformer are extracted as the prototype learning objects. The MIE module extracts features from different layers of the Swin-Transformer, representing different granularities. Low-granularity features contain a wealth of texture detail information, while high-granularity features contain a wealth of high-level semantic information. The features extracted by the MIE module are flattened and concatenated to obtain the prototype feature vector extracted by the visual model. Subsequently, L... focus The loss (cross-entropy loss) forces the visual model to extract as many fine-grained features as possible from the F2, F3, and F4 layers, and applies a prototype learning strategy to this vector, namely prototype distillation and prototype exchange. Prototype distillation involves fusing the extracted prototype with prototypes of the same type from the prototype library and concatenating them with the global feature g extracted from F4, then classifying them using a classifier head and optimizing with the L0 (cross-entropy loss) loss function. Prototype exchange involves exchanging the extracted prototype with prototypes of different types from the prototype library in a certain proportion, concatenating them with the global feature g, classifying them again using a classifier head, and optimizing with the L0 (cross-entropy loss) loss function. mix The loss function (purity and maturity) is used to calculate the loss. This training strategy completes the learning and updating of the prototype library.

[0045] During the testing phase, not only are the features extracted by the visual model used for classification, but the extracted features are also matched with the prototypes trained in the prototype library, i.e., prototype matching (cosine similarity). The probability distribution obtained by the visual model classification is fused with the probability distribution of the prototype matching to achieve reliable probability fusion, and finally the image classification is completed.

[0046] Please see Figure 2 , Figure 2The flowchart of an image classification method based on a multi-granularity hybrid fine-grained prototype network provided in Embodiment 1 of the present invention is shown, specifically including steps S01 to S04.

[0047] Step S01: Acquire the image, input the image into the Swin-Transformer backbone network of the visual model, output feature maps of different granularities, and input the feature maps of the last three layers into the MIE module respectively to extract the low-level fine-grained features of different granularities. Concatenate the low-level fine-grained features to obtain the prototype vector.

[0048] Specifically, the feature maps of the last three layers contain high-level semantics, which are... Figure 1 It can be seen that feature map F2 is input into the corresponding MIE module MIE1, feature map F3 into the corresponding MIE module MIE2, and feature map F4 into the corresponding MIE module MIE3, and then feature extraction is performed in the corresponding MIE modules. Understandably, the visual model uses the Swin-Transformer backbone network as a feature extractor and extracts feature maps with different receptive fields and multi-granularity information from its different layers. Simultaneously, MIE modules are designed to extract prototype representations of objects at different granularities. Specifically, the MIE module includes a discriminative feature downsampling submodule, a partial sampling compression submodule, and a cross-attention fusion submodule, arranged sequentially. The discriminative feature downsampling submodule is used to downsample to the size of the last layer feature map through 2D convolution; the partial sampling compression submodule is used to learn fine-grained features of the object that have key features and are not limited to rectangular regions; the cross-attention fusion submodule is used to perform cross-attention fusion processing on the output of the partial sampling compression submodule and the feature map of the deepest layer of the Swin-Transformer backbone network. Guided by the high-level semantic information contained in the deepest feature maps, the MIE module can extract the most critical fine-grained features contained in each layer of feature maps and use them for the final prototype learning.

[0049] It should be noted that in the discriminative feature downsampling submodule, the feature map is first downsampled to the size of the last layer feature map using 2D convolution, then the sampled feature map is subjected to squeeze excitation (SE-Block), and finally the resulting feature map is normalized using Layernorm. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of the extrusion excitation module. Specifically, the first step is to input the feature map X (shaped as follows). ,in , For spatial dimensions, (Number of channels), through a conversion operation The intermediate feature map U (shaped as) is obtained. ).

[0050] Furthermore, a compression operation is performed on the intermediate feature map U to reduce its spatial dimension ( The information is compressed into global statistics along the channel dimension, and then the compressed vector is activated through a fully connected layer. Figure 1 middle To another The transformation, including weights W, learns the dependencies between channels, obtaining the weight of each channel. Finally, the channel weights obtained from the activation are recalibrated with the original intermediate feature map U (multiplied channel by channel) to obtain the recalibrated feature map. .

[0051] After processing by the discriminative feature downsampling submodule, preliminary discriminative information cues can be sampled at different granularities, but the feature map still contains a large amount of redundant information. To obtain concise discriminative features guided by high-level semantic information, it is first partially sampled. The channel dimension of the downsampled feature map is compressed through a Liner layer, allowing the visual model to learn features truly beneficial for classification. Then, the compressed feature map is fused with the feature map of the deepest layer of the Swin-Transformer backbone network using cross-attention. The expression for the cross-attention fusion process is as follows:

[0052] ;

[0053] ;

[0054] Where softmax(•) is the softmax function, q(•) is the query generator function, k(•) is the key generator function, and X... i Let F4 be the feature map of the i-th layer, F4 be the feature map of the deepest layer of the Swin-Transformer backbone network, A be the fusion weight, and p be the fused feature. Understandably, the purpose of cross-attention fusion is to allow the deepest high-level semantic information of the visual model to guide the compressed discriminative features obtained by downsampling at different granularities. Different granularities all use the deepest high-level semantic information, which also enhances the semantic consistency of features from different layers. Thus, we can obtain simplified discriminative features at different granularities, which can then be concatenated to serve as a template for subsequent prototype learning.

[0055] Step S02: Use a prototype distillation strategy to refine the prototypes in the prototype library, wherein the prototype vectors are fused with the prototypes with real category labels in the prototype library, and this classification is used to optimize the prototypes in the prototype library.

[0056] In this embodiment of the invention, the prototype vector is fused with the prototypes of the real category labels in the prototype library based on similarity, and the distilled and fused features are used for classification prediction. Cross-entropy loss is used, and the prototypes in the prototype library are constrained and updated through backpropagation. The expression for the prototype distillation strategy is as follows:

[0057] ;

[0058] in, For prototypes of the same type obtained from the prototype library, p is the fused feature, q(•) is the query generator function, and k(•) is the key generator function. This allows us to obtain prototypes of the corresponding type refined using real-type distillation, which are then used to train refinement using CELOSS cross-entropy loss.

[0059] Step S03: Use a prototype exchange strategy to force the visual model to learn the differences between different prototypes. In this step, the prototype vector is mixed and exchanged with prototypes with non-real category labels in the prototype library in a certain proportion and used for classification. The model is then constrained and updated using a preset loss function. Finally, a robust prototype representation of multi-granular fine-grained low-level objects is obtained and stored in the prototype library.

[0060] Specifically, the prototype swapping strategy involves swapping the prototype of the real category label with prototypes of different categories according to the generation ratio. The swapped features are then used to predict a probability distribution. A new loss function is used to constrain the classification loss after the swap, and the loss for each swapped category is assigned according to the prototype swapping ratio. For example, if the real category prototype is swapped with the swapped category by 30%, then there is a 70% probability of predicting the real category and a 30% probability of predicting the swapped category.

[0061] The expression for the prototype exchange strategy is as follows:

[0062] ;

[0063] in, The prototype after the exchange is represented by IP, which indicates purity, i.e., the exchange ratio. For the prototype of the current type in the prototype library, This represents the different types of prototypes currently in the prototype library. In the code, IP is a 0-1 matrix, signifying the retention of information. This yields prototypes with swapped IP ratios. These are then classified using a classification head and a new loss function, i.e., a pre-defined loss function. The expression for the pre-defined loss function is:

[0064] ;

[0065] Where L is the preset loss function, y1 and y2 are the labels of different categories, P1 is the adaptive weight of category y1 during training, and P2 is the adaptive weight of category y2 during training. Let y1 be the cross-entropy loss function. Let y2 be the cross-entropy loss function.

[0066] It should be noted that by using the aforementioned loss function for training, the model can learn the differences between different types of prototypes, thus addressing the challenge of small inter-class differences and large intra-class differences in current fine-grained image classification tasks. To improve the smoothness of the prototype exchange process, an adaptive training strategy is introduced, implemented through information purity and maturity. The MT module is used to measure the stability of the visual model's exchange at different training stages.

[0067] ;

[0068] ;

[0069] ;

[0070] Where P1 is the adaptive weight of class y1 during training, P2 is the adaptive weight of class y2 during training, IP is purity, and MT is maturity, used to control the adaptation during training. This represents the current training round number. The total number of training rounds. `<prototype>` is a hyperparameter that represents the number of rounds before prototype swapping stops. As training progresses, the MT module suppresses the impact of prototype swapping, allowing the visual model to be trained without swapping prototypes. In this way, the visual model can smoothly learn the differences between discriminative features of the corresponding category and those of different categories, aided by the prototypes of the swapped features.

[0071] Step S04: Perform prototype matching between the prototype vector and the prototype representations corresponding to each type of prototype in the prototype library, and perform reliable fusion with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution.

[0072] Prototype matching, also known as similarity matching, typically employs distance- or inner product-based metrics. For example, cosine similarity, Euclidean distance, or dot product can be used. Trustworthy fusion can employ weighted fusion, probability product / average, or attention-based fusion.

[0073] It should be noted that the image classification method based on multi-granularity hybrid fine-grained prototype networks described above has the following beneficial effects:

[0074] 1. Existing technologies primarily use the output of the last layer of the model for feature processing. This approach causes the model to ignore discriminative features derived from different levels of information, such as low-level contour and texture information. The fine-grained image classification model of this invention uses a MIE module to extract fine-grained features at different granularities. Simultaneously, a squeezing excitation module is introduced to ensure the richness of the learned features and the maximum recovery and preservation of discriminative information. Furthermore, an interaction between high-level semantic information and information at different granularities is designed to ensure semantic consistency across different granularities of the MIE. Thus, the features obtained from the MIE include both detailed information from low-level features and learned semantic information from high-level features.

[0075] 2. Existing technologies primarily focus on improving model accuracy through different attention methods and feature extraction techniques, neglecting the biggest problem facing current deep learning models in high-risk domains: interpretability. This invention innovatively introduces a prototype learning mechanism into fine-grained image classification, mapping discriminative features of different categories to a prototype domain. It learns the underlying representation of each category and uses it to aid classification, significantly enhancing the interpretability of deep learning models. The prototype represents the "central" or "typical" features of a category. The training process forces the model to learn the most discriminative and robust feature representations for each category, focusing on key attributes that distinguish the current class from other classes (especially neighboring classes). This directly addresses the problem of "minor inter-class differences."

[0076] 3. To address the challenge of small inter-class differences and large intra-class differences in fine-grained tasks, a prototype learning mechanism is introduced. This mechanism maps key features of different subclasses to a prototype space, and prototype learning strategies, namely prototype distillation and prototype exchange, are designed. In this space, the prototypes of similar but different classes (especially similar classes) are far apart. This perfectly aligns with the requirement of fine-grained classification to accurately distinguish similar categories. Constraining the prototypes using these two strategies ensures the accuracy and robustness of the learned prototypes.

[0077] In summary, the image classification method based on multi-granularity hybrid fine-grained prototype networks in the above embodiments of the present invention extracts low-level fine-grained features of different granularities from different layers of the Swin-Transformer backbone network of the visual model using the MIE module, and uses these features as the source for learning corresponding class prototypes; it uses a prototype distillation strategy to refine the prototypes in the prototype library, making the prototypes more accurately represent the low-level features of the object; in order to enable the visual model to learn more robust fine-grained representations, a prototype exchange strategy is used, that is, exchanging prototypes of different types, so that the model learns the differences in representations between different types; finally, the prototype vectors are matched with the prototype representations corresponding to the prototypes of each type in the prototype library, and the results are reliably fused with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution, thereby improving classification accuracy and efficiency.

[0078] Example 2

[0079] Please see Figure 4 , Figure 4 This is a structural block diagram of an image classification system based on a multi-granularity hybrid fine-grained prototype network, provided in Embodiment 2 of the present invention. This image classification system 200 based on the multi-granularity hybrid fine-grained prototype network is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0080] Specifically, the image classification system 200 based on a multi-granularity hybrid fine-grained prototype network includes: an acquisition module 21, a thinning module 22, an exchange module 23, and a matching module 24, wherein:

[0081] The acquisition module 21 is used to acquire images, input the images into the Swin-Transformer backbone network of the visual model, output feature maps of different granularities, and input the feature maps of the last three layers into the MIE module to extract the low-level fine-grained features of different granularities. The low-level fine-grained features are then concatenated to obtain the prototype vector. The MIE module includes a discriminative feature downsampling submodule, a partial sampling compression submodule, and a cross-attention fusion submodule, which are set sequentially. The discriminative feature downsampling submodule is used to downsample to the size of the last layer feature map through 2D convolution. The partial sampling compression submodule is used to learn fine-grained features of objects that have key features and are not limited to rectangular regions. The cross-attention fusion submodule is used to perform cross-attention fusion processing on the output of the partial sampling compression submodule and the feature map of the deepest layer of the Swin-Transformer backbone network. In addition, in the discriminative feature downsampling submodule, the feature map is first downsampled to the size of the last layer feature map using 2D convolution, then the sampled feature map is squeezed and excited, and finally the obtained feature map is normalized using Layernorm. The expression for the cross-attention fusion processing is:

[0082] ;

[0083] ;

[0084] Where softmax(•) is the softmax function, q(•) is the query generator function, k(•) is the key generator function, and X... i Let F4 be the feature map of the i-th feature map, F4 be the feature map of the deepest layer of the Swin-Transformer backbone network, A be the fusion weight, and p be the fused feature.

[0085] The refinement module 22 is used to refine the prototypes in the prototype library using a prototype distillation strategy. This involves fusing the prototype vectors with prototypes of the true category labels in the prototype library for classification to optimize the prototypes in the library. The module also performs similarity fusion between the prototype vectors and prototypes of the true category labels in the prototype library, and performs classification prediction on the distilled and fused features. Cross-entropy loss is used, and backpropagation is employed to constrain and update the prototypes in the prototype library. The expression for the prototype distillation strategy is:

[0086] ;

[0087] in, For prototypes of the same kind obtained from the prototype library, p is the fused feature, q(•) is the query generator function, and k(•) is the key generator function;

[0088] The exchange module 23 is used to force the visual model to learn the differences between different prototypes using a prototype exchange strategy. Specifically, the prototype vectors are mixed and exchanged with prototypes in the prototype library that do not have true category labels in a certain proportion, and then used for classification. A preset loss function is used for constraint and updating, ultimately obtaining robust prototype representations of multi-granularity, fine-grained low-level objects, which are stored in the prototype library. The expression for the prototype exchange strategy is:

[0089] ;

[0090] in, The prototype after the exchange, IP represents purity. For the prototype of the current type in the prototype library, For the different types of prototypes currently in the prototype library, the expression for the preset loss function is:

[0091] ;

[0092] Where L is the preset loss function, y1 and y2 are the labels of different categories, P1 is the adaptive weight of category y1 during training, and P2 is the adaptive weight of category y2 during training. Let y1 be the cross-entropy loss function. Let y2 be the cross-entropy loss function;

[0093] The matching module 24 is used to perform prototype matching between the prototype vector and the prototype representations corresponding to each type of prototype in the prototype library, and to perform reliable fusion with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution.

[0094] Example 3

[0095] In another aspect, the present invention also proposes an electronic device, please refer to [link to relevant documentation]. Figure 5 The image shown is an electronic device according to Embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, it implements the image classification method based on a multi-granularity hybrid fine-grained prototype network as described above.

[0096] In some embodiments, the processor 10 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 20 or process data, such as executing access restriction programs.

[0097] The memory 20 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 20 can include both internal and external storage units of the electronic device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or will be output.

[0098] It should be pointed out that, Figure 5 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0099] This invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image classification method based on a multi-granularity hybrid fine-grained prototype network as described above.

[0100] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0101] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0102] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0104] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. An image classification method based on multi-granularity hybrid fine-grained prototype networks, characterized in that, The method, implemented using a visual model, includes: The image is acquired and input into the Swin-Transformer backbone network of the visual model. It outputs feature maps of different granularities and inputs the feature maps of the last three layers into the MIE module to extract the fine-grained features of different granularities. The fine-grained features of the bottom layer are concatenated to obtain the prototype vector. The prototype distillation strategy is used to refine the prototypes in the prototype library, wherein the prototype vectors are fused with the prototypes of the real category labels in the prototype library, and the prototypes in the prototype library are classified in this way to optimize the prototypes in the prototype library. The prototype exchange strategy forces the visual model to learn the differences between different prototypes. The prototype vector is mixed and exchanged with prototypes with non-real category labels in the prototype library in a certain proportion and used for classification. It is constrained and updated with a preset loss function. Finally, a robust prototype representation of multi-granular fine-grained low-level objects is obtained and stored in the prototype library. The prototype vector is matched with the prototype representations corresponding to each type of prototype in the prototype library, and then reliably fused with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution. The MIE module includes a discriminative feature downsampling submodule, a partial sampling compression submodule, and a cross-attention fusion submodule arranged sequentially. The discriminative feature downsampling submodule is used to downsample to the size of the last layer feature map through 2D convolution. The partial sampling compression submodule is used to learn fine-grained features of the object that have key features and are not limited to rectangular regions. The cross-attention fusion submodule is used to perform cross-attention fusion processing on the output of the partial sampling compression submodule and the feature map of the deepest layer of the Swin-Transformer backbone network. The prototype vector is fused with the prototypes of the true category labels in the prototype library based on similarity, and the fused features are used for classification prediction. Cross-entropy loss is used, and the prototypes in the prototype library are updated through backpropagation constraints. The expression for the prototype distillation strategy is as follows: ； in, For prototypes of the same kind obtained from the prototype library, p is the fused feature, q(•) is the query generator function, and k(•) is the key generator function; The expression for the prototype exchange strategy is: ； in, The prototype after the exchange, IP represents purity. For the prototype of the current type in the prototype library, This refers to the different types of prototypes currently in the prototype library.

2. The image classification method based on multi-granularity hybrid fine-grained prototype networks according to claim 1, characterized in that, In the discriminative feature downsampling submodule, the feature map is first downsampled to the size of the last layer feature map using 2D convolution, then the sampled feature map is squeezed and excited, and finally the obtained feature map is normalized using Layernorm.

3. The image classification method based on multi-granularity hybrid fine-grained prototype networks according to claim 2, characterized in that, The expression for the cross-attention fusion processing is: ； ； Where softmax(•) is the softmax function, q(•) is the query generator function, k(•) is the key generator function, and X... i Let F4 be the feature map of the i-th layer, F4 be the feature map of the deepest layer of the Swin-Transformer backbone network, A be the fusion weight, and p be the fused feature.

4. The image classification method based on multi-granularity hybrid fine-grained prototype networks according to claim 3, characterized in that, The expression for the preset loss function is: ； Where L is the preset loss function, y1 and y2 are the labels of different categories, P1 is the adaptive weight of category y1 during training, and P2 is the adaptive weight of category y2 during training. Let y1 be the cross-entropy loss function. Let y2 be the cross-entropy loss function.

5. An image classification system based on a multi-granularity hybrid fine-grained prototype network, characterized in that, For implementing the image classification method based on a multi-granularity hybrid fine-grained prototype network as described in any one of claims 1-4, the system comprises: The acquisition module is used to acquire images. It inputs the images into the Swin-Transformer backbone network of the visual model and outputs feature maps of different granularities. The feature maps of the last three layers are then input into the MIE module to extract the fine-grained features of different granularities. The fine-grained features of the bottom layer are then concatenated to obtain the prototype vector. The refinement module is used to refine the prototypes in the prototype library using a prototype distillation strategy, wherein the prototype vectors are fused with the prototypes with real category labels in the prototype library, and the prototypes in the prototype library are classified in this way to optimize the prototypes in the prototype library. The exchange module is used to force the visual model to learn the differences between different prototypes using a prototype exchange strategy. The prototype vector is mixed and exchanged with prototypes with non-real category labels in the prototype library in a certain proportion and used for classification. It is constrained and updated with a preset loss function and finally obtains a robust prototype representation of multi-granular fine-grained low-level objects, which is stored in the prototype library. The matching module is used to perform prototype matching between the prototype vector and the prototype representations corresponding to each type of prototype in the prototype library, and to perform reliable fusion with the prediction results of the classification head of the visual model to obtain the final predicted probability distribution.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the image classification method based on a multi-granularity hybrid fine-grained prototype network as described in any one of claims 1-4.

7. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the image classification method based on a multi-granularity hybrid fine-grained prototype network as described in any one of claims 1-4.

Images