Method, device and equipment for identifying interpretable fine-grained image and medium

Abstract

The invention provides an interpretable fine-grained image recognition method, device and equipment and a medium, and relates to the technical field of fine-grained image recognition, and the method adopts a double-branch structure frame, and combines feature fusion and prototype learning to carry out fine-grained image classification. Wherein the feature extraction layer captures a complex mode and long-distance dependence in an image through a self-attention mechanism, meanwhile, a feature selection fusion module is introduced, information marks of prototype learning are effectively selected, and it is ensured that the learned prototype pays more attention to the prospect. The prototype learning layer designs a double-branch structure, processes and learns a global prototype and a local prototype of an image respectively, improves the accuracy of the network, and explains a model classification result from a global perspective. And the full connection layer realizes a final classification decision, and obtains a final classification probability through weighted sum. The objective of the invention is to solve the problem of black box characteristics of an existing deep learning model in fine-grained image classification and improve the application credibility of the model in the high-risk field.

Description

Technical Field

[0001] The present invention relates to the technical field of fine-grained image recognition, and in particular to an explainable fine-grained image recognition method, device, equipment and medium. Background Art

[0002] In the field of artificial intelligence, fine-grained image recognition is a challenging task that requires algorithms to be able to identify different subcategories within the same large category that are very similar in visual features. With the development of deep learning technology, especially the application of convolutional neural networks (CNN) and visual transformers (ViT), the accuracy of fine-grained image classification has been significantly improved. However, the black-box nature of deep learning models limits their application in high-risk areas such as medical decision-making, autonomous driving, and toxic biological identification, because in these areas, the decision-making process of the model needs to be highly explainable and credible.

[0003] Among them, the goal of the field of computer vision is to simulate and expand the functions of the human visual system so that computers can extract useful information from images and videos. The fine-grained image classification task is a core problem in computer vision, which requires the algorithm to be able to identify subcategories with subtle differences in the same large category. Although deep learning models have made progress in this field, the opacity of their internal information processing makes the decision-making process of the model difficult for humans to understand and explain, which to some extent limits its application in high-risk fields.

[0004] To solve this problem, the market currently proposes ex ante interpretability algorithms and ex post interpretability algorithms. Ex ante interpretability algorithms are designed to make the model structure itself interpretable, or the model's reasoning process is understandable to humans, such as linear / logistic regression models and decision tree models. Ex post interpretability algorithms are designed for trained models to explain the model's behavior logic and decision basis. However, these methods either sacrifice the accuracy of the model or provide explanations that are not completely credible.

[0005] The prototype learning-based method is a research direction proposed in recent years. It learns the prototype part representing the typical features of each class for each class, and identifies the category by calculating the similarity between the features of the input image and the prototype. This method is inspired by the human visual system and can provide a certain degree of interpretability. However, existing prototype learning-based methods usually use convolutional neural networks as the backbone network for extracting features, which makes the extracted features easy to lose detailed features, and the prototype learning is not accurate enough and is prone to focusing on the background.

[0006] In view of this, this application is filed. Summary of the invention

[0007] The present invention provides an explainable fine-grained image recognition method, device, equipment and medium, which can at least partially improve the above-mentioned problems.

[0008] To achieve the above object, the present invention adopts the following technical solutions: An interpretable fine-grained image recognition method, comprising: Obtain an image x to be identified, input the image x into a feature extraction layer for feature extraction preprocessing, and generate an output vector; The category encoding vector of the output vector is used as the global feature and input into the global branch of the prototype learning layer for calculation to obtain the similarity score between the global feature and the global prototype; Obtaining N local features output by the feature extraction layer, and selecting K preset foreground information related features for local prototype learning, to obtain similarity scores between the local features and the local prototype; A fully connected layer is used to perform probability calculation processing on the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype to obtain a final classification probability, and an image recognition result is generated based on the classification probability.

[0009] The present invention also provides an interpretable fine-grained image recognition device, which comprises: A feature extraction unit is used to obtain an image x to be identified, input the image x into a feature extraction layer for feature extraction preprocessing, and generate an output vector; The global branch unit is used to input the category encoding vector of the output vector as a global feature into the global branch of the prototype learning layer for calculation to obtain a similarity score between the global feature and the global prototype; A local branch unit, used to obtain the local features output by the N feature extraction layers, and select K preset foreground information related features for local prototype learning to obtain a similarity score between the local features and the local prototype; The fully connected unit is used to use the fully connected layer to perform probability calculation processing on the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype to obtain the final classification probability, and generate an image recognition result based on the classification probability.

[0010] The present invention also provides an interpretable fine-grained image recognition device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the interpretable fine-grained image recognition method described in any one of the above.

[0011] The present invention also provides a readable storage medium storing a computer program, which can be executed by a processor of a device where the storage medium is located to implement the interpretable fine-grained image recognition method as described in any one of the above.

[0012] In summary, the interpretable fine-grained image recognition method aims to improve the accuracy and interpretability of deep learning models in fine-grained image classification tasks, especially in high-risk areas such as medical decision-making and autonomous driving. This method effectively combines feature fusion and prototype learning by designing a dual-branch structural framework to achieve a deep understanding of the global and local features of the image.

[0013] Specifically, in the feature extraction layer, this method adopts the Transformer architecture and uses the self-attention mechanism to capture the global features and local detail information in the image. In particular, a feature selection fusion module is introduced, which can select important features from each layer of the Transformer and fuse these features to enhance the model's recognition ability of foreground features while reducing the interference of background features. The prototype learning layer is the core of this method. It includes a global branch and a local branch, which are responsible for learning the global prototype and local prototype of the image respectively. The global branch focuses on the overall information of the image, while the local branch focuses on the details and local features of the image. This dual-branch structure enables the model to understand the image from different scales, improving the accuracy of classification and the interpretability of the model. Finally, the fully connected layer performs a weighted sum of the similarity scores obtained by the global branch and the local branch to obtain the final classification probability.

[0014] In general, the fine-grained image recognition method not only improves the classification accuracy of the model, but also enhances the interpretability of the model, making it more suitable for applications in fields that require high model interpretability. By combining prototype learning and feature fusion, this method provides a new solution for the field of fine-grained image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of an explainable fine-grained image recognition method provided by the first embodiment of the present invention; Figure 2 is a schematic diagram of the overall structure of the explainable fine-grained image recognition method provided by the first embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a feature selection fusion module of an interpretable fine-grained image recognition method provided by an embodiment of the present invention; Figure 4 is a schematic diagram of visualization results of a local prototype before and after using an ARWS module provided by an embodiment of the present invention; Figure 5is a schematic diagram of a global prototype visualization result provided by an embodiment of the present invention; Figure 6 is a reasoning process diagram of the explainable fine-grained image recognition method provided by an embodiment of the present invention; Figure 7 It is a module diagram of an explainable fine-grained image recognition device provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] refer to Figure 1 , Figure 2 , Figure 6 As shown, the first embodiment of the present invention discloses an interpretable fine-grained image recognition method, which can be executed by an interpretable fine-grained image recognition device (hereinafter referred to as a recognition device), and in particular, by one or more processors in the recognition device to implement the following method: See also Figure 3 , Figure 4 , S1, obtain the image x to be identified, input the image x into the feature extraction layer for feature extraction preprocessing, and generate an output vector; Specifically, step S1 includes: obtaining an image x to be identified, , where the size of image x is , H is the image height, W is the image width, C is the number of image channels, is a matrix; Divide the image x into N patches of equal size and tile the patches to a length of Sequence images , , where the patch size is , is the resolution of each image block, is the number of blocks finally generated; For the sequence of images Perform linear projection processing to generate a feature sequence; Add the category encoding vector to the feature sequence , as the encoding feature of image x; Add a positional encoding vector , connect the two-dimensional position information of image x to the one-dimensional feature sequence to obtain the visual vector , ,in, is the first image sequence, is the second image sequence, is the Nth image sequence, E is the unit matrix, is the product of the first image sequence and the identity matrix, is the product of the second image sequence and the identity matrix, is the product of the Nth image sequence and the identity matrix; The visual vector Input into the module of stacking L Transformer encoders to get the output vector of the lth layer Transformer encoder , , , where each Transformer encoder includes one or more self-attention heads and a multi-layer perceptron, LN represents the normalization operation, MSA represents multi-head self-attention, and MLP represents a multi-layer perceptron. is the output vector of the l-1th layer Transformer encoder; The output vector The category encoding vector of , input to the global branch, and output vector Fusion features As local features, input to the local branch.

[0018] Preferably, the calculation steps of the multiple self-attention head results of the l-th layer Transformer encoder are specifically as follows: The visual sequence input to the l-th layer Transformer encoder Perform linear transformation to obtain three identical visual sequences, which are called query vector q, key vector k, and value vector v; Attention weight based on the h-th self-attention head , use the softmax function to calculate the similarity between the query vector q and all key vectors k, and multiply each value vector v by its corresponding attention weight to obtain a weighted value vector, and take the sum of all weighted value vectors as the output vector. The calculation formula is: , , ,in, is the linear projection matrix of the h-th self-attention head, W is the linear projection matrix, is the length of the h-th self-attention head, T is the transpose of the matrix, and A is the self-attention weight map; Calculate the results of H self-attention heads in parallel, add them up, and get the final multi-head self-attention result .

[0019] Preferably, the fusion feature The calculation steps are as follows: Get the self-attention weight map of the hth attention head of the lth layer Transformer encoder , , ,in, is the attention value between the i-th visual sequence and the j-th visual sequence, The attention value between the category encoding vector and all visual sequences; Based on self-attention weight map , calculate the attention score standard value matrix of the l-th layer Transformer encoder, and its recursive formula is: , ,in, is the identity matrix, H is the number of attention heads; set up is the standard value of the attention score of the i-th visual sequence and the category encoding vector, and the order of i from 1 to N is Before sorting, filtering The visual sequence with the highest attention score standard value is taken as the important visual sequence of each layer, which is expressed as: ,in, is the i-th important visual sequence in the l-th layer Transformer encoder, is the number of important visual sequences selected; Get the fused features , Represents the features selected by the ARWS module from the L-1th layer.

[0020] In this embodiment, the interpretable fine-grained image recognition method mainly consists of three parts: feature extraction layer, prototype learning layer and fully connected layer. The feature extraction layer uses the ViT structure, because ViT can capture complex patterns and long-distance dependencies in the image through its self-attention mechanism, thereby effectively extracting global features and local detail information, providing a rich information basis for subsequent prototype learning and classification decisions. The feature selection fusion module designed by this method is added to the feature extraction layer, and important features are selected at each layer of ViT, and finally the selected features are fused.

[0021] Specifically, in this embodiment, the feature extraction module adopts the Transformer architecture. First, the image x to be identified is obtained, and its size is , which is the height, width and number of color channels. The image x is divided into N patches of uniform size, each of which is , forming a sequence image. Then, the sequence image is linearly projected, i.e., Patch Embedding operation, to generate a feature sequence , and at the same time add category encoding vectors and position encoding vectors to the feature sequence to form a visual vector, effectively integrating the two-dimensional position information into the one-dimensional feature sequence.

[0022] Specifically, the class token is used as the encoding feature of the model for the input image, which can represent the global features of learning and is used for subsequent global prototype learning. Unlike convolutional neural networks, a position encoding vector (Position Embedding) is also required to add the two-dimensional position information of the image to the one-dimensional sequence, so that the model can use the prior information of the image. Finally, the input image x is encoded into a series of visual tokens.

[0023] The visual vector is then input into a module that stacks L Transformer encoders, each of which contains multiple self-attention heads and multi-layer perceptrons. Among them, multi-head self-attention is the core of the Transformer architecture. In each layer of Transformer encoder, the visual sequence is processed by linear transformation to generate a query vector q, a key vector k, and a value vector v. Based on the self-attention weights, the similarity between q and k is calculated using the softmax function, and v is multiplied by the corresponding attention weight, which can emphasize the information that is more important to the current element, and obtain a weighted value vector. The sum of all weighted value vectors is used as the output vector. This process calculates the results of H self-attention heads in parallel, and then adds them up to obtain the final multi-head self-attention result. This process can be regarded as the degree of attention assigned to each element in the input sequence when considering the information of all other elements.

[0024] In the calculation of fusion features, the self-attention weight map A of the hth attention head of the Transformer encoder of the lth layer is obtained. Based on this weight map, the attention score standard value matrix is ​​calculated and recursively updated until convergence. Then, the visual sequences with the highest attention score standard values ​​of the first M are selected as the important visual sequences of each layer. These sequences are represented as important visual information for subsequent feature fusion. Among them, the feature selection fusion module is called Attention Rollout Weight Selection (ARWS). A new attention score standard attention rollout is introduced in this module. This value is calculated using the attention score provided by ViT, and no additional parameters are introduced, which can better select discriminative local features. Using the ARWS module in each layer of the network can better utilize the low-level, middle-level and high-level information of the network.

[0025] The role of the ARWS module is to select the first K important features from each Transformer Layer. The way to select important features is to introduce the attention rollout value. An important structure used for feature learning in the Transformer architecture is the multi-head attention mechanism. In this way, the features selected by the ARWS module from the L-1 layer are fused to form global features and local features. Global features are identified by category encoding vectors, while local features are identified by fused features. This method not only improves the accuracy of image recognition, but also enhances the model's ability to capture local details of the image, especially when dealing with complex and diverse image data.

[0026] Among them, attention weights are usually considered to correspond to feature importance, but recent studies have shown that attention weights do not necessarily correspond to the importance of input features. Another important assumption is that the original attention lacks identifiability in higher layers. Therefore, the attention rollout value is introduced in the ARWS module to select the most important features in each layer. For the ViT model, the lth (l 1) The attention rollout matrix of the layer is recursively defined as: , The initial attention rollout value The class token is usually used for the final classification and collects the most information in the image. Therefore, a token with a higher attention score than the class token is considered to be more helpful for classification, which means that the token is more discriminative and more important. Similarly, it can be concluded that a token with a higher attention rollout value than the class token is more important.

[0027] See also Figure 5 , S2, the category encoding vector of the output vector is used as the global feature and input into the global branch of the prototype learning layer for calculation to obtain the similarity score between the global feature and the global prototype; Specifically, step S2 includes: predefining A learnable global prototype ,and Learnable local prototypes ; Get the global features output by the feature extraction layer , calculate the global feature The similarity score with each global prototype is calculated as: ,in, is the ith global prototype, Global feature The similarity score with the i-th global prototype, is the smoothing term.

[0028] In this embodiment, the prototype learning layer includes two branches, one is a global branch and the other is a local branch, which are used to learn the global prototype and the local prototype respectively. A learnable global prototype and Learnable local prototypes, which are learned by the model during training and can represent the core features of different categories. These prototypes represent the typical visual features of their respective categories and can eventually be visualized as part of the input image.

[0029] After obtaining the global feature vector from the feature extraction layer, the similarity score between the global feature vector and each predefined global prototype is calculated. This calculation is done through a specific calculation formula to quantify the similarity between the global feature and each global prototype.

[0030] S3, obtaining N local features output by the feature extraction layer, and selecting K preset foreground information related features for local prototype learning, to obtain similarity scores between the local features and the local prototype; Specifically, step S3 includes: extracting the self-attention weight map , and select the first K visual vectors with the highest attention scores; Set a binary mask , the attention score of the i-th position encoding vector to the category encoding vector The softmax function in the formula is modified as follows: , to preserve the vision sequence saved using the foreground save mask, where is the j-th binary mask, is the kth binary mask; The visual sequence saved using the foreground preservation mask is processed with each local prototype to calculate its similarity score, and after maximum pooling, the similarity score of the local prototype is obtained. The calculation formula is: ,in, represents the K selected visual vectors with the highest attention scores, Indicates that each time from the visual vector A single visual vector selected from is the i-th local prototype, is the similarity score between the local feature and the i-th local prototype.

[0031] In this embodiment, since the features related to the foreground information in the local features are the key to classification and can better represent the prototype of a category, in the local branch, first, from the N local features output from the feature extraction layer, K features related to the foreground information are selected to learn the local prototype. In order to avoid introducing new parameters, the module is committed to discovering key local areas from the existing self-attention weight map of the Transformer, so as to select local features that are beneficial to the classification results for learning local prototypes. The ViT network adds a category encoding class token, which aggregates the global features of the image and can be used as a feature representation of the entire image for classification. The importance of the foreground information of the network to network classification is self-evident. Therefore, local features that are more important to the category encoding are selected from the self-attention weights of the category encoding, and the selected features can basically represent the foreground information of the network.

[0032] Specifically, in this embodiment, this step focuses on obtaining N local features from the local features output by the feature extraction layer, and selects K features related to the foreground information for local prototype learning to obtain the similarity score between the local features and the local prototype. This process first involves extracting the self-attention weight map and identifying the top K visual vectors with the highest attention scores, which are considered to be the most representative and can capture the key foreground information in the image. In order to further optimize this selection process, the method sets a binary mask, which is used to modify the softmax function to retain those visual sequences related to the foreground information and with higher attention scores. Specifically, the j-th binary mask and the k-th binary mask are used to adjust the attention score formula so that the model can focus more on the foreground information, thereby improving the accuracy and relevance of local features.

[0033] Next, these visual sequences saved by the foreground preservation mask are used to calculate the similarity scores of each local prototype. This calculation is completed through a specific formula, which involves quantifying the similarity between each visual vector and the local prototype. After the maximum pooling operation, the similarity score of the local prototype is obtained. This step helps to extract the most representative information from multiple local features and enhance the model's ability to recognize local details.

[0034] S4, using a fully connected layer to perform probability calculation processing on the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype to obtain a final classification probability, and based on the classification probability, generate an image recognition result.

[0035] Specifically, step S4 includes: inputting the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype into the fully connected layer respectively, using the softmax function as the activation function for calculation, and obtaining the global branch probability and local branching probability , the calculation formula is: , ,in, , , , These are the parameters of the fully connected layer; For the global branch probability and local branching probability Perform weighted sum processing to obtain the final classification probability, the formula is: ,in, is the weight coefficient of the global branch, is the weight coefficient of the local branch.

[0036] In this embodiment, a fully connected layer is used to perform probability calculation processing on the similarity scores between the global features and the global prototype, and the similarity scores between the local features and the local prototype, to obtain the final classification probability, and generate the image recognition result based on this probability. This step is a key link in the entire image recognition process, which integrates local and global information to improve the accuracy and robustness of recognition.

[0037] Specifically, the similarity scores between global features and global prototypes and the similarity scores between local features and local prototypes are input into the fully connected layer respectively. In the fully connected layer, the softmax function is used as the activation function for calculation to obtain the global branch probability and the local branch probability. This calculation involves the parameters of the fully connected layer, including weights and biases, which are optimized during the training process to best represent the relationship between categories. The softmax function can convert any real-valued vector into a probability distribution, so that the value of each element is between 0 and 1, and the sum of all elements is 1, which makes the softmax function very suitable for multi-classification problems. Next, the global branch probability and the local branch probability are weighted and processed to obtain the final classification probability. This step involves the weighting coefficient of the global branch and the weight coefficient of the local branch , these two coefficients determine the influence of global and local information in the final classification decision. By adjusting these coefficients, the importance of global and local features can be balanced according to the needs of different application scenarios.

[0038] Specifically, in this embodiment, ProtoPNet is used as the basic model for interpretable fine-grained image classification, and the effectiveness of this method is verified by a fine-grained image public dataset. For the convenience of expression, this method records the feature selection fusion module as ARWS and the global branch in the prototype learning layer as GB. The experimental results of using only the ProtoPNet model, the ProtoPNet + GB model, and the final model of this method, ProtoPNet + GB + ARWS, are compared to verify the effectiveness of this method, as shown in Table 1. Among them, the evaluation indicator of model accuracy is accuracy. The proportion of the number of images correctly recognized by the model to all images. The results show that compared with the initial model, the accuracy of this method on the CUB, Dogs and Cars datasets has increased by 5.77%, 5.75%, and 3.12%, respectively.

[0039] Table 1 Recognition accuracy results of each method

[0040] In summary, the interpretable fine-grained image recognition achieves full-process automation from image feature extraction to classification probability calculation through deep learning technology and Transformer architecture. In the feature extraction stage, the input image is first divided into multiple patches and linear projection is performed to generate a feature sequence. Subsequently, the category encoding vector and the position encoding vector are added to the feature sequence to form a visual vector, which is input into multiple stacked Transformer encoder modules to obtain a deep feature representation. Each Transformer encoder contains a multi-head self-attention mechanism and a multi-layer perceptron, which can effectively capture the global and local features of the image. Secondly, in the prototype learning stage, the global branch and the local branch are used to process the global features and the local features respectively, and the key information of the image is further extracted by calculating the similarity scores between these features and the preset global prototype and local prototype. This process uses the self-attention weight map and binary mask technology to ensure that the model focuses on the most important foreground information in the image. Finally, in the classification probability calculation stage, the global and local similarity scores are processed using a fully connected layer, converted into probability distributions through the softmax function, and combined with weighted sum processing to obtain the final classification probability. This approach not only improves classification accuracy, but also enhances the model's ability to recognize local details of images, especially when dealing with complex and diverse image data.

[0041] Compared with the existing technology, the interpretable fine-grained image recognition method has the following advantages: The dual-branch structure is used to improve the training process of the model, and the global prototype and local prototype of the category are learned separately, instead of only generating local prototypes. The two prototypes add higher interpretability to the network, and the global branch can be used to guide the training of the local branch, so that the accuracy of the network is further improved. A feature selection fusion module is added to the feature extraction layer, so that the network selects more important features for prototype learning, which further improves the representation ability of the network, and through this module, the prototype learning is more focused on the foreground rather than the background. A new feature importance evaluation calculation method is used, which is more effective than the traditional method of using attention scores to evaluate feature importance. In short, the interpretable fine-grained image recognition method improves the accuracy and robustness of image recognition, enhances the generalization ability of the model, and enables it to maintain a high recognition accuracy when facing new and unseen data. In addition, the method also improves the flexibility and adaptability of the model because it can balance the importance of global and local features according to the needs of different application scenarios. Finally, this method provides a new solution for the field of image recognition, which has broad application prospects and practical value.

[0042] See also Figure 7 The second embodiment of the present invention provides an interpretable fine-grained image recognition device, which includes: The feature extraction unit 201 is used to obtain an image x to be identified, input the image x into a feature extraction layer for feature extraction preprocessing, and generate an output vector; A global branch unit 202 is used to input the category encoding vector of the output vector as a global feature into the global branch of the prototype learning layer for calculation to obtain a similarity score between the global feature and the global prototype; The local branch unit 203 is used to obtain the local features output by the N feature extraction layers, and select K preset foreground information related features to perform local prototype learning to obtain a similarity score between the local features and the local prototype; The fully connected unit 204 is used to use the fully connected layer to perform probability calculation processing on the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype to obtain the final classification probability, and generate an image recognition result based on the classification probability.

[0043] The third embodiment of the present invention provides an interpretable fine-grained image recognition device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the interpretable fine-grained image recognition method described in any one of the above.

[0044] A fourth embodiment of the present invention provides a readable storage medium storing a computer program, which can be executed by a processor of a device where the storage medium is located to implement the interpretable fine-grained image recognition method as described in any one of the above.

[0045] Exemplarily, the above-mentioned devices and process steps can be implemented by a computer program. The computer program can be divided into one or more units. The one or more units are stored in the memory and executed by the processor to complete the present invention.

[0046] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0047] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the present invention by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0048] Wherein, if the electronic device or printer integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0049] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0050] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An interpretable fine-grained image recognition method, characterized in that: include: Obtain an image x to be identified, input the image x into a feature extraction layer for feature extraction preprocessing, and generate an output vector; The category encoding vector of the output vector is used as the global feature and input into the global branch of the prototype learning layer for calculation to obtain the similarity score between the global feature and the global prototype; Obtaining N local features output by the feature extraction layer, and selecting K preset foreground information related features for local prototype learning, to obtain similarity scores between the local features and the local prototype; A fully connected layer is used to perform probability calculation processing on the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype to obtain a final classification probability, and an image recognition result is generated based on the classification probability.

2. The interpretable fine-grained image recognition method according to claim 1, characterized in that: Get the image x to be identified, input the image x into the feature extraction layer for feature extraction preprocessing, and generate an output vector, specifically: Get the image x to be recognized, , where the size of image x is , H is the image height, W is the image width, C is the number of image channels, is a matrix; Divide the image x into N patches of equal size and tile the patches to a length of Sequence images of , , where the patch size is , is the resolution of each image block, is the number of blocks finally generated; For the sequence of images Perform linear projection processing to generate a feature sequence; Add the category encoding vector to the feature sequence , as the encoding feature of image x; Add a positional encoding vector , connect the two-dimensional position information of image x to the one-dimensional feature sequence to obtain the visual vector , ,in, is the first image sequence, is the second image sequence, is the Nth image sequence, E is the unit matrix, is the product of the first image sequence and the identity matrix, is the product of the second image sequence and the identity matrix, is the product of the Nth image sequence and the identity matrix; The visual vector Input to the module of stacking L Transformer encoders to get the output vector of the lth layer Transformer encoder , , , where each Transformer encoder includes one or more self-attention heads and a multi-layer perceptron, LN represents the normalization operation, MSA represents multi-head self-attention, and MLP represents a multi-layer perceptron. is the output vector of the l-1th layer Transformer encoder; The output vector The category encoding vector of , input to the global branch, and output vector Fusion features As local features, input to the local branch.

3. The interpretable fine-grained image recognition method according to claim 2, characterized in that: The calculation steps of multiple self-attention head results of the l-th layer Transformer encoder are as follows: The visual sequence input to the l-th layer Transformer encoder Perform linear transformation to obtain three identical visual sequences, which are called query vector q, key vector k, and value vector v; Attention weight based on the h-th self-attention head , use the softmax function to calculate the similarity between the query vector q and all key vectors k, and multiply each value vector v by its corresponding attention weight to obtain a weighted value vector, and take the sum of all weighted value vectors as the output vector. The calculation formula is: , , ,in, is the linear projection matrix of the h-th self-attention head, W is the linear projection matrix, is the length of the h-th self-attention head, T is the transpose of the matrix, and A is the self-attention weight map; Calculate the results of H self-attention heads in parallel, add them up, and get the final multi-head self-attention result .

4. The interpretable fine-grained image recognition method according to claim 3, characterized in that: Fusion Features The calculation steps are as follows: Get the self-attention weight map of the hth attention head of the lth layer Transformer encoder , , ,in, is the attention value between the i-th visual sequence and the j-th visual sequence, The attention value between the category encoding vector and all visual sequences; Based on self-attention weight map , calculate the attention score standard value matrix of the l-th layer Transformer encoder, and its recursive formula is: , ,in, is the identity matrix, H is the number of attention heads; set up is the standard value of the attention score of the i-th visual sequence and the category encoding vector, and the order of i from 1 to N is Before sorting, filtering The visual sequence with the highest attention score standard value is taken as the important visual sequence of each layer, which is expressed as: ,in, is the i-th important visual sequence in the l-th layer Transformer encoder, is the number of important visual sequences selected; Get the fused features , Represents the features selected by the ARWS module from the L-1th layer.

5. The interpretable fine-grained image recognition method according to claim 4, characterized in that: The category encoding vector of the output vector is used as the global feature and input into the global branch of the prototype learning layer for calculation to obtain the similarity score between the global feature and the global prototype, which is: Predefined A learnable global prototype ,and Learnable local prototypes ; Get the global features output by the feature extraction layer , calculate the global feature The similarity score with each global prototype is calculated as: ,in, is the ith global prototype, Global feature The similarity score with the i-th global prototype, is the smoothing term.

6. The interpretable fine-grained image recognition method according to claim 5, characterized in that: Obtain N local features output by the feature extraction layer, and select K preset foreground information related features for local prototype learning to obtain the similarity score between the local features and the local prototype, specifically: Extracted from the attention weight map , and select the first K visual vectors with the highest attention scores; Set a binary mask , the attention score of the i-th position encoding vector to the category encoding vector The softmax function in the formula is modified as follows: , to preserve the vision sequence saved using the foreground save mask, where is the j-th binary mask, is the kth binary mask; The visual sequence saved using the foreground preservation mask is processed with each local prototype to calculate its similarity score, and after maximum pooling, the similarity score of the local prototype is obtained. The calculation formula is: ,in, represents the K selected visual vectors with the highest attention scores, Indicates that each time from the visual vector A single visual vector selected from is the i-th local prototype, is the similarity score between the local feature and the i-th local prototype.

7. The interpretable fine-grained image recognition method according to claim 6, characterized in that: The similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype are subjected to probability calculation using a fully connected layer to obtain the final classification probability, specifically: The similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype are respectively input into the fully connected layer, and the softmax function is used as the activation function for calculation to obtain the global branch probability and local branching probability , the calculation formula is: , ,in, , , , These are the parameters of the fully connected layer; For the global branch probability and local branching probability Perform weighted sum processing to obtain the final classification probability, the formula is: ,in, is the weight coefficient of the global branch, is the weight coefficient of the local branch.

8. An interpretable fine-grained image recognition device, characterized in that: include: A feature extraction unit is used to obtain an image x to be identified, input the image x into a feature extraction layer for feature extraction preprocessing, and generate an output vector; The global branch unit is used to input the category encoding vector of the output vector as a global feature into the global branch of the prototype learning layer for calculation to obtain a similarity score between the global feature and the global prototype; A local branch unit, used to obtain the local features output by the N feature extraction layers, and select K preset foreground information related features for local prototype learning to obtain a similarity score between the local features and the local prototype; The fully connected unit is used to use the fully connected layer to perform probability calculation processing on the similarity scores between the global feature and the global prototype and the similarity scores between the local feature and the local prototype to obtain the final classification probability, and generate an image recognition result based on the classification probability.

9. An interpretable fine-grained image recognition device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the interpretable fine-grained image recognition method as described in any one of claims 1 to 7 is implemented.

10. A readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor of the device where the storage medium is located to implement the interpretable fine-grained image recognition method as described in any one of claims 1 to 7.

Images