An Interpretable Fine-Grained Image Classification Method and System Based on Improved Neural Prototype Trees

Abstract

This invention belongs to the field of image processing technology and provides an interpretable fine-grained image classification method and system based on an improved neural prototype tree. The invention designs an interpretable fine-grained image classification model including a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation of the image through the multi-granularity feature extraction layer, generating a depth feature map. The prototype layer calculates the similarity between prototypes and patches based on the depth feature map, finding the patch closest to the prototype. Each prototype is replaced with the potential patch closest to the prototype and visualized as evidence. The soft neural binary decision tree layer trains the prototypes and uses a background prototype removal mechanism to eliminate erroneous prototypes. The selected prototypes are used to make prototype path decisions, resulting in the image classification result. This invention solves the problems of incomplete deep feature extraction, insufficient deep feature mining, and easy classification errors in traditional models.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an interpretable fine-grained image classification method and system based on an improved neural prototype tree. Background Technology

[0002] Fine-grained image classification is a highly challenging research topic in computer vision. In recent years, it has been applied to fields such as medicine, finance, and autonomous driving. However, these fields, while focusing on model accuracy, place even greater emphasis on model interpretability. For example, a medical diagnostic system must be interpretable and transparent to gain the trust of patients and doctors. Therefore, interpretability is crucial for fine-grained image classification tasks. Early applications of ex post-interpretable models in fine-grained image classification included a novel visualization technique introduced by MD Zeiler et al. to help understand the function of intermediate feature layers and the operation of the classifier. R. Selvaraju's Grad-weighted Class Activation Map (Grad-CAM) highlights important regions in the image used for predicting concepts, enabling visual interpretation. Ex post-interpretation provides interpretability for the training process and decisions after training is complete. However, most ex post-interpretable methods are more or less inaccurate, and their biggest drawback is the inability to identify which part of the model provides incorrect explanatory evidence.

[0003] The inventors have observed that in recent years, procedurally interpretable models have been continuously proposed. These models often provide more convincing explanations. For example, Chen et al. proposed ProtoPNet, a prototype part network that analyzes images by finding prototypes and combines evidence from prototypes and image parts to provide classification criteria. This was the first prototype network interpretable framework based on case-based reasoning. While ProtoPNet provides interpretable evidence for classification, its prototype training strategy needs improvement, resulting in an excessive number of prototypes. ProtoTree introduces soft neural decision trees to train prototypes, improving the prototype training strategy and reducing the number of prototypes. However, these models are not comprehensive enough in extracting deep features from images and cannot fully mine deep features. Furthermore, the models learn incorrect background prototypes for prototype path decisions, easily leading to classification errors. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes an interpretable fine-grained image classification method and system based on an improved neural prototype tree. Specifically, it presents an interpretable fine-grained image classification model, MBC-Prototree, which enhances the ability to extract deep image features by employing a multi-granularity feature fusion network. This network can extract deep features at different granularities. Furthermore, a novel background prototype removal mechanism (BPRM) is designed to reduce the impact of erroneous background prototypes on image classification. Finally, to enhance the model's generalization ability, a novel loss function is designed, incorporating both leaf node loss functions and fully connected layer loss functions.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] In a first aspect, the present invention provides an interpretable fine-grained image classification method based on an improved neural prototype tree, comprising:

[0007] Obtain the image to be classified;

[0008] Based on the acquired image to be classified and the preset interpretable fine-grained image classification model, the classification result is obtained;

[0009] The interpretable fine-grained image classification model includes a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation of the image through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, and finds the patch closest to the prototype. Each prototype is replaced with the potential patch closest to the prototype and visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to remove erroneous prototypes. The prototype path decision is made based on the selected prototypes to obtain the image classification result.

[0010] Furthermore, the Inception-v4 and Inception-ResNet-v2 networks are introduced to obtain multi-granularity feature representations of the image. The features are convolved and pooled in parallel by convolution kernels of different sizes. The Res2Net network is used to generate a combination of multiple receptive fields at a fine-grained level.

[0011] Furthermore, using unbiased generalized convolution operations, the Euclidean distance between the prototype and the receptive field is calculated by sliding the prototype over the receptive field; minimum pooling is used to select the patch in the receptive field that is closest to the prototype; and the distance between the patch that is closest to the prototype and the prototype is used to determine the extent to which the prototype exists in the input image.

[0012] Furthermore, the patch closest to the prototype is selected as the prototype projection to approximate the prototype; each prototype is projected onto the nearest latent feature block in the same class as the corresponding prototype.

[0013] Furthermore, the image is segmented into patches with the same shape as the original layer patches. A primary color judgment method is used to select background patches, and the top three colors in each patch are compared for similarity. If the top three colors are similar, the image patch is considered a background patch. Color features are used as clustering vectors for hierarchical clustering of the background patches. Representative patches from each cluster are selected to construct a background patch dataset.

[0014] Furthermore, the soft neural binary decision tree layer includes a set of internal nodes, a set of leaf nodes, and a set of edge nodes; internal nodes represent prototypes, leaf nodes represent predicted classes, and edges represent the routing probabilities from the current layer prototype to the child node prototype; the probability of routing through the child node is calculated by comparing the potential patch of each prototype with the prototype itself, and all nodes are involved by traversing all edges to obtain the predicted class probability; during the training of the soft neural binary decision tree, when each internal node is updated, the similarity score between the image patch corresponding to the updated prototype and the background patch in the background patch dataset is calculated. If the image patch of the updated prototype is similar to a background patch in the background patch dataset, the internal node needs to be updated again.

[0015] Furthermore, the model is trained using the leaf node loss function and the fully connected layer loss function, and time-varying weights are assigned to the leaf node loss function and the fully connected layer loss function respectively. The time-varying weights are then used to optimize the classification decision.

[0016] Secondly, the present invention also provides an interpretable fine-grained image classification system based on an improved neural prototype tree, comprising:

[0017] The image acquisition module is configured to acquire images to be classified.

[0018] The classification module is configured to obtain classification results based on the acquired image to be classified and a preset interpretable fine-grained image classification model.

[0019] The interpretable fine-grained image classification model includes a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, finding the patch closest to the prototype. Each prototype is replaced with the nearest potential patch and visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to eliminate erroneous prototypes. The selected prototypes are used to make prototype path decisions, resulting in image classification results. Prototypes are selected to provide evidence for image classification. The model is trained using leaf node loss functions and fully connected layer loss functions, and time-varying weights are assigned to the leaf node loss function and fully connected layer loss function respectively to optimize image classification decisions and obtain classification results.

[0020] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the interpretable fine-grained image classification method based on the improved neural prototype tree described in the first aspect.

[0021] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the interpretable fine-grained image classification method based on the improved neural prototype tree described in the first aspect.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. This invention designs an interpretable fine-grained image classification model comprising a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation of the image through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, finding the patch closest to the prototype. Each prototype is replaced with the potential patch closest to the prototype, and the result is visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to eliminate incorrect prototypes. The selected prototypes are used to make prototype path decisions, resulting in the image classification result. This solves the problems of incomplete deep feature extraction, insufficient deep feature mining, and easy classification errors in traditional models.

[0024] 2. This invention introduces the Inception-v4 network and the Inception-ResNet-v2 network. The Inception-v4 network performs better in extracting features at a wider level and sets a larger depth feature output, while the Inception-ResNet-v2 performs better in extracting deeper network features and sets a smaller depth feature output. This results in feature outputs of different granularities for extracting multi-granular features, thereby improving the ability to extract image depth features.

[0025] 3. In this invention, the image is segmented into patches with the same shape as the prototype layer patch. A primary color judgment method is used to select background patches, and the top three colors in each patch are compared for similarity. If all three colors are similar, the image patch is considered a background patch. Color features are used as clustering vectors for hierarchical clustering of the background patches. Representative patches from each cluster are selected to construct a background patch dataset. When the internal nodes of the soft neural binary decision tree are updated, the similarity score between the updated prototype image patch and a background patch in the background patch dataset is calculated. If the updated prototype image patch is similar to a background patch in the background patch dataset, the internal nodes need to be updated again. This novel background prototype removal mechanism reduces the impact of erroneous background prototypes on image classification.

[0026] 4. This invention designs a new loss function that includes both a leaf node loss function and a fully connected layer loss function. Time-varying weights are assigned to the leaf node loss function and the fully connected layer loss function respectively. The leaf node loss and the fully connected loss are combined and used together as the objective function in the multi-granularity image classification training process. This can make up for the shortcomings of single loss function optimization, which is prone to bias, and enhance the generalization ability of the model. Attached Figure Description

[0027] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0028] Figure 1 This is a flowchart of the model in Embodiment 1 of the present invention;

[0029] Figure 2 This is a schematic diagram of the Inception module in Embodiment 1 of the present invention;

[0030] Figure 3 This is a schematic diagram of the kernel module of Res2Net in Embodiment 1 of the present invention;

[0031] Figure 4 This is the generalized convolution of Embodiment 1 of the present invention;

[0032] Figure 5 A prototype visualization of Embodiment 1 of the present invention;

[0033] Figure 6 This is a background patch clustering principle diagram of Embodiment 1 of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0036] Example 1:

[0037] This embodiment provides an interpretable fine-grained image classification method based on an improved neural prototype tree, including:

[0038] Obtain the image to be classified;

[0039] Based on the acquired image to be classified and the preset interpretable fine-grained image classification model, the classification result is obtained;

[0040] The interpretable fine-grained image classification model includes a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation of the image through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, and finds the patch closest to the prototype. Each prototype is replaced with the potential patch closest to the prototype and visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to remove erroneous prototypes. The prototype path decision is made based on the selected prototypes to obtain the image classification result.

[0041] Specifically, the image to be classified obtains a feature representation of the image through the multi-granularity feature extraction layer, generating a deep feature map; the prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, and finds the patch closest to the prototype; each prototype is replaced with the potential patch closest to the prototype and visualized as evidence; the soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to remove erroneous prototypes; the prototype path decision is made based on the selected prototypes to obtain the image classification result; this solves the problems of incomplete deep feature extraction, insufficient deep feature mining, and easy classification errors in traditional models.

[0042] like Figure 1As shown, the fine-grained image classification model in this embodiment can be explained by including three layers: a multi-grained feature extraction network layer, a prototype layer, and a soft neural binary decision tree layer.

[0043] First, the image is processed by the multi-granularity feature extraction layer to obtain a feature representation of the image, generating a deep feature map. Then, the deep feature map generated by the multi-granularity feature extraction layer is input into the prototype layer, which calculates the similarity between the prototype and the patch, finds the patch closest to the prototype, and provides evidence for image classification. Next, the soft neural binary decision tree layer trains the prototype and makes prototype path decisions. Here, a background prototype removal mechanism is added to optimize the prototype path decision. At the same time, a fully connected layer loss function and a leaf node loss function are combined to improve the generalization ability of the model during training. Finally, the combination of the multi-granularity feature extraction layer, the prototype layer, and the soft neural binary decision tree layer fuses the network into a probabilistic prediction.

[0044] Specific implementation process:

[0045] First, the input image enters the multi-granularity feature extraction layer f(x; ω), resulting in D two-dimensional (H×W) feature maps, where ω represents the trainable parameters of f. The transformation is as follows:

[0046] f(x;ω)--->R H×W×D (1)

[0047] Traditional convolutional networks neglect the importance of parallel processing of image features. To address this issue, this embodiment introduces the Inception-v4 and Inception-ResNet-v2 networks. The Inception modules unique to Inception-v4 and Inception-ResNet-v2 networks are as follows: Figure 2 As shown.

[0048] Features processed by the Inception module can undergo parallel convolution and pooling operations with kernels of different sizes. The Inception-v4 network performs better at extracting features at wider levels by setting larger depth feature outputs, while Inception-ResNet-v2 performs better at extracting deeper network features by setting smaller depth feature outputs. By introducing Inception-v4 and Inception-ResNet-v2, deeper and wider depth features can be extracted, resulting in better feature extraction performance.

[0049] The Res2Net network can generate multiple receptive fields at a fine-grained level without increasing the number of layers, thus improving the effective receptive field and avoiding redundant information. The kernel module structure of Res2Net is as follows: Figure 3 As shown. After a 1×1 convolution, the feature map is divided into four parts. The first part is simple: x1 is not processed and is directly passed to y1. In the second part, x2 is divided into two lines after a 3×3 convolution: one continues forward to y2, and the other is passed to x3, so the third line receives information from the second line. In the third part, x3 is divided into two lines after a 3×3 convolution: one continues forward to y3, and the other is passed to x4, so the fourth line receives information from the third line. In the fourth part, x4 is convolved with a 3×3 layer and then passed to y4.

[0050] The image is processed through three different multi-granularity feature extraction layers, which divide the image into depth feature blocks of different sizes, allowing for the extraction of features at different granularities. During computation, the prototype can obtain receptive fields of different sizes. This is used for subsequent comparison of the prototype with depth features and for prototype visualization. The multi-granularity feature extraction layers compensate for the limitations of the single receptive field of the neural prototype tree.

[0051] Then, the image is processed by the multi-granularity feature extraction layer to generate a feature map, which is then input into the prototype layer. The prototype layer finds the patch closest to the prototype, providing evidence for image classification. The prototype is defined as a trainable tensor of shape (H1×W1×D), where H1≤H and W1≤W. Furthermore, an unbiased generalized convolutional form can be used, where each prototype p... n ∈P acts as the kernel, and is computed by sliding over the receptive field z of shape (H×W×D). n The Euclidean distance between the current receiving domain z and its current receiving domain z is called a patch, such as... Figure 4 As shown.

[0052] Mini-pooling is applied to select the z closest to the prototype p. n The patch has a shape of (H1×W1×D), and the calculation formula is as follows:

[0053]

[0054] Recent potential patches and prototype p n The distance between them determines the extent to which the prototype exists in the input image.

[0055] After selecting prototypes, prototype visualization is performed. To facilitate visualization of decision interpretability, a projection operation is performed on the prototypes. Similar to ProtoPNet, the closest patch to the prototype is selected as the prototype projection to approximate the prototype, achieving the purpose of prototype visualization. Each prototype is projected onto the nearest latent feature block in the same class as that prototype, allowing us to view any prototype that contributes to image classification decisions. The following updates have been performed:

[0056]

[0057] In this embodiment, using Indicates the most recent patch The corresponding training image, prototype p n It can be visualized as The patch. A 2D similarity map is created using f, containing p. n and Similarity score of all patches in the data.

[0058]

[0059] Where (i, j) represents the pattern(z) Position; s n Representing similarity graphs using bicubic interpolation pairs Upsample the input shape; p n Visualized as The rectangular block, in the same location as the most recent potential patch; prototype visualization as... Figure 5 As shown.

[0060] Finally, the output of the prototype layer is used as the input of the soft neural binary decision tree layer. The soft neural binary decision tree layer consists of a set of internal nodes N, a set of leaf nodes L, and a set of edge nodes E. Internal nodes represent tensor prototypes, leaf nodes represent predicted classes, and edges represent the routing probabilities from the current layer prototype to its child node prototypes. In the soft neural binary decision tree, the number of prototypes to be learned, i.e., |p|, depends on the size of the tree. A binary tree structure is initialized by defining a maximum height h, which will create 2 n 2 leaves and 2 n -1 prototype. Therefore, the computational complexity of learning prototype P increases exponentially with the height h of the tree.

[0061] Some ProtoTree prototypes appear to focus on the background, meaning that ProtoTree might exploit learned biases due to the prototype design's tendency to make incorrect prototype path decisions. For example, ProtoTree might use green leaves to distinguish between a grey owl and a black tern. Therefore, this embodiment designs a novel Background Prototype Removal Mechanism (BPRM) to reduce the impact of erroneous prototype selection. First, the images in the dataset are segmented into patches of the same size as the shape (H×W) at the prototype layer, and a dominant color judgment method is used to select background color patches. Since the background patch color is usually relatively uniform, the top three colors in the image patch are selected, and their similarity is compared. If the top three colors are similar, then the image patch can be considered a background patch. Next, hierarchical clustering is performed on the image patches. Specifically, color features from the R, G, and B color channels are used as clustering vectors, and a multidimensional histogram is calculated. This embodiment sets a distance threshold to control the clustering process. Then, the image patch closest to the center of each cluster is selected as the representative patch for that cluster, and these are used to construct the background patch dataset. The principle diagram of the clustering process is as follows: Figure 6 As shown.

[0062] Background patch clustering groups background patches of the same type together, such as... Figure 6 As shown, gray lake patches are clustered together, green grass patches are clustered together, and blue sky patches are clustered together.

[0063] During prototype tree training, when each internal node is updated, the similarity between the corresponding image patches is calculated to update the prototype and background patch datasets. If the patch of the updated prototype is highly similar to the background patch, it means that the learned prototype is an incorrect prototype, and the internal nodes of the tree should be updated again.

[0064] In the soft neural binary decision tree, node n uses soft routing, and the fuzzy weights for routing z to its two child nodes are set within [0, 1], and a probabilistic interpretation is given to them. Based on this probabilistic terminology, the following definition is provided: and p n The similarity between them, therefore the probability of sample z being routed through the internal nodes of the right edge is as follows:

[0065]

[0066] Where z represents the depth feature map; p n This represents the prototype. The expression indicates the degree to which sample z reaches the right leaf node of node n.

[0067] Therefore, the probability of sample z passing through the left edge can be derived from the routing probability of sample z through the internal nodes of the right edge as follows:

[0068] p e(n,n.left) (z)=1-p e(n,n.right) (z) (6)

[0069] Sample z traverses all edges, reaching each leaf node l∈L with a certain probability. Path p e Let represent the edge sequence from the root node to the leaf node l. The probability that sample z reaches a leaf node is denoted by π. l , is path p e The product of the probabilities of each side. The probability function formula for sample z reaching leaf l is as follows:

[0070]

[0071] Each leaf node l∈L carries a trainable parameter k. l , representing the distribution of the k classes that need to be learned in this leaf. The softmax function. For k l Normalization is performed to obtain the final predicted class probability distribution of the input image x. The latent representation z = f(x; ω) is traversed through all edges of the tree, such that all leaves are related to the final prediction. Contributed. Obtained the class probability distribution of leaf l. Predicted class probability distribution. yes:

[0072]

[0073] Simultaneously, it is necessary to learn the contribution of leaf nodes and their distribution; this is a global learning problem. k is learned through backpropagation along with ω and P. The update function for the leaf node parameter k is:

[0074]

[0075] Where t represents a training period; ⊙ represents element-wise multiplication; This represents a division between elements; the result is a vector of size K, representing the class distribution in leaf l.

[0076] Model Training:

[0077] Before model training, pre-trained Inception-v4, Inception-ResNet-v2, and Res2Net models in the multi-granularity feature extraction layer are required. During training, these three networks process the image in parallel, extracting multi-granularity feature patches of different sizes. Next, the prototype layer finds the patch closest to the prototype, providing evidence for image classification. Finally, a soft neural binary decision tree layer is used for prototype path decision-making to predict the class probability distribution of the input image. During training, the prototype p is learned. n ∈P, the leaf class distribution c and CNN parameters ω.

[0078] To improve the model's generalization ability, both leaf node loss functions and fully connected layer loss functions are used for model training, and time-varying weights are used for classification decisions. Different time-varying weights are assigned to the fully connected layer loss and the leaf node loss, respectively. Through continuous experimentation, appropriate time-varying weight parameters are assigned to the fully connected layer loss and the leaf node loss, thereby achieving higher image classification accuracy. The model's loss function is as follows:

[0079]

[0080] Where, α t The time-varying weights representing the loss of the leaf nodes of the tree; β t The time-varying weights represent the fully connected loss; k represents the leaf node parameters; t represents the round number. Represents the loss of the leaf nodes; CrsEnt(y full ,y label ) represents the loss of the fully connected layer.

[0081] Finally, leveraging the architectural advantages of different complementary networks, and based on the image processing results from the three feature extraction networks, a fusion probability prediction score is calculated. This fusion strategy effectively avoids the limitation of inaccurate prototype selection by a single network, making the prototypes selected through training more accurate. The formula for the fusion probability prediction score is as follows:

[0082]

[0083] in, denoted by , n represents the probability distribution of the predicted classes of the three feature extraction networks; n represents the number of backbone networks; and x represents the prediction score of a single network.

[0084] In the experiments, MBC-Prototree was applied to two authoritative fine-grained datasets, CUB-200 and FGVC-aircraft, and a medical dataset, chest x-ray. Subsequent experiments divided these three datasets into training, testing, and validation sets, and compared them with a series of baseline models to validate the model's effectiveness.

[0085] The model in this embodiment was compared with other baseline models on datasets CUB-200, FGVC-aircraft, and chest x-ray, and the experimental results showed a significant improvement.

[0086] Example 2:

[0087] This embodiment provides an interpretable fine-grained image classification system based on an improved neural prototype tree, including:

[0088] The image acquisition module is configured to acquire images to be classified.

[0089] The classification module is configured to obtain classification results based on the acquired image to be classified and a preset interpretable fine-grained image classification model.

[0090] The interpretable fine-grained image classification model includes a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation of the image through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, and finds the patch closest to the prototype. Each prototype is replaced with the potential patch closest to the prototype and visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to remove erroneous prototypes. The prototype path decision is made based on the selected prototypes to obtain the image classification result.

[0091] The working method of the system is the same as that of the interpretable fine-grained image classification method based on the improved neural prototype tree in Example 1, and will not be repeated here.

[0092] Example 3:

[0093] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the interpretable fine-grained image classification method based on an improved neural prototype tree as described in Embodiment 1.

[0094] Example 4:

[0095] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the interpretable fine-grained image classification method based on an improved neural prototype tree as described in Embodiment 1.

[0096] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. An interpretable fine-grained image classification method based on an improved neural prototype tree, characterized in that, include: Obtain the image to be classified; Based on the acquired image to be classified and the pre-defined interpretable fine-grained image classification model, the classification result is obtained; the Inception-v4 network and the Inception-ResNet-v2 network are introduced to obtain multi-granular feature representations of the image, and the features are convolved and pooled in parallel by convolution kernels of different sizes; the Res2Net network is used to generate a combination of multiple receptive fields at a fine-grained level. The interpretable fine-grained image classification model includes a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, finding the patch closest to the prototype. Each prototype is replaced with the nearest potential patch and visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to eliminate erroneous prototypes. Based on the selected prototypes, a prototype path decision is made to obtain the image classification result. The image is segmented into patches with the same shape as the original layer patches. The primary color judgment method is used to select background patches. The top three colors in the patches are selected for similarity comparison. If the top three colors are similar, the image patch is regarded as a background patch. The background patches are clustered hierarchically using color features as cluster vectors. Representative patches from each cluster are selected to construct the background patch dataset. The soft neural binary decision tree layer consists of a set of internal nodes, a set of leaf nodes, and a set of edge nodes. Internal nodes represent prototypes, leaf nodes represent predicted classes, and edges represent the routing probabilities from the current prototype to its child prototypes. The probability of a route passing through a child node is calculated by comparing the potential patch of each prototype with the prototype itself. By traversing all edges to involve all nodes, the predicted class probability is obtained. During the training of the soft neural binary decision tree, when each internal node is updated, the similarity score between the image patch corresponding to the updated prototype and the background patch in the background patch dataset is calculated. If the image patch of the updated prototype is similar to a background patch in the background patch dataset, the internal node needs to be updated again.

2. The interpretable fine-grained image classification method based on improved neural prototype trees as described in claim 1, characterized in that, Using unbiased generalized convolution operations, the Euclidean distance between the prototype and the receptive field is calculated by sliding the prototype over the receptive field; minimum pooling is used to select the patch in the receptive field that is closest to the prototype; and the distance between the patch that is closest to the prototype and the prototype determines the extent to which the prototype exists in the input image.

3. The interpretable fine-grained image classification method based on an improved neural prototype tree as described in claim 2, characterized in that, The patch closest to the prototype is selected as the prototype projection to approximate the prototype; each prototype is projected onto the nearest latent feature block in the same class as the corresponding prototype.

4. The interpretable fine-grained image classification method based on improved neural prototype trees as described in claim 1, characterized in that, The model is trained using the leaf node loss function and the fully connected layer loss function, and time-varying weights are assigned to the leaf node loss function and the fully connected layer loss function respectively. The time-varying weights are then used to optimize the classification decision.

5. An interpretable fine-grained image classification system based on an improved neural prototype tree, characterized in that, include: The image acquisition module is configured to acquire images to be classified. The classification module is configured to: obtain classification results based on the acquired image to be classified and a preset interpretable fine-grained image classification model; introduce Inception-v4 and Inception-ResNet-v2 networks to obtain multi-granular feature representations of the image, and perform parallel convolution and pooling operations on the features using convolution kernels of different sizes; and utilize the Res2Net network to generate a combination of multiple receptive fields at a fine-grained level. The interpretable fine-grained image classification model includes a multi-granularity feature extraction layer, a prototype layer, and a soft neural binary decision tree layer. The image to be classified obtains a feature representation through the multi-granularity feature extraction layer, generating a deep feature map. The prototype layer calculates the similarity between the prototype and the patch based on the deep feature map, finding the patch closest to the prototype. Each prototype is replaced with the nearest potential patch and visualized as evidence. The soft neural binary decision tree layer trains the prototype and uses a background prototype removal mechanism to eliminate erroneous prototypes. Based on the selected prototypes, a prototype path decision is made to obtain the image classification result. The image is segmented into patches with the same shape as the original layer patches. The primary color judgment method is used to select background patches. The top three colors in the patches are selected for similarity comparison. If the top three colors are similar, the image patch is regarded as a background patch. The background patches are clustered hierarchically using color features as cluster vectors. Representative patches from each cluster are selected to construct the background patch dataset. The soft neural binary decision tree layer consists of a set of internal nodes, a set of leaf nodes, and a set of edge nodes. Internal nodes represent prototypes, leaf nodes represent predicted classes, and edges represent the routing probabilities from the current prototype to its child prototypes. The probability of a route passing through a child node is calculated by comparing the potential patch of each prototype with the prototype itself. By traversing all edges to involve all nodes, the predicted class probability is obtained. During the training of the soft neural binary decision tree, when each internal node is updated, the similarity score between the image patch corresponding to the updated prototype and the background patch in the background patch dataset is calculated. If the image patch of the updated prototype is similar to a background patch in the background patch dataset, the internal node needs to be updated again.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps of the interpretable fine-grained image classification method based on an improved neural prototype tree as described in any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the interpretable fine-grained image classification method based on an improved neural prototype tree as described in any one of claims 1-4.

Images