Fine-grained image recognition and classification method and system, storage medium and equipment

Abstract

The invention discloses a fine-grained image recognition and classification method and system, a storage medium and equipment, and the method comprises the steps: obtaining a fine-grained image, constructing an image data set, and importing the image data set into a visual model; importing each stage feature map into a feature pyramid network module to obtain a first target feature map fusing shallow layer information; importing the first target feature map into a weak supervision sampling module, screening out target features, and stacking a target vector and a second target feature map based on the target features; importing the second target feature map and the initial prediction value which is not processed by the activated function into a neural decision forest module, respectively obtaining a target prediction value corresponding to each granularity, and storing the target prediction value into an output sequence; and importing the output sequence into a credible multi-view classification module, and fusing predicted value distribution of a plurality of opinions to obtain a final fused classification result. According to the invention, the recognition and classification precision of the fine-grained image is improved, so that the fine-grained image classification is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a fine-grained image recognition classification method, system, storage medium and device. Background Art

[0002] Fine-grained image classification (FGVC) is a challenging task, whose main goal is to discover and classify objects based on discriminative regions and features of the image. This task is more difficult when distinguishing classes that are very similar in appearance. Since most images have high inter-class similarity and low intra-class differences, as well as inconsistencies such as environment and posture, both humans and machines face recognition bottlenecks.

[0003] In the past, convolutional neural networks (CNNs) have achieved remarkable results in this field. However, with the introduction of the Visual Transformer (ViT) model, ViT has surpassed CNN in multiple visual tasks and has become a new frontier technology. In recent years, the application of ViT in FGVC tasks has gradually increased. For example, some models use attention mechanisms to extract local features of images and optimize their discrimination capabilities. However, whether it is a CNN or ViT model, its essence is based on a black-box learning mechanism, and its model has the problem of interpretability.

[0004] In fine-grained image classification tasks, the model needs to identify the image category from tiny visual differences. This type of task is highly challenging because in fine-grained images, the visual features of different categories often differ slightly, and the presence of background and irrelevant areas may also interfere with the model's focus on key parts, affecting the classification effect. In order to improve the performance of the model in these tasks, the key is to help the model find the most discriminative parts or "key patches" in the image. The current mainstream methods include methods based on part positioning, attention, and feature channels. These methods each optimize the model's ability to focus on discriminative areas in different aspects, and can usually be used in combination to improve classification results, but these methods are usually not explainable, and people do not know why the model focuses on the so-called discriminative areas.

[0005] The fusion methods in fine-grained image classification include feature-level fusion and decision-level fusion. Feature-level fusion is to integrate the features of different layers or different models during the feature extraction process, such as splicing, weighting or fusion through attention mechanism, so that the model can better capture local details and global semantic information, and improve the ability to recognize subtle differences. Decision-level fusion is to integrate the results of multiple models or multiple inferences in the classification decision stage, and obtain the final prediction result through voting, averaging or weighting, so as to improve the robustness and classification accuracy of the overall system. For decision-level fusion, direct fusion is too simple. If the appropriate fusion weights or strategies are not found, it may lead to unsatisfactory fusion effect and even reduce classification accuracy, especially in fine-grained classification tasks, and the fusion method lacks interpretability. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a fine-grained image recognition and classification method, system, storage medium and device, aiming to improve the recognition and classification accuracy of fine-grained images so as to make the fine-grained image classification more accurate.

[0007] A first aspect of the present invention is to provide a fine-grained image recognition and classification method, the method comprising: Obtain an image dataset corresponding to a pre-collected fine-grained image, and import the image dataset into a Transformer-based visual model; Importing each stage feature map into the feature pyramid network module of the visual model to obtain a first target feature map that integrates shallow information; Importing the first target feature map into the weakly supervised sampling module of the visual model, screening out target features that are useful for classification, and stacking them into a target vector and a second target feature map based on the target features; Importing the second target feature map and the initial prediction value that has not been processed by the activation function into the neural decision forest module of the visual model, respectively obtaining the target prediction value corresponding to each granularity and storing them in the output sequence; The output sequence is imported into the credible multi-view classification module of the visual model, and the predicted value distributions of multiple opinions are fused to obtain a final fused classification result.

[0008] According to one aspect of the above technical solution, the step of importing each stage feature map into the feature pyramid network module of the visual model to obtain a first target feature map that integrates shallow information includes: Importing the feature map of each stage in the image data set into the feature pyramid network module of the visual model; The deep features containing semantic information and the shallow features containing spatial information are fused layer by layer through the feature pyramid network module to obtain a first target feature map.

[0009] According to one aspect of the above technical solution, the first target feature map is imported into the weakly supervised sampling module of the visual model, target features that are beneficial to classification are screened out, and the step of stacking the target features into a target vector and a second target feature map includes: Importing the first target feature map into the weakly supervised sampling module of the visual model, and performing category prediction on the pixel values, i.e., feature points, of the first target feature map through the fully connected layer in the weakly supervised sampling module; Performing multi-classification probability conversion processing on the prediction results, if the highest prediction probability of the feature point is greater than a preset threshold, determining the feature point as a target feature that is helpful for classification; Feature stacking is performed based on the target features, and a target vector and a second target feature map are output.

[0010] According to one aspect of the above technical solution, the method further includes: If the highest prediction probability of the feature point is less than the preset threshold, it is regarded as an invalid feature point whose contribution to fine-grained classification is less than a preset value, and the invalid feature point is filtered out.

[0011] According to one aspect of the above technical solution, the step of importing the second target feature map and the initial prediction value not processed by the activation function into the neural decision forest module of the visual model, obtaining the target prediction value corresponding to each granularity and storing it in the output sequence includes: The neural decision forest module is composed of multiple neural decision trees, each of which is composed of multiple nodes, and each node hierarchically routes input features through a hierarchical structure so that the visual model captures fine-grained information in the second target feature map.

[0012] According to one aspect of the above technical solution, the step of importing the output sequence into the trusted multi-view classification module of the visual model, fusing the predicted value distributions of multiple opinions, and obtaining the final fused classification result includes: importing the output sequence into a trusted multi-view classification module of the vision model; In the credible multi-view classification module, the category probability distribution is modeled by variational Dirichlet distribution, and Dempster-Shafer is used to integrate evidence to fuse the predicted value distributions of multiple opinions to obtain the final fused classification result.

[0013] A second aspect of the present invention is to provide a fine-grained image recognition and classification system, which is applied to the method described in the above technical solution, and the system comprises: An image data import module, used to obtain an image data set corresponding to a pre-collected fine-grained image, and import the image data set into a Transformer-based visual model; A shallow information fusion module, used for importing the feature map of each stage into the feature pyramid network module of the visual model to obtain a first target feature map that fuses the shallow information; A pixel feature screening module, used for importing the first target feature map into the weakly supervised sampling module of the visual model, screening out target features that are useful for classification, and stacking the target features into a target vector and a second target feature map based on the target features; A decision module, used for importing the second target feature map and the initial prediction value that has not been processed by the activation function into the neural decision forest module of the visual model, respectively obtaining the target prediction value corresponding to each granularity and storing it in an output sequence; The classification module is used to import the output sequence into the trusted multi-view classification module of the visual model, fuse the predicted value distributions of multiple opinions, and obtain a final fused classification result.

[0014] According to one aspect of the above technical solution, the shallow information fusion module is specifically used for: Importing the feature map of each stage in the image data set into the feature pyramid network module of the visual model; The deep features containing semantic information and the shallow features containing spatial information are fused layer by layer through the feature pyramid network module to obtain a first target feature map.

[0015] A third aspect of the present invention is to provide a readable storage medium having a computer program stored thereon, which implements the method described in the above technical solution when executed by a processor.

[0016] The fourth aspect of the present invention is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the above technical solution when executing the computer program.

[0017] Compared with the prior art, the fine-grained image recognition and classification method, system, storage medium and device shown in the present invention have the following beneficial effects: The present invention proposes a fine-grained image recognition and classification method for fine-grained image classification. While taking into account the interpretability of the model, multi-granularity information is used to improve the training efficiency of the visual model and guide the model to find discriminative areas faster, thereby improving the accuracy of the model. A transformer-based visual model is used as a feature extractor, and feature maps of multi-granularity information with different receptive fields output by different levels are extracted. After the feature pyramid, each layer of granularity feature map is finally screened for key features, and the multi-granularity information containing key features that are beneficial to classification is sent to an interpretable prototype decision tree forest. Finally, the decision forest uses an interpretable dynamic decision fusion algorithm to form a prediction result with predictive robustness to form the final classification, thereby improving the recognition and classification accuracy of fine-grained images and making the classification more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 Schematic diagram of the process of a fine-grained image recognition and classification method in one embodiment of the present invention; Figure 2 4 is a structural block diagram of a fine-grained image recognition and classification system in one embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. Several embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0020] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0022] Embodiment 1 See also Figure 1 The first embodiment of the present invention provides a fine-grained image recognition and classification method, the method comprising steps S10 to S50: Step S10, obtaining an image dataset corresponding to the pre-collected fine-grained image, and importing the image dataset into a Transformer-based visual model.

[0023] First of all, it should be explained that fine-grained images refer to images with subtle visual differences. These differences are mainly reflected in different subcategories under the same category. Since the visual differences between subcategories are very small, fine-grained image classification is difficult and requires high-resolution images and complex algorithms to achieve accurate classification. For example, different species of birds, car models, dog breeds, etc.

[0024] In this embodiment, fine-grained images of the target object are first collected, and then an image dataset is constructed based on the fine-grained images collected at different time stages. The image dataset is imported into a Transformer-based visual model, namely the Swin Transformer model.

[0025] Among them, the above-mentioned visual model includes a feature pyramid network module, a weakly supervised sampling module, a neural decision forest module and a trusted multi-view classification module.

[0026] Step S20, importing each stage feature map into the feature pyramid network module of the visual model to obtain a first target feature map that integrates shallow information.

[0027] In this embodiment, the step of importing each stage feature map into the feature pyramid network module of the visual model to obtain a first target feature map integrating shallow information includes: Importing the feature map of each stage in the image data set into the feature pyramid network module of the visual model; The deep features containing semantic information and the shallow features containing spatial information are fused layer by layer through the feature pyramid network module to obtain a first target feature map.

[0028] First of all, it should be explained that the Transformer-based visual model has four stages, which are equivalent to four feature extraction steps, which are used to extract features from the image. The feature map extracted by each stage is sent to the feature pyramid for processing, and the multi-level feature map information is fused to obtain the first target feature map that integrates the shallow information.

[0029] Among them, the feature pyramid network module, namely the FPN module (FPN), is a module used for multi-scale feature fusion and is often used in image processing tasks to enhance the model's ability to recognize objects of different scales.

[0030] Specifically, the FPN module constructs a bottom-up feature pyramid to fuse deep features including semantic information with shallow features including spatial information layer by layer.

[0031] In the specific implementation, the FPN module extracts feature maps from different levels of the backbone network of the visual model, first performs dimensionality reduction projection so that the feature maps corresponding to different levels have the same feature dimension, then aligns the feature maps of different resolutions through upsampling operation, and adds features layer by layer to obtain the first target feature map that integrates shallow information.

[0032] The feature pyramid network module shown in this embodiment can effectively integrate multi-scale information in the image, so that the visual model can retain key global semantic information and capture subtle local features when processing fine-grained tasks, thereby improving the accuracy and robustness of overall classification or detection.

[0033] Step S30, importing the first target feature map into the weakly supervised sampling module of the visual model, screening out target features that are useful for classification, and stacking them into a target vector and a second target feature map based on the target features.

[0034] In this embodiment, the first target feature map is imported into the weakly supervised sampling module of the visual model, target features that are useful for classification are screened out, and the step of stacking the target features into a target vector and a second target feature map includes: Importing the first target feature map into the weakly supervised sampling module of the visual model, and performing category prediction on the pixel values, i.e., feature points, of the first target feature map through the fully connected layer in the weakly supervised sampling module; Performing multi-classification probability conversion processing on the prediction results, if the highest prediction probability of the feature point is greater than a preset threshold, determining the feature point as a target feature that is helpful for classification; Feature stacking is performed based on the target features, and a target vector and a second target feature map are output.

[0035] The method further comprises: If the highest prediction probability of the feature point is less than the preset threshold, it is regarded as an invalid feature point whose contribution to fine-grained classification is less than a preset value, and the invalid feature point is filtered out.

[0036] In this embodiment, the first target feature map is imported into the weakly supervised sampling module of the visual model, and target features that are beneficial to classification are screened out. The step of stacking the target features into a target vector and a second target feature map is specifically implemented by a weakly supervised sampling module (WSS). In the WSS module, the pixel values ​​of the first target feature map are called feature points. The WSS module adopts a simple and efficient design, and performs category prediction on each feature point through a fully connected layer.

[0037] Specifically, after the prediction results are subjected to multi-classification probability conversion processing (softmax processing), if the highest prediction probability of a feature point exceeds a preset threshold, the feature point is considered to be an effective feature that helps with classification, that is, the above-mentioned target feature, which will be retained and used in subsequent feature fusion steps.

[0038] On the contrary, if the highest prediction probability of a feature point does not reach the preset threshold, it is regarded as a feature point that contributes less to the fine-grained classification, and is determined to be an invalid feature point, and then filtered out.

[0039] By adopting the above-mentioned threshold-based feature point screening method, the WSS module can automatically screen out the most discriminative feature points for fine-grained image classification, effectively reducing the computational burden of the visual model and helping the visual model to better focus on the discriminative key parts in the fine-grained image, thereby improving the classification accuracy.

[0040] Step S40, importing the second target feature map and the initial prediction value that has not been processed by the activation function into the neural decision forest module of the visual model, respectively obtaining the target prediction value corresponding to each granularity and storing them in the output sequence.

[0041] In this embodiment, a neural decision forest (Neural Decision Forest) composed of multiple neural decision trees is first constructed to enhance the overall robustness of the model. Each neural decision tree consists of multiple nodes, and the nodes hierarchically route the input features through a hierarchical structure, so that the visual model can capture the fine-grained information in the second target feature map. Each neural decision tree in the neural decision forest independently learns different feature distributions and discrimination rules, thereby improving the model's overall adaptability to complex data and classification accuracy. By integrating the decisions of multiple neural decision trees, the neural decision forest can effectively balance the bias and variance of a single tree model, improve the resistance to noise and sample bias, and ultimately achieve robust and accurate classification of the input image.

[0042] Step S50, importing the output sequence into the credible multi-view classification module of the visual model, fusing the predicted value distributions of multiple opinions, and obtaining a final fused classification result.

[0043] In this embodiment, the step of importing the output sequence into the trusted multi-view classification module of the visual model, fusing the predicted value distributions of multiple opinions, and obtaining the final fused classification result includes: importing the output sequence into a trusted multi-view classification module of the vision model; In the credible multi-view classification module, the category probability distribution is modeled by variational Dirichlet distribution, and Dempster-Shafer is used to integrate evidence to fuse the predicted value distributions of multiple opinions to obtain the final fused classification result.

[0044] Among them, in this embodiment, a trusted multi-view classification module (Trusted Multi-View Classification, TMC) is used to effectively integrate multi-granularity information and improve the reliability and credibility of decision-making.

[0045] In this embodiment, unlike the traditional feature or output level fusion method, the TMC module fuses the evidence information from each perspective at the evidence level, thereby achieving stable multi-view integration with more reasonable uncertainty estimation. In this embodiment, the category probability distribution is modeled by variational Dirichlet distribution, and the Dempster-Shafer theory is used for evidence integration to ensure that the multi-view information is optimized in terms of interpretability and credibility.

[0046] Specifically, the TMC module combines the uncertainty of each view and adaptively allocates view weights during the visual model training process, making the fused decision more reliable, more robust, and theoretically guaranteed. In addition, the TMC module not only does not require additional calculations or network changes, but also provides evidence-based integration for classification of each granularity through subjective uncertainty, thereby significantly improving classification accuracy and interpretability.

[0047] In summary, compared with the prior art, the fine-grained image recognition and classification method shown in this embodiment has the following beneficial effects: This embodiment proposes a fine-grained image recognition and classification method for fine-grained image classification. While taking into account the interpretability of the model, multi-granularity information is used to improve the training efficiency of the visual model and guide the model to find discriminative areas faster, thereby improving the accuracy of the model. A transformer-based visual model is used as a feature extractor, and feature maps of multi-granularity information with different receptive fields output by different levels are extracted. After the feature pyramid, each layer of granularity feature map is finally screened for key features, and the multi-granularity information containing key features that are beneficial to classification is sent to an interpretable prototype decision tree forest. Finally, the decision forest uses an interpretable dynamic decision fusion algorithm to form a prediction result with predictive robustness to form the final classification, thereby improving the recognition and classification accuracy of fine-grained images and making the classification more accurate.

[0048] Embodiment 2 See also Figure 2 The second embodiment of the present invention provides a fine-grained image recognition and classification system, which includes: an image data import module 10, a shallow information fusion module 20, a pixel feature screening module 30, a decision module 40 and a classification module 50.

[0049] The image data import module 10 is used to obtain an image data set corresponding to a pre-collected fine-grained image, and import the image data set into a Transformer-based visual model.

[0050] First of all, it should be explained that fine-grained images refer to images with subtle visual differences. These differences are mainly reflected in different subcategories under the same category. Since the visual differences between subcategories are very small, fine-grained image classification is difficult and requires high-resolution images and complex algorithms to achieve accurate classification. For example, different species of birds, car models, dog breeds, etc.

[0051] In this embodiment, fine-grained images of the target object are first collected, and then an image dataset is constructed based on the fine-grained images collected at different time stages. The image dataset is imported into a Transformer-based visual model, namely the Swin Transformer model.

[0052] Among them, the above-mentioned visual model includes a feature pyramid network module, a weakly supervised sampling module, a neural decision forest module and a trusted multi-view classification module.

[0053] The shallow information fusion module 20 is used to import the feature map of each stage into the feature pyramid network module of the visual model to obtain a first target feature map that fuses the shallow information.

[0054] In this embodiment, the shallow information fusion module 20 is specifically used for: Importing the feature map of each stage in the image data set into the feature pyramid network module of the visual model; The deep features containing semantic information and the shallow features containing spatial information are fused layer by layer through the feature pyramid network module to obtain a first target feature map.

[0055] Among them, the feature pyramid network module, namely the FPN module (FPN), is a module used for multi-scale feature fusion and is often used in image processing tasks to enhance the model's ability to recognize objects of different scales.

[0056] Specifically, the FPN module constructs a bottom-up feature pyramid to fuse deep features including semantic information with shallow features including spatial information layer by layer.

[0057] In the specific implementation, the FPN module extracts feature maps from different levels of the backbone network of the visual model, first performs dimensionality reduction projection so that the feature maps corresponding to different levels have the same feature dimension, then aligns the feature maps of different resolutions through upsampling operation, and adds features layer by layer to obtain the first target feature map that integrates shallow information.

[0058] The feature pyramid network module shown in this embodiment can effectively integrate multi-scale information in the image, so that the visual model can retain key global semantic information and capture subtle local features when processing fine-grained tasks, thereby improving the accuracy and robustness of overall classification or detection.

[0059] The pixel feature screening module 30 is used to import the first target feature map into the weakly supervised sampling module of the visual model, screen out target features that are beneficial to classification, and stack them into a target vector and a second target feature map based on the target features.

[0060] In this embodiment, the pixel feature screening module 30 is specifically used for: Importing the first target feature map into the weakly supervised sampling module of the visual model, and performing category prediction on the pixel values, i.e., feature points, of the first target feature map through the fully connected layer in the weakly supervised sampling module; Performing multi-classification probability conversion processing on the prediction results, if the highest prediction probability of the feature point is greater than a preset threshold, determining the feature point as a target feature that is helpful for classification; Feature stacking is performed based on the target features, and a target vector and a second target feature map are output.

[0061] In this embodiment, the pixel feature screening module 30 is also used for: If the highest prediction probability of the feature point is less than the preset threshold, it is regarded as an invalid feature point whose contribution to fine-grained classification is less than a preset value, and the invalid feature point is filtered out.

[0062] In this embodiment, during the use of the weakly supervised sampling module (WSS), the pixel values ​​of the first target feature map are called feature points. The WSS module adopts a simple and efficient design to predict the category of each feature point through a fully connected layer.

[0063] Specifically, after the prediction results are subjected to multi-classification probability conversion processing (softmax processing), if the highest prediction probability of a feature point exceeds a preset threshold, the feature point is considered to be an effective feature that helps with classification, that is, the above-mentioned target feature, which will be retained and used in subsequent feature fusion steps.

[0064] On the contrary, if the highest prediction probability of a feature point does not reach the preset threshold, it is regarded as a feature point that contributes less to the fine-grained classification, and is determined to be an invalid feature point, and then filtered out.

[0065] By adopting the above-mentioned threshold-based feature point screening method, the WSS module can automatically screen out the most discriminative feature points for fine-grained image classification, effectively reducing the computational burden of the visual model and helping the visual model to better focus on the discriminative key parts in the fine-grained image, thereby improving the classification accuracy.

[0066] The decision module 40 is used to import the second target feature map and the initial prediction value that has not been processed by the activation function into the neural decision forest module of the visual model, obtain the target prediction value corresponding to each granularity respectively and store them in the output sequence.

[0067] In this embodiment, a neural decision forest (Neural Decision Forest) composed of multiple neural decision trees is first constructed to enhance the overall robustness of the model. Each neural decision tree consists of multiple nodes, and the nodes hierarchically route the input features through a hierarchical structure, so that the visual model can capture the fine-grained information in the second target feature map. Each neural decision tree in the neural decision forest independently learns different feature distributions and discrimination rules, thereby improving the model's overall adaptability to complex data and classification accuracy. By integrating the decisions of multiple neural decision trees, the neural decision forest can effectively balance the bias and variance of a single tree model, improve the resistance to noise and sample bias, and ultimately achieve robust and accurate classification of the input image.

[0068] The classification module 50 is used to import the output sequence into the trusted multi-view classification module of the visual model, fuse the predicted value distributions of multiple opinions, and obtain the final fused classification result.

[0069] In this embodiment, the classification module 50 is specifically used for: importing the output sequence into a trusted multi-view classification module of the vision model; In the credible multi-view classification module, the category probability distribution is modeled by variational Dirichlet distribution, and Dempster-Shafer is used to integrate evidence to fuse the predicted value distributions of multiple opinions to obtain the final fused classification result.

[0070] Among them, in this embodiment, a trusted multi-view classification module (Trusted Multi-View Classification, TMC) is used to effectively integrate multi-granularity information and improve the reliability and credibility of decision-making.

[0071] In this embodiment, unlike the traditional feature or output level fusion method, the TMC module fuses the evidence information from each perspective at the evidence level, thereby achieving stable multi-view integration with more reasonable uncertainty estimation. In this embodiment, the category probability distribution is modeled by variational Dirichlet distribution, and the Dempster-Shafer theory is used for evidence integration to ensure that the multi-view information is optimized in terms of interpretability and credibility.

[0072] Specifically, the TMC module combines the uncertainty of each view and adaptively allocates view weights during the visual model training process, making the fused decision more reliable, more robust, and theoretically guaranteed. In addition, the TMC module not only does not require additional calculations or network changes, but also provides evidence-based integration for classification of each granularity through subjective uncertainty, thereby significantly improving classification accuracy and interpretability.

[0073] In summary, compared with the prior art, the fine-grained image recognition and classification system shown in this embodiment has the following beneficial effects: This embodiment proposes a fine-grained image recognition and classification system for fine-grained image classification. While taking into account the interpretability of the model, multi-granularity information is used to improve the training efficiency of the visual model and guide the model to find discriminative areas faster, thereby improving the accuracy of the model. A transformer-based visual model is used as a feature extractor, and feature maps of multi-granularity information with different receptive fields output by different levels are extracted. After the feature pyramid, each layer of granularity feature map is finally screened for key features, and the multi-granularity information containing key features that are beneficial to classification is sent to an interpretable prototype decision tree forest. Finally, the decision forest uses an interpretable dynamic decision fusion algorithm to form a prediction result with predictive robustness to form the final classification, thereby improving the recognition and classification accuracy of fine-grained images and making the classification more accurate.

[0074] Embodiment 3 A third embodiment of the present invention provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the above embodiment is implemented.

[0075] Embodiment 4 A fourth embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the above embodiments when executing the computer program.

[0076] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0077] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A fine-grained image recognition and classification method, characterized in that: The method comprises: Obtain an image dataset corresponding to a pre-collected fine-grained image, and import the image dataset into a Transformer-based visual model; Importing each stage feature map into the feature pyramid network module of the visual model to obtain a first target feature map that integrates shallow information; Importing the first target feature map into the weakly supervised sampling module of the visual model, screening out target features that are useful for classification, and stacking them into a target vector and a second target feature map based on the target features; Importing the second target feature map and the initial prediction value that has not been processed by the activation function into the neural decision forest module of the visual model, respectively obtaining the target prediction value corresponding to each granularity and storing them in the output sequence; The output sequence is imported into the credible multi-view classification module of the visual model, and the predicted value distributions of multiple opinions are fused to obtain a final fused classification result.

2. The fine-grained image recognition and classification method according to claim 1, characterized in that: The step of importing each stage feature map into the feature pyramid network module of the visual model to obtain a first target feature map that integrates shallow information includes: Importing the feature map of each stage in the image data set into the feature pyramid network module of the visual model; The deep features containing semantic information and the shallow features containing spatial information are fused layer by layer through the feature pyramid network module to obtain a first target feature map.

3. The fine-grained image recognition and classification method according to claim 1, characterized in that: The step of importing the first target feature map into the weakly supervised sampling module of the visual model, screening out target features that are useful for classification, and stacking the target features into a target vector and a second target feature map comprises: Importing the first target feature map into the weakly supervised sampling module of the visual model, and performing category prediction on the pixel values, i.e., feature points, of the first target feature map through the fully connected layer in the weakly supervised sampling module; Performing multi-classification probability conversion processing on the prediction results, if the highest prediction probability of the feature point is greater than a preset threshold, determining the feature point as a target feature that is helpful for classification; Feature stacking is performed based on the target features, and a target vector and a second target feature map are output.

4. The fine-grained image recognition and classification method according to claim 3 is characterized in that: The method further comprises: If the highest prediction probability of the feature point is less than the preset threshold, it is regarded as an invalid feature point whose contribution to fine-grained classification is less than a preset value, and the invalid feature point is filtered out.

5. The fine-grained image recognition and classification method according to claim 1, characterized in that: The step of importing the second target feature map and the initial prediction value not processed by the activation function into the neural decision forest module of the visual model, obtaining the target prediction value corresponding to each granularity and storing them in the output sequence includes: The neural decision forest module is composed of multiple neural decision trees, each of which is composed of multiple nodes, and each node hierarchically routes input features through a hierarchical structure so that the visual model captures fine-grained information in the second target feature map.

6. The fine-grained image recognition and classification method according to claim 1, characterized in that: The step of importing the output sequence into the trusted multi-view classification module of the visual model, fusing the predicted value distributions of multiple opinions, and obtaining the final fused classification result comprises: importing the output sequence into a trusted multi-view classification module of the vision model; In the credible multi-view classification module, the category probability distribution is modeled by variational Dirichlet distribution, and Dempster-Shafer is used to integrate evidence to fuse the predicted value distributions of multiple opinions to obtain the final fused classification result.

7. A fine-grained image recognition and classification system, characterized in that: The method applied to any one of claims 1 to 6, wherein the system comprises: An image data import module, used to obtain an image data set corresponding to a pre-collected fine-grained image, and import the image data set into a Transformer-based visual model; A shallow information fusion module, used for importing the feature map of each stage into the feature pyramid network module of the visual model to obtain a first target feature map that fuses the shallow information; A pixel feature screening module, used for importing the first target feature map into the weakly supervised sampling module of the visual model, screening out target features that are useful for classification, and stacking the target features into a target vector and a second target feature map based on the target features; A decision module, used for importing the second target feature map and the initial prediction value that has not been processed by the activation function into the neural decision forest module of the visual model, respectively obtaining the target prediction value corresponding to each granularity and storing it in an output sequence; The classification module is used to import the output sequence into the trusted multi-view classification module of the visual model, fuse the predicted value distributions of multiple opinions, and obtain a final fused classification result.

8. The fine-grained image recognition and classification system according to claim 7, characterized in that: The shallow information fusion module is specifically used for: Importing the feature map of each stage in the image data set into the feature pyramid network module of the visual model; The deep features containing semantic information and the shallow features containing spatial information are fused layer by layer through the feature pyramid network module to obtain a first target feature map.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images