Fine-grained visual classification method and system based on self-evaluation classifier
By employing a self-evaluation classifier method for preliminary classification and secondary evaluation of fine-grained visual classification, and combining word embedding and attention mechanisms, useless information regions are identified and removed. This solves the problems of accuracy and fuzzy category distinction in fine-grained visual classification, achieving higher classification accuracy.
Patent Information
- Application Number
- CN202411077169.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2025-01-07
AI Technical Summary
Existing fine-grained visual classification methods are not very accurate, rely on manual annotation, have low accuracy rates, and struggle to accurately detect separated bounding boxes. The question remains how to improve the accuracy of fine-grained visual classification and its ability to distinguish between fuzzy categories.
A self-evaluation classifier-based approach is adopted. A coarse-grained classifier is used for initial classification, and a joint representation generation method is constructed. The output of the word embedding method and the preprocessed image data are jointly represented. A GRU network is used to generate word embedding vectors and generate a joint feature matrix. The computation process is optimized through a unitized attention mechanism, and a secondary evaluation is performed to identify and remove useless information regions and background noise.
It effectively improves the accuracy of fine-grained visual classification, reduces misclassification of fuzzy categories, enhances the model's ability to distinguish fuzzy categories, and improves classification performance.
Smart Images
Figure CN119273957A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a fine-grained visual classification method and system based on a self-assessment classifier. Background Art
[0002] The fine-grained visual classification task aims to classify images belonging to the same category. Compared with general image classification tasks, fine-grained image classification is more challenging for three main reasons: first, the intra-class differences are large, and objects belonging to the same category present significantly different postures and perspectives; second, the subtle inter-class differences, objects belonging to different categories may be very similar, except for some subtle differences, for example, the color style of a bird's head can usually determine its fine-grained category; third, the limitation of training data, labeling fine-grained categories usually requires expertise and a lot of annotation time; for these reasons, it is not a simple task to obtain accurate classification results using only the most advanced CNNs, such as VGG and ResNet; recent studies have shown that the key solution to fine-grained classification is to find informative regions in multiple object parts and extract discriminative features.
[0003] A popular approach to learning object parts is based on manual annotation; however, annotating fine-grained regions is time-consuming, making this approach impractical; some improvements utilize unsupervised or weakly supervised learning to localize informative object parts or region-of-interest bounding boxes; although this is a promising approach to addressing the problem of manually annotating fine-grained regions, these approaches suffer from drawbacks such as low accuracy, high cost in training / inference phases, and difficulty in accurately detecting separated bounding boxes.
[0004] Existing research in the field of FGVC mainly focuses on three aspects: first, enhancing the attention and positioning capabilities of CNN, relying on component position or attribute annotations to improve classification accuracy, such as through geometric priors and bilinear pooling methods; second, by maximizing the entropy of the output probability distribution, making the model prediction more dispersed and improving the handling of intra-class differences, such as multi-attention multi-class loss methods and attention regularization; third, using unsupervised or weakly supervised learning to reduce dependence on pixel-level annotations, which is challenging but effective, such as introducing adversarial learning and Gaussian mixture models; however, these methods often ignore predictions between fuzzy categories, which may lead to misclassification. Summary of the invention
[0005] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.
[0006] In view of the above-mentioned problems, the present invention is proposed.
[0007] Therefore, the technical problems solved by the present invention are: the existing fine-grained visual classification has low accuracy, relies on manual labeling, has low accuracy, and is difficult to accurately detect separated bounding boxes, and how to improve the accuracy of fine-grained visual classification and the ability to distinguish between fuzzy categories.
[0008] To solve the above technical problems, the present invention provides the following technical solutions: a fine-grained visual classification method and system based on a self-evaluation classifier, comprising collecting image data, performing preliminary classification on the preprocessed image data through a coarse-grained classifier; constructing a joint representation generation method, performing a joint representation on the output result of the word embedding method and the preprocessed image data; based on the joint representation result, performing a secondary evaluation on the preliminary classification result, generating a fine-grained classification result, and identifying and removing information-rich areas and background noise that are useless for fine-grained classification.
[0009] As a preferred solution of the fine-grained visual classification method and system based on self-evaluation classifier described in the present invention, the preprocessing includes preprocessing the input image, generating a feature map F through a CNN deep convolutional neural network as a baseline, and processing the convolution feature map F through a fully connected layer to obtain a visual feature V.
[0010] As a preferred solution of the fine-grained visual classification method and system based on the self-evaluation classifier described in the present invention, the preliminary classification includes using the CNN main classifier to perform preliminary classification on the visual features V to obtain the top k prediction results C k , set C k The k categories with the highest probability are called Top-k categories.
[0011] As a preferred solution of the fine-grained visual classification method and system based on self-evaluation classifier described in the present invention, the word embedding method includes encoding the Top-k category labels to generate a word embedding set E k ′, use the GRU network to process each class label and generate a word embedding vector E k .
[0012] As a preferred solution of the fine-grained visual classification method and system based on the self-evaluation classifier described in the present invention, the joint representation generation method includes: k Perform joint calculation to generate the joint feature J and the attention map of the Top-k categories Construct the joint feature matrix, expressed as:
[0013]
[0014] Where T is the transpose of the feature matrix, is a third-order learnable tensor, vec is a vectorized operator, ×1 is the tensor modular product along the first mode, and ×2 is the tensor modular product along the second mode;
[0015] Use the unitized attention mechanism to optimize the calculation process, expressed as:
[0016]
[0017] Among them, f is the number of superpixels in the convolutional feature map F, k is the number of word embeddings, is the first Line The elements of the column, is a smaller learnable tensor, F i is the i-th channel of the feature map F, (E k ) j is the word embedding E k The j-th word embedding vector of The real number field is used to define the dimension of vectors and matrices, d e is the dimension of word embedding, d f is the number of channels of the feature map, d j is the dimension of the joint feature.
[0018] As a preferred solution of the fine-grained visual classification method and system based on the self-evaluation classifier described in the present invention, the secondary evaluation of the preliminary classification results includes secondary evaluation of the Top-k categories based on the joint features to generate the fine-grained classification results C k ', use the cross entropy loss function to train the classifier and optimize the Top-k prediction results.
[0019] As a preferred solution of the fine-grained visual classification method and system based on the self-evaluation classifier described in the present invention, the method of identifying and removing the information-rich areas and background noise that are useless for fine-grained classification includes using the attention map Identify useless background areas, generate Dropping masks, and recalculate the attention distribution of the remaining areas.
[0020] Another object of the present invention is to provide a fine-grained visual classification method and system based on a self-evaluation classifier, which can generate fine-grained classification results by performing a secondary evaluation on the preliminary classification results based on the joint representation results, identify and remove information-rich areas and background noise that are useless for fine-grained classification, and solve the current fine-grained visual classification tasks. The problems of large intra-class differences, subtle inter-class differences, and limited training data lead to low classification accuracy, as well as the problem of reliance on manual labeling, low accuracy, high cost, and difficulty in accurately detecting separated bounding boxes.
[0021] As a preferred solution of the fine-grained visual classification method and system based on self-evaluation classifier described in the present invention, it includes: an image data analysis module, a label embedding joint module, and a self-evaluation optimization module; the image data analysis module is used to collect image data and perform preliminary classification on the preprocessed image data through a coarse-grained classifier; the label embedding joint module is used to construct a joint representation generation method to jointly represent the output result of the word embedding method and the preprocessed image data; the self-evaluation optimization module is used to perform a secondary evaluation on the preliminary classification result based on the joint representation result, generate a fine-grained classification result, and identify and remove information-rich areas and background noise that are useless for fine-grained classification.
[0022] A computer device includes a memory and a processor, wherein the memory stores a computer program, and wherein the processor executes the computer program to implement a step of a fine-grained visual classification method and system based on a self-assessment classifier.
[0023] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of a fine-grained visual classification method and system based on a self-assessment classifier are implemented.
[0024] Beneficial effects of the present invention: The fine-grained visual classification method and system based on a self-evaluation classifier provided by the present invention re-evaluates the Top-k classification results with the highest probability through a self-evaluation classifier, effectively reducing the misclassification of k fuzzy categories that are difficult to distinguish, and improving the accuracy of classification; using a word embedding method and a feature map for joint representation, and introducing an attention map at the superpixel block level, the model's ability to distinguish between fuzzy categories is enhanced; a new Dropping method is proposed, which uses an attention map to identify and remove areas that cause misclassification and useless background noise, further improving the classification effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0026] Figure 1 An overall flow chart of a fine-grained visual classification method and system based on a self-assessment classifier provided by one embodiment of the present invention;
[0027] Figure 2A visualization diagram of a fine-grained visual classification method based on a self-assessment classifier and different Dropping technologies in the system provided by the second embodiment of the present invention;
[0028] Figure 3 A second embodiment of the present invention provides a fine-grained visual classification method based on a self-assessment classifier and a visualization diagram of the positioning results of attention-guided Localization, random Localization and SAC Localization in an image of the system;
[0029] Figure 4 A module diagram of a fine-grained visual classification method and system based on a self-assessment classifier provided in the third embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0031] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a fine-grained visual classification method and system based on a self-assessment classifier, comprising:
[0032] S1: Collect image data and perform preliminary classification on the preprocessed image data through a coarse-grained classifier.
[0033] Furthermore, the preprocessing includes preprocessing the input image, generating a feature map F through a CNN deep convolutional neural network as a baseline, and processing the convolutional feature map F through a fully connected layer to obtain a visual feature V.
[0034] It should be noted that the preliminary classification includes using the CNN main classifier to perform preliminary classification on the visual features V and obtain the top k prediction results C k , set C k The k categories with the highest probability are called Top-k categories.
[0035] It should also be noted that TCCS, as a coarse-grained classifier, accepts images as input for processing. Each input image is first processed by a deep convolutional neural network (Deep CNN) to generate a feature map F. The size of the feature map F is d f ×m×n,d fis the number of channels of the feature map, m and n are the height and width of the feature map respectively, is the dimension of the feature map F; the generation of visual features V, which is obtained by applying a fully connected layer after the convolution feature map F, d v is the size of V, is the dimension of the visual feature V; preliminary classification, use the main classifier of the backbone network to perform preliminary classification on the visual feature V and obtain the top k prediction results C k ; Top-k prediction, assuming there are N categories in the coarse-grained dataset, the top k prediction results C k ={C1,…,C k} are all predicted categories C N , those classes with the highest confidence scores.
[0036] S2: Construct a joint representation generation method to jointly represent the output results of the word embedding method and the preprocessed image data.
[0037] Furthermore, the word embedding method includes encoding the Top-k category labels to generate a word embedding set E k ′, use the GRU network to process each class label and generate a word embedding vector E k .
[0038] It should be noted that the joint representation generation method includes combining the feature map F and the word embedding vector E k Perform joint calculation to generate the joint feature J and the attention map of the Top-k categories Construct the joint feature matrix, expressed as:
[0039]
[0040] Where T is the transpose of the feature matrix, is a third-order learnable tensor, vec is a vectorized operator, ×1 is the tensor modular product along the first mode, and ×2 is the tensor modular product along the second mode.
[0041] Use the unitized attention mechanism to optimize the calculation process, expressed as:
[0042]
[0043] Among them, f is the number of superpixels in the convolutional feature map F, k is the number of word embeddings, is the first Line The elements of the column, is a smaller learnable tensor, F i is the i-th channel of the feature map F, (E k )j is the word embedding E k The j-th word embedding vector of The real number field is used to define the dimension of vectors and matrices, d e is the dimension of word embedding, d f is the number of channels of the feature map, d j is the dimension of the joint feature.
[0044] It should also be noted that the joint feature matrix optimization process, d F =(d f ×f), d E =(d e ×k), d E is the dimension of the word embedding multiplied by the number of word embeddings, d F is the number of channels of the feature map multiplied by the number of superpixels, for The dimension of The memory usage is too large, which is not practical for learning, so a divide-and-conquer strategy can be used to reduce it. size but still ensure the same learning effect by relying on a unitized attention mechanism; specifically, let J p is the joint representation of the pth channel from the pth pair of inputs, where each pair of input channels comes from different inputs, and the set of channels for all input pairs is actually k (E k ) j and f F i , i∈[1,f]j=[1,k], that is, F and E k There are f×k combinations of two modalities; therefore, the joint representation J can be represented by the joint representation J of the channels of all input pairs p The approximate substitution is used to derive the formula of the present invention, wherein the invention features are is a ratio Smaller size of learnable tensors.
[0045] S3: Based on the joint representation results, the preliminary classification results are evaluated again to generate fine-grained classification results, and the information-rich areas and background noise that are useless for fine-grained classification are identified and removed.
[0046] Furthermore, a secondary evaluation of the preliminary classification results includes a secondary evaluation of the Top-k categories based on the joint features to generate a fine-grained classification result C k ', use the cross entropy loss function to train the classifier and optimize the Top-k prediction results.
[0047] It should be noted that methods for identifying and removing information-rich regions and background noise that are useless for fine-grained classification include using attention maps Identify useless background areas, generate Dropping masks, and recalculate the attention distribution of the remaining areas.
[0048] It should also be noted that the top-k categories are evaluated again based on the joint features to obtain the second top-k prediction C k ′,C k ′={C1′,…,C k ′}. Intuitively, C k ′ is the top k classification results after self-evaluation; SAC is used as a fine-grained classifier to generate a second prediction to re-evaluate the ambiguous classification results; in practice, the cross entropy loss function can be used to train this classifier to obtain the top k prediction results; through back propagation, the feature map F and the joint feature J from the CNN backbone network will be more useful for fine-grained classification; by training coarse-grained and fine-grained classifiers at the same time, the method of the present invention can increase the discriminability of the feature map; therefore, the Top-k ambiguous categories can be reduced and the overall fine-grained classification accuracy can be improved; using the attention map Identify useless background areas and generate Dropping masks. Dropping is a new Dropping mechanism introduced by this invention. By using the current Top-k attention map To remove those areas with high similarity scores or similar within the class, and recalculate the attention distribution between the remaining areas and Top-k, to improve the classification accuracy; and it is worth emphasizing that these proposed areas are not necessarily background noise, but may also be part of the identified object itself that is necessary for classification, but these areas do not have discriminability for fine-grained classification tasks.
[0049] Example 2, reference Figure 2-3 , which is an embodiment of the present invention, provides a fine-grained visual classification method and system based on a self-evaluation classifier. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.
[0050] This example is conducted on an NVIDIA 2080ti GPU 22GB RAM*2, and quantitative analysis is performed on multiple data sets and different parameters. The model is trained using the stochastic gradient descent algorithm with a momentum of 0.9, a maximum number of epochs of 80, a weight decay of 0.00001, a minimum batch size of 24, an initial learning rate of 0.001, and an exponential decay rate of 0.9 after every 2 epochs. Based on the verification results, the number of Top-k fuzzy classes is set to 10, and the parameters are α is 0.1 and 0.5 respectively, and the word embedding vector E kThe dimension is 1024, the feature map F and the word embedding vector E k The union of represents the dimension of the joint feature J is 1024.
[0051] Referring to Table 1, the performance of Dropping based on SAC is compared with Dropping of other data enhancement methods on the CUB-200-2011 dataset. Weakly supervised learning bilinear attention network (WS-BAN) and weakly supervised data enhancement network (WS-DAN) are used as the backbone networks in the experiment; it is worth noting that this embodiment follows the hyperparameters and evaluation indicators mentioned in WS-DAN for fair comparison; specifically, the Dropping proposed in the present invention improves the accuracy of WS-BAN and WS-DAN by 2.1% and 0.9%, respectively; these experimental results show that Dropping in the method of the present invention successfully removes the ambiguous areas between the Top-k classes and achieves relatively competitive results compared with other enhancement methods.
[0052] Table 1 Performance comparison of different models with different enhancement methods
[0053] Method Accuracy(%) WS-BAN(Backbone) 86.4 WS-BAN+Random Cropping 86.8 WS-BAN+Random Dropping 86.7 WS-BAN+Random Cropping&Dropping 87.2 WS-BAN+Attention Cropping 87.8 WS-BAN+Attention Dropping 87.4 WS-BAN+Attention Cropping&Dropping 88.4 WS-BAN+Dropping(ours) 88.5 WSDAN(Backbone) 89.4 WSDAN+Dropping(ours) 90.3
[0054] Referring to Table 2, Dropping is an important step in the method of the present invention and also an important step in other FGVC tasks. In practice, the Dropped region can enhance the distinction between the Top-k classes, but may also damage the distinction between the original classification and the remaining classes. Therefore, the hyperparameters can be used in the method of the present invention. To control this problem; Table 2 shows the results of the present invention on the WS-DAN baseline under different conditions on the CUB-200-2011 dataset; if If it is set too small, the effectiveness of Dropping will not be enough to improve the overall performance; If the value is set too large, Dropping may mistakenly remove features that are discriminative for different images of the same category, thereby reducing accuracy. Through experimental results, it can be observed that the Dropping threshold It shows consistent performance on different CNN backbone networks and fine-grained datasets.
[0055] Table 2 Performance comparison of different Dropping thresholds
[0056]
[0057] Referring to Table 3, the experimental results show that by introducing the SAC of the present invention into some recently studied fine-grained classifiers, better classification results are obtained on the CUB-200-2011, Stanford Dogs, FGVC Aircraft datasets and the datasets collected by the present invention.
[0058] Table 3 Performance comparison of different models under different datasets
[0059]
[0060] The data set sources include the Caltech-UCSD Birds-200-2011 (CUB-200-2011) data set, which is the most commonly used data set for fine-grained visual classification tasks; it contains 11,788 images belonging to 200 subcategories of birds, of which 5,994 are used for training and 5,794 are used for testing; each image has detailed annotations: 1 subcategory label, 15 part locations, 312 binary attributes, and 1 bounding box; these text information comes from Reed et al.; the CUB-200-2011 data set is extended by collecting fine-grained natural language descriptions; ten single-sentence descriptions are collected for each image, and these natural language descriptions are collected through the Amazon Mechanical Turk (AMT) platform, requiring at least 10 words, without any information about subcategories and actions.
[0061] The Stanford Dogs dataset is a dataset widely used for fine-grained image classification tasks, especially in the field of dog breed identification. The dataset contains images of 120 different dog breeds, providing rich and diverse training and test samples. There are a total of about 20,580 images, each of which is annotated with the corresponding dog breed, and bounding box annotations are provided for each dog. Since the visual differences between some breeds are very subtle, this makes this dataset an ideal benchmark for evaluating the performance of fine-grained classification algorithms.
[0062] The FGVC Aircraft dataset is another dataset commonly used for fine-grained image classification tasks, mainly for aircraft model recognition; this dataset contains images of various aircraft models, providing diverse training and test samples; there are about 10,000 images in total, covering 102 different aircraft models, and the corresponding aircraft model is annotated for each image; due to the subtle differences between aircraft models, this makes this dataset highly challenging and is often used for benchmarking fine-grained classification algorithms to evaluate their ability to distinguish similar objects.
[0063] The dataset collected by the present invention is an image dataset of 34 bird species crawled from Baidu, each species contains about 200 images, 32 of these bird species are not in the 200 species of the CUB-200-2011 dataset, and each image has a corresponding label. This dataset provides rich and diverse training and test samples, which is suitable for fine-grained image classification tasks. Since the visual features of birds are relatively small, this makes the dataset highly challenging when evaluating the performance of classification algorithms in fine-grained classification tasks; by using this dataset, machine learning models can be effectively trained and tested to improve their accuracy and robustness in bird species recognition tasks.
[0064] This embodiment performs a qualitative analysis of visual evaluation on the CUB-200-2011 dataset based on the WS-DAN architecture.
[0065] Figure 2 The visual comparison of different Dropping methods is shown: (a) is the input image, (b) is random Dropping, (c) is attention Dropping, (d) is the Dropping module distribution diagram of the present invention, and (e) is the Dropping of the present invention. Different from random Dropping that may erase the entire object or only erase the background, and attention Dropping that increases generalization ability by erasing discriminative information areas, the Dropping method proposed in the present invention can efficiently identify and erase information areas that cause misclassification.
[0066] Figure 3 A comparison between attention-guided Localization, random Localization and the SACLocalization of the present invention is shown to demonstrate the effectiveness of the positioning method in SAC, where (a) is the input image, (b) is the SACLocalization result, (c) is the Localization result of attention Cropping, and (d) is the positioning result of random Cropping. The dark border represents the true value, and the light border represents the predicted result. Unlike random Localization that may erase the useful information area of the object from the image, and attention-guided Localization that erases the background and retains the object in the image, SAC's Localization can efficiently identify useful information areas for fine-grained classification between difficult-to-distinguish categories.
[0067] The simulation results show that the proposed fine-grained visual classification method and system based on the self-evaluation classifier can identify and remove useless information-rich areas and background noise for fine-grained classification, further improve the classification effect, and verify the effectiveness of the proposed design method in the results. Therefore, our invention is creative.
[0068] Example 3, reference Figure 4 , is an embodiment of the present invention, and provides a fine-grained visual classification method and system based on a self-evaluation classifier, including an image data analysis module, a label embedding joint module, and a self-evaluation optimization module.
[0069] The image data analysis module is used to collect image data and perform preliminary classification of the preprocessed image data through a coarse-grained classifier; the label embedding joint module is used to construct a joint representation generation method to jointly represent the output results of the word embedding method and the preprocessed image data; the self-evaluation optimization module is used to perform a secondary evaluation of the preliminary classification results based on the joint representation results, generate fine-grained classification results, and identify and remove useless information-rich areas and background noise for fine-grained classification.
[0070] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0072] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.
[0073] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
[0074] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A fine-grained visual classification method and system based on a self-evaluation classifier, characterized in that: include: Collect image data, and perform preliminary classification on the preprocessed image data through a coarse-grained classifier; Construct a joint representation generation method to jointly represent the output of the word embedding method and the preprocessed image data; Based on the joint representation results, the preliminary classification results are evaluated for the second time to generate fine-grained classification results, and the information-rich areas and background noise that are useless for fine-grained classification are identified and removed.
2. The fine-grained visual classification method and system based on a self-assessment classifier as claimed in claim 1, characterized in that: The preprocessing includes preprocessing the input image, generating a feature map F through a CNN deep convolutional neural network as a baseline, and processing the convolutional feature map F through a fully connected layer to obtain a visual feature V.
3. The fine-grained visual classification method and system based on a self-evaluation classifier as claimed in claim 2, characterized in that: The preliminary classification includes using the CNN main classifier to perform preliminary classification on the visual feature V to obtain the top k prediction results C k , set C k The k categories with the highest probability are called Top-k categories.
4. The fine-grained visual classification method and system based on a self-evaluation classifier as claimed in claim 3, characterized in that: The word embedding method includes encoding the Top-k category labels to generate a word embedding set E k ′, use the GRU network to process each class label and generate a word embedding vector E k .
5. The fine-grained visual classification method and system based on a self-evaluation classifier as claimed in claim 4, characterized in that: The joint representation generation method includes combining the feature map F and the word embedding vector E k Perform joint calculation to generate the joint feature J and the attention map of the Top-k categories Construct the joint feature matrix, expressed as: J T =(T×1vec(F))×2vec(E k ) Where T is the transpose of the feature matrix, T is a third-order learnable tensor, vec is a vectorized operator, ×1 is the tensor modular product along the first mode, and ×2 is the tensor modular product along the second mode; Use the unitized attention mechanism to optimize the calculation process, expressed as: Among them, f is the number of superpixels in the convolutional feature map F, k is the number of word embeddings, is the first Line The elements of the column, is a smaller learnable tensor, F i is the i-th channel of the feature map F, (E k ) j is the word embedding E k The j-th word embedding vector of The real number field is used to define the dimension of vectors and matrices, d e is the dimension of word embedding, d f is the number of channels of the feature map, d j is the dimension of the joint feature.
6. The fine-grained visual classification method and system based on a self-assessment classifier as claimed in claim 5, characterized in that: The secondary evaluation of the preliminary classification results includes secondary evaluation of the Top-k categories based on the joint features to generate a fine-grained classification result C k ′, use the cross entropy loss function to train the classifier and optimize the Top-k prediction results.
7. The fine-grained visual classification method and system based on a self-evaluation classifier as claimed in claim 6, characterized in that: The method for identifying and removing information-rich areas and background noise that are useless for fine-grained classification includes using an attention map Identify useless background areas, generate Dropping masks, and recalculate the attention distribution of the remaining areas.
8. A fine-grained visual classification method and system using the self-assessment classifier according to any one of claims 1 to 7, characterized in that: It includes image data analysis module, label embedding joint module, and self-evaluation optimization module; The image data analysis module is used to collect image data and perform preliminary classification on the preprocessed image data through a coarse-grained classifier; The label embedding joint module is used to construct a joint representation generation method to jointly represent the output results of the word embedding method and the preprocessed image data; The self-evaluation optimization module is used to perform a secondary evaluation on the preliminary classification results based on the joint representation results, generate fine-grained classification results, and identify and remove information-rich areas and background noise that are useless for fine-grained classification.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the fine-grained visual classification method and system based on a self-assessment classifier according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the fine-grained visual classification method and system based on a self-assessment classifier according to any one of claims 1 to 7 are implemented.