A fine-grained visual recognition method based on confusion perception and candidate recall
By constructing a structured thinking chain through self-confusion supervised fine-tuning and candidate recall perceptual reinforcement learning, the problems of insufficient candidate coverage and inaccurate calibration of multimodal large language models in fine-grained visual recognition are solved, thereby improving recognition accuracy and coverage. It is applicable to both closed and open world recognition.
Patent Information
- Application Number
- CN202611627600.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal large language models suffer from insufficient candidate coverage and inaccurate candidate calibration in fine-grained visual recognition, which causes the model to miss target categories or select incorrect candidate categories during the comparison process, thus affecting recognition accuracy.
By employing self-confusion supervised fine-tuning and candidate recall-aware reinforcement learning, a structured thought chain is constructed to mine the actual confusion categories of the model and optimize the candidate set. Self-confusion supervised fine-tuning is used to mine targeted confusion categories, construct a self-confusion supervised trajectory, and optimize the candidate set through candidate recall-aware reinforcement learning to generate a correctness-driven reward function to update the model strategy.
It improves the effectiveness of candidate calibration, reduces the chance of target categories being missed before comparison, and enhances the accuracy and coverage of fine-grained visual recognition, making it suitable for both closed and open-world recognition scenarios.
Smart Images

Figure CN123119218A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a fine-grained visual recognition method based on confusion perception and candidate recall, belonging to the fields of computer vision and artificial intelligence. Background Technology
[0002] Fine-grained visual recognition is a crucial task in computer vision, requiring the differentiation of subcategories that are almost identical in global appearance and can only be distinguished by local detail cues. Early fine-grained recognition methods relied on specially designed local representations and region selection mechanisms; visual-language pre-training techniques aligned image content with category descriptions, utilizing textual information to assist in fine-grained discrimination. In recent years, multimodal large language models have placed category names and visual evidence in the same generative space, enabling them not only to output predicted categories but also to simultaneously generate local visual evidence for distinguishing similar categories, providing a novel interactive discriminative paradigm for fine-grained visual recognition.
[0003] To further enhance the fine-grained visual recognition capabilities of multimodal large language models, existing work has introduced a structured chain-of-thought approach, enabling the model to reason according to a four-stage process: first analyzing the image, then enumerating candidate subcategories, followed by comparing candidates, and finally predicting the category. Meanwhile, reinforcement learning with verifiable rewards strengthens this reasoning process by optimizing the rule-verifiable final answer. A typical example is the Fine-R1 method, which applies reinforcement learning optimizers such as TAPO to the four-stage fine-grained reasoning trajectory. However, the supervision method of this type of approach is still essentially centered on the final answer: although the reasoning trajectory includes a candidate subcategory field, the content and coverage of this field are not directly optimized. This means that even if the model generates a seemingly reasonable comparison process, it may still construct an incorrect candidate set, or even miss the target category before the comparison begins.
[0004] Specifically, existing methods suffer from two interrelated problems: candidate coverage and candidate calibration. The candidate coverage problem refers to the possibility that the candidate set may omit the target class before comparisons begin; when subsequent comparisons revolve around existing candidate classes, this omission weakens the basis for correct discrimination. The candidate calibration problem refers to the situation where the candidate set fails to reflect the true confusion class of the model's strategy itself, containing only semantically or taxonomically similar classes.
[0005] Existing candidate construction methods typically employ externally selected, semantically similar, or taxonomically nearest candidate categories, or candidate categories provided by the teacher model. These candidates fail to align with the actual error patterns of the pedestal strategy, making it difficult for the comparison process to specifically eliminate the true confusion of the model.
[0006] Therefore, there is an urgent need for a fine-grained visual recognition method that can uncover the true confusion of the base strategy, retain the target category before comparison, and optimize candidate coverage and candidate calibration under the premise of prioritizing the correctness of the final answer, so as to solve the above-mentioned problems of multimodal large language models in fine-grained visual recognition. Summary of the Invention
[0007] To address the issues of insufficient candidate coverage and failure of candidate categories to reflect actual confusion in fine-grained visual recognition using multimodal large language models, this invention proposes a fine-grained visual recognition method based on confusion perception and candidate recall. First, self-confusion-supervised fine-tuning enables the model to learn to compare the target category with its own true confusion category. Then, candidate recall-aware reinforcement learning, with a correctness-driven reward, jointly optimizes the final answer's correctness and target category recall without modifying the underlying policy update rules.
[0008] The technical solution of the present invention includes the following steps: (1) Construct a structured fine-grained reasoning format to organize the fine-grained visual recognition process of multimodal large language models into a four-stage structured thinking chain of "visual analysis-candidate sub-category-comparison-prediction".
[0009] (2) Self-confusion supervised fine-tuning is a base strategy for pre-trained multimodal large language models. It uses open-world cues without candidate lists to sample training images multiple times, counts the number of times each real category is predicted as other categories, and mines the directional confusion categories of each real category. The target category and its directional confusion categories are combined to form a candidate field, construct a self-confusion supervised fine-tuning trajectory, and obtain the first training model through teacher-forced supervised fine-tuning.
[0010] (3) The candidate recall perception reinforcement learning generates a structured inference trajectory with the first training model as the initial strategy, and parses out the final prediction and the restricted candidate set. It calculates the selection signal representing whether the final prediction is correct and the coverage signal representing whether the target category is retained in the restricted candidate set. It constructs a correctness-dominated reward function with the selection signal as the main component and the coverage signal as the auxiliary component, which is used for reinforcement learning strategy update to obtain the second training model.
[0011] (4) The inference application inputs the image to be identified and the identification question into the second training model, generates a four-segment structured inference trajectory and outputs the final fine-grained prediction result. During the inference stage, the true category label or reward of the image to be identified is not accessed.
[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention improves the effectiveness of candidate calibration by mining directional confusion categories from the actual prediction errors of the base model, so that the candidate comparison content in the supervision trajectory corresponds to the error pattern of the model strategy itself. This improves the method of selecting candidate categories based solely on external semantic similarity or taxonomic relationships. 2. This invention uses a candidate recall-based reward system to encourage the target category to remain in a limited candidate set, prioritizing the correctness of the final answer. This provides direct training feedback for candidate coverage and reduces the chance of the target category being missed before comparison. 3. The reward function of this invention is a trajectory-level score rather than a policy update rule. It can be used without modifying the update logic of reinforcement learning optimizers such as GRPO, DAPO, and TAPO. Furthermore, it does not require access to the real labels and rewards during the inference phase, making it suitable for fine-grained visual recognition in both closed and open worlds. Attached Figure Description
[0013] Figure 1 This is a block diagram illustrating the principle of the fine-grained visual recognition method based on confusion perception and candidate recall of the present invention.
[0014] Figure 2 This is a structured output diagram of the fine-grained visual recognition method and comparison method based on confusion perception and candidate recall of the present invention. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0016] First, the output format and symbols of the structured thought chain for fine-grained visual recognition are explained. The fine-grained visual recognition process of the multimodal large language model is organized into a four-stage structured thought chain: "visual analysis - candidate sub-category - comparison - prediction". Given an image... Model strategy The generated output is represented by equation (1): (1) in, o This represents a structured reasoning trajectory. Visual analysis is indicated. Represents an ordered list of candidate subcategories. c 1 , ..., c m This indicates the candidate subcategories arranged in order. mIndicates the length of the candidate subcategory list. This indicates a comparison of candidate categories. This indicates the final fine-grained prediction. Indicates the input image. π Represents the model strategy.
[0017] To ensure consistent evaluation of candidate coverage, a category normalization mapping is employed. Valid, complete leaf label strings (i.e., complete fine-grained category names) are mapped to standard category labels within the domain of the fine-grained visual recognition task, while invalid strings are mapped to invalid symbols; the candidate list is parsed sequentially. Invalid entries are discarded, duplicate labels are removed, and the top four valid candidate categories are retained to form a restricted candidate set. This restricted candidate set fixes the candidate budget used for reward calculation and candidate-level evaluation.
[0018] in, This represents the normalized mapping of the above categories. This indicates the domain to which the fine-grained visual recognition task belongs. This represents the restricted candidate set obtained after normalization, deduplication, and retention of the first four valid categories in the original order.
[0019] like Figure 1 As shown, a fine-grained visual recognition method based on confusion perception and candidate recall includes the following steps: (1) Constructing a structured, fine-grained reasoning format: In the specific implementation, the four-segment structured thinking chain uses labels to mark the four segments: visual analysis, candidate subcategories, comparison, and prediction are placed in separate locations. <analysis>< / analysis> , <options>< / options> , <comparison>< / comparison> and <prediction>< / prediction> Within the tag, and ultimately with <answer>< / answer> The labels provide the prediction results.
[0020] (2) Self-confusion supervision fine-tuning, such as Figure 1 As shown in stage 1, this step addresses the candidate calibration problem by first performing targeted obfuscation mining and then constructing self-obfuscation trajectories, so that the candidate set reflects the true obfuscation category of the base strategy itself.
[0021] Base strategy Multiple sampling is performed on the training images using open-world cues without a candidate list; for the target category And another category The statistical base strategy is based on the real label as The image is predicted as The number of times is calculated according to formula (2): (2) in, Representation of domain The training image set, Represents training images, label(x) This indicates the true category label of the image. Indicates base strategy Prediction of training images, This represents the directional confusion count from the target category to another category, with the vertical line on the right indicating the number of elements in the set.
[0022] Based on the directional confusion count, select the top [number] [item]. Non-target categories are used as target categories The directional confusion category is calculated according to formula (3): (3) in, Represents the set of targeted obfuscation categories for the target category. Y d This represents a set of standard category labels within a domain. This indicates the number of confusion categories selected, with TopK representing the operation of selecting the aforementioned number of categories based on confusion count. Unlike taxonomic nearest neighbors or embedding nearest neighbors, targeted confusion categories are determined by the actual prediction error of the base strategy; and Statistically count them separately, and they are not required to be equal.
[0023] Then, a self-confusing supervised trajectory is constructed. For the true category is... For each training image, the target category and the directional confusion category are combined to form a candidate field, which is calculated according to equation (4): (4) in, C SC (y) A self-obfuscating candidate field representing the target category. This represents the set of targeted obfuscation categories for the target category. The set union operation is used to combine the target category with its obfuscation categories. The vertical bar indicates the number of categories in the candidate field. The candidate field contains the target category and at most three obfuscation categories, consistent with the budget of retaining the first four valid candidate categories.
[0024] Constructing a structured target trajectory And maintain its candidate budget and limited candidate set. Consistent.
[0025] in, Visual evidence representing training images, This indicates the comparison between the target category and the obfuscated categories in the candidate field; the structured target trajectory uses the target category as the final prediction.
[0026] The resulting set of trajectories As a supervisory signal, the first training model is obtained by fine-tuning the model through teacher-mandated supervision, minimizing the negative log-likelihood loss of the supervised target trajectory. .
[0027] in, Represents the set of self-confused supervised trajectories. This represents the strategy of the first trained model obtained through self-obfuscation supervision fine-tuning.
[0028] (3) Reinforcement learning for candidate recall perception, such as Figure 1 As shown in stage 2, this step addresses the candidate coverage problem, starting from the first trained model. The system generates a structured inference trajectory, parses the trajectory into a final prediction and a restricted candidate set, calculates the trajectory-level reward signal, and uses it for policy updates.
[0029] For the true label is The image, set Indicates from A trajectory. Applying candidate parsing rules yields a restricted candidate set. With standardized prediction Let the selection signal and coverage signal be defined respectively. The selection signal is used to measure whether the model has selected the correct target category, and is calculated according to equation (5): (5) in, r acc (o) Indicates a selection signal. This represents the final prediction after the above category normalization mapping. Indicates the true category label, This is an indicator function that takes a value of 1 if the condition is true and 0 otherwise.
[0030] The coverage signal is used to measure whether the target category is still available during the candidate comparison stage, and is calculated according to equation (6): (6) in, r rec (o) Indicates coverage signal, This represents a limited set of candidates that retains the first four valid candidate categories. Indicates the true category label, This represents the indicator function. The selection signal and the coverage signal are calculated separately; the former reflects the correctness of the final answer, and the latter reflects the retention of the target category in the candidate set.
[0031] The correctness-driven reward function combines the two signals mentioned above and calculates them according to equation (7): (7) in, R AR (o) Indicates trajectory-level reward. r acc (o) Indicates a selection signal. r rec (o) Indicates coverage signal, Represents the weight coefficient of the correctness sub-item and satisfies .constraint The reward for correctly predicting the trajectory is significantly higher than that for incorrectly predicting the trajectory. Therefore, candidate recall will not offset incorrect final answers.
[0032] When the final answers of two trajectories are equally correct, the difference in their rewards is determined by the difference in the candidate recalls, calculated according to equation (8): (8) in, o 1 and o 2 Let each represent a structured reasoning trajectory to be compared. Δr rec This represents the difference between the coverage signal of the previous trajectory and the coverage signal of the next trajectory. This represents the weight coefficient of the correctness sub-item. When the weight coefficient is less than 1, the candidate recall can further differentiate the reward among trajectories with the same final answer correctness; when the weight coefficient is equal to 1, the weight of the candidate recall sub-item is 0, and the reward is determined only by the final answer correctness.
[0033] This reward function, presented as a trajectory-level score rather than a policy update rule, can be provided to trajectory-unfold-based reinforcement learning optimizers, including GRPO, DAPO, and TAPO. The selected optimizer uses the trajectory-level reward as a basis for optimization. The second trained model is then updated without changing the policy update rules of the selected optimizer.
[0034] (4) The second training model is used to receive the image to be recognized and the recognition question, and generate a structured inference trajectory. It outputs the final fine-grained prediction result without accessing the ground truth label of the image to be identified or the training reward. Open-world recognition does not provide a list of candidate categories; closed-world evaluation uses the given candidate label settings.
[0035] Figure 2 The structured output diagrams of the method and comparison method of the present invention are given, respectively showing the visual analysis, candidate sub-categories, candidate comparison and final prediction fields, which are used to illustrate the organization of the candidate category construction and comparison process.
[0036] The multimodal large language model in this invention is an image-language or video-language large model, which adopts a visual encoder-projector-large language model architecture, including the Qwen2.5-VL series model.
[0037] In the specific implementation, Qwen2.5-VL-32B was used as the teacher model for the structured thinking chain, and Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct were used as student backbone models. Full-parameter supervised fine-tuning was performed using LLaMA-Factory, with both the 3B and 7B models fine-tuned for 10 epochs. Subsequently, reinforcement learning was performed using the models obtained from the supervised fine-tuning as the initial policy, with the 3B and 7B models trained for 10 and 5 epochs respectively. All training was conducted on four NVIDIA A100 GPUs. The correctness-driven reward function used weights... Calculate according to formula (9): (9) in, R AR (o) Indicates trajectory-level reward. r acc (o) Indicates a selection signal. r rec (o) The coverage signal is represented by 0.85 and 0.15, which represent the weights of the correctness sub-item and the candidate recall sub-item, respectively. In this embodiment, the candidate recall sub-item only takes effect when the structured output is complete, the candidate list does not exceed four, and they are not stacked.
[0038] To verify the effectiveness of the method of this invention, it was validated on the Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct models. Thirteen fine-grained visual recognition methods were selected as comparison methods, covering image-text matching-based visual language encoders, general multimodal large language models, and inference-based multimodal large language models, specifically: Method 1: CLIP proposed by Radford et al. Reference "Radford, A.; Kim, JW; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. LearningTransferable Visual Models from Natural Language Supervision. In Proceedings of the International Conference on Machine Learning.” Method 2: EVA-CLIP proposed by Sun et al., see reference "Sun, Q.; Fang, Y.; Wu, L.; Wang, X.; and Cao, Y. 2023. EVA-CLIP: Improved Training Techniques for CLIP atScale. arXiv preprint arXiv:2303.15389." Method 3: SigLIP proposed by Zhai et al., see reference "Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE / CVF International Conference on ComputerVision." Method 4: SigLIP2 proposed by Tschannen et al., see reference "Tschannen, M.; Gritsenko, A.; Wang, X.; et al. 2025. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXivpreprint arXiv:2502.14786." Method 5: Idefics2 proposed by Laurençon et al., see reference "Laurençon, H.; Tronchon, L.; Cord, M.; and Sanh, V. 2024. What matters when building vision-language models? In Advances in Neural Information Processing Systems." Method 6: Idefics3 proposed by Laurençon et al., see reference "Laurençon, H.; Marafioti, A.; Sanh, V.; and Tronchon, L. 2024. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637." Method 7: LLaVA-v1.6 (LLaVA-NeXT) proposed by Liu et al., reference "Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, YJ 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge." Method 8: LLaVA-OneVision proposed by Li et al., see reference "Li, B.; Zhang, Y.; Guo, D.; et al. 2024. LLaVA-OneVision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.03326." Method 9: InternVL 2.5 proposed by Chen et al., see reference "Chen, Z.; Wang, W.; Cao, Y.; et al. 2024. Expanding Performance Boundaries of Open-Source MultimodalModels with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271." Method 10: Qwen2-VL proposed by Wang et al., reference "Wang, P.; Bai, S.; Tan, S.; et al. 2024. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv preprint arXiv:2409.12191." Method 11: Qwen2.5-VL proposed by Bai et al., reference "Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al.2025a. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923." Method 12: DeepPerception proposed by Ma et al., reference: "Ma, X.; Ding, Z.; Luo, Z.; Chen, C.; Guo, Z.; Wong, DF; Feng, X.; and Sun, M. 2025. DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding. arXiv preprint arXiv:2503.12797." Method 13: Fine-R1 proposed by He et al., see reference "He, H.; Geng, Z.; and Peng, Y. 2026. Fine-R1: Make Multimodal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning." The benchmarks include: FGVC-Aircraft, a fine-grained dataset for aircraft model recognition; CUB-200-2011, a fine-grained dataset for bird recognition; Stanford Cars, a fine-grained dataset for car recognition; Stanford Dogs, a fine-grained dataset for dog recognition; Oxford 102 Flowers, a fine-grained dataset for flower recognition; and Oxford-IIITPets, a fine-grained dataset for pet recognition. For each dataset, 60% of the data is classified as seen categories and 40% as unseen categories. A single model is jointly trained across six domains, with four training images used for each seen category.
[0039] The closed-world evaluation report demonstrates the exact matching accuracy given candidate labels; the open-world evaluation requires the model to generate fine-grained categories without providing a list of candidate categories, and uses the SigLIP text encoder to measure the relative semantic similarity between the generated labels and the ground truth labels. The 3B and 7B models obtained using the method of this invention are denoted as CoCo-R1-3B and CoCo-R1-7B, respectively.
[0040] As shown in Tables 1 and 3, the method proposed in this invention achieves high overall accuracy in closed-world recognition on six fine-grained visual recognition benchmarks. The overall accuracy of CoCo-R1-7B is 95.32%, an improvement of 6.61 percentage points compared to Fine-R1-7B's 88.71%; the average accuracy for unseen categories improves from 85.70% to 93.16%. The overall accuracy of CoCo-R1-3B is 88.99%, higher than Fine-R1-3B's 85.19%, indicating that this invention achieves improvements at both the 3B and 7B model scales listed.
[0041] In the table, airplanes, birds, cars, dogs, flowers, and pets correspond to the six evaluation criteria mentioned above. The "average" in Tables 1 and 2 represents the average result of this method across the six domains under the corresponding category divisions; the overall averages for the closed world and the open world are listed in Table 3 for easy comparison. The row containing the method of this invention is shown in bold.
[0042] Table 1. Comparison of fine-grained recognition accuracy in closed worlds (%; categories seen)
[0043] Continued from Table 1: Comparison of Fine-Grained Recognition Accuracy in Closed Worlds (%; No Category Found)
[0044] As shown in Tables 2 and 3, in open-world identification, the overall relative semantic similarity of CoCo-R1-7B is 78.36%, an increase of 3.56 percentage points compared to Fine-R1-7B's 74.80%. The average relative semantic similarity for the unidentified category increased from 66.97% to 68.91%. The overall relative semantic similarity of CoCo-R1-3B and Fine-R1-3B are 74.09% and 67.32%, respectively, a difference of 6.77 percentage points. The results in the tables are reported according to the aforementioned open-world evaluation method and are not equivalent to exact matching results under a closed-world setting.
[0045] Table 2. Comparison of relative semantic similarity in open worlds (%; categories seen)
[0046] Table 2 (Continued) Comparison of relative semantic similarity in open worlds (%; No categories found)
[0047] Table 3. Overall average comparison between closed and open worlds (%)
[0048] Candidate calibration ablation experiments show that, while maintaining consistency in pedestal model, trajectory budget, supervised fine-tuning hyperparameters, and evaluation settings, replacing the candidates provided by the teacher model with pedestal strategy self-confusion candidates improves the results for unseen categories in the open world from 58.66% to 62.89%, a difference of 4.23 percentage points. This indicates that matching the candidate category with the actual confusion relationship of the pedestal strategy is helpful for identification.
[0049] Ablation experiments on the perceived reward of candidate recall showed that, under the matched TAPO, DAPO and GRPO optimizer settings, adding a candidate recall reward brought improvements of 1.75, 1.98 and 1.24 percentage points, respectively, indicating that candidate recall can serve as a trajectory-level reward signal independent of the optimizer update rules.
[0050] In supplementary experiments on the Qwen3-VL-4B backbone model, the accuracy of the closed world improved from 74.38% to 90.20%, and the accuracy of the open world improved from 51.23% to 74.44%, indicating that the present invention is also applicable under this backbone model and the listed evaluation settings.
[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A fine-grained visual recognition method based on confusion perception and candidate recall, characterized in that, Includes the following steps: S1. Construct a structured fine-grained reasoning format: Organize the fine-grained visual recognition process of the multimodal large language model into a four-stage structured thinking chain of "visual analysis - candidate sub-category - comparison - prediction"; S2, Self-Confusion Supervised Fine-Tuning: For the pre-trained multimodal large language model base policy, open-world prompts without candidate lists are used to sample the training images multiple times. The number of times each real category is predicted as another category by the base policy is counted. The directed confusion categories of each real category are mined, and the target category and its directed confusion categories are combined to form a candidate field. A self-confusion supervised fine-tuning trajectory is constructed, and the model is supervised and fine-tuned to obtain the first training model. S3. Candidate Recall Perception Reinforcement Learning: Using the first training model as the initial policy, a structured inference trajectory is generated, and the final prediction and the restricted candidate set are parsed out. The selection signal representing whether the final prediction is correct and the coverage signal representing whether the target category is retained in the restricted candidate set are calculated respectively. The correctness-dominated reward function is constructed with the selection signal as the main component and the coverage signal as the auxiliary component. This function is used for reinforcement learning policy updates to obtain the second training model. S4. Inference Application: Input the image to be identified and the identification question into the second training model, generate a four-segment structured inference trajectory and output the final fine-grained prediction result. During the inference stage, the true category label or reward of the image to be identified is not accessed.
2. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, In step S1, the four-segment structured thinking chain uses labels to annotate the four segments, with visual analysis, candidate subcategories, comparison, and prediction placed separately. <analysis>< / analysis> , <options>< / options> , <comparison>< / comparison> and <prediction>< / prediction> Within the tag, and ultimately with <answer>< / answer> The labels provide the prediction results.
3. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, In step S1, when parsing the candidate sub-category list, a category normalization mapping is used to map the valid complete fine-grained category names to standard category labels within the domain of the fine-grained visual recognition task. Invalid strings are discarded, the candidate list is parsed in the original order, duplicate labels are removed, and the first four valid candidate categories are retained as a restricted candidate set. .
4. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, In step S2, when mining targeted obfuscation categories, for the target category And another category Statistical base strategy In the real label The image is predicted as The number of times is calculated as follows: ; in, Representation of domain The training image set, Represents training images, label(x) This represents the true class label of the training image. This represents the prediction of the pedestal strategy for the image. This represents the directional confusion count from the target category to another category, with the vertical line on the right indicating the number of elements in the set; Select the one with the largest count One category as the target category The targeted obfuscation categories are calculated as follows: ; in, Represents the set of targeted obfuscation categories for the target category. Y d This represents a set of standard category labels within a domain. This indicates the number of obfuscation categories selected, and TopK represents the operation of selecting the aforementioned number of categories with the highest obfuscation count; the directional obfuscation categories have directionality. and Statistically count them separately, and they are not required to be equal.
5. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, In step S2, when constructing the self-confusing supervised fine-tuning trajectory, the target category is... Together with its targeted obfuscation category, candidate fields are constructed, and the calculation methods are as follows: ; in, C SC (y) A self-obfuscating candidate field representing the target category. This represents the set of targeted obfuscation categories for the target category. The set union operation is used to merge the target category with its obfuscation categories. The vertical bar indicates the number of categories in the candidate field, and the total number of candidate categories does not exceed four. Constructing a structured target trajectory Candidate budget and limited candidate set Maintain consistency and obtain the first training model through teacher-mandated supervision and fine-tuning; in, Visual evidence representing training images, This indicates the comparison between the target category and the obfuscated categories in the candidate field.
6. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, In step S3, each structured reasoning trajectory is processed. Define the selection signal and the coverage signal. The selection signal is used to measure whether the model has correctly selected the target category. The calculation methods are as follows: ; in, r acc (o) Indicates a selection signal. This represents the final prediction result after category normalization mapping. Category normalization mapping maps valid, complete, fine-grained category names to standard category labels within the domain of the fine-grained visual recognition task, and maps invalid strings to invalid symbols. Indicates the true category label, An indicator function that takes the value 1 if the condition is true and 0 otherwise; The coverage signal is used to measure whether the target category is still available during the candidate comparison stage, and is calculated as follows: ; in, r rec (o) Indicates coverage signal, This represents a limited set of candidates that retains the first four valid candidate categories. Indicates the true category label, This indicates an indicator function.
7. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, The correctness-driven reward function in step S3 is calculated as follows: ; in, R AR (o) Indicates trajectory-level reward. r acc (o) Indicates a selection signal. r rec (o) Indicates coverage signal, Represents the weight coefficient of the correctness sub-item and satisfies The reward for the trajectory with the correct final answer is strictly higher than that for the trajectory with the incorrect final answer. When the weight coefficient is less than 1 and the final answer of the two trajectories is the same, the reward is further distinguished by the candidate recall signal. When the weight coefficient is equal to 1, the reward is determined only by the correctness of the final answer.
8. The fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, The reward function in step S3 is a trajectory-level score rather than a policy update rule, which can be directly used by a trajectory-based reinforcement learning optimizer. The reinforcement learning optimizer includes any one of GRPO, DAPO, and TAPO, and does not need to access the real labels and rewards during the inference phase.
9. A fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, The multimodal large language model is an image-language or video-language large model, which adopts an architecture composed of a visual encoder, a projector and a large language model, including the Qwen2.5-VL series model; the fine-grained visual recognition task includes closed world recognition and open world recognition.
10. A fine-grained visual recognition method based on confusion perception and candidate recall according to claim 1, characterized in that, The correctness-driven reward function in step S3 combines a correctness sub-item weight of 0.85 and a candidate recall sub-item weight of 0.15, calculated as follows: ; in, R AR (o) Indicates trajectory-level reward. r acc (o) Indicates a selection signal. r rec (o) This indicates a coverage signal; candidate recall sub-items only take effect when the structured output is complete, the candidate list has no more than four items and they are not stacked.
