A collaborative target detection and recognition method based on large and small models of thought chains

Abstract

The present invention belongs to the field of target detection and recognition technology, and specifically relates to a method for collaborative target detection and recognition based on large and small models of thought chains. In this invention, the small model is responsible for most of the easy-to-detect targets, reducing the computational pressure of the large model, and the large model is responsible for doubtful samples. By combining visual and language multimodal reasoning, the overall false detection and missed detection rates are reduced. Confidence assessment is performed through joint probability, and automatic screening and manual review of uncertain results are achieved to ensure the reliability of key results. The "pseudo-thinking chain + pseudo-label" method, with the help of the reasoning and labels generated by the model itself, reduces the dependence on manually labeled data, and only manually confirms low-confidence samples, narrowing the scope of human intervention and significantly saving labor costs. The introduction of phrase-level feature descriptors provides the model with more fine-grained semantic information, improving the ability to identify complex target attributes and states.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and specifically relates to a method for collaborative target detection and recognition based on large and small models of thought chains. Background Art

[0002] Existing object detection and recognition technologies typically use two types of models: lightweight small models (such as YOLO-World and Tiny-YOLO), which enable real-time, high-frequency inference. However, these models have limited capacity and expressiveness, and often perform poorly for small objects, occluded objects, or fine-grained differences between categories in complex scenes. Large, high-capacity multimodal models (such as CLIP, Flamingo, and BEiT-3) offer powerful vision-language joint understanding and reasoning capabilities, enabling more accurate detection and classification of difficult-to-distinguish objects. However, these models are computationally expensive and have high inference latency, making them unsuitable for large-scale, real-time deployment.

[0003] On the other hand, the current mainstream object detection labeling process is highly manual: It requires people to manually label detection boxes and categories in large numbers of images, which is costly and time-consuming. Furthermore, relying solely on automated model outputs fails to provide a complete "decision-making process" or "reasoning process," resulting in a lack of interpretability in detection results and hindering subsequent model optimization and security audits.

[0004] To improve the efficiency, robustness, and interpretability of the system, the industry has begun exploring various collaborative mechanisms, including "small model + large model," "automatic labeling + manual confirmation," and "Chain-of-Thought." However, most methods focus only on a single link and fail to achieve a balance among speed, accuracy, cost, and interpretability. Summary of the Invention

[0005] The purpose of the present invention is to provide a collaborative target detection and recognition method for large and small models based on thought chains, which can be real-time and efficient, accurate and reliable, with low labeling cost, continuous adaptation, explainable and easy to audit, and strong scalability.

[0006] The technical solutions adopted by the present invention are as follows:

[0007] A method for collaborative target detection and recognition based on large and small models of thought chains, comprising:

[0008] The small model performs high-frequency target detection on the input image, and directly outputs the target detection box coordinates and confidence scores for recognition results with confidence levels higher than a preset threshold. The questionable detection results with confidence levels lower than the threshold are transmitted to the multimodal large model for re-identification.

[0009] Re-identify questionable detection results through a multimodal large model;

[0010] The confidence score of the detection results is evaluated based on the joint probability of the output sequence of the multimodal large model. The target detection box coordinates and category information are retained for high-confidence detection results, and a manual verification interface is established for low-confidence detection results.

[0011] Based on the preset prompt word template, the high-confidence results of the multimodal large model are integrated with the manual verification results to generate a pseudo-thought chain;

[0012] The target detection box coordinates of the large model's high-confidence detection results are combined with the target detection box coordinates of the manual verification results to convert them into detection pseudo labels, and the pseudo thought chain is decomposed into phrase-level feature descriptors to construct category pseudo labels;

[0013] Integrate pseudo thought chains, detection pseudo labels, and category pseudo labels to construct a thought chain target detection dataset;

[0014] The MindChain object detection dataset is regularly used to fine-tune both small and large models.

[0015] In a preferred embodiment, the multimodal large model is a dedicated model fine-tuned for the target detection task, whose input is the image to be detected and the target category text, and the output is the target category text and the detection box coordinates.

[0016] In a preferred embodiment, the confidence score determination condition is:

[0017] When the confidence score satisfies S low ≤S≤S high When , it is determined to be an ambiguous and questionable detection result, that is, a low-confidence detection result;

[0018] When the confidence score satisfies S>S high When , it is determined to be a credible recognition result, that is, a high-confidence detection result;

[0019] Among them, S represents the confidence score, S low Expressed as the lower limit of the confidence score, S high The upper limit of the confidence score.

[0020] In a preferred embodiment, the confidence score of the joint probability evaluation detection result of the multimodal large model output sequence is calculated as follows:

[0021] ;

[0022] Among them, I represents the output image, X represents the input problem, Y represents the output sequence, i represents the number of the probability value of the multimodal large model output sequence token, N is the length of the multimodal large model output sequence, p(Y i) is the probability value of the i-th token in the multimodal large model output sequence.

[0023] In a preferred embodiment, the multimodal large model generates a prompt word template through a preset pseudo thought chain to generate a pseudo thought chain.

[0024] In a preferred embodiment, in the pseudo-thought chain generation step, a preset prompt word template is used to guide the multimodal large model distributed reasoning process.

[0025] In a preferred embodiment, the construction of detection pseudo labels and category pseudo labels is expressed as:

[0026] ;

[0027] Among them, D pseudo Represented as a pseudo-label set, I i Denoted as the i-th image, T i Denoted as the i-th phrase-level feature descriptor, D i Represented as the coordinate information of the i-th detection box.

[0028] In a preferred embodiment, the phrase-level feature descriptor is obtained by decomposing the pseudo thought chain guided by prompt words.

[0029] In a preferred embodiment, the thought chain target detection dataset includes pseudo thought chains, detection pseudo labels, and category pseudo labels.

[0030] The present invention also provides a system for detecting and identifying large and small models in a collaborative manner based on thought chains, which is used in the above-mentioned method for detecting and identifying large and small models in a collaborative manner based on thought chains, comprising:

[0031] The small model inference module performs high-frequency target detection on the input image through the small model. The recognition results with confidence levels above the preset threshold are directly output as the target detection box coordinates and confidence scores. The doubtful detection results with confidence levels below the threshold are transmitted to the multimodal large model for re-identification.

[0032] The large model reasoning module uses a multimodal large model to re-identify questionable detection results;

[0033] Large model confidence estimation: evaluates the confidence score of the detection result based on the joint probability of the multimodal large model output sequence. For high-confidence detection results, the target detection box coordinates and category information are retained, and a manual verification interface is established for low-confidence detection results.

[0034] The manual confirmation module receives the recognition results that need manual confirmation sent by the manual verification interface and confirms the recognition results through manual intervention;

[0035] The pseudo thought chain generation module, based on a preset prompt word template, fuses the high-confidence results of the multimodal large model with the manual verification results to generate a pseudo thought chain;

[0036] The pseudo-label collaborative generation module converts the target detection box coordinates of the large model's high-confidence detection results and the target detection box coordinates of the manual verification results into detection pseudo-labels, and decomposes the pseudo-thought chain into phrase-level feature descriptors to construct category pseudo-labels;

[0037] The dataset building module integrates pseudo thought chains, detection pseudo labels, and category pseudo labels to construct a thought chain target detection dataset;

[0038] The fine-tuning learning module regularly uses the Thinking Chain object detection dataset to fine-tune the small model and the large model.

[0039] The technical effects achieved by the present invention are:

[0040] In this invention, the small model is responsible for most of the easy-to-detect targets, reducing the computational pressure of the large model, and the large model is responsible for questionable samples. By combining visual and language multimodal reasoning, the overall false detection and missed detection rates are reduced, and confidence assessment is performed through joint probability to achieve automatic screening and manual review of uncertain results, ensuring the reliability of key results. The "pseudo-thinking chain + pseudo-label" method, with the help of reasoning and labels generated by the model itself, reduces the dependence on manually labeled data, only manually confirms low-confidence samples, narrows the scope of human intervention, and significantly saves labor costs. The introduction of phrase-level feature descriptors provides the model with finer-grained semantic information, improves the ability to identify complex target attributes and states, and the pseudo-thinking chain fully records the reasoning process of the model, providing a traceable "thinking" path for detection and recognition results, which is helpful for future audits and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method provided by the present invention;

[0042] Figure 2 This is a system module diagram provided by the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it constitute a separate or selective embodiment that is mutually exclusive of other embodiments.

[0046] Secondly, the present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention in detail, for the sake of convenience, the schematic diagrams are only examples and should not limit the scope of protection of the present invention.

[0047] Please see the attached Figure 1 As shown, a method for collaborative target detection and recognition based on large and small models of thought chains is provided, including:

[0048] S1. Perform high-frequency target detection on the input image using a small model. Recognition results with confidence levels above a preset threshold are directly output as target detection box coordinates and confidence scores. Doubtful detection results with confidence levels below the threshold are transmitted to the multimodal large model for re-identification.

[0049] S2, re-identify the questionable detection results through a multimodal large model;

[0050] S3. Evaluate the confidence score of the detection results based on the joint probability of the multimodal large model output sequence. For high-confidence detection results, retain the target detection box coordinates and category information, and establish a manual verification interface for low-confidence detection results.

[0051] S4. Based on the preset prompt word template, the high-confidence results of the multimodal large model are integrated with the manual verification results to generate a pseudo-thought chain;

[0052] S5. Convert the target detection box coordinates of the high-confidence detection results of the large model and the target detection box coordinates of the manual verification results into detection pseudo labels, and decompose the pseudo thought chain into phrase-level feature descriptors to construct category pseudo labels;

[0053] S6. Integrate pseudo-thought chains, detection pseudo-labels, and category pseudo-labels to construct a thought chain target detection dataset.

[0054] S7. Regularly use the Thinking Chain object detection dataset to fine-tune the small model and the large model.

[0055] As in steps S1 to S7 above, a lightweight YOLO-World small model is used to perform real-time and batch target detection on the input image. For each detection box output by the model, if the confidence is greater than the preset threshold, its coordinates and category results are directly retained. If the confidence is less than or equal to the threshold, the result is considered "doubtful" and is transferred to the multimodal large model. The doubtful area of ​​the small model (image fragment + preliminary category prediction of the small model) is input into the fine-tuned multimodal large model. The multimodal large model combines visual and language cues (target category text, problem description, etc.) to re-identify and output more accurate target detection box coordinates and category information. By calculating the joint probability of the output sequence of the multimodal large model, the re-identification results are reset. Confidence scoring: if the confidence of the multimodal large model is greater than the threshold, it is directly retained; if it is lower than the threshold, the detection result is reviewed by professionals through the manual confirmation module, and the high-confidence results of the multimodal large model and the results after manual confirmation are uniformly input into the prompt word template to guide the multimodal large model to "think step by step" and generate the corresponding reasoning process text, namely the "pseudo-thinking chain". This thinking chain records in detail how the model understands image information, how to judge categories and positions based on the context, etc., which is helpful for subsequent feature extraction and detection of pseudo-labels. The retained high-confidence target box coordinates and the manually confirmed box coordinates are converted into standard target detection annotation formats and category pseudo-labels. The text generated based on the pseudo-thinking chain is analyzed using vocabulary mapping and phrase-level feature analysis. Solution, extract the key "attribute_state_relationship" information into 3-5 ≤6-word descriptors as category feature labels for incremental learning, integrate the pseudo-thinking chain (complete reasoning process text), detection pseudo-labels and category pseudo-labels to form a "thinking chain target detection dataset". The dataset contains both visual positioning information and semantic features that can be used for incremental learning of the model, realizing multimodal and multi-level knowledge fusion. The above dataset is regularly put into fine-tuning training of small and large models (offline adjustment is possible). By continuously supplementing pseudo-labels and thinking chain information, the accuracy and robustness of the two models in target detection and recognition tasks are gradually improved. The small model is responsible for most of the easy-to-detect targets, reducing the calculation pressure of the large model, and the large model is responsible for storage. For suspicious samples, combined with visual and language multimodal reasoning, the overall false detection and missed detection rates are reduced. Confidence assessment is performed through joint probability, which enables automatic screening and manual review of uncertain results to ensure the reliability of key results. The "pseudo-thinking chain + pseudo-label" method uses the reasoning and labels generated by the model itself to reduce dependence on manually labeled data. Only low-confidence samples are manually confirmed, which reduces the scope of human intervention and significantly saves labor costs. The introduction of phrase-level feature descriptors provides the model with finer-grained semantic information and improves the ability to identify complex target attributes and states. The pseudo-thinking chain fully records the reasoning process of the model, providing a traceable "thinking" path for detection and recognition results, which is helpful for future audits and optimizations.

[0056] In a preferred embodiment, the multimodal large model is a dedicated model fine-tuned for the target detection task, whose input is the image to be detected and the target category text, and its output is the target category text and the detection box coordinates.

[0057] As mentioned above, the spatial feature map of the image to be detected is first extracted through a pre-trained convolution or visual Transformer network (such as ViT, ResNet, etc.), and the target category text (such as "broken insulator") is converted into a semantic vector through word embedding or text encoder (such as BERT, CLIP text branch, LLM embedding layer), and the image features and text features are fused in the spatial-semantic dimension. Common methods include: using the text vector as the query (Query) on the visual feature map and performing multi-head self-attention interaction; splicing the two features in the channel dimension or position dimension and sending them to the subsequent Transformer layer for joint encoding, based on the general multimodal pre-trained model (such as CLIP, BEIT-3, F Lamingo, etc.), and end-to-end fine-tuning is performed on the labeled object detection dataset. After fine-tuning, the model receives an image and one or more "candidate category" texts in the inference stage. The model decoder (usually a Transformer decoder or a specific detection head) outputs in parallel: category text, which is scored and the best matching category is selected based on the input category list; detection box coordinates, which regress and predict the position of each category in the image, given in the format of (x_center, y_center, width, height). Compared with the traditional separate feature extraction + detector architecture, the fine-tuned multimodal model completes feature fusion and detection in a single network, reducing manually designed intermediate modules and improving inference efficiency and accuracy.

[0058] In a preferred embodiment, the confidence score determination condition is:

[0059] When the confidence score satisfies S low ≤S≤S high When , it is determined to be an ambiguous and questionable detection result, that is, a low-confidence detection result;

[0060] When the confidence score satisfies S>S high When , it is determined to be a credible recognition result, that is, a high-confidence detection result;

[0061] Among them, S represents the confidence score, S low Expressed as the lower limit of the confidence score, S high The upper limit of the confidence score.

[0062] As mentioned above, for each test (regardless of small or large models), a confidence score S will be output, which represents the confidence of the model in the test result. When the confidence score meets S low ≤S≤Shigh When the confidence score satisfies S>S, it is judged as an ambiguous and questionable detection result, that is, a low-confidence detection result, which needs to be further re-identified / verified by a multimodal large model or manually. high When , the recognition result is determined to be credible, that is, the high-confidence detection result, which can be directly output and used for subsequent pseudo-label construction and dataset fine-tuning. low and S high The value depends on the specific data characteristics. Generally, the initial value is 0.4 or 0.6. This maximizes the use of the model's high-confidence judgment, improves overall processing efficiency, and concentrates precious computing power and human resources on the samples that most need verification to avoid waste.

[0063] In a preferred embodiment, the confidence score of the joint probability evaluation test result of the multimodal large model output sequence is calculated as follows:

[0064] ;

[0065] Among them, I represents the output image, X represents the input problem, Y represents the output sequence, i represents the number of the probability value of the multimodal large model output sequence token, N is the length of the multimodal large model output sequence, p(Y i ) is the probability value of the i-th token in the multimodal large model output sequence.

[0066] As mentioned above, when the multimodal large model answers the composite output of "target category text + detection box coordinates", it generates a sequence word by word (or token by token). For the i-th token, the model gives a conditional probability. By multiplying the conditional probabilities of all tokens, the "joint probability" of the entire output sequence is obtained. This value is used as the confidence score of the detection result, reflecting the model's overall confidence in the entire prediction sequence (category + coordinates). If a key token (such as a coordinate value or category label) is assigned an extremely low probability, the product will drop rapidly, thereby classifying the sample as "low confidence" or "doubtful". Conversely, if the probabilities of all tokens are high, it means that the model is relatively confident in all aspects, and the overall value will fall into the "high confidence" range. An example of a typical large model target detection confidence score calculation process is as follows:

[0067] Input: For [image] and question [“Where is the insulator? Broken or not?”];

[0068] Output: ["0.13 0.1 0.24 0.2 Broken insulator"]" & ["0.8 0.95 0.93 0.99 0.93"];

[0069] ;

[0070] By calculating token by token, we not only consider whether the category is correct, but also take into account the consistency of coordinate value generation, thus making the overall quality assessment of the test results more comprehensive and rigorous.

[0071] In a preferred embodiment, the multimodal large model generates a pseudo thought chain by generating a prompt word template through a preset pseudo thought chain.

[0072] As mentioned above, during the system initialization or fine-tuning phase, a "pseudo-thought chain generation template" is first compiled. This template contains step-by-step guidance for question-answer thinking. For example, "Please simulate a thoughtful expert and describe their detailed thinking process when answering the following questions, including how to analyze image features, how to combine prior knowledge, how to locate the target position, and how to determine the category." This template defines the format requirements of the thought chain, such as "step-by-step description," "include visual features," and "include spatial relationships." In actual operation, the high-confidence results of the small or large model (image fragments, category labels, coordinate information) are spliced ​​with the template text to form the final prompt for the large model:

[0073] Image: {cut or full image description};

[0074] Question: "Where is the insulator located? Broken or unbroken?";

[0075] Known answer: "Broken insulator, coordinates (x, y, w, h)";

[0076] In this way, the model not only sees the image and the answer, but also clearly knows to output the "thinking process". After receiving the combined prompt, the multimodal large model uses the self-attention mechanism and cross-modal fusion to gradually generate a "pseudo-thinking chain" text in the Transformer decoder. The output content is usually divided into several thinking steps, such as "first notice the ring structure of the insulator... then locate the upper left corner... finally determine that the crack width meets the damage characteristics..." The generated text can be further extracted through rules or simple NLP scripts. Key sentences are converted into subsequent phrase-level feature descriptors or pseudo-labels to complete the closed loop of annotated data construction. The preset template ensures that the output ideas are output according to the same framework each time the large model is called, reducing the parsing difficulty caused by the diversity of thought chain formats.

[0077] In a preferred embodiment, in the pseudo thought chain generation step, a preset prompt word template is used to guide the multimodal large model distributed reasoning process.

[0078] As mentioned above, during the system design phase, a set of prompt word templates covering all aspects of multimodal reasoning were compiled to clearly guide the model to "distributed reasoning" (step-by-step reasoning) when generating thought chains. An example is shown below:

[0079] 1) How to identify the visual features of the target;

[0080] 2) How to extract location information;

[0081] 3) How to judge the status based on the category text;

[0082] 4) How to integrate the above information to draw the final conclusion;

[0083] Combine the above template with the image to be processed, category prompt, coordinate information and other instance content to form a complete Prompt:

[0084] image: {image_id};

[0085] Category hint: "Broken Insulator";

[0086] Coordinates: (x, y, w, h);

[0087] Driven by the attention mechanism, the model fuses image and text features. According to the "four-step" requirement in the template, the model generates the internal thinking of each step in turn, and the output format is as follows:

[0088] "First, focus on the gray-brown ring structure..."

[0089] "Then locate the upper left quadrant of the image..."

[0090] "Then, combine the concept of 'damage' to determine the shape of the crack..."

[0091] “The area was eventually identified as a broken insulator…”

[0092] The model calculates attention weights in a distributed manner according to the steps, with each step focusing on a different subtask to avoid missing context caused by one-time generation. Each reasoning step can be independently verified and extracted, providing clear boundaries for subsequent phrase feature decomposition.

[0093] A typical input and output example is as follows:

[0094] Input: Given an image and the question "Where is the insulator? Is it broken or not?", and the answer "0.13 0.1 0.24 0.2 Broken insulator", please simulate a smart and thoughtful person and give a detailed thought process when answering this question, including how to understand the information in the image, how to combine the information in the image to understand the problem, how to recall relevant knowledge, how to organize the answer, etc., to form a complete thought chain.

[0095] Output: The insulator in the picture is dark brown and consists of multiple stacked ring structures. It is located in the upper left corner of the image and has obvious white cracks on the surface, indicating damage.

[0096] In a preferred embodiment, the construction of detection pseudo labels and category pseudo labels is expressed as:

[0097] ;

[0098] Among them, D pseudo Represented as a pseudo-label set, I i Denoted as the i-th image, T i Denoted as the i-th phrase-level feature descriptor, D i Represented as the coordinate information of the i-th detection box;

[0099] D i An example is as follows:

[0100] ;

[0101] T i An example is as follows:

[0102] ;

[0103] Above, define the pseudo label set , n represents the number of images in the data. Using the pseudo-thinking chain generated by the preset template, phrases in the format of "attribute_state_relationship" (3-5, each ≤ 6 words) are extracted through regularized parsing or lightweight NLP modules. These phrases contain both visual features (such as "dark brown cracks") and spatial relationships (such as "upper left quadrant"), which are used to guide the model to learn the correspondence between semantics and positioning. The high-confidence output from the small model or large model and the manually confirmed detection boxes are uniformly converted into the format required by YOLO-World. The phrase descriptor provides fine-grained semantic guidance for the model, and the detection box provides precise positioning supervision for the model, ultimately forming a multimodal training sample that integrates "visual positioning + semantic features".

[0104] In a preferred embodiment, the phrase-level feature descriptor is obtained by decomposing the pseudo thought chain guided by prompt words.

[0105] As mentioned above, a set of "Phrase Decomposition" prompt word templates are pre-written to explicitly require the model to decompose complete pseudo-thought chains into phrases in the "attribute_state_relationship" format. For example, "You are a multimodal feature deconstructor and need to convert pseudo-thought chains into phrase-level features suitable for incremental learning of the object detection model."

[0106] Process according to the following rules:

[0107] Output requirements:

[0108] - Generate 3-5 phrases, each ≤ 6 words, conforming to the [attribute_state_relationship] structure;

[0109] -Must include visual features (color / texture) and spatial features (coordinates / pose);

[0110] - Phrases need to be mapped to the YOLO-World open vocabulary, and new words need to be added with the "extension tag";

[0111] The pseudo thought chain (multi-step reasoning text) generated by the large model is spliced ​​with the above template to form the final prompt for the large model. Based on the prompt, the multimodal large model automatically extracts and outputs several phrases using self-attention and cross-modal fusion mechanisms, such as:

[0112] "dark brown_with cracks_upper left quadrant";

[0113] "ring structure_stacked_near center";

[0114] "white crack_3px wide_along the ring seam";

[0115] The output phrase list is the phrase-level feature descriptor of the image, which can be directly mapped to the YOLO-World open vocabulary or marked as "extended tag" to support new concepts. Each phrase has a maximum of six words and a unified structure. It can be directly used for incremental fine-tuning of small models and quickly adapt to new scenarios and new categories.

[0116] In a preferred embodiment, the thought chain target detection dataset includes pseudo thought chains, detection pseudo labels, and category pseudo labels.

[0117] As mentioned above, each sample records the reasoning process text generated by the original large model under the guidance of preset prompt words. During training, it can be used as a "self-supervisory" signal to guide the model to learn the internal logical path from image features to category judgment, and convert the high-confidence (or manually confirmed) detection box coordinates and categories into a standard target detection annotation format (such as "class, x_center, y_center, width, height"), which is directly used as the model's positioning and classification supervision signal for training the detection head. The phrase-level feature descriptor ("attribute_state_relationship" format) obtained by the pseudo-thinking chain through the prompt word template decomposition serves as additional semantic supervision, allowing the model to learn not only "what this box is" but also "why it is", and strengthen the category feature representation in the multimodal fusion layer. In a single iteration, the detection head uses the detection pseudo-label for position and category prediction, and the fusion module or auxiliary language head uses the phrase description for semantic alignment. The self-supervised loss can use the pseudo-thinking chain text for additional contrastive learning or regression tasks. A typical example is as follows:

[0118] Data i=I i &C i &D i &T i ;

[0119] Among them, Data i It is represented as a thought chain target detection dataset, C represents the detection category, such as "broken insulator", D represents the detection box coordinates, such as "0.1 0.15 0.2 0.25", and T represents the thought chain, such as "the insulator in the picture is dark brown, composed of multiple stacked ring structures, located in the upper left position of the image, with obvious white cracks on the surface, and is judged to be broken."

[0120] Please see the attached Figure 2 As shown, the present invention also provides a system for collaborative target detection and recognition of large and small models based on thought chain, which is used in the above-mentioned collaborative target detection and recognition method of large and small models based on thought chain, comprising:

[0121] The small model inference module performs high-frequency target detection on the input image through the small model. The recognition results with confidence levels above the preset threshold are directly output as the target detection box coordinates and confidence scores. The doubtful detection results with confidence levels below the threshold are transmitted to the multimodal large model for re-identification.

[0122] The large model reasoning module uses a multimodal large model to re-identify questionable detection results;

[0123] Large model confidence estimation: evaluates the confidence score of the detection result based on the joint probability of the multimodal large model output sequence. For high-confidence detection results, the target detection box coordinates and category information are retained, and a manual verification interface is established for low-confidence detection results.

[0124] The manual confirmation module receives the recognition results that need manual confirmation sent by the manual verification interface and confirms the recognition results through manual intervention;

[0125] The pseudo thought chain generation module, based on a preset prompt word template, fuses the high-confidence results of the multimodal large model with the manual verification results to generate a pseudo thought chain;

[0126] The pseudo-label collaborative generation module converts the target detection box coordinates of the large model's high-confidence detection results and the target detection box coordinates of the manual verification results into detection pseudo-labels, and decomposes the pseudo-thought chain into phrase-level feature descriptors to construct category pseudo-labels;

[0127] The dataset building module integrates pseudo thought chains, detection pseudo labels, and category pseudo labels to construct a thought chain target detection dataset;

[0128] The fine-tuning learning module regularly uses the Thinking Chain object detection dataset to fine-tune the small model and the large model.

[0129] As mentioned above, the small model reasoning module deploys a lightweight YOLO-World small model to perform high-speed and frequent preliminary detection on the original image. If the confidence is higher than the preset threshold, the target detection box coordinates and confidence score are directly output. If the confidence is lower than the threshold, the doubtful detection result is transmitted to the multimodal large model for re-identification. The large model reasoning module deploys a multimodal large model to receive doubtful image fragments and category prompts, and re-predicts the target box and category through visual-language joint encoding and decoding to make up for the shortcomings of the small model in complex scenes. The large model confidence estimation is The output sequence generated by the large model calculates the joint probability, which is used to measure the credibility of the entire "category + coordinate" sequence. If the confidence score is higher than the preset threshold, it is judged as high confidence and enters the pseudo-label generation. If the confidence score is lower than the threshold, it triggers manual confirmation. The manual confirmation module provides a set of manual verification interfaces for low-confidence results, manually reviews and corrects the test results to ensure the accuracy of key samples. The pseudo-thinking chain generation module inputs the high-confidence results of the large model and the manual confirmation results into the preset prompt word template, calls the large model to generate a pseudo-thinking chain, and records the step-by-step reasoning process. The pseudo-label collaborative generation module integrates high-confidence and manually confirmed box coordinates and categories, outputs standardized detection pseudo-labels, and decomposes pseudo-thought chains into phrase-level feature descriptors using prompt words to form category-level semantic labels. The dataset construction module integrates pseudo-thought chains, detection pseudo-labels, and category pseudo-labels into a thought chain target detection dataset for subsequent model training. The fine-tuning learning module regularly uses the above dataset to incrementally fine-tune the small and large models, allowing them to continuously adapt to new samples and new scenarios, continuously improving detection and recognition performance. The small model undertakes the vast majority of simple detection tasks to ensure high throughput; the large model and manual processing only handle difficult samples, which not only improves overall accuracy but also avoids unnecessary resource consumption. Resetting the confidence threshold and manual verification mechanism ensures that only highly confident results are automatically included in training and application, significantly reducing false detections and missed detections. Training data is automatically constructed through pseudo-thought chains and pseudo-labels, and manual intervention is only performed on low-confidence samples, significantly saving labeling time and labor costs. The pseudo-thought chain records the model's reasoning logic at every step in detail, providing a complete "thinking log" for system auditing, troubleshooting, and security compliance.

[0130] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.

Claims

1. A method for collaborative target detection and recognition based on large and small models of thought chains, characterized in that: include: The small model performs high-frequency target detection on the input image, and directly outputs the target detection box coordinates and confidence scores for recognition results with confidence levels higher than a preset threshold. The questionable detection results with confidence levels lower than the threshold are transmitted to the multimodal large model for re-identification. Re-identify questionable detection results through a multimodal large model; The confidence score of the detection results is evaluated based on the joint probability of the output sequence of the multimodal large model. The target detection box coordinates and category information are retained for high-confidence detection results, and a manual verification interface is established for low-confidence detection results. Based on the preset prompt word template, the high-confidence results of the multimodal large model are integrated with the manual verification results to generate a pseudo-thought chain; The target detection box coordinates of the large model's high-confidence detection results are combined with the target detection box coordinates of the manual verification results to convert them into detection pseudo labels, and the pseudo thought chain is decomposed into phrase-level feature descriptors to construct category pseudo labels; Integrate pseudo thought chains, detection pseudo labels, and category pseudo labels to construct a thought chain target detection dataset; The MindChain object detection dataset is regularly used to fine-tune both small and large models.

2. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 1 is characterized in that: The multimodal large model is a special model fine-tuned for the target detection task. Its input is the image to be detected and the target category text, and its output is the target category text and the detection box coordinates.

3. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 1 is characterized in that: The confidence score determination conditions are: When the confidence score satisfies S low ≤S≤S high When , it is determined to be an ambiguous and questionable detection result, that is, a low-confidence detection result; When the confidence score satisfies S>S high When , it is determined to be a credible recognition result, that is, a high-confidence detection result; Among them, S represents the confidence score, S low Expressed as the lower limit of the confidence score, S high The upper limit of the confidence score.

4. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 3 is characterized in that: The calculation formula for the confidence score of the joint probability evaluation detection result of the multimodal large model output sequence is: ； Among them, I represents the output image, X represents the input problem, Y represents the output sequence, i represents the number of the probability value of the multimodal large model output sequence token, N is the length of the multimodal large model output sequence, p(Y i ) is the probability value of the i-th token in the multimodal large model output sequence.

5. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 1 is characterized in that: The multimodal large model generates prompt word templates through preset pseudo-thought chains to generate pseudo-thought chains.

6. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 5 is characterized in that: In the pseudo-thought chain generation step, a preset prompt word template is used to guide the distributed reasoning process of the multimodal large model.

7. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 1 is characterized in that: The construction of detection pseudo labels and category pseudo labels is expressed as: ； Among them, D pseudo Represented as a pseudo-label set, I i Denoted as the i-th image, T i Denoted as the i-th phrase-level feature descriptor, D i Represented as the coordinate information of the i-th detection box.

8. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 7 is characterized in that: The phrase-level feature descriptor is obtained by decomposing the pseudo-thought chain guided by prompt words.

9. The method for collaborative target detection and recognition based on large and small models of thought chains according to claim 1 is characterized in that: The thought chain target detection dataset includes pseudo thought chains, detection pseudo labels, and category pseudo labels.

10. A system for collaborative target detection and recognition based on large and small models of thought chains, applied to the method for collaborative target detection and recognition based on large and small models of thought chains according to any one of claims 1 to 9, characterized in that: include: The small model inference module performs high-frequency target detection on the input image through the small model. The recognition results with confidence levels above the preset threshold are directly output as the target detection box coordinates and confidence scores. The doubtful detection results with confidence levels below the threshold are transmitted to the multimodal large model for re-identification. The large model reasoning module uses a multimodal large model to re-identify questionable detection results; Large model confidence estimation: evaluates the confidence score of the detection result based on the joint probability of the multimodal large model output sequence. For high-confidence detection results, the target detection box coordinates and category information are retained, and a manual verification interface is established for low-confidence detection results. The manual confirmation module receives the recognition results that need manual confirmation sent by the manual verification interface and confirms the recognition results through manual intervention; The pseudo thought chain generation module, based on a preset prompt word template, fuses the high-confidence results of the multimodal large model with the manual verification results to generate a pseudo thought chain; The pseudo-label collaborative generation module converts the target detection box coordinates of the large model's high-confidence detection results and the target detection box coordinates of the manual verification results into detection pseudo-labels, and decomposes the pseudo-thought chain into phrase-level feature descriptors to construct category pseudo-labels; The dataset building module integrates pseudo thought chains, detection pseudo labels, and category pseudo labels to construct a thought chain target detection dataset; The fine-tuning learning module regularly uses the Thinking Chain object detection dataset to fine-tune the small model and the large model.

Images