COT data labeling method
Through the multi-model automatic comparison and intelligent audit mechanism, the problems of limited user calls and unstable quality in the traditional COT data annotation process are solved, efficient and automated COT data annotation are achieved, and data quality and consistency are improved.
Patent Information
- Application Number
- CN202510529995.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-25
- Publication Date
- 2025-08-05
AI Technical Summary
The traditional COT data annotation process relies on a third-party open model, the number of user calls is limited, the data generation efficiency is low, the format adjustment is cumbersome, the quality is unstable, and it is difficult to obtain high-quality COT data.
The multi-model automatic comparison and intelligent audit mechanism are adopted to mark the COT question data through models such as DeepSeek-R1, ChatGPT and Qwen2.5-Math, and automatically split the tasks in parallel. Combined with the knowledge base and field adaptive optimization, dynamically adjust the model weight and confidence, check the format and problem-solving steps to determine the optimal annotation result.
It realizes the automation of COT data annotation, improves data quality and readability, reduces manual intervention, and improves data generation efficiency and consistency.
Smart Images
Figure CN120429641A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data annotation, and in particular to a COT data annotation method. Background Art
[0002] High-quality datasets play a crucial role in the training and optimization of artificial intelligence models. The Chain of Thought (COT) dataset is used to train and evaluate the model's reasoning capabilities, which is particularly crucial for tasks such as mathematical and logical reasoning. However, the traditional COT data annotation process relies on third-party open models. Most AI-generated services require membership privileges, limiting the number of daily calls available to ordinary users and impacting data collection efficiency. Data generation is inefficient, requiring manual input of prompt words and recording of information, a cumbersome and error-prone process. The presentation is also unfriendly, with some AI-generated content not being uniformly formatted, requiring additional formatting using tools like Markdown and LaTeX, adding significant workload. The quality of the results generated by different AI models varies widely, making it difficult to directly obtain high-quality COT data content. Summary of the Invention
[0003] In view of this, the purpose of the present invention is to provide a COT data labeling method to overcome the problems existing in the current prior art.
[0004] To achieve the above objectives, the present invention adopts the following technical solutions:
[0005] On the one hand, the present application provides a COT data annotation method, comprising:
[0006] Obtain COT topic data, and call multiple models to label the COT topic data;
[0007] Compare the annotation results of all the models and determine the optimal annotation result through an intelligent review mechanism.
[0008] Furthermore, in the above method, the multiple models include at least: DeepSeek-R1, ChatGPT and Qwen2.5-Math.
[0009] Furthermore, the method described above, wherein obtaining COT topic data and calling multiple models to label the COT topic data, includes:
[0010] Get COT question data;
[0011] Retrieving corresponding content in the knowledge base according to the COT topic data;
[0012] Based on the COT topic data and the retrieved corresponding content, multiple models are called to label the COT topic data.
[0013] Furthermore, the method described above, wherein the calling of multiple models to label the COT topic data based on the COT topic data and the retrieved corresponding content, includes:
[0014] Determining a COT topic category based on the COT topic data;
[0015] Determine the model to be called according to the COT question category;
[0016] According to the COT topic data and the retrieved corresponding content, the COT topic data is labeled by the model to be called.
[0017] Furthermore, the method described above, wherein the COT topic data and the retrieved corresponding content are labeled by the to-be-called model, includes:
[0018] Searching the historical best annotation result database based on the COT topic data;
[0019] If the best historical annotation result is not found, the annotation task of the COT topic data is split into multiple subtasks;
[0020] According to the COT topic data and the retrieved corresponding content, all the subtasks are labeled at the same time by calling multiple models to be called.
[0021] Furthermore, the method described above, wherein the comparison of the labeling results of all the models and the determination of the optimal labeling result through an intelligent review mechanism, includes:
[0022] Obtaining annotation results of all the models;
[0023] Performing formatting checks and problem-solving step integrity checks on all the annotation results, determining annotation results that fail the formatting checks or the problem-solving step integrity checks, and determining corresponding models;
[0024] The large language model automatically adjusts the prompt of the corresponding model according to the preset strategy, and calls the adjusted model for annotation to obtain the final annotation result;
[0025] Marking the annotation results that pass the formatting check and the problem-solving step integrity check as final annotation results;
[0026] Calculating the similarity of each of the final annotation results, and performing consistency judgment on the final annotation results based on the similarity;
[0027] According to the result of the consistency judgment, an optimal labeling result is determined among all the final labeling results.
[0028] Furthermore, in the above method, determining the optimal labeling result among all the final labeling results based on the result of the consistency judgment includes:
[0029] If the consistency judgment result of the final annotation result is inconsistent, determining the voting weight of each model corresponding to the final annotation result;
[0030] Determining the optimal annotation result in the final annotation results according to the voting weights;
[0031] If the consistency judgment result of the final annotation results is consistent, the credibility of each final annotation result is calculated, and each final annotation result is sorted according to the confidence weight through a weighted confidence algorithm, and the final annotation result with the highest credibility is determined as the optimal annotation result.
[0032] Furthermore, the above method further includes:
[0033] The voting weight and confidence weight of each final annotation result are dynamically adjusted according to the accuracy of the historical annotation results of the model corresponding to the final annotation result and the adaptability of the current COT question data task domain.
[0034] Furthermore, the above method further includes:
[0035] The optimal annotation result is self-checked by a large language model. After the self-check passes, the credibility scoring algorithm is used to evaluate whether to accept the current annotation result based on the historical annotation results and model performance of the model corresponding to the optimal annotation result;
[0036] If not accepted, the next final annotation result will be determined as the optimal annotation result according to the ranking.
[0037] Furthermore, the above method further includes:
[0038] A visual data management interface is introduced, through which the labeled data can be queried, filtered and modified.
[0039] The beneficial effects of the present invention are:
[0040] This application first obtains COT question data and uses multiple models to annotate the COT question data. It then compares the annotation results of all models and uses an intelligent review mechanism to determine the optimal annotation result. In this application, through automatic multi-model comparison, intelligent review, and dynamic optimization, COT data annotation is automated, improving data quality and readability. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0042] Figure 1 This is a flowchart provided by an embodiment of a COT data labeling method of the present invention. DETAILED DESCRIPTION
[0043] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.
[0044] Figure 1 This is a flow chart of an embodiment of a COT data annotation method provided by the present invention. Figure 1 , this embodiment may include the following steps:
[0045] S1. Obtain COT question data and call multiple models to label the COT question data;
[0046] S2. Compare the labeling results of all models and determine the optimal labeling result through an intelligent review mechanism.
[0047] Understandably, after users upload a COT problem, the system automatically calls on multiple models, including DeepSeek-R1, ChatGPT, and Qwen2.5-Math, to generate information such as the reasoning process, problem-solving steps, key knowledge points, and difficulty level. The system then automatically compares the results generated by these models and uses an intelligent review mechanism to automatically select the best-quality result.
[0048] Preferably, the multiple models include at least: DeepSeek-R1, ChatGPT and Qwen2.5-Math.
[0049] Preferably, step S1 includes:
[0050] Get COT question data;
[0051] Retrieve corresponding content in the knowledge base based on COT question data;
[0052] Based on the COT topic data and the retrieved corresponding content, multiple models are called to label the COT topic data.
[0053] Understandably, traditional models may generate content with logical loopholes or missing knowledge. We introduce a knowledge enhancement mechanism that allows the model to reference additional knowledge bases to improve annotation quality. Before annotation, relevant content is automatically retrieved from knowledge bases (such as mathematical theorem libraries and logical rule libraries). This retrieved content is then combined with optimized prompts to further enhance the accuracy and rigor of the model's generated results.
[0054] Preferably, based on the COT topic data and the retrieved corresponding content, multiple models are called to annotate the COT topic data, including:
[0055] Determine the COT topic category based on the COT topic data;
[0056] Determine the model to be called based on the COT question category;
[0057] According to the COT topic data and the corresponding content retrieved, the COT topic data is labeled through the model to be called.
[0058] Understandably, COT data spans diverse fields (such as mathematics, logical reasoning, physics, and economics), and different models perform differently in these areas. We introduced a domain-adaptive optimization algorithm to automatically assign the most appropriate model based on the problem category. When handling math problems, Qwen2.5-Math and DeepSeek-R1 are prioritized; when handling logical reasoning problems, ChatGPT is prioritized. By combining historical data, we continuously optimize the model assignment strategy to improve annotation quality and efficiency.
[0059] Preferably, based on the COT topic data and the retrieved corresponding content, the COT topic data is annotated by the model to be called, including:
[0060] Based on the COT question data, search the historical best annotation result database;
[0061] If the best historical annotation result is not found, the annotation task of the COT question data is split into multiple subtasks;
[0062] According to the COT topic data and the corresponding content retrieved, all subtasks are labeled at the same time by calling multiple waiting models.
[0063] Understandably, traditional COT data processing is often performed serially, resulting in limited speed. In this embodiment, a single task is automatically split into multiple subtasks, which are then processed in parallel using multithreading to improve efficiency. Simultaneously invoking multiple models for calculations reduces latency and increases throughput. For similar problems, the best-quality results from the past are cached to reduce repeated calculations.
[0064] Preferably, step S2 includes:
[0065] Get the annotation results of all models;
[0066] Perform formatting checks and problem-solving step integrity checks on all annotation results, identify annotation results that fail the formatting checks or problem-solving step integrity checks, and determine the corresponding models;
[0067] The large language model automatically adjusts the prompt of the corresponding model according to the preset strategy, and calls the adjusted model for annotation to obtain the final annotation result;
[0068] Mark the annotation results that pass the formatting check and the problem-solving step completeness check as the final annotation results;
[0069] Calculate the similarity of each final annotation result, and make consistency judgment on the final annotation result based on the similarity;
[0070] According to the consistency judgment result, the optimal annotation result is determined among all the final annotation results.
[0071] It is understandable that formatting checks specifically ensure the correct structure of LaTeX formulas, Markdown, tables, and code. Problem-solving step integrity checks specifically use rule matching or machine learning methods to determine whether reasoning steps are missing or incoherent. If the annotation results have defects such as incoherent reasoning steps or formatting errors, the large language model is used to automatically adjust the prompt according to a preset strategy, and the model is called again for content generation to further improve the quality of the annotation results. This adjustment allows for more precise annotation of key points in the problem, thereby improving the consistency and accuracy of the content generated by the model.
[0072] For annotation results generated by multiple models, we first evaluate their consistency by calculating similarity (e.g., cosine similarity, Jaccard similarity, etc.). If multiple models produce similar problem-solving processes and answers, the annotations are considered more reliable. The higher the consistency between the models, the more reliable the final result. Significant differences in the annotation results require further review.
[0073] Preferably, according to the consistency judgment result, the optimal annotation result is determined among all the final annotation results, including:
[0074] If the consistency judgment result of the final annotation result is inconsistent, the voting weight of the model corresponding to each final annotation result is determined;
[0075] According to the voting weight, the optimal annotation result is determined in the final annotation result;
[0076] If the consistency judgment result of the final annotation result is consistent, the credibility of each final annotation result is calculated, and each final annotation result is sorted according to the confidence weight through the weighted confidence algorithm, and the final annotation result with the highest credibility is determined as the optimal annotation result.
[0077] It is understood that for final annotation results that are consistent, the credibility of each final annotation result is evaluated based on historical data, model confidence, and answer consistency. A weighted confidence algorithm is used to sort multiple generated results by weight, and results with higher confidence are given priority.
[0078] For final annotation results with significant discrepancies, a voting mechanism is used to determine the final decision. The "voting weight" of each model is dynamically adjusted based on factors such as its historical performance, generation quality, and task domain adaptability. This dynamic voting mechanism effectively optimizes the accuracy of the final selection.
[0079] Preferably, it also includes:
[0080] The voting weight and confidence weight of each final annotation result are dynamically adjusted based on the accuracy of the historical annotation results of the model corresponding to the final annotation result and the adaptability of the current COT question data task domain.
[0081] As you can understand, after multiple votes and evaluations, each model's weight is dynamically adjusted based on its performance in different tasks. For example, if one model excels on math problems, while another performs better on logical reasoning, the system intelligently assigns the task to the most appropriate model, optimizing the quality of pre-labeling. Based on each model's success rate and accuracy, its frequency of use in future tasks is adjusted, ensuring that each model is used more frequently in its area of expertise, thereby improving overall labeling results.
[0082] Preferably, it also includes:
[0083] The optimal annotation result is self-checked through a large language model. After the self-check passes, the credibility scoring algorithm is used to evaluate whether to accept the current annotation result based on the historical annotation results and model performance of the model corresponding to the optimal annotation result.
[0084] If not accepted, the next final annotation result will be determined as the optimal annotation result according to the ranking.
[0085] It's understandable that the optimal annotation result is first self-checked using a large language model to verify the correctness and logical rigor of the reasoning process. Then, based on historical data and model performance, a credibility scoring algorithm is used to assess whether to accept the current result. If the current result is deemed unreliable, further optimization is initiated or another model result is selected.
[0086] Preferably, it also includes:
[0087] A visual data management interface is introduced, through which the labeled data can be queried, filtered and modified.
[0088] Understandably, traditional Excel storage of annotated data is not intuitive for querying and managing. This embodiment introduces a visual data management interface that supports querying, filtering, and modifying annotated data. It supports Markdown and LaTeX rendering, allowing you to preview formulas, reasoning processes, problem-solving steps, and more directly in the interface. It also statistically analyzes the quality of annotated data and provides optimization suggestions for model generation.
[0089] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.
[0090] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is at least two.
[0091] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.
[0092] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0093] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0094] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
[0095] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0096] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0097] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A COT data annotation method, characterized in that: include: Obtain COT topic data, and call multiple models to label the COT topic data; Compare the annotation results of all the models and determine the optimal annotation result through an intelligent review mechanism.
2. The method according to claim 1, characterized in that The multiple models include at least: DeepSeek-R1, ChatGPT and Qwen2.5-Math.
3. The method according to claim 2, characterized in that The obtaining of COT topic data and calling multiple models to label the COT topic data include: Get COT question data; Retrieving corresponding content in the knowledge base according to the COT topic data; Based on the COT topic data and the retrieved corresponding content, multiple models are called to label the COT topic data.
4. The method according to claim 3, characterized in that The method of calling multiple models to label the COT topic data based on the COT topic data and the retrieved corresponding content includes: Determining a COT topic category based on the COT topic data; Determine the model to be called according to the COT question category; According to the COT topic data and the retrieved corresponding content, the COT topic data is labeled by the to-be-called model.
5. The method according to claim 4, characterized in that The step of labeling the COT topic data by using the to-be-called model according to the COT topic data and the retrieved corresponding content includes: Searching the historical best annotation result database based on the COT topic data; If the best historical annotation result is not found, the annotation task of the COT topic data is split into multiple subtasks; According to the COT topic data and the retrieved corresponding content, all the subtasks are labeled at the same time by calling multiple models to be called.
6. The method according to claim 5, characterized in that Comparing the annotation results of all the models and determining the optimal annotation result through an intelligent review mechanism includes: Obtaining annotation results of all the models; Performing formatting checks and problem-solving step integrity checks on all the annotation results, determining annotation results that fail the formatting checks or the problem-solving step integrity checks, and determining corresponding models; The large language model automatically adjusts the prompt of the corresponding model according to the preset strategy, and calls the adjusted model for annotation to obtain the final annotation result; Marking the annotation results that pass the formatting check and the problem-solving step integrity check as final annotation results; Calculating the similarity of each of the final annotation results, and performing consistency judgment on the final annotation results based on the similarity; According to the result of the consistency judgment, an optimal labeling result is determined among all the final labeling results.
7. The method according to claim 6, characterized in that Determining the optimal labeling result among all the final labeling results based on the consistency judgment result includes: If the consistency judgment result of the final annotation result is inconsistent, determining the voting weight of the model corresponding to each final annotation result; Determining the optimal annotation result in the final annotation results according to the voting weights; If the consistency judgment result of the final annotation results is consistent, the credibility of each final annotation result is calculated, and each final annotation result is sorted according to the confidence weight through a weighted confidence algorithm, and the final annotation result with the highest credibility is determined as the optimal annotation result.
8. The method according to claim 7, characterized in that Also includes: The voting weight and confidence weight of each final annotation result are dynamically adjusted according to the accuracy of the historical annotation results of the model corresponding to the final annotation result and the adaptability of the current COT question data task domain.
9. The method according to claim 8, characterized in that Also includes: The optimal annotation result is self-checked by a large language model. After the self-check passes, the credibility scoring algorithm is used to evaluate whether to accept the current annotation result based on the historical annotation results and model performance of the model corresponding to the optimal annotation result; If not accepted, the next final annotation result will be determined as the optimal annotation result according to the ranking.
10. The method according to claim 9, characterized in that Also includes: A visual data management interface is introduced, through which the labeled data can be queried, filtered and modified.