Error sample contrast learning enhancement-based reasoning method and system
By using a contrastive learning enhancement method, erroneous samples are transformed into negative constraint signals, which locate and correct erroneous inference fragments. This solves the problem of single training signals caused by the discarding of erroneous samples in existing technologies, and improves the inference stability and robustness of large language models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING KNOWLEDGE ATLAS TECHNOLOGY CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-15
AI Technical Summary
In the training process of existing large language models, erroneous samples are treated as noise or discarded directly, resulting in a single training signal and a lack of explicit identification and avoidance of erroneous inference paths, which affects the robustness and reliability of inference.
By using a contrastive learning enhancement method, erroneous samples are transformed into usable negative constraint signals. The same large language model is used to locate erroneous segments in the erroneous samples and generate corresponding correct segments. "Eroneous-correct" comparison samples are constructed, and cosine similarity matching is used to locate the erroneous landing point in the inference chain and apply contrastive learning constraints at that position.
It significantly improves the model's ability to identify and avoid erroneous reasoning patterns, enhances reasoning stability and robustness, and is suitable for continuous iterative training and engineering deployment on large-scale reasoning data.
Smart Images
Figure CN122047482A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology and relates to a reasoning method and system, particularly a reasoning method and system based on error sample contrastive learning enhancement. Background Technology
[0002] With the rapid development of Large Language Models (LLMs), their applications in mathematical reasoning, logical deduction, code generation, complex instruction execution, and reasoning-intensive generation tasks are becoming increasingly widespread. In these tasks, models not only need to output the final answer but are also often required to provide the intermediate reasoning process to enhance interpretability and verifiability. The stability, correctness, and generalization ability of the reasoning process directly determine the usability and reliability of the model in real-world scenarios.
[0003] However, in real-world data construction and model training, inference datasets inevitably contain a large number of samples with reasoning processes. These include samples where both the reasoning process and the final answer are correct, as well as samples where the reasoning process contains errors or where there are partial errors in the reasoning chain. Error samples may manifest as: calculation errors at an intermediate step, logical jumps, incorrect intermediate conclusions, improper derivation order, or subsequent chain errors caused by early biases. Faced with these error samples, existing training paradigms typically tend to "learn only correct samples," treating error samples as noise and eliminating or weakening them. While this strategy can avoid direct contamination of model training by obvious errors, it also results in the model lacking the ability to explicitly identify and avoid erroneous reasoning paths. Similar error patterns may still repeatedly occur during the reasoning process, affecting the robustness and reliability of the inference.
[0004] To address the aforementioned issues, existing research and engineering practices have attempted to improve the quality of inference training from perspectives such as data cleaning, preference learning, or comparative training. Currently, commonly used implementation schemes mainly fall into the following categories: 1. Inference data cleaning and filtering methods.
[0005] This type of method filters samples in the inference dataset through rules, weak supervision, or model scoring, retaining only those inference samples judged as correct or of high quality for supervised fine-tuning (SFT), while discarding or reducing the weight of erroneous samples. This method is simple to implement and has low risk, but it is essentially still "denoising learning," failing to fully utilize the counterexample information contained in erroneous inference samples. This can easily lead to a single training signal and a lack of targeted suppression of erroneous inference bifurcation points in the model.
[0006] 2. A positive and negative sample training method based on preference alignment.
[0007] Some methods construct "better answer / worse answer" preference pairs and employ preference learning or ranking training strategies to make the model more inclined to output better solutions. While this method can improve the quality of answers at the result level, in reasoning augmentation tasks, preference samples often focus on comparing the merits of the "final answer" or the "overall reasoning chain," lacking a fine-grained mechanism for locating and correcting local error segments in the reasoning chain, and making it difficult to directly suppress the recurrence of certain high-frequency erroneous ideas in the reasoning process.
[0008] 3. Error-correcting training method based on control samples.
[0009] Some methods attempt to pair erroneous samples with their corrected versions (or manually labeled correct versions), enabling the model to learn the transition from error to correctness through comparative learning or error-corrective fine-tuning. While these methods have some "error notebook" concept, existing implementations often lack a stable mechanism for "how erroneous fragments are aligned with the model's own inference state": the position of erroneous inference fragments in the model's generation chain may not be fixed, and their expression may also differ, making it difficult to accurately apply error correction signals at key points in the model's inference chain, resulting in unstable training effects.
[0010] Therefore, although the above methods improve the quality of inference training to some extent, they still have the following obvious shortcomings in the area of "mistake sample exploitation": (1) The training supervision signal is too simple and lacks a clear negative constraint mechanism.
[0011] Most methods still focus on "learning only correct samples," discarding incorrect samples or using them only for weak filtering. This results in the model mainly learning "what is correct," but lacking explicit learning signals for "what is wrong and which ideas should be avoided," making it difficult to form clear error avoidance boundaries.
[0012] (2) Lack of fragment-level error correction and localization mechanism for reasoning process.
[0013] Existing solutions often judge the quality at the overall sample level, but rarely locate, extract and correct key error segments in the reasoning process. This results in training being unable to effectively suppress local error patterns in the reasoning chain, especially in complex reasoning tasks where the phenomenon of "errors in the middle process but the model still outputs in a self-consistent manner" is prone to occur.
[0014] (3) Lack of a controllable training mechanism for “aligning ideas”.
[0015] There is positional drift and representational difference between erroneous reasoning fragments and model-generated reasoning chains. Existing methods often lack stable alignment means to determine "which position in the model reasoning chain corresponds to the erroneous idea landing point", making it difficult to accurately apply negative constraints to key reasoning states, which can easily lead to problems such as scattered error correction signals, misalignment, or training non-convergence.
[0016] (4) The value of the erroneous samples has not been systematically transformed into trainable comparison targets.
[0017] Error-reasoning samples naturally contain usable counterexample information, but existing technologies lack a comparative learning target (such as InfoNCE) that can be directly optimized by constructing "error fragment - correct fragment" into a comparative learning target, thus failing to achieve a clear optimization direction of "getting closer to the correct and staying away from the error" at the conceptual level.
[0018] In summary, existing inference training methods either over-rely on data cleaning, wasting information from erroneous samples, or lack precise mechanisms for locating local errors in the inference process and aligning thought processes, making it difficult to explicitly suppress the recurrence of erroneous inference patterns during the training phase. Therefore, there is an urgent need for a method that can transform erroneous inference processes into usable negative constraint signals without discarding erroneous samples, and then implement precise correction at key points in inference through thought alignment and contrastive learning. This would improve the inference stability, error avoidance ability, and overall robustness of large language models in complex inference tasks. Summary of the Invention
[0019] To overcome the shortcomings of existing technologies, this invention proposes a reasoning method and system based on error sample contrastive learning enhancement. It transforms error samples from noise in the traditional sense into usable negative constraint signals, enabling the model not only to know what is correct, but also to clearly identify what is wrong and the reasoning paths that should be avoided. This significantly improves reasoning stability, error avoidance ability, and overall robustness, and has high engineering practical value and promotion prospects.
[0020] To achieve the above objectives, the present invention provides the following technical solution: A reasoning method based on error sample contrastive learning enhancement, characterized by the following steps: S1: Obtain the inference dataset The inference dataset Each sample contains a question / instruction. Reasoning process text and the final answer ; S2: For each sample, extract its reasoning process text. Separate steps into sequences based solely on newline characters. ,in, The reasoning process text is described. The first in Line text, The reasoning process text is described. Total number of lines of text in the document; S3: For each sample, determine its question / instruction. Step sequence and the final answer The data is input into a large language model, which judges its correctness, and then processes the inference dataset based on the judgment results. Divided into correct sets and error set ; S4: For the error set Each sample in the dataset is input into a large language model, which then locates the erroneous lines and provides the correct way to write them. S5: For the error set For each sample in the sample, its question / instruction The input is fed into a large language model, which then generates its own reasoning process text. , It is the text describing one's own reasoning process. The first in Line text, It is the text describing one's own reasoning process. Total number of lines of text in the document; S6: Find the text of your own reasoning process. The row most similar to the erroneous row ; S7: During the update phase of the model to be optimized, comparative learning is used to make the model in the most similar row... Stay away from the wrong process and get closer to the right process.
[0021] Preferably, in step S6, the reasoning process text is found using cosine similarity. The row most similar to the erroneous row .
[0022] Preferably, step S6 specifically includes: S61: Combine the text of the erroneous line with the text of your own reasoning process. Each line of text in the document is represented as a vector: , , , In the formula, It is a vector representation of the text of the erroneous line; This is the text of the line containing the error; It is the text describing one's own reasoning process. The first in Vector representation of line text; It is the text describing one's own reasoning process. The first in Line text; S62: Calculate cosine similarity: , In the formula, yes and cosine similarity, yes transpose, yes and dot product, yes The length of the mold, yes The modulus length; S63: Select the row that is most similar to the erroneous row. : .
[0023] Preferably, in step S7, during the update phase of the model to be optimized, the loss used is: , In the formula, It is the temperature coefficient; It is an anchor point and ; It is the text describing one's own reasoning process. The first in A line is a vector representation of the text of the most similar line. It is a vector representation of the text of the erroneous line; It is the vector representation of the correct way to write the erroneous line.
[0024] Preferably, in step S4, the error set is... Each sample in the dataset is input into the large language model as a question / instruction. Reasoning process text and the final answer Or, for the question / instruction Step sequence and the final answer .
[0025] Preferably, it further includes: S8: Utilizing the correct set Each sample in the dataset is used to fine-tune the model to be optimized.
[0026] Preferably, in step S8, fine-tuning is performed using a supervised fine-tuning method.
[0027] Furthermore, the present invention also provides an inference system based on error sample contrastive learning enhancement, characterized in that it includes: The inference dataset acquisition module is used to acquire the inference dataset. The inference dataset Each sample contains a question / instruction. Reasoning process text and the final answer ; The inference process text module is used to segment the text by line breaks. It is used to extract the inference process text for each sample. Separate steps into sequences based solely on newline characters. ,in, The reasoning process text is described. The first in Line text, The reasoning process text is described. Total number of lines of text in the document; The sample correctness judgment module is used to judge the question / instruction of each sample. Step sequence and the final answer The data is input into a large language model, which judges its correctness, and then processes the inference dataset based on the judgment results. Divided into correct sets and error set ; Error sample processing module, which is used to process the error set Each sample in the dataset is input into a large language model, which then locates the erroneous lines and provides the correct way to write them. Error sample reasoning process generation module, which is used for the error set For each sample in the sample, its question / instruction The input is fed into a large language model, which then generates its own reasoning process text. , It is the text describing one's own reasoning process. The first in Line text, It is the text describing one's own reasoning process. Total number of lines of text in the document; The most similar line determination module is used to find the text of its own reasoning process. The row most similar to the erroneous row ; The contrastive learning correction module is used during the update phase of the model to be optimized to correct the model's behavior in the most similar rows through contrastive learning. Stay away from the wrong process and get closer to the right process.
[0028] Furthermore, the present invention also provides an inference device based on error sample contrastive learning enhancement, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the error sample contrastive learning-enhanced inference method as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the inference method based on error sample contrastive learning enhancement as described above.
[0029] Compared with existing methods that commonly employ "supervised fine-tuning (SFT) with only correct samples" or "directly filtering out incorrect samples as noise" for training inference data, the inference method and system proposed in this invention, based on error inference comparison and thought alignment, has significant advantages in data utilization efficiency, error avoidance capability, training signal refinement, and inference stability. These advantages are specifically reflected in the following aspects: (1) Make full use of error samples to overcome the limitations of “learning only the correct and filtering the error”.
[0030] Existing inference training schemes often focus on data cleaning, typically using only correct samples for fine-tuning or discarding incorrect samples to avoid contaminating the training. While this approach reduces the risk posed by incorrect samples, it also results in a single training supervision signal, causing the model to primarily learn "what is correct" without explicit constraints on "what is wrong and which inference paths should be avoided."
[0031] To address the aforementioned issues, this invention transforms error samples from traditionally perceived noise objects into usable negative training signals: by using the same large language model to locate error segments in error samples and generate corresponding correct segments, a "error-correct" comparison sample is constructed, enabling error samples to participate in training in the form of contrastive constraints. This fundamentally avoids the problem of wasting error samples and achieves joint enhancement of "learning correctly + avoiding errors".
[0032] (2) The thought alignment mechanism is more accurate, avoiding the dispersion and misapplication of error correction signals.
[0033] Existing training methods based on error correction or preference alignment often compare the merits of different samples at the overall sample level or directly replace the original answer with the corrected answer for training. They lack a stable mechanism for locating and aligning the "key error points" in the reasoning process. Since erroneous ideas may appear in different expressions and positions in the model generation chain, error correction signals are easily applied in a scattered manner, leading to unstable training results.
[0034] This invention ensures consistent reasoning granularity by requiring a large language model to generate a reasoning chain with the same number of lines as the original reasoning process after line breaks. Based on this, it uses a cosine similarity matching model to locate the position of the most similar segment in the reasoning chain to the erroneous segment, and applies contrastive learning constraints only at this key position. This achieves precise suppression of erroneous ideas and targeted reinforcement of correct ideas, significantly improving the effectiveness and controllability of error correction signals.
[0035] (3) Explicitly bring the correct answer closer and push the wrong answer further away to improve error avoidance ability and reasoning robustness.
[0036] Supervised fine-tuning based solely on correct samples can typically only improve a model’s ability to fit the correct inference template, but it is difficult to prevent the model from repeatedly heading to a certain type of error bifurcation point at key inference nodes. This is especially true in complex inference tasks, where problems such as error chain recurrence and local self-consistency but overall error are more likely to occur.
[0037] This invention constructs an anchor-positive-negative contrast relationship at the matched inference endpoints and employs contrastive learning loss such as InfoNCE. This ensures that if the model encounters an inference state highly similar to an erroneous segment during the inference process, it is explicitly pulled towards the correct segment and pushed away from the erroneous segment. This mechanism establishes clear decision boundaries at the "thought level," significantly enhancing the model's ability to identify and avoid erroneous inference patterns, thereby improving inference stability, robustness, and reliability.
[0038] (4) It is more suitable for large-scale inference data training and engineering iteration, improving data utilization and training efficiency.
[0039] Existing "error filtering" strategies often require additional data cleaning processes and manual verification costs. Furthermore, when the proportion of error samples is high or the data sources are complex, the amount of usable data can decrease significantly, affecting the upper limit of model training. Some error-correction training schemes also rely on manually labeled high-quality correction chains, which is difficult to scale.
[0040] This invention employs a single large language model to complete self-checking for correctness, error segment location, and correct segment generation, forming an automated error notebook-style training closed loop that can significantly improve the density of usable training signals without discarding data. Through a joint training method of "correct sample SFT + error sample comparison and correction," this invention maintains the stability of correct learning while introducing error comparison constraints at a low additional cost. It is suitable for continuous iterative training and engineering deployment of large-scale inference data, and has high practical value and promising prospects for promotion. Attached Figure Description
[0041] Figure 1 This is a flowchart of the inference method based on error sample contrastive learning enhancement of the present invention.
[0042] Figure 2 This is a schematic diagram of the inference system based on error sample contrastive learning enhancement according to the present invention.
[0043] Figure 3 This is a block diagram of the inference device based on error sample contrastive learning enhancement according to the present invention. Detailed Implementation
[0044] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof in this invention is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.
[0045] With the widespread application of Large Language Models (LLMs) in reasoning-intensive tasks such as mathematical reasoning, logical inference, code generation, and complex instruction execution, the improvement of model reasoning capabilities increasingly relies on continuous training and fine-tuning with high-quality reasoning datasets. However, during the process of collecting and constructing real-world data, reasoning data often contains both correct and incorrect reasoning samples: errors may occur at intermediate calculation steps, logical jumps, incorrect intermediate conclusions, or deviations in the reasoning path. If there is a lack of effective mechanisms to utilize or correct such erroneous reasoning, the model may still repeat similar erroneous ideas after training, exhibiting unstable reasoning, recurrence of error chains, and insufficient ability to avoid "error bifurcation points," thus affecting the final reliability and generalization ability of the reasoning.
[0046] To alleviate the above problems, existing technologies typically employ a "data cleaning / filtering" approach: first, the inference data is screened, retaining only samples deemed correct for supervised fine-tuning (SFT), or erroneous samples are discarded directly to avoid noise contamination. However, existing solutions generally suffer from the following technical limitations: (1) Error samples are directly discarded, resulting in a single training signal: Most methods only use correct reasoning samples for learning, so that the model mainly obtains positive supervision of "what is right", but lacks explicit constraints on "what is wrong and the reasoning path that should be avoided", making it difficult to form clear reasoning boundaries and error avoidance capabilities.
[0047] (2) Retaining correct samples alone cannot suppress the recurrence of incorrect thinking: Even if all training data is correct, the model may still fall into the same thinking branch as the historical error during inference (e.g., incorrect intermediate conclusions / incorrect derivation methods). Since no "error comparison signal" is introduced during training, the model lacks a targeted correction mechanism and is prone to systematic errors at key nodes.
[0048] (3) The exploitation of erroneous reasoning lacks a "thought-level alignment" mechanism: During the reasoning process, erroneous thought fragments may appear in different positions or have different forms of expression in different samples and different generation chains. Existing solutions have difficulty establishing a stable association between "erroneous fragments located in the dataset" and "the corresponding thought points in the model's own reasoning chain", thus making it impossible to accurately suppress and correct erroneous thought processes.
[0049] (4) The value of erroneous data has not been transformed into trainable negative constraints: Erroneous inference samples essentially contain "negative sample / counterexample" information, which can be used to guide the model away from erroneous inference patterns. However, existing training paradigms are mostly focused on "denoising" and lack a mechanism to systematically transform erroneous samples into comparative learning targets, resulting in insufficient data utilization efficiency.
[0050] To address the aforementioned technical bottlenecks, this invention proposes a reasoning method based on error-reasoning comparison and contrastive learning enhancement. Its core idea includes: (1) In the inference data construction stage, the same large language model is used to perform self-checking on the inference samples, and the samples are divided into correct sets and incorrect sets; the incorrect samples are further located to locate the incorrect segments in their inference process, and the correct writing corresponding to the incorrect segments is generated, thereby constructing a "correct segment - correct segment" comparison training signal. (2) In the model reasoning chain construction stage, the large language model is required to generate a reasoning chain with the same number of steps as the original reasoning process after line breaks for the same question, so as to ensure consistent reasoning granularity and facilitate alignment at the "thought fragment" scale. (3) In the thought alignment stage, cosine similarity is used to match the erroneous fragment with each step of the inference chain generated by the model, locate the landing point in the model inference chain that is most similar to the erroneous thought, and construct the anchor-positive-negative contrast learning sample at the location. (4) During the model update stage, contrastive learning loss such as InfoNCE is introduced so that when the model encounters a reasoning state that is highly similar to the erroneous fragment, it is explicitly pulled toward the corresponding correct fragment and pushed away from the erroneous fragment, thereby suppressing the erroneous reasoning mode and strengthening the correct reasoning mode at the level of thinking. At the same time, the correct samples are still used for positive learning using conventional SFT to achieve joint training of "learning the right + avoiding the wrong".
[0051] Through the above-mentioned solution, this invention can overcome the problems of existing inference training methods such as "only learning the correct ones, being unable to identify and avoid erroneous paths, wasting erroneous data, and lacking thought-level alignment and correction": it transforms erroneous samples from noise in the traditional sense into usable negative constraint signals, so that the model not only knows "what is correct", but also clearly understands "what is wrong and the inference paths that should be avoided", thereby significantly improving inference stability, error avoidance ability and overall robustness, and has high engineering practical value and promotion prospects.
[0052] Figure 1 A flowchart of the inference method based on error sample contrastive learning enhancement of the present invention is shown. Figure 1 As shown, the inference method based on error sample contrastive learning enhancement of the present invention includes the following steps: S1: Obtaining the inference dataset.
[0053] Obtain the inference dataset The inference dataset Each sample in the document contains a question and / or instructions. Reasoning process text and the final answer Each sample is stored uniformly as {Q, S, A}.
[0054] In this invention, the inference dataset It can be any existing inference dataset capable of training and optimizing large language models; this invention does not limit it.
[0055] S2: Divide the reasoning process text by line breaks.
[0056] For the inference dataset For each sample in the text, the reasoning process is described. Separate steps into sequences based solely on newline characters. .in, The reasoning process text is described. The first in Line text, The reasoning process text is described. The total number of lines of text in the document. That is, the text describing the reasoning process, bounded only by newline characters. It is divided into multiple steps, with each line considered as a reasoning step, and the number of lines used as the granularity of the reasoning chain.
[0057] Preferably, the text of the reasoning process can be removed. The blank lines in the sequence of steps, that is, the blank lines in the sequence of steps. The text does not include blank lines.
[0058] S3: Judgment of whether the sample is correct or incorrect.
[0059] For the inference dataset For each sample in the sample, its question / instruction Step sequence and the final answer The data is input into a large language model, which judges its correctness, and then processes the inference dataset based on the judgment results. Divided into correct sets and error set .
[0060] That is, for the aforementioned inference dataset For each sample in the dataset, a large language model is used to analyze the question / instruction. Step sequence and the final answer The system performs consistency and correctness checks, outputting a conclusion that the sample is "correct" or "incorrect," and divides the inference dataset accordingly. Samples that output "correct" are considered correct samples and are assigned to the correct set. Samples that output "error" are considered error samples and are included in the error set. .
[0061] In this invention, the correct set The correct samples in the dataset can be directly used for supervised fine-tuning of the model to be optimized, thereby enhancing its correct reasoning ability. The error set D... - Error samples are not discarded, but used for subsequent error localization and comparative learning to correct errors.
[0062] Many existing general-purpose language models can perform self-checking and grading output using appropriate prompts / templates without additional training. Therefore, the general-purpose language model used can be one with strong reasoning and evaluation / correction capabilities. Of course, to obtain better judgment results, the general-purpose language model can be further trained. This training is an existing technology and will not be described in detail here for simplicity.
[0063] S4: Error sample handling.
[0064] For the error set Each sample is input into a large language model, which then locates the erroneous lines and provides the correct way to write them.
[0065] In this invention, the input to the large language model is a question / instruction. Reasoning process text and the final answer It can also be a question / instruction Step sequence and the final answer The output is a wrong snippet and a correct snippet.
[0066] The error snippet (wrong_snippet) is a key erroneous line / inference fragment located from the reasoning steps of the text in the reasoning process of the error sample. It is used as a negative sample in contrastive learning (an erroneous line of thought that the model needs to avoid). The correct snippet (correct_snippet) is the corresponding correct writing / correction fragment generated for the error snippet (wrong_snippet). It is used as a positive sample in contrastive learning (a correct line of thought that the model needs to approach).
[0067] The large language model used is the same as the one used in step S3. Similarly, a general-purpose large language model with strong reasoning and evaluation / correction capabilities can be selected, and the self-checking and grading output can be completed through appropriate prompts / templates. Of course, in order to obtain better error location and grading results, the general-purpose large language model can also be further trained.
[0068] S5: Error sample inference process generation.
[0069] For the error set For each sample in the sample, its question / instruction The input is fed into a large language model, which then generates its own reasoning process text. , It is the text describing one's own reasoning process. The first in Line text, It is the text describing one's own reasoning process. The total number of lines of text in the document.
[0070] In this invention, the error set For each sample, only the question / instruction needs to be entered. The requirement is for the large language model to generate the reasoning chain, that is, its own reasoning process text: .
[0071] It should be noted that in this invention, appropriate prompts / templates are needed to ensure that the large language model outputs its own reasoning process text. It must be exactly the same as Line, that is, the text of the reasoning process with erroneous samples. The number of rows is consistent, and each row represents one step, so as to unify the granularity of reasoning and facilitate the alignment and correction of thought.
[0072] S6: Error sample inference process generation.
[0073] Find the text of your own reasoning process. The row most similar to the erroneous row That is, in the text of its own reasoning process In the text, find the line that most closely resembles the incorrect line in the text.
[0074] In this invention, the text describing one's reasoning process can be found through cosine similarity matching. The row most similar to the erroneous row And it specifically includes: 1. Vector representation.
[0075] The text of the erroneous line and the text of one's own reasoning process. Each line of text in the document is represented as a vector: , , .
[0076] In the formula, It is a vector representation of the text of the erroneous line; It is the text of the erroneous line, that is, the erroneous fragment; It is the text describing one's own reasoning process. The first in Vector representation of line text; It is the text describing one's own reasoning process. The first in Line text.
[0077] 2. Calculate the cosine similarity.
[0078] Calculate the vector representation of the text of the erroneous line. and the text of one's own reasoning process The first in Vector representation of line text Cosine similarity between them: .
[0079] In the formula, yes and cosine similarity, yes transpose, yes and dot product, yes The length of the mold, yes The length of the module.
[0080] 3. Select the most similar position.
[0081] The row most similar to the erroneous row is selected based on cosine similarity. : .
[0082] That is, the text of one's own reasoning process. The first in Vector representation of line text Vector representation of the text of the erroneous line The cosine similarity is the highest, that is, the text of its own reasoning process. The first in The "thought fragment" of the line is closest to the erroneous thought process in the line that is incorrect.
[0083] Therefore, in this invention, it is not required to describe one's own reasoning process in text. Text of the reasoning process with erroneous samples Instead of corresponding line by line at fixed positions, the text compares similarity in its own reasoning process. Find the text of the reasoning process related to the erroneous sample. The position of the reasoning step closest to the erroneous segment in the text is used as the correction point.
[0084] S7: Comparative learning and correction.
[0085] During the update phase of the model to be optimized, contrastive learning is used to make the model in the most similar row... The point of correction is to move away from the erroneous process and closer to the correct process.
[0086] Specifically, during the update phase of the model to be optimized, the loss used is: .
[0087] In the formula, It is the temperature coefficient, which controls the strength of the "gap".
[0088] For anchor points and ; It is the text describing one's own reasoning process. The first in A line is a vector representation of the text of the most similar line.
[0089] Negative samples and It is a vector representation of the text of the erroneous line, that is, .
[0090] It is a positive sample and It is the vector representation of the correct way to write the erroneous line, that is, .
[0091] At the point of correction, that is, in the text of one's own reasoning process. The first in Minimize this loss in the process. It will make Closer further away In other words, by minimizing the above loss. This causes the model to be optimized to encounter erroneous segments during the inference process. Highly similar reasoning states (i.e., falling near incorrect lines of thought) will be explicitly pulled towards the corresponding correct segment. At the same time, it was pushed away from the erroneous segment. This allows for the suppression of erroneous reasoning patterns and the strengthening of correct reasoning patterns at the "thinking level".
[0092] This invention, based on the idea of contrastive learning, transforms erroneous samples from noise or objects to be removed in traditional training into usable negative constraint signals. Even if there are erroneous reasoning processes in the inference samples, these samples can still serve as important training data to improve the reasoning ability of the large model to be optimized. This is because learning only correct samples can only let the model "know what is correct," but it is difficult for the model to "know which reasoning paths are wrong and should be avoided." By introducing contrastive learning between erroneous and correct segments, the large model to be optimized can form clearer decision boundaries and reasoning preferences, more stably avoid erroneous bifurcation points at key reasoning nodes, and thus improve the reliability and robustness of reasoning.
[0093] Furthermore, this invention may include: S8: Fine-tuning.
[0094] Using the correct set Each sample in the model is used to fine-tune the model to be optimized, for example, through supervised fine-tuning (SFT).
[0095] Therefore, this invention, through the joint training method of "correct sample SFT + incorrect sample contrast learning correction", maintains the stability of correct learning and introduces error control constraints at a low additional cost. It is suitable for continuous iterative training and engineering deployment of large-scale inference data and has high practical value and promotion prospects.
[0096] Figure 2 A schematic diagram illustrating the structure of the inference system based on error sample contrastive learning enhancement according to the present invention is shown. Figure 2 As shown, the inference system based on error sample contrastive learning enhancement of the present invention includes: 1. Inference Dataset Acquisition Module.
[0097] The inference dataset acquisition module is used to acquire the inference dataset. The inference dataset Each sample contains a question / instruction. Reasoning process text and the final answer .
[0098] 2. Divide the reasoning process text module by line breaks.
[0099] The text module for segmenting the reasoning process by line breaks is used to segment the reasoning process text for each sample. Separate steps into sequences based solely on newline characters. ,in, The reasoning process text is described. The first in Line text, The reasoning process text is described. The total number of lines of text in the document.
[0100] 3. Sample correctness judgment module.
[0101] The sample correctness judgment module is used to classify each sample and its question / instruction. Step sequence and the final answer The data is input into a large language model, which judges its correctness, and then processes the inference dataset based on the judgment results. Divided into correct sets and error set .
[0102] 4. Error Sample Handling Module.
[0103] The error sample processing module is used to process the error set. Each sample is input into a large language model, which then locates the erroneous lines and provides the correct way to write them.
[0104] 5. Error Sample Reasoning Process Generation Module.
[0105] The error sample reasoning process generation module is used to generate the error set. For each sample in the sample, its question / instruction The input is fed into a large language model, which then generates its own reasoning process text. , It is the text describing one's own reasoning process. The first in Line text, It is the text describing one's own reasoning process. The total number of lines of text in the document.
[0106] 6. The module for determining the most similar row.
[0107] The most similar line determination module is used to find the text of its own reasoning process. The row most similar to the erroneous row .
[0108] 7. Comparative learning and correction module.
[0109] The contrastive learning correction module is used during the update phase of the model to be optimized to correct the model's behavior in the most similar rows through contrastive learning. Stay away from the wrong process and get closer to the right process.
[0110] Preferably, the inference system based on error sample contrastive learning enhancement of the present invention further includes: 8. Fine-tuning module.
[0111] The fine-tuning module is used to utilize the correct set Each sample in the dataset is used to fine-tune the model to be optimized.
[0112] This invention can be widely applied to scenarios such as mathematical reasoning, logical inference, code generation, complex instruction execution, and intelligent agent reasoning chain construction based on large language models. It is especially suitable for training environments where it is difficult to guarantee the full correctness of reasoning data, but it is desirable to maximize data utilization and improve reasoning reliability.
[0113] By introducing a mechanism for comparing and learning to correct errors using erroneous samples, this invention can transform erroneous samples into important training signals for improving reasoning ability without discarding data. This enables large models to form clearer correct / erroneous boundaries and more stable reasoning preferences, thereby improving the reliability, consistency, and engineering practicality of the model in complex reasoning tasks. It has good value for large-scale training and promising prospects for industrial applications.
[0114] Furthermore, this invention also provides an inference device based on error sample contrastive learning enhancement. For example... Figure 3 As shown, the inference device based on error sample contrastive learning enhancement of the present invention includes: a memory 11 for storing one or more programs; one or more processors 12; when the one or more programs are executed by the one or more processors 12, the one or more processors 12 implement the inference method based on error sample contrastive learning enhancement of the present invention. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the inference method based on error sample contrastive learning enhancement in the present invention.
[0115] The computer-readable storage medium includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this invention, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0116] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention can be implemented in hardware, software modules executed by a processor, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.
[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. A reasoning method based on error sample contrastive learning enhancement, characterized in that, Includes the following steps: S1: Obtain the inference dataset The inference dataset Each sample contains a question / instruction. Reasoning process text and the final answer ; S2: For each sample, extract its reasoning process text. Separate steps into sequences based solely on newline characters. ,in, The reasoning process text is described. The first in Line text, The reasoning process text is described. Total number of lines of text in the document; S3: For each sample, determine its question / instruction. Step sequence and the final answer The data is input into a large language model, which judges its correctness, and then processes the inference dataset based on the judgment results. Divided into correct sets and error set ; S4: For the error set Each sample in the dataset is input into a large language model, which then locates the erroneous lines and provides the correct way to write them. S5: For the error set For each sample in the sample, its question / instruction The input is fed into a large language model, which then generates its own reasoning process text. , It is the text describing one's own reasoning process. The first in Line text, It is the text describing one's own reasoning process. Total number of lines of text in the document; S6: Find the text of your own reasoning process. The row most similar to the erroneous row ; S7: During the update phase of the model to be optimized, comparative learning is used to make the model in the most similar row... Stay away from the wrong process and get closer to the right process.
2. The inference method based on error sample contrastive learning enhancement according to claim 1, characterized in that, In step S6, the text of one's own reasoning process is found using cosine similarity. The row most similar to the erroneous row .
3. The inference method based on error sample contrastive learning enhancement according to claim 2, characterized in that, Step S6 specifically includes: S61: Combine the text of the erroneous line with the text of your own reasoning process. Each line of text in the document is represented as a vector: ， , ， In the formula, It is a vector representation of the text of the erroneous line; This is the text of the line containing the error; It is the text describing one's own reasoning process. The first in Vector representation of line text; It is the text describing one's own reasoning process. The first in Line text; S62: Calculate cosine similarity: ， In the formula, yes and cosine similarity, yes transpose, yes and dot product, yes The length of the mold, yes The modulus length; S63: Select the row that is most similar to the erroneous row. : 。 4. The inference method based on error sample contrastive learning enhancement according to claim 3, characterized in that, In step S7, during the update phase of the model to be optimized, the loss used is: ， In the formula, It is the temperature coefficient; It is an anchor point and ; It is the text describing one's own reasoning process. The first in A line is a vector representation of the text of the most similar line. It is a vector representation of the text of the erroneous line; It is the vector representation of the correct way to write the erroneous line.
5. The inference method based on error sample contrastive learning enhancement according to claim 1, characterized in that, In step S4, the error set Each sample in the dataset is input into the large language model as a question / instruction. Reasoning process text and the final answer Or, for the question / instruction Step sequence and the final answer .
6. The inference method based on error sample contrastive learning enhancement according to any one of claims 1-5, characterized in that, Further includes: S8: Utilizing the correct set Each sample in the dataset is used to fine-tune the model to be optimized.
7. The inference method based on error sample contrastive learning enhancement according to claim 6, characterized in that, In step S8, fine-tuning is performed using a supervised fine-tuning method.
8. A reasoning system based on error sample contrastive learning enhancement, characterized in that, include: The inference dataset acquisition module is used to acquire the inference dataset. The inference dataset Each sample contains a question / instruction. Reasoning process text and the final answer ; The inference process text module is used to segment the text by line breaks. It is used to extract the inference process text for each sample. Separate steps into sequences based solely on newline characters. ,in, The reasoning process text is described. The first in Line text, The reasoning process text is described. Total number of lines of text in the document; The sample correctness judgment module is used to judge the question / instruction of each sample. Step sequence and the final answer The data is input into a large language model, which judges its correctness, and then processes the inference dataset based on the judgment results. Divided into correct sets and error set ; Error sample processing module, which is used to process the error set Each sample in the dataset is input into a large language model, which then locates the erroneous lines and provides the correct way to write them. Error sample reasoning process generation module, which is used for the error set For each sample in the sample, its question / instruction The input is fed into a large language model, which then generates its own reasoning process text. , It is the text describing one's own reasoning process. The first in Line text, It is the text describing one's own reasoning process. Total number of lines of text in the document; The most similar line determination module is used to find the text of its own reasoning process. The row most similar to the erroneous row ; The contrastive learning correction module is used during the update phase of the model to be optimized to correct the model's behavior in the most similar rows through contrastive learning. Stay away from the wrong process and get closer to the right process.
9. An inference device based on error sample contrastive learning enhancement, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the inference method based on error sample contrastive learning enhancement as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the inference method based on error sample contrastive learning enhancement as described in any one of claims 1-7.