A large language model knowledge correction method based on adaptive contrast learning
By dividing the knowledge quadrant of a large language model using an adaptive contrastive learning method and setting thresholds, contrastive learning data is constructed. This solves the problem of insufficient model handling of known but uncertain issues, improves the honesty and credibility of the model, and reduces hallucination phenomena. It has important applications, especially in the medical and legal fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL
- Filing Date
- 2025-01-07
- Publication Date
- 2026-04-14
AI Technical Summary
Existing fine-tuning methods are insufficient in handling large language models that have known but are uncertain issues, causing the model to tend to respond with a rejection response, affecting the model's honesty and credibility, which may have serious consequences, especially in critical fields such as medicine and law.
An adaptive contrastive learning method is adopted to divide the knowledge of the large language model into four quadrants, set upper and lower thresholds to identify the knowledge boundary of the model, construct contrastive learning data, and enhance the model's ability to master knowledge and maintain its generative ability by bringing positive samples closer and pushing negative samples further away.
It significantly improves the honesty and credibility of the model, enabling more accurate identification and processing of the model's knowledge state, reducing illusion phenomena, and improving the performance of the response authenticity index.
Smart Images
Figure CN119808952B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to artificial intelligence natural language processing technology, and in particular to a knowledge correction method for large language models based on adaptive contrastive learning. Background Technology
[0002] In the field of Natural Language Processing (NLP), Large Language Models (LLMs) have achieved breakthroughs in various downstream tasks such as machine translation, text generation, and question answering by learning rich linguistic statistical features and semantic information through pre-training on massive amounts of text data. The success of these models is mainly attributed to their large parameter scale and efficient utilization of massive amounts of data. However, LLMs may generate content that is inconsistent with reality or contextual logic when generating text; this phenomenon is called "illusion," as shown in Table 1. Illusion not only affects user experience but may also lead to serious consequences in critical fields such as medicine and law. LLMs learn a wealth of knowledge from human society and perform various tasks based on this knowledge. Among mainstream techniques, Supervised Finetuning (SFT) aligns LLMs with real-world human knowledge, playing a crucial role in achieving tasks expected by humans. After collecting a large amount of labeled corpus, researchers fine-tune pre-trained language models to enable them to learn from carefully selected and labeled human knowledge and apply it to unseen tasks. In the SFT process, model performance can be improved by increasing the scale of fine-tuning data and collecting complex data across multiple domains and tasks, but this may sacrifice the model's generality by focusing on domain-specific fine-tuning. In addition to diverse data, instruction fine-tuning methods design various instruction-response pairs, enabling the model to learn response patterns and content that better align with human instruction expectations, making LLM more realistic, believable, and useful.
[0003] Table 1 Examples of "illusions" generated by the large language model
[0004]
[0005] One approach is to enhance honesty by mixing rejection responses into the fine-tuning data during the instruction fine-tuning phase. Illusions typically involve logical fallacies (model inference errors), factual errors (the model confidently asserts non-existent facts), and data-driven bias (dominant data can cause model output to be skewed in certain directions). Other research has also explored the issue of model honesty, suggesting that a model's credibility can be improved when it can honestly answer "I don't know." For example, Cheng et al., during the instruction fine-tuning phase, identified the difference between the knowledge in the instruction fine-tuning data and the model's internal stored knowledge by repeatedly sampling and setting thresholds. They then instructed the model to reject responses to questions outside its internal stored knowledge, divided the data into four quadrants, and proposed a novel honesty assessment metric based on the question's position within each quadrant.
[0006] However, the existing technology has the following areas for improvement: the current fine-tuning method does not adequately handle some problems that were originally known but are uncertain when using SFT, which causes the fine-tuned model to tend to respond with a rejection response when answering these uncertain responses.
[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0008] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a knowledge correction method for large language models based on adaptive contrastive learning.
[0009] To achieve the above objectives, the present invention adopts the following technical solution:
[0010] A knowledge correction method for large language models based on adaptive contrastive learning includes the following steps:
[0011] S1. Knowledge Quadrant Division: The knowledge of the large language model is divided into four quadrants, including model knows what it knows, model does not know what it knows, model knows what it does not know, and model does not know what it does not know, in order to identify the model's mastery of knowledge;
[0012] S2. Knowledge Boundary Detection: Based on knowledge quadrant division, the knowledge boundary of the model is detected by setting an upper threshold for identifying the model's high confidence in its knowledge and a lower threshold for identifying the model's lack of confidence in its knowledge, so as to determine the model's confidence measure for each problem and thus identify the model's knowledge boundary.
[0013] S3. Construction of contrastive learning data: Based on the knowledge boundary of the probe, collect knowledge data that the model knows and does not know, and construct contrastive learning data, including positive samples and negative samples;
[0014] S4. Adaptive Contrastive Learning: Based on the construction of contrastive learning data, an adaptive contrastive learning strategy is adopted for model learning. The adaptive contrastive learning strategy enhances the model's ability to master knowledge by bringing positive samples closer and pushing negative samples further away, while maintaining the model's generative ability.
[0015] Furthermore, step S2 specifically includes:
[0016] Filter questions from different question-answering datasets;
[0017] For each question, the response of the large language model is sampled multiple times;
[0018] Calculate the accuracy of the response to each question, and use this as a measure of the model's confidence in each question;
[0019] Set upper and lower threshold values as hyperparameters;
[0020] When a model’s confidence in a problem exceeds an upper limit threshold, it is considered that the model has a complete grasp of the knowledge, and is defined as the model knowing the knowledge it knows.
[0021] When a model’s confidence in a problem falls between the upper and lower thresholds, it is defined as a model not knowing the knowledge it possesses.
[0022] When the model's confidence in a problem is below the lower limit threshold, it is considered that the model lacks this knowledge and is defined as the model not knowing the knowledge it does not know.
[0023] The knowledge boundary of the model is defined through the evaluation of the above thresholds and confidence metrics.
[0024] Furthermore, step S3 specifically includes:
[0025] Based on the knowledge boundaries determined in step S2, the problems are divided into two categories: known and unknown, which correspond to the knowledge that the model knows and the knowledge that the model does not know, respectively.
[0026] For questions in the first quadrant where the model knows what it knows, select questions with an accuracy above the upper threshold IK, where the model’s correct answers are designed as positive samples.
[0027] Two types of negative samples are constructed for the positive samples in the first quadrant: one is the model's answer "I don't know", which is used to push further in contrastive learning to enhance the model's confidence; the other is the wrong answer generated by the large language model, which is used to help the model forget the wrong knowledge.
[0028] For knowledge that the model does not know, namely the third and fourth quadrants, select questions with an accuracy rate below the upper threshold, where negative samples are incorrect answers generated by the large language model;
[0029] For questions where the model is unaware of the knowledge it possesses (i.e., the second quadrant), select questions with accuracy above the lower threshold but below the upper threshold, and set questions with correct answers as positive samples to enhance the model's understanding of uncertain knowledge.
[0030] For knowledge that the model does not know (i.e., the fourth quadrant), select questions with accuracy below the lower threshold, design corresponding questions and "I don't know" answers as positive samples, in order to encourage the model to admit "I don't know" when faced with knowledge it does not know.
[0031] Different comparative data are constructed for different quadrants. The comparative data for each quadrant includes input questions and answers, and covers anchor points, positive samples, and negative samples.
[0032] Further, in step S4, the adaptive contrastive learning strategy specifically includes:
[0033] Instruction pairs for constructing positive and negative samples are obtained from the three knowledge quadrants: the first quadrant (where the model knows what it knows), the second quadrant (where the model does not know what it knows), and the third quadrant (where the model knows what it does not know).
[0034] Adaptive comparison instructions are used for fine-tuning, which enhances the model's ability to grasp knowledge by bringing positive samples closer and pushing negative samples further away.
[0035] By employing different data usage strategies, positive and negative samples in different quadrants are adaptively calculated.
[0036] Furthermore, in step S4, the adaptive contrastive learning strategy further includes:
[0037] The contrastive loss function is used to calculate the contrastive loss between positive and negative samples, so that the model learns to close the distance with positive samples and widen the distance with negative samples in the feature space, thereby enhancing the ability to identify correct knowledge.
[0038] The cross-entropy loss between the model output and the expected answer is calculated to preserve the language generation capability of the large language model;
[0039] The overall learning objective function is formed by combining the contrastive loss and the cross-entropy loss.
[0040] By optimizing the overall learning objective function, the large language model is fine-tuned to achieve knowledge correction.
[0041] Furthermore, the contrastive loss function uses cosine similarity to measure the similarity between the model output and positive samples, and encourages the model output to maximize the similarity with positive samples and minimize the similarity with negative samples through negative log-likelihood. The value of the contrastive loss function is determined by the cosine similarity between positive and negative samples, where the similarity of positive samples is increased and the similarity of negative samples is decreased, thereby making the model more inclined to the correct answer during training.
[0042] Furthermore, the contrast loss function introduces a temperature hyperparameter to adjust the sensitivity of similarity. The temperature hyperparameter is used to control the scaling of the exponential function in the contrast loss function, thereby affecting the relative importance of the distance between positive and negative samples.
[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned knowledge correction method for a large language model based on adaptive contrastive learning.
[0044] A computer program product includes a computer program that, when executed by a processor, implements the knowledge correction method for a large language model based on adaptive contrastive learning.
[0045] The present invention has the following beneficial effects:
[0046] This invention proposes an innovative knowledge correction method for large language models based on adaptive contrastive learning. This method, by meticulously dividing knowledge quadrants, can more accurately identify and process the model's performance under different knowledge states. The method not only retains the model's known knowledge but also reinforces knowledge that the model is uncertain or lacks confidence in. Through a contrastive learning strategy, the model learns to appropriately express "I don't know" when faced with unknown knowledge. By adding a new partitioning threshold, this invention can correct the model's knowledge with finer granularity, thereby improving the model's honesty and credibility. During supervised fine-tuning, the adaptive contrastive learning strategy enhances the model's ability to identify correct knowledge by bringing positive samples closer and pushing negative samples further away, while maintaining the model's language generation capabilities. Experimental results show that using this adaptive contrastive learning-based fine-tuning method significantly improves the model's performance on the authenticity index, surpassing various existing methods. This invention has significant practical application value and significance for alleviating the "illusion" phenomenon that may occur when large language models generate text, especially in key fields such as medicine and law.
[0047] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description
[0048] Figure 1This is a flowchart illustrating the overall process of the knowledge correction method for large language models based on adaptive contrastive learning, as described in this invention.
[0049] Figure 2 This is a schematic diagram illustrating the implementation principle of the knowledge correction method for large language models based on adaptive contrastive learning, as described in an embodiment of the present invention.
[0050] Figure 3 This is an example of a repeated sampling boundary in an embodiment of the present invention.
[0051] Figure 4 This is an example of comparative learning data used in an embodiment of the present invention. Detailed Implementation
[0052] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.
[0053] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0054] Current fine-tuning methods using SFT do not adequately handle some issues where the model was originally known but uncertain. This leads to the fine-tuned model tending to reject responses when given uncertain information. We hypothesize that there is some knowledge that the model is uncertain about. Therefore, based on this hypothesis, we set a new threshold to divide knowledge into three categories: knowledge that the model knows it already knows, knowledge that the model doesn't know it already knows, and knowledge that the model doesn't know it already doesn't know. We then use corresponding contrastive learning strategies in each quadrant to help the model retain known knowledge, reinforce uncertain knowledge, and forget unknown knowledge.
[0055] This invention proposes a knowledge correction method for large language models based on adaptive contrastive learning. It mainly involves two key points: First, it adds a new threshold to segment the knowledge that the model knows but is uncertain about, allowing for more granular knowledge correction. Second, it proposes an adaptive contrastive learning method to enhance the supervised fine-tuning process of large language models, enabling the model to retain known knowledge, consolidate uncertain knowledge, and forget unknown knowledge. Through threshold segmentation, model knowledge is divided into categories: knowledge the model knows it already knows, knowledge the model doesn't know it already knows, and knowledge the model doesn't know it doesn't know. The adaptive contrastive learning approach is used to enhance the honesty fine-tuning stage of large language models.
[0056] See Figure 1This invention provides a knowledge correction method for large language models based on adaptive contrastive learning, comprising the following steps:
[0057] Step S1: Knowledge Quadrant Division: Divide the knowledge of the Large Language Model (LLM) into four quadrants, including the model knowing what it knows, the model not knowing what it knows, the model knowing what it does not know, and the model not knowing what it does not know, in order to identify the model's mastery of knowledge.
[0058] Step S2, Knowledge Boundary Detection: Based on the knowledge quadrant division, the knowledge boundary of the model is detected by setting an upper threshold for identifying the model's high confidence in its knowledge and a lower threshold for identifying the model's lack of confidence in its knowledge, so as to determine the model's confidence measure for each problem and thus identify the model's knowledge boundary.
[0059] In a preferred embodiment, step S2 specifically includes: filtering questions from different question-answering datasets; sampling the response of the large language model multiple times for each question; calculating the accuracy of the response to each question as a confidence measure of the model for each question; setting two thresholds: an upper threshold (IK, the rate at which the model knows the answer) and a lower threshold (IDK, the rate at which the model does not know the answer) as hyperparameters; when the model's confidence measure for a question exceeds the upper threshold IK, it is considered that the model fully grasps the knowledge, corresponding to the model knowing what it knows; when the model's confidence measure for a question is between IK and IDK, it is considered that the model possesses the knowledge but may not be fully aware of it, which may lead to occasional incorrect answers, corresponding to the model not knowing what it knows; when the model's confidence measure for a question is below the lower threshold IDK, it is considered that the model lacks the knowledge, resulting in a low correct answer rate, corresponding to the model not knowing what it does not know; through the evaluation of the above thresholds and confidence measures, the knowledge boundary of the model is defined, including the knowledge that the model knows, the knowledge that the model does not know, and the knowledge that the model does not know.
[0060] Step S3: Constructing comparative learning data: Based on the detected knowledge boundaries, collect known and unknown knowledge data of the model, and construct comparative learning data, including positive and negative samples, to enhance the model's mastery of knowledge.
[0061] In a preferred embodiment, step S3 specifically includes: classifying the questions into two categories, known and unknown, based on the knowledge boundaries determined in step S2, corresponding to knowledge that the model knows and knowledge that the model does not know, respectively; for knowledge that the model knows (first quadrant), selecting questions with an accuracy higher than the upper threshold IK, where the model's correct answers are designed as positive samples; constructing two types of negative samples for the positive samples in the first quadrant: one is the large language model's answer "I don't know," used to push back in contrastive learning to enhance the model's confidence; the other is the incorrect answers generated by the large language model, used to help the model forget incorrect knowledge; for knowledge that the model does not know (third and fourth quadrants), selecting questions with an accuracy lower than the upper threshold IK. In this model, negative samples are incorrect answers generated by the large language model. For knowledge that the model doesn't know (quadrant 2), questions with accuracy above the lower threshold IDK but below the upper threshold IK are selected, and questions with correct answers are set as positive samples to enhance the model's understanding of uncertain knowledge. For knowledge that the model doesn't know (quadrant 4), questions with accuracy below the lower threshold IDK are selected, and corresponding questions and "I don't know" answers are designed as positive samples to encourage the model to admit "I don't know" when faced with knowledge it doesn't possess. Different contrastive data are constructed for different quadrants, where x represents the input question and y represents the answer, including anchor points, positive samples, and negative samples. The main difference lies in the choice of y. The construction of these contrastive data provides a data foundation for the subsequent adaptive contrastive learning strategy, enabling the calculation of adaptive contrastive loss and model optimization in later steps.
[0062] Step S4, Adaptive Contrastive Learning: Based on the construction of contrastive learning data, an adaptive contrastive learning strategy is adopted for learning. The adaptive contrastive learning strategy enhances the model's ability to master knowledge by bringing positive samples closer and pushing negative samples further away, while maintaining the generation ability of the large language model.
[0063] In a preferred embodiment, the adaptive contrastive learning strategy specifically includes: obtaining instruction pairs for constructing positive and negative samples from three knowledge quadrants: the model knows what it knows (first quadrant), the model does not know what it knows (second quadrant), and the model knows what it does not know (third quadrant); performing adaptive contrastive instruction fine-tuning to enhance the model's knowledge mastery by bringing positive samples closer and pushing negative samples further away; adaptively calculating positive and negative samples in different quadrants through different data usage strategies; enabling the model to think about different knowledge more specifically, and enhancing the model's ability to retain known knowledge, consolidate uncertain knowledge, and forget unknown knowledge.
[0064] In a preferred embodiment, the adaptive contrastive learning strategy further includes: calculating the contrastive loss between positive and negative samples using a contrastive loss function, so that the model learns to narrow the distance with positive samples and widen the distance with negative samples in the feature space, thereby enhancing the ability to identify correct knowledge; calculating the cross-entropy loss between the model output and the expected answer to maintain the language generation capability of the large language model; combining the contrastive loss and the cross-entropy loss to form an overall learning objective function; and fine-tuning the large language model by optimizing the overall learning objective function to achieve refined knowledge management and correction. Preferably, the contrastive loss function uses cosine similarity to measure the similarity between the model output and positive samples, and encourages the model output to maximize the similarity with positive samples and minimize the similarity with negative samples through the form of negative log-likelihood; the value of the contrastive loss function is determined by the cosine similarity between positive and negative samples, wherein the similarity of positive samples is increased and the similarity of negative samples is decreased, thereby making the model more inclined to the correct answer during training. Furthermore, the contrast loss function introduces a temperature hyperparameter to adjust the sensitivity of similarity. The temperature hyperparameter is used to control the scaling of the exponential function in the contrast loss function, thereby affecting the relative importance of the distance between positive and negative samples.
[0065] Experiments have shown that the method of this invention can significantly improve the performance of the model's response realism index, which is of great significance for alleviating the illusion problem of large language models.
[0066] The specific principles, algorithm examples, and experimental verifications of the embodiments of the present invention are further described below.
[0067] To obtain a more honest and trustworthy Large Language Model (LLM) assistant, the LLM should possess two capabilities: not only should it be able to refuse to answer unknown questions, but it should also be able to correctly distinguish whether it truly understands the knowledge related to the question. This means the model needs to be aware of what it knows and what it doesn't know. The LLM's knowledge awareness ability can be represented by a knowledge quadrant. The knowledge quadrant is divided into four quadrants: knowing what it knows, knowing what it doesn't know, not knowing what it knows, and not knowing what it doesn't know. The vertical axis represents the model's self-perception (i.e., what the model believes it knows or doesn't know), and the horizontal axis represents the model's actual knowledge mastery (i.e., whether the model truly masters a specific piece of knowledge). Therefore, as... Figure 2As shown, the knowledge in the first quadrant, "Knowing What You Know," represents the AI's knowledge of what it knows, while the knowledge in the fourth quadrant, "Not Knowing What You Don't Know," represents the AI's ignorance of what it knows. The same logic applies to the other quadrants. Ideally, the LLM provides a correct answer to a question, which falls under the "Knowing What You Know" quadrant. However, when faced with illusion problems, the LLM fails to recognize its lack of knowledge and provides an incorrect answer, falling under the "Not Knowing What You Don't Know" quadrant. The horizontal axis represents the LLM's specific knowledge mastery, which is limited by the LLM's training process and knowledge capabilities. To alleviate the illusion problem and meet the needs of honest LLM systems, this invention focuses more on the vertical "Knowing What You Don't Know / Not Knowing" ability. This invention considers providing correct answers to known questions and refusing to respond to unknown questions as acceptable behaviors for honest LLMs. Therefore, the goal of this invention in alleviating the LLM illusion problem is to move more input instructions and responses from the lower two quadrants to the upper two quadrants.
[0068] Knowledge boundary detection of the model:
[0069] Different question-answering datasets contain various types of knowledge and expressions, making it challenging to evaluate whether a model truly possesses specific knowledge, as the models' confidence levels vary. To help models better identify their knowledge boundaries and assess their ability to answer relevant questions, this invention sets two thresholds: an upper threshold (IK, the rate at which the model knows the knowledge) and a lower threshold (IDK, the rate at which the model does not know the knowledge). These correspond to three types of knowledge: the model knows what it knows (knows what it already knows), the model does not know what it knows (does not know what it already knows / is unsure of its knowledge), and the model does not know what it does not know (does not know what it does not know).
[0070] The dataset is categorized into questions, and LLM is sampled multiple times to obtain responses. Accuracy is calculated as a measure of the model's confidence in each question. The number of sampled responses, along with thresholds IK and IDK, are used as hyperparameters. Figure 3 As shown, when confidence exceeds IK, the model is considered to have fully mastered the knowledge. If confidence is between IK and IDK, the model possesses the knowledge but may not be aware of it, leading to occasional incorrect answers. Confidence below IDK indicates a lack of knowledge, resulting in a low rate of correct answers.
[0071] By defining these knowledge boundaries, contrastive learning data is constructed, and the model is adaptively adjusted based on these knowledge types. This process aims to help the model retain known knowledge, reinforce uncertain knowledge, and forget unknown knowledge. In subsequent steps, more knowledge is transferred from the third and fourth quadrants to the first and second quadrants through adaptive contrastive learning instruction tuning, thereby improving the model's honesty while ensuring it provides helpful answers.
[0072] Contrastive learning data construction:
[0073] Based on the defined knowledge boundaries, known and unknown knowledge data for the model were collected, corresponding to the right and left quadrants of the knowledge, respectively. Next, comparative learning data was constructed for this knowledge, such as... Figure 4 As shown.
[0074] Known problems with the model:
[0075] In this section, questions with an accuracy above the upper threshold IK belong to the first quadrant of knowledge where "the model knows what it knows." For a given question, the model's correct answer is designed as a positive sample. There are two types of negative samples: one is the model's answer "I don't know," which can be pushed further in contrastive learning instruction tuning to enhance the model's confidence and avoid situations where the model knows something but thinks it doesn't. The other type of negative sample is an incorrect answer generated by the LLM (Learning Learning Model), used to help the LLM forget incorrect knowledge during training. It is worth noting that the selection of samples is related to the specific parameters of the threshold. When the upper threshold IK is chosen as 1.0, it means that only questions that the model answers correctly in all sampled instances are considered known questions. In this case, there will be no corresponding incorrect answer negative samples in the negative sample construction, because questions with incorrect answers have already been classified as questions the model doesn't know.
[0076] Unknown problems in the model:
[0077] Questions with an accuracy rate below the upper threshold IK, including "the model doesn't know it knows" and "the model doesn't know it doesn't know," fall into the bottom two quadrants of the knowledge quadrant and are therefore considered "the model doesn't know." Negative samples for these two types of questions are incorrect answers generated by the LLM. In constructing positive samples, for questions with an accuracy rate above the lower threshold IDK, this invention considers the model to actually possess the relevant knowledge but is unsure whether it knows it; this falls under the category of "the model doesn't know it knows." Questions with correct answers are designated as positive samples, aiming to enhance the model's grasp of uncertain knowledge by introducing this knowledge. For questions with an accuracy rate below the lower threshold IDK, this invention considers the model to lack the relevant knowledge; this falls under the category of "the model doesn't know it doesn't know." This invention designs corresponding questions and "I don't know" answers as positive samples. By narrowing the distance between these positive samples, the model can be better encouraged to admit "I don't know" when faced with knowledge it lacks, thereby enhancing the honesty and credibility of the LLM's answers.
[0078] To calculate the adaptive contrastive loss, different contrastive data need to be constructed for different quadrants. Specifically, this invention constructs different instruction data I = (x, y), where x is the input question and y is the answer. In the contrastive setup, in addition to the anchor point I, there is also a positive sample I. + and negative sample I- The main difference lies in the choice of y, which will be explained in detail in a later section.
[0079] Adaptive contrastive learning:
[0080] In the "Contrastive Learning Data Construction Section," instruction pairs for constructing positive and negative instances of known and unknown knowledge of the model are obtained from the three quadrants, followed by adaptive contrastive instruction fine-tuning. Compared to traditional SFT, the contrastive instruction fine-tuning method enhances the model's knowledge mastery by bringing positive instances closer and pushing negative instances further away. In the adaptive contrastive learning strategy, different data usage strategies are designed for the positive and negative instances corresponding to knowledge in different quadrants, which can adaptively calculate positive and negative samples in different quadrants. In this way, the model can think about different knowledge more specifically, better enhancing its ability to retain known knowledge, consolidate uncertain knowledge, and forget unknown knowledge.
[0081] For knowledge that is known and confirmed to be known in the model (such as...) Figure 4 The Quadrant-1 shown has comparative data I. IKIK , and The i-th sample in a training batch includes the original input. (its y) i (This is the standard answer) and positive examples (its y) i (This is the correct answer before LLM) and negative instances (its y) i (This is either "I don't know" or an incorrect answer before LLM). Positive examples. Zoom in and bring the negative instance closer. To calculate the contrast loss, this invention designs a contrast loss function, as shown in the formula:
[0082]
[0083] in τ is the cosine similarity, and τ is the temperature hyperparameter.
[0084] To maintain the generative capability of LLM, a standard cross-entropy loss was designed for each instruction pair according to the SFT training method, as shown in the formula:
[0085]
[0086] Where l is the length of the input and the desired output. It is y i The kth mark, yes Previous markings.
[0087] By combining the two loss functions mentioned above, the overall learning objective function is obtained, as shown in Equation 4.20:
[0088]
[0089] Here, detach indicates that the loss value is separated from the computation graph and therefore treated as a scalar, and λ is the upper bound of the weights.
[0090] Experiments demonstrate that this invention employs an adaptive contrastive learning-based fine-tuning method during the instruction fine-tuning stage. This method allows the model to retain known knowledge, consolidate uncertain knowledge, and forget unknown knowledge. The model, after this fine-tuning approach, significantly improves the performance of the response authenticity index and surpasses various existing methods. This invention is of great significance for improving the honesty of large language models and alleviating the illusion problem in large language models.
[0091] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.
[0092] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.
[0093] This invention also provides a processor that executes a computer program, at least performing the methods described above.
[0094] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.
[0095] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0096] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0097] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0098] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0099] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0100] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.
[0101] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.
[0102] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.
[0103] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.
Claims
1. A knowledge correction method for large language models based on adaptive contrastive learning, applied to text generation, characterized in that... Includes the following steps: S1. Knowledge Quadrant Division: The knowledge of the large language model is divided into four quadrants, including model knows what it knows, model does not know what it knows, model knows what it does not know, and model does not know what it does not know, in order to identify the model's mastery of knowledge; S2. Knowledge Boundary Detection: Based on knowledge quadrant division, the knowledge boundary of the model is detected by setting an upper threshold for identifying the model's high confidence in its knowledge and a lower threshold for identifying the model's lack of confidence in its knowledge, so as to determine the model's confidence measure for each problem and thus identify the model's knowledge boundary. Step S2 specifically includes: filtering questions from different question-answering datasets; sampling the response of the large language model multiple times for each question; calculating the accuracy of the response for each question, using this as a confidence measure of the model for each question; setting upper and lower thresholds as hyperparameters; when the model's confidence measure for a question exceeds the upper threshold, it is considered that the model fully grasps the knowledge, defined as the model knowing what it knows; when the model's confidence measure for a question is between the upper and lower thresholds, it is defined as the model not knowing what it knows; when the model's confidence measure for a question is below the lower threshold, it is considered that the model lacks this knowledge, defined as the model not knowing what it does not know; defining the model's knowledge boundary through the above threshold and confidence measure evaluation. S3. Construction of contrastive learning data: Based on the knowledge boundary of the probe, collect knowledge data that the model knows and does not know, and construct contrastive learning data, including positive samples and negative samples; S4. Adaptive Contrastive Learning: Based on the construction of contrastive learning data, an adaptive contrastive learning strategy is adopted for model learning. The adaptive contrastive learning strategy enhances the model's ability to master knowledge by bringing positive samples closer and pushing negative samples further away, while maintaining the model's generative ability.
2. The knowledge correction method for large language models based on adaptive contrastive learning as described in claim 1, characterized in that, Step S3 specifically includes: Based on the knowledge boundaries determined in step S2, the problems are divided into two categories: known and unknown, which correspond to the knowledge that the model knows and the knowledge that the model does not know, respectively. For questions in the first quadrant where the model knows what it knows, select questions with an accuracy above the upper threshold IK, where the model’s correct answers are designed as positive samples. Two types of negative samples are constructed for the positive samples in the first quadrant: one is the model's answer "I don't know", which is used to push further in contrastive learning to enhance the model's confidence; the other is the incorrect answer generated by the large language model, which is used to help the model forget incorrect knowledge. For knowledge that the model does not know, namely the third and fourth quadrants, select questions with an accuracy rate below the upper threshold, where negative samples are incorrect answers generated by the large language model; For questions where the model is unaware of the knowledge it possesses (i.e., the second quadrant), select questions with accuracy above the lower threshold but below the upper threshold, and set questions with correct answers as positive samples to enhance the model's understanding of uncertain knowledge. For knowledge that the model does not know (i.e., the fourth quadrant), select questions with accuracy below the lower threshold, design corresponding questions and "I don't know" answers as positive samples to encourage the model to admit "I don't know" when faced with knowledge it does not know. Different comparative data are constructed for different quadrants. The comparative data for each quadrant includes input questions and answers, and covers anchor points, positive samples, and negative samples.
3. The knowledge correction method for large language models based on adaptive contrastive learning as described in claim 1, characterized in that, In step S4, the adaptive contrastive learning strategy specifically includes: Instruction pairs for constructing positive and negative samples are obtained from the three knowledge quadrants: the first quadrant (where the model knows what it knows), the second quadrant (where the model does not know what it knows), and the third quadrant (where the model knows what it does not know). Adaptive comparison instructions are used for fine-tuning, which enhances the model's ability to grasp knowledge by bringing positive samples closer and pushing negative samples further away. By employing different data usage strategies, positive and negative samples in different quadrants are adaptively calculated.
4. The knowledge correction method for large language models based on adaptive contrastive learning as described in any one of claims 1 to 3, characterized in that, In step S4, the adaptive contrastive learning strategy further includes: The contrastive loss function is used to calculate the contrastive loss between positive and negative samples, so that the model learns to close the distance with positive samples and widen the distance with negative samples in the feature space, thereby enhancing the ability to identify correct knowledge. The cross-entropy loss between the model output and the expected answer is calculated to preserve the language generation capability of the large language model; The overall learning objective function is formed by combining the contrastive loss and the cross-entropy loss. By optimizing the overall learning objective function, the large language model is fine-tuned to achieve knowledge correction.
5. The knowledge correction method for large language models based on adaptive contrastive learning as described in claim 4, characterized in that, The contrastive loss function uses cosine similarity to measure the similarity between the model output and positive samples, and encourages the model output to maximize the similarity with positive samples and minimize the similarity with negative samples through negative log-likelihood. The value of the contrastive loss function is determined by the cosine similarity between positive and negative samples, where the similarity of positive samples is increased and the similarity of negative samples is decreased, thereby making the model more inclined to the correct answer during training.
6. The knowledge correction method for large language models based on adaptive contrastive learning as described in claim 5, characterized in that, The contrast loss function introduces a temperature hyperparameter to adjust the sensitivity of similarity. The temperature hyperparameter is used to control the scaling of the exponential function in the contrast loss function, thereby affecting the relative importance of the distance between positive and negative samples.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the knowledge correction method for large language models based on adaptive contrastive learning as described in any one of claims 1 to 6.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the knowledge correction method for large language models based on adaptive contrastive learning as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Industrial large model-oriented fine tuning data generation method, computer program and terminal
CN119004100A
Knowledge fusion method and device based on knowledge graph and large language model
CN119227785A