A training method and system for improving the honesty of a large model with a small amount of labeled data

Abstract

The application discloses a training method and system for improving the honesty of a large model with a small amount of labeled data, and a lightweight network is newly added to assist the LLM in self-labeling data, thereby significantly expanding the data scale available for model training on the basis of limited labeled data, so that the model can still be fully trained and optimized under the condition of limited data resources. A close cooperative working mechanism is formed between the lightweight network and the LLM, the lightweight network generates reliable labeling information for unlabeled data by performing honesty evaluation on the answers generated by the LLM, this process not only fully utilizes the probability evaluation ability of the LLM itself, but also combines the quantitative evaluation of the lightweight network on the honesty, so that the model can learn a more honest answer strategy under the condition of limited data, and the purpose of enhancing the honesty of the LLM by using a small amount of data is achieved.

Description

A training method and system for improving the honesty of large models with a small amount of labeled data. Technical Field

[0001] This invention relates to the field of computer and network technology, and specifically to a training method and system that requires a small amount of labeled data to improve the honesty of a large model. Background Technology

[0002] In today's digital age, the honesty of LLMs (Large Language Models) is of paramount importance. With the rapid development of artificial intelligence, LLMs are increasingly widely used across various fields, and their output information has a profound impact on user decision-making, knowledge acquisition, and social cognition. In this context, the honesty of LLMs has become a key factor in ensuring their reliability and credibility. For example, in high-risk scenarios such as healthcare, law, and finance, an honest LLM can provide professionals with accurate information support, avoiding serious consequences caused by misinformation. Honesty, as one of the "3H" (Helpfulness, Honesty, Harmless) criteria for large language model alignment, has gradually become a research focus in recent years.

[0003] Honesty is a crucial dimension for evaluating the performance and reliability of an LLM (Leveled Model). Specifically, when faced with a user's question, if the LLM possesses the necessary knowledge and ability to answer it, it should provide a precise and accurate answer. However, when faced with a question beyond its knowledge base or one for which it cannot provide a definitive answer, the LLM needs to honestly acknowledge its uncertainty about the answer, avoiding potentially misleading information. For example, when a user asks, "What are the three primary colors?", if the model's knowledge base explicitly contains information about primary colors, an honest and appropriate answer would be, "The primary colors are red, green, and blue." However, if the model has not learned or cannot accurately recall this knowledge, an honest response would be something like, "Sorry, I'm not sure about that answer."

[0004] Currently, there are two main solutions for improving the honesty of LLM (Limited Model Learning): cue engineering and model fine-tuning. Cue engineering primarily involves carefully designing structured instructions, such as guiding statements like "when encountering unknown domains, explicitly express uncertainty," and integrating them into the interaction with the model. The advantage of this approach is that it can achieve a limited improvement in honesty without modifying the model's parameters. It's like providing the model with a set of behavioral guidelines, allowing it to respond relatively honestly to different situations.

[0005] Model fine-tuning approaches based on supervised tuning and direct preference optimization achieve more significant improvements by adjusting the model's parameters. The key to these methods is constructing a question-answering pair dataset with knowledge state annotations. This means explicitly labeling the knowledge state corresponding to each question—whether it's known knowledge, unknown knowledge, or partially known knowledge. Then, the model is trained using this dataset, enabling it to flexibly choose appropriate answer strategies based on its knowledge mastery, thus providing more honest and reasonable responses to various questions.

[0006] However, despite some progress in improving LLM honesty, both approaches have significant limitations. For cue engineering, the improvement is constrained by the fact that the underlying model itself was not specifically trained for honesty. Because the underlying model was not initially optimized for honesty during training, cue engineering alone can only achieve relatively limited improvements in honesty, much like making minor repairs to an existing building—it's difficult to fundamentally change its structure and performance.

[0007] While model fine-tuning methods can achieve superior results in improving the honesty of LLMs, they face the core challenge of a scarcity of high-quality training data. This is particularly true for complex problems with diverse or subjective answers. On the one hand, the diversity of answers makes it difficult to exhaustively represent all reasonable and possible responses, making the construction of a comprehensive and accurate training dataset extremely challenging. On the other hand, the lack of objective and unified evaluation criteria makes it difficult to accurately judge whether the model's performance in answering such questions is sufficiently honest and reasonable. More importantly, different LLM architectures exhibit significant differences in their training data distribution and parameter scale, resulting in unique cognitive blind spots. This necessitates the customized construction of corresponding training sets for each specific target LLM to accommodate its unique architectural characteristics and cognitive blind spots. This process often requires substantial computational resources and significant manpower for annotation, and the high cost severely restricts the scalability and adaptability of this approach in different scenarios, posing numerous challenges in practical applications. Summary of the Invention

[0008] Therefore, it is necessary to provide a high-efficiency, low-cost training method and system that requires a small amount of labeled data to improve the honesty of large models.

[0009] A training method that requires a small amount of labeled data to improve the honesty of a large model includes the following steps:

[0010] Step 1: Use a predetermined amount of labeled data to perform supervised fine-tuning of the LLM and standardize the output format of the LLM;

[0011] Step 2: Collect the output of the fine-tuned LLM on the labeled data, as well as the hidden state corresponding to each reply;

[0012] Step 3: Label each collected response using an honesty metric to construct preference data;

[0013] Step 4: Train a lightweight network using preference data; train a lightweight network that uses the hidden state as input to predict the honesty score of the corresponding response.

[0014] Step 5: For each unlabeled question, select the answer with the highest prediction score from the lightweight network and the probability weighted average of the LLM as the labeled data;

[0015] Step six: Train the LLM using labeled data. Through training, the LLM learns to meet the honesty requirement.

[0016] Preferably, step one, which involves using a predetermined amount of labeled data to perform supervised fine-tuning of the LLM and standardizing the output format of the LLM, includes the following specific steps:

[0017] Step 1.1, set the format of the labeled data as follows: ,in, For the question, In reply;

[0018] Step 1.2: Select a pre-trained model and input the labeled data into the pre-trained model;

[0019] Step 1.3: For questions that the model can answer correctly, use the model's response directly. For questions that the model cannot answer correctly, add the string "I am not sure." to the end of the original model response.

[0020] Preferably, step two, which involves collecting the output of the fine-tuned LLM on the labeled data and the specific steps for obtaining the hidden state corresponding to each response, includes:

[0021] Step 2.1: Input the question into the model, and the model generates multiple responses through beam search;

[0022] Step 2.2: Collect the hidden state of the last token of each reply after each transformer block in the large model.

[0023] Preferably, step three, which involves labeling each collected response using an honesty index to construct preference data, includes the following specific steps:

[0024] Step 3.1: Establish a labeling strategy by assigning different weights based on whether the model's response is correct and whether it contains uncertain statements.

[0025] Step 3.2: According to the annotation strategy, an honesty index is marked for each reply;

[0026] Step 3.3: Optimize the data format by grouping different outputs corresponding to the same input into a comparison group to construct preference data.

[0027] Preferably, step four, training a lightweight network using preference data, includes the following specific steps:

[0028] Step 4.1, for the labeled score For each question, all answers are subjected to softmax normalization, and the normalized labeled score is calculated. The calculation formula is shown in equation (1):

[0029] (1);

[0030] Where n is the number of answers to each question, and the value ranges from 10 to 40;

[0031] Step 4.2, calculate the normalized score. Compared with lightweight network prediction scores The cross-entropy loss between them is calculated using the formula shown in equation (2):

[0032] (2).

[0033] Step 4.3: Train different lightweight networks for the hidden states of each Transformer block in the Transformer model, and test the accuracy of the lightweight networks trained on different layers using a validation set.

[0034] Step 4.4: Select the lightweight network with the highest accuracy and collect the hidden states of the corresponding layer of the lightweight network.

[0035] Preferably, step five, which involves selecting the answer with the highest prediction score from the lightweight network and the probability-weighted average of the LLM for each unlabeled question, includes the following specific steps:

[0036] Step 5.1: Input the question into LLM and generate multiple responses using beam search, with each response corresponding to a probability value;

[0037] Step 5.2: Input the hidden state of the specific layer corresponding to the last token of each reply into the lightweight network to generate a prediction score;

[0038] The specific layer refers to the output layer corresponding to the hidden state used by the lightweight network during input training;

[0039] Step 5.3: Select the answer with the highest predicted score from the lightweight network and the highest probability linear weight from the LLM as the labeled data.

[0040] Furthermore, a training system that requires a small amount of labeled data to improve the honesty of a large model, for implementing the training method described above that requires a small amount of labeled data to improve the honesty of a large model, includes:

[0041] The LLM module is used to generate multiple answers for unlabeled data and output the probability value and hidden state information for each answer. Based on the built-in probability model of the LLM module, each generated answer is assigned a specific probability value, which reflects the model's self-evaluation of the credibility of the answer.

[0042] A lightweight network is used to evaluate honesty scores for each answer. By learning and analyzing the hidden state information obtained from the LLM module, it identifies honesty-related features and patterns in the answers, providing additional quantitative indicators for the honesty of the answers, enabling the model to judge the reliability of the answers more comprehensively and objectively.

[0043] Preferably, the lightweight network adopts a multi-layer MLP structure, with each MLP layer sequentially connected. The MLP consists of an input layer, a hidden layer, and an output layer, with fully connected neurons between layers. It learns on limited labeled data and is trained through a learning ranking framework to identify the most honest answer among the candidate answers generated by the LLM.

[0044] Preferably, the probability evaluation value output by the LLM module is combined with the honesty evaluation of the lightweight network. By weighting the probability and honesty scores, a comprehensive score is obtained, and then the answer with the highest comprehensive score is selected as the labeled data.

[0045] The aforementioned training method and system, which requires a small amount of labeled data to improve the honesty of a large model, incorporates a lightweight network to assist the LLM in self-labeling the data. This significantly expands the scale of data available for model training within a limited set of labeled data, enabling the model to be adequately trained and optimized even with limited data resources. A close collaborative mechanism exists between the lightweight network and the LLM. The lightweight network evaluates the honesty of the responses generated by the LLM, generating reliable labels for unlabeled data. This process not only fully utilizes the probabilistic evaluation capabilities of the LLM but also combines the lightweight network's quantitative evaluation of honesty. This allows the model to learn more honest answering strategies under limited data conditions, achieving the goal of enhancing the honesty of the LLM with limited data. The algorithm of this invention is simple, easy to implement, low-cost, and readily scalable. Attached Figure Description

[0046] Figure 1 is a flowchart of a training method for improving the honesty of a large model using a small amount of labeled data, according to an embodiment of the present invention.

[0047] Figure 2 is a schematic diagram of the structure of a training system that requires a small amount of labeled data to improve the honesty of a large model according to an embodiment of the present invention. Detailed Implementation

[0048] This embodiment takes a training method and system that requires a small amount of labeled data to improve the honesty of a large model as an example. The invention will be described in detail below with reference to specific embodiments and accompanying drawings.

[0049] Please refer to Figure 1, which illustrates a training method for improving the honesty of a large model using a small amount of labeled data, according to an embodiment of the present invention. The method specifically includes the following steps:

[0050] Step one involves supervised fine-tuning of the LLM using a predetermined amount of labeled data, standardizing the LLM's output format for this task. This step performs initial optimization of the model, making it better suited to the needs of the specific task and laying the foundation for subsequent processing. The format of each labeled data point is as follows: ,in For the question, For questions that the model can answer correctly, use the model's response directly. For questions that the model cannot answer correctly, add the string "I am not sure." to the end of the original model's response.

[0051] Specifically, in this embodiment, 2000 labeled data points are used to perform supervised fine-tuning of the LLM and standardize the output format of the LLM.

[0052] Step two involves collecting the output of the fine-tuned LLM on labeled data, as well as the hidden state corresponding to each answer. After the first step of training is completed, the question is input into the model, which generates multiple responses through beam search. Simultaneously, we collect the output of the last token of each response after each transformer block in the larger model.

[0053] Specifically, the hidden state is a dynamic vector representation generated within the Transformer model at each position in the sequence after each layer of processing, containing the token at that position and its contextual information. As important information within the model, the hidden state provides rich semantic and feature support for subsequent honesty evaluation.

[0054] Step three involves labeling the collected responses using honesty metrics to construct preference data. This step provides clear guidance for model training by quantitatively evaluating the honesty of the responses, ensuring that the model can learn more honest answering strategies.

[0055] First, for each response, an annotation strategy is established, assigning different weights based on whether the model's response is correct and whether it contains uncertain statements. The annotation strategy is as follows:

[0056] If the model can answer the question correctly and does not express uncertainty, it scores 3.

[0057] If the model gives an incorrect answer and contains uncertain statements, the score is 2.

[0058] If the model gives a correct answer and contains statements of uncertainty, it scores 1.

[0059] If the model gives an incorrect answer and does not contain any statements of uncertainty, the score is 0.

[0060] We define an expression of uncertainty as one that includes the string "I am not sure." after the response.

[0061] Then, based on the labeling strategy, each reply is labeled with an honesty index.

[0062] Specifically, different responses to the same question are sorted and scored, with high weights assigned to answers that meet the criteria of accuracy, security, and logic, and low weights assigned to those that do not.

[0063] Finally, the data format is optimized by grouping different outputs corresponding to the same input into a comparison group to construct preference data.

[0064] Step four involves training a lightweight network that takes the hidden states as input and estimates the honesty score of the corresponding responses. The specific calculation process will be described below. First, for the labeled scores... For each question, softmax normalization is applied to the answer, and the normalized labeled score is... The calculation formula is:

[0065] (1);

[0066] Where n is the number of answers to each question.

[0067] Then calculate the normalized score. Compared with lightweight network prediction scores The cross-entropy loss between them is calculated using the following formula:

[0068] (2);

[0069] In equation (2), n has the same meaning as n in equation (1), and its value range is generally set to 10~40.

[0070] Since the Transformer model consists of several Transformer blocks, we train different lightweight networks for the hidden states of each Transformer block. We then test the accuracy of these lightweight networks on a validation set, selecting the one with the highest accuracy for subsequent operations. In these subsequent operations, we only need to collect the hidden states of the corresponding layer. The collected hidden states are used as input to the subsequent lightweight networks. The lightweight networks are designed to achieve efficient evaluation of the honesty of responses with minimal computational resource consumption, further improving the model's honesty performance.

[0071] Step 5: For each unlabeled question, select the answer with the highest predicted score from the lightweight network and the highest probability weighted score from the LLM as the labeled data.

[0072] The specific steps are as follows: Input the question into a large model, generate multiple responses using beam search, each response corresponding to a probability value. Input the hidden state of a specific layer corresponding to the last token of each response into a lightweight network to generate a prediction score. Here, the specific layer refers to the output layer corresponding to the hidden state used by the lightweight network during input training.

[0073] The answer with the highest linear weighted score from the lightweight network and the probability from the LLM is selected as the labeled data. This process generates reliable annotation information for unlabeled data by combining the evaluation results of the lightweight network and the probability of the LLM, thereby expanding the data scale and alleviating the data scarcity problem.

[0074] Step six involves retraining the LLM using labeled data. Through this training process, the LLM learns a more honest response strategy, thereby outputting more authentic and reliable information in subsequent generation processes.

[0075] Step 7: After the optimization and training steps above, we can finally obtain an LLM that performs well in terms of honesty. Its output is more trustworthy and can better meet users' needs for accurate information.

[0076] According to the method described above, operators can effectively improve the honesty of LLM and make its output more reliable by relying on only a small amount of labeled data. This method is not only technically innovative, but also demonstrates high practical value in real-world applications, providing a feasible solution to the problem of LLM's difficulty in generating truly reliable output under limited supervision.

[0077] Figure 2 illustrates the basic architecture of a training system for improving the honesty of large models using a small amount of labeled data, according to an embodiment of the present invention. The architecture consists of two core modules: an LLM (Large Language Model) and a lightweight network, along with the labeling process of these two core modules in the training method for improving the honesty of large models. The LLM module not only provides input to the subsequent lightweight network but also generates responses for filtering. The prediction score of the lightweight network and the probability of each response generated by the LLM work together to provide strong support for determining the final labeled data. The lightweight network employs a multi-layer MLP (Multi-Layer Perceptron) structure. It should be noted that in this embodiment, a three-layer MLP is preferred, but this is not a fixed pattern; in other embodiments, different numbers of MLP layers can be selected according to actual needs. From a functional perspective, the MLP layers are sequentially connected, and the differences in function between each layer are not significant. The choice of three layers is based on a comprehensive consideration of both neural network fitting performance and computational efficiency. The functions of each module and unit will be described in detail below.

[0078] For unlabeled data, LLM generates multiple responses. It outputs the probability value and hidden state information for each response. Based on the LLM's built-in probabilistic model, each generated response is assigned a specific probability value. This probability value, to a certain extent, intuitively reflects the model's own assessment of the response's credibility—essentially a self-assessment of the response's credibility. This crucial process lays the foundation for subsequent honesty evaluation by the lightweight network, enabling it to filter out more honest responses. Furthermore, the hidden states output by LLM serve as input to the lightweight network, providing a data foundation for subsequent evaluation processes.

[0079] The primary responsibility of the lightweight network is to evaluate the honesty score for each answer. Specifically, it receives the hidden states from the LLM output as input. By deeply learning and analyzing the hidden states and other relevant information obtained from the LLM, it can accurately identify features and potential patterns closely related to honesty in the answer. This evaluation stage provides additional and explicit quantitative indicators for the honesty of the answer, helping the entire model to more comprehensively and objectively determine the reliability of the answer.

[0080] By combining the probability of each answer with its corresponding honesty score, the response with the highest weighted value is selected as the labeled data. This process combines the probabilistic evaluation of LLM with the honesty evaluation of the lightweight network, ensuring that the selected answer has both high probability and excellent honesty performance. Specifically, a comprehensive score is obtained by weighting or combining the probability and honesty score, and the answer with the highest comprehensive score is selected as the labeled data. This selection mechanism effectively utilizes the strengths of both models, providing high-quality data support for model training.

[0081] This invention demonstrates numerous significant advantages and positive effects. First, the lightweight and efficient honest alignment framework proposed in this invention significantly reduces data annotation costs compared to other schemes that rely on large-scale labeled data. In practical applications, data annotation often requires substantial manpower, resources, and time, and the annotation quality is also affected by factors such as the professional level of the annotators. This invention, by relying on only a small amount of labeled data, can achieve effective training and optimization of LLM, thereby significantly reducing resource investment in data collection and annotation. This not only makes the training process more economical and efficient but also reduces reliance on professional annotation teams, providing more resource-constrained users and enterprises with the possibility of applying LLM, further promoting the popularization and development of LLM technology.

[0082] Secondly, the framework of this invention exhibits ease of training during the training process, requiring fewer additional network parameters for data annotation and consequently reducing the resources needed for training. In existing technologies, training some complex models often requires substantial computational and storage resources, placing high demands on hardware and increasing energy consumption and equipment costs. This invention, by simplifying network parameters, makes the training process simpler and faster, reducing hardware resource consumption. This not only improves training efficiency but also reduces energy consumption and equipment costs, aligning with current green and energy-saving development trends and demonstrating significant environmental and social benefits.

[0083] Furthermore, this invention introduces a lightweight network that learns on limited labeled data and is trained using a ranking framework, enabling it to identify the most honest responses from candidate responses generated by LLM. This innovative design not only significantly reduces the need for labeled data, effectively alleviating the data scarcity problem, but also demonstrates strong performance advantages in practical applications. Even with limited data, the lightweight network maintains good learning performance, accurately assesses the honesty of candidate responses, and improves the model's output quality. In addition, the introduction of the lightweight network makes the entire system more flexible and efficient, enabling it to quickly adapt to different application scenarios and changing needs, providing users with a more personalized and precise service experience.

[0084] To address the issue of lacking clear reference answers, this invention combines the scores predicted by a lightweight network with the prediction probabilities of an LLM (Limited Linear Model) to generate pseudo-labels. This mechanism has significant implications and advantages. It extends honest alignment to unlabeled data, greatly reducing reliance on manual annotation. In practical applications, unlabeled data often constitutes the majority of the dataset, and this invention can fully utilize this data, further improving the model's training performance. By generating pseudo-labels, the model can maintain honesty even without explicit reference answers, improving its generalization ability and adaptability. This mechanism also enables the model to self-learn and self-optimize, allowing it to continuously improve and refine in practical applications, better meeting the growing needs of users.

[0085] Finally, this invention fine-tunes the base LLM using a synthetic dataset, combining the specific scoring capability of lightweight networks with the generalization capability of LLM. This avoids dependence on specific model architectures and creates a unified training process adaptable to various architectures. This mechanism not only improves the model's versatility and flexibility but also provides ample room for further optimization and improvement. By combining the specific scoring capability of lightweight networks with the generalization capability of LLM, the model maintains honesty while possessing stronger generalization and adaptability, better capable of handling various complex application scenarios and changing requirements. Furthermore, this mechanism provides new ideas and methods for subsequent research and development, promoting the further development and innovation of LLM technology.

[0086] In summary, through the aforementioned mechanism, this invention enables efficient LLM training and optimization without relying on large-scale labeled data, providing an innovative and efficient technical solution for honest alignment of language models. This solution not only boasts significant technical advantages but also demonstrates broad applicability and positive effects in practical applications, providing strong support for the intelligent development of various industries, and possesses important practical significance and broad application prospects.

[0087] The core innovation of this invention lies in the addition of a lightweight network to effectively reduce data annotation costs. This lightweight network is designed and positioned to assist the LLM in self-annotating data, thereby significantly expanding the scale of data available for model training based on limited labeled data. This design cleverly solves the problem of data scarcity, enabling the model to be adequately trained and optimized even with limited data resources.

[0088] The challenge of this invention lies in how to enhance the honesty of LLM using limited data. To address this challenge, a close collaborative mechanism is established between the lightweight network and the LLM. The lightweight network evaluates the honesty of the responses generated by the LLM, generating reliable annotations for unlabeled data. This process not only fully utilizes the probabilistic evaluation capabilities of the LLM itself but also combines the lightweight network's quantitative evaluation of honesty, enabling the model to learn more honest answering strategies under limited data conditions.

[0089] It should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method that requires a small amount of labeled data to improve the honesty of a large model, characterized in that, The specific steps include: Step 1, supervising and fine-tuning the LLM using a predetermined amount of labeled data to standardize the LLM's output format; Step 2, collecting the output of the fine-tuned LLM on the labeled data, as well as the hidden state corresponding to each response; Step 3, labeling each collected response using an honesty index to construct preference data; Step 4, training a lightweight network using the preference data; training a lightweight network, taking the hidden state as input, to predict the honesty score of the corresponding response; Step 5, for each unlabeled question, selecting the answer with the highest probability weighting between the lightweight network's predicted score and the LLM's probability as labeled data; Step 6, training the LLM using the labeled data, enabling the LLM to learn to meet honesty requirements; wherein, the specific steps of Step 1, supervising and fine-tuning the LLM using a predetermined amount of labeled data to standardize the LLM's output format, include: Step 1.1, setting the format of the labeled data as... ,in, For the question, For the response; Step 1.2, select a pre-trained model and introduce the labeled data into the pre-trained model; Step 1.3, for questions that the model can answer correctly, directly use the model's response; for questions that the model cannot answer correctly, add the string "I am not sure." to the end of the original model's response; Step 3, for each collected response, label it with an honesty index to construct preference data. The specific steps include: Step 3.1, establish a labeling strategy, setting different weights based on whether the model's response is correct and whether it contains uncertain expressions; Step 3.2, label each response with an honesty index according to the labeling strategy; Step 3.3, optimize the data format, grouping different outputs corresponding to the same input into a comparison group to construct preference data.

2. The training method for improving the honesty of a large model with a small amount of labeled data as described in claim 1, characterized in that, Step 2, collecting the output of the fine-tuned LLM on the labeled data, and the specific steps for the hidden state corresponding to each response, include: Step 2.1, inputting the question into the model, and the model generating multiple responses through beam search; Step 2.2, collecting the hidden state of the last token of each response after each transformer block in the large model.

3. The training method as described in claim 1, which requires a small amount of labeled data to improve the honesty of a large model, is characterized in that... Step four, training a lightweight network using preference data, includes the following steps: Step 4.1, for the labeled scores... For each question, all answers are subjected to softmax normalization, and the normalized labeled score is calculated. The calculation formula is shown in equation (1): (1); where n is the number of answers to each question, ranging from 10 to 40; Step 4.2, calculate the normalized score. Compared with lightweight network prediction scores The cross-entropy loss between them is calculated using the formula shown in equation (2): (2); Step 4.3, train different lightweight networks for the hidden states of each Transformer block in the Transformer model, and test the accuracy of the lightweight networks trained on different layers through the validation set; Step 4.4, select the lightweight network with the highest accuracy and collect the hidden states of the corresponding layer of the lightweight network.

4. The training method as described in claim 1, which requires a small amount of labeled data to improve the honesty of a large model, is characterized in that... Step five, which selects the answer with the highest predicted score from the lightweight network and the highest probability weighted score from the LLM for each unlabeled question, includes the following steps: Step 5.1: Input the question into the LLM and generate multiple responses using beam search, each response corresponding to a probability value; Step 5.2: Input the hidden state of a specific layer corresponding to the last token of each response into the lightweight network to generate a predicted score; wherein, the specific layer refers to the output layer corresponding to the hidden state used by the lightweight network during input training; Step 5.3: Select the answer with the highest predicted score from the lightweight network and the highest probability weighted score from the LLM as the labeled data.

5. A training system that requires a small amount of labeled data to improve the honesty of a large model, used to implement the training method for improving the honesty of a large model requiring a small amount of labeled data as described in any one of claims 1-4, characterized in that, include: The LLM module is used to generate multiple answers for unlabeled data and output the probability value and hidden state information for each answer. The built-in probabilistic model based on the LLM module assigns a specific probability value to each generated answer, reflecting the model's self-evaluation of the answer's credibility. A lightweight network is used to evaluate the honesty score for each answer. By learning and analyzing the hidden state information obtained from the LLM module, the model identifies honesty-related features and patterns in the answers, providing additional quantitative indicators for the honesty of the answers. This allows the model to more comprehensively and objectively judge the reliability of the answers.

6. The training system as described in claim 5, which requires a small amount of labeled data to improve the honesty of a large model, is characterized in that... The lightweight network adopts a multi-layer MLP structure, with each MLP layer sequentially connected. The MLP consists of an input layer, a hidden layer, and an output layer, with fully connected neurons between layers. We learn from limited labeled data and train by learning a ranking framework to identify the most honest answers among the candidate answers generated by LLM.

7. The training system as described in claim 5, which requires a small amount of labeled data to improve the honesty of a large model, is characterized in that... The probability evaluation value output by the LLM module is combined with the honesty evaluation of the lightweight network. By weighting the probability and honesty score, a comprehensive score is obtained, and then the answer with the highest comprehensive score is selected as the labeled data.

Images