Training method and system for improving honesty of large model by using small amount of annotated data

Abstract

The invention discloses a training method and system for improving the integrity of a large model by using a small amount of annotated data, and the method comprises the steps: adding a lightweight network, which is used for assisting an LLM to carry out the self-annotation of data, thereby remarkably enlarging the scale of data which can be used for model training on the basis of limited annotated data, and improving the integrity of the model. Therefore, the model can still be fully trained and optimized under the condition of limited data resources. A compact cooperative work mechanism is formed between the lightweight network and the LLM, the lightweight network performs honesty evaluation on the answer generated by the LLM and generates reliable annotation information for the unannotated data, the process not only fully utilizes the probability evaluation capability of the LLM, but also combines quantitative evaluation of the lightweight network on honesty, so that the reliability of the system is improved, and the reliability of the system is improved. Therefore, the model can learn a more honest answer strategy under the condition of limited data, and the purpose of enhancing the honesty of the LLM by using a small amount of data is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer and network technologies, and in particular to a training method and system for improving the honesty of a large model using a small amount of labeled data. Background Art

[0002] In today's digital age, the honesty of Large Language Models (LLMs) is of paramount importance. With the rapid development of artificial intelligence (AI), LLMs are increasingly being used in various fields, and the information they output has a profound impact on user decision-making, knowledge acquisition, and social cognition. In this context, the honesty of LLMs has become a key factor in ensuring their reliability and credibility. For example, in high-risk scenarios such as healthcare, law, and finance, an honest LLM can provide professionals with accurate information and avoid the serious consequences of misinformation. Honesty, as one of the "3H" (Helpfulness, Honesty, Harmless) principles of Large Language Model alignment, has become a research focus in recent years.

[0003] The honesty of LLM is one of the important dimensions to measure its performance and reliability. Specifically, when faced with questions raised by users, if LLM has the corresponding knowledge reserves and capabilities to answer the question, then it should be able to provide users with accurate and correct answers; and when LLM faces questions that are beyond its own knowledge or cannot give an exact correct answer with its existing capabilities, it needs to honestly indicate its uncertainty about the answer to the user and avoid providing potentially misleading information. For example, when a user asks a question like "What are the three primary colors?", if the model's knowledge base clearly contains relevant knowledge about the three primary colors, an honest and appropriate answer can be "The three primary colors are red, green, and blue." However, if the model has not learned or cannot accurately recall this knowledge, then an honest response should be something like "Sorry, I'm not sure about this answer." Currently, there are two main approaches to improving LLM honesty: hint engineering and model fine-tuning. Hint engineering involves carefully crafting structured instructions, such as guiding statements like "explicitly express uncertainty when encountering unknown areas," and incorporating them into the model's interactions. The advantage of this approach is that it can achieve a limited improvement in honesty without modifying the model's parameters. It's like providing the model with a set of behavioral guidelines that allow it to respond relatively honestly in different situations.

[0004] Model fine-tuning approaches based on supervised fine-tuning and direct preference optimization achieve more significant improvements by adjusting model parameters. The key to these approaches is constructing a dataset of question-answer pairs with labeled knowledge states. Specifically, each question must clearly indicate the knowledge state corresponding to each question, such as known knowledge, unknown knowledge, or partially known knowledge. This dataset is then used to train the model, enabling it to flexibly select appropriate answer strategies based on its knowledge base, resulting in more honest and reasonable responses to various questions.

[0005] However, despite progress in improving LLM honesty, both approaches face significant bottlenecks. Hint engineering's effectiveness is limited by the fact that the underlying model itself is not specifically trained for honesty. Because the underlying model isn't specifically trained for honesty, the honesty improvements achieved through hint engineering alone are relatively limited. This is like making minor repairs to an existing building, failing to fundamentally alter its structure and performance.

[0006] While model fine-tuning methods can achieve superior results in improving the honesty of LLMs, they face the core challenge of a shortage of high-quality training data. This is particularly true for complex questions with diverse or subjective answers. On the one hand, the sheer variety of answers makes it difficult to exhaust all plausible and possible answers, making it extremely challenging to construct comprehensive and accurate training datasets. On the other hand, the lack of objective, unified evaluation criteria makes it difficult to accurately determine whether a model's performance in answering such questions is sufficiently honest and reasonable. Furthermore, different LLM architectures exhibit significant differences in their training data distribution and parameter sizes, resulting in unique cognitive blind spots. This requires customizing the training dataset for each specific target LLM to accommodate its unique architectural characteristics and cognitive blind spots. This process often requires significant computational resources and labor-intensive labeling. The high cost significantly hinders the scalability of this approach and its adaptability to diverse scenarios, making its practical application challenging. Summary of the Invention

[0007] In view of this, it is necessary to provide an efficient and low-cost training method and system that requires a small amount of labeled data to improve the honesty of large models.

[0008] A training method that requires a small amount of labeled data to improve the honesty of a large model. The specific steps include: Step 1: Use a predetermined amount of labeled data to fine-tune the LLM and standardize the output format of the LLM. Step 2: Collect the output of the fine-tuned LLM on the labeled data, as well as the hidden state corresponding to each response; Step 3: Label each collected response using the honesty index to construct preference data; Step 4: Use the preference data to train a lightweight network. Train a lightweight network that takes the hidden state as input and estimates the honesty score of the corresponding response. Step 5: For each unlabeled question, select the answer with the highest prediction score of the lightweight network and the probability weighted by the LLM as the labeled data; Step 6: Use the labeled data to train the LLM. Through training, the LLM learning can meet the honesty requirement.

[0009] Preferably, the step 1 of fine-tuning the LLM using a predetermined amount of labeled data and standardizing the output format of the LLM includes: Step 1.1, set the format of the annotation data to ,in, For the problem, For reply; Step 1.2: Select a pre-trained model and introduce the labeled data into the pre-trained model; In step 1.3, for questions that the model can answer correctly, use the model's response directly. For questions that the model cannot answer correctly, add the string "I am not sure." to the end of the original model's response.

[0010] Preferably, in step 2, the specific steps of collecting the output of the fine-tuned LLM on the labeled data and the hidden state corresponding to each response include: Step 2.1: Input the question to the model, and the model generates multiple responses through beam search. In step 2.2, collect the hidden state of the last token of each reply after each transformer block in the large model.

[0011] Preferably, in step three, each collected response is labeled using an honesty indicator, and the specific steps of constructing preference data include: Step 3.1: Establish a labeling strategy and set different weights based on whether the model's response is correct and whether it contains uncertainty statements; Step 3.2: Label each reply with an honesty indicator based on the labeling strategy; In step 3.3, data format optimization is performed, different outputs corresponding to the same input are grouped into comparison groups, and preference data is constructed.

[0012] Preferably, the specific steps of step 4, using the preference data to train the lightweight network, include: Step 4.1, for the annotation score All answers to each question are normalized using softmax, and the normalized annotation scores are The calculation formula is shown in formula (1): (1); in, n The number of answers to each question ranges from 10 to 40; Step 4.2, calculate the normalized score Prediction score with lightweight network The cross entropy loss between , the calculation formula of the loss function is shown in formula (2): (2).

[0013] In step 4.3, train different lightweight networks for the hidden states of each transformer block in the transformer model, and test the accuracy of the lightweight networks trained at different layers using the validation set. In step 4.4, the lightweight network with the highest accuracy is selected and the hidden state of the corresponding layer of the lightweight network is collected.

[0014] Preferably, the specific steps of step 5, for each unlabeled question, selecting the answer with the highest prediction score of the lightweight network and the highest probability weight of the LLM as the labeled data, include: Step 5.1: Input the question into the LLM and generate multiple responses through beam search, each of which corresponds to a probability value. In step 5.2, the hidden state of the specific layer corresponding to the last token of each reply is input into the lightweight network to generate a prediction score; The specific layer refers to the output layer corresponding to the hidden state used by the lightweight network for input training; In step 5.3, the answer with the highest linear weighting of the lightweight network's prediction score and the LLM's probability is selected as the labeled data.

[0015] And, a training system for improving the honesty of a large model with a small amount of labeled data is provided, which is used to implement the training method for improving the honesty of a large model with a small amount of labeled data as described above, comprising: The LLM module is used to generate multiple answers for unlabeled data and output the probability value and hidden state information corresponding to each answer. Based on the built-in probability model of the LLM module, a specific probability value is assigned to each generated answer, and the probability value reflects the model's self-assessment of the answer's credibility. A lightweight network is used to assess the honesty score of each answer. By learning and analyzing the hidden state information obtained from the LLM module, it identifies features and patterns related to honesty in the answers, providing additional quantitative indicators for the honesty of the answers, allowing the model to judge the reliability of the answers more comprehensively and objectively.

[0016] Preferably, the lightweight network adopts a multi-layer MLP structure, each layer of MLP is connected sequentially, the MLP consists of an input layer, a hidden layer and an output layer, and the neurons between layers are fully connected; learning is performed on limited labeled data, and training is performed through a learning ranking framework to identify the most honest answer among the candidate answers generated by the LLM.

[0017] Preferably, the probability evaluation value output by the LLM module is combined with the honesty evaluation of the lightweight network, and a comprehensive score is obtained by weighted calculation of the probability and honesty score, and then the answer with the highest comprehensive score is selected as the labeled data.

[0018] In the above-mentioned training method and system that requires a small amount of labeled data to improve the honesty of a large model, a lightweight network is newly added to assist the LLM in self-labeling the data, thereby significantly expanding the scale of data available for model training based on limited labeled data, so that the model can still be fully trained and optimized when data resources are limited. A close collaborative working mechanism is formed between the lightweight network and the LLM. The lightweight network generates reliable labeling information for unlabeled data by evaluating the honesty of the answers generated by the LLM. This process not only makes full use of the probability evaluation ability of the LLM itself, but also combines the quantitative evaluation of honesty by the lightweight network, so that the model can learn a more honest answer strategy under limited data conditions, thereby achieving the purpose of enhancing the honesty of the LLM with a small amount of data. The algorithm of the present invention is simple, easy to implement, low-cost, and easy to promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a training method for improving the honesty of a large model that requires a small amount of labeled data according to an embodiment of the present invention.

[0020] Figure 2 2 is a schematic diagram of the structure of a training system for improving the honesty of a large model using a small amount of labeled data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] This embodiment takes the training method and system that requires a small amount of labeled data to improve the honesty of a large model as an example. The present invention will be described in detail below with reference to specific embodiments and drawings.

[0022] See also Figure 1, showing a training method provided by an embodiment of the present invention that requires a small amount of labeled data to improve the honesty of a large model, specifically comprising the following steps: Step 1: Use a predetermined amount of labeled data to fine-tune the LLM and standardize the output format of the LLM on this task. This step performs preliminary optimization on the model so that it can better adapt to the needs of the specific task and lay the foundation for subsequent processing. The format of each labeled data is ,in For the problem, For replies, for questions that the model can answer correctly, use the model's reply directly. For questions that the model cannot answer correctly, add the string "I am not sure." after the original model's reply.

[0023] Specifically, in this embodiment, 2000 labeled data are used to supervise and fine-tune the LLM and standardize the output format of the LLM.

[0024] Step 2: Collect the output of the fine-tuned LLM on the labeled data, as well as the hidden state corresponding to each answer. After the first step of training is complete, the question is fed to the model, which generates multiple responses through beam search. We also collect the output of the last token of each response after each transformer block in the larger model.

[0025] Specifically, the hidden state is a dynamic vector representation of the token at each position in the sequence, generated after each layer of processing within the Transformer model. It contains information about the token at that position and its context. As crucial information within the model, the hidden state provides rich semantic and feature support for subsequent honesty assessments.

[0026] Step three: Label the collected answers using an honesty metric to construct preference data. This step provides clear guidance for model training by quantitatively assessing the honesty of the answers, ensuring that the model learns more honest answering strategies.

[0027] First, for each response, a labeling strategy is established, and different weights are set based on whether the model's response is correct and whether it contains uncertainty. The labeling strategy is: If the model correctly answers the question and expresses no uncertainty, it receives a score of 3.

[0028] If the model gives an incorrect answer and contains uncertainty statements, it is scored as 2.

[0029] If the model gives the correct answer and contains uncertainty statements, it is scored as 1.

[0030] If the model gives an incorrect answer and does not include an uncertainty statement, the score is 0.

[0031] We define uncertainty statements as responses followed by the string “I am not sure.”

[0032] Then, based on the labeling strategy, each reply is labeled with an honesty indicator.

[0033] Specifically, different responses to the same question are sorted and scored, and high weights are set for responses that meet the requirements of accuracy, security, and logic; otherwise, low weights are set.

[0034] Finally, the data format is optimized, and different outputs corresponding to the same input are grouped into comparison groups to construct preference data.

[0035] Step 4: Train a lightweight network that takes the hidden state as input and estimates the honesty score of the corresponding reply. The following describes the specific calculation process. First, for the labeling score The softmax normalization process is performed on the answer to each question, and the normalized annotation score is The calculation formula is: (1); in, n The number of responses for each question.

[0036] Then calculate the normalized score Prediction score with lightweight network The cross entropy loss between , the loss function is calculated as: (2); Among them, in formula (2), n and (1) n The value range is generally set to 10~40.

[0037] Because the Transformer model consists of several layers of Transformer blocks, we train different lightweight networks for the hidden states of each layer. We then test the accuracy of the trained lightweight networks at different layers on a validation set and select the lightweight network with the highest accuracy for subsequent operations. In these operations, we only collect the hidden states of the corresponding layers. The collected hidden states serve as input to the subsequent lightweight networks. The lightweight networks are designed to efficiently evaluate the honesty of answers with minimal computational resources, further improving the model's honesty performance.

[0038] Step 5: For each unlabeled question, select the answer with the highest prediction score of the lightweight network and the probability weighted by the LLM as the labeled data.

[0039] The specific steps are as follows: A question is fed into the large model, and beam search is used to generate multiple responses, each with a corresponding probability value. The hidden state of a specific layer corresponding to the last token in each response is fed into the lightweight network to generate a prediction score. The specific layer refers to the output layer corresponding to the hidden state used during input training of the lightweight network.

[0040] The answer with the highest linear weighting between the lightweight network's prediction score and the LLM's probability is selected as the labeled data. This process combines the lightweight network's evaluation results with the LLM's probability to generate reliable annotation information for the unlabeled data, expanding the data scale and alleviating the data scarcity problem.

[0041] Step 6: Use the labeled data to retrain the LLM. Through this training process, the LLM can learn a more honest answer strategy, thereby outputting more authentic and reliable information in the subsequent generation process.

[0042] Step 7. After optimization and training in the above steps, we can finally obtain an LLM that performs well in terms of honesty. Its output is more trustworthy and can better meet the user's demand for accurate information.

[0043] Using this method, operators can effectively improve the integrity of LLMs, making their outputs more reliable, by relying on only a small amount of labeled data. This process is not only technically innovative but also demonstrates high practical value in real-world applications, providing a practical solution to the difficulty LLMs face in generating reliable outputs under limited supervision.

[0044] like Figure 2Figure 1 illustrates the basic architecture of a training system for improving the honesty of large models using a small amount of labeled data, according to an embodiment of the present invention. This architecture consists of two core modules: an LLM (Large Language Model) and a lightweight network, as well as the labeling process used by these two core modules in a training method for improving the honesty of large models. The LLM module not only provides input for the subsequent lightweight network but also generates responses for screening. The lightweight network's prediction scores work in conjunction with the probability of each response generated by the LLM to provide strong support for determining the final labeled data. The lightweight network utilizes a multi-layer MLP (Multi-Layer Perceptron) structure. It should be noted that in this embodiment, the lightweight network is preferably a three-layer MLP, but this is not a fixed pattern. In other embodiments, different numbers of MLP layers can be used based on actual needs. From a functional perspective, the MLP layers are connected sequentially, and the functional differences between the layers themselves are not significant. The choice of three layers was made after a comprehensive balance between neural network fitting performance and computational efficiency. The functions of each module and unit are detailed below.

[0045] For unlabeled data, the LLM generates multiple responses. It also outputs a probability value and hidden state information for each response. Based on the LLM's built-in probabilistic model, each generated response is assigned a specific probability value. This probability value intuitively reflects the model's own assessment of the response's credibility, serving as a self-assessment of the response's credibility. This critical process lays the foundation for the subsequent honesty evaluation performed by the lightweight network, enabling it to select the most honest responses. Furthermore, the hidden state output by the LLM serves as input to the lightweight network, providing the data foundation for subsequent evaluation processes.

[0046] The lightweight network's primary responsibility is to assess the honesty score of each answer. Specifically, it receives the hidden state output from the LLM as input. By deeply learning and analyzing the hidden state and other relevant information obtained from the LLM, it can accurately identify features and underlying patterns in the answer that are closely related to honesty. This evaluation provides an additional, clear quantitative indicator of the honesty of the answer, helping the entire model to more comprehensively and objectively determine the reliability of the answer.

[0047] Combining the probability of each answer and its corresponding honesty score, the response with the highest weighted value is selected as the labeled data. This process combines the probability assessment of the LLM with the honesty assessment of the lightweight network to ensure that the selected responses have both high probability and high honesty. Specifically, the probability and honesty score are weighted or combined to obtain a comprehensive score, and the response with the highest comprehensive score is selected as the labeled data. This selection mechanism effectively leverages the strengths of both models and provides high-quality data support for model training.

[0048] The invention has shown many significant advantages and positive effects. First of all, the lightweight and efficient honest alignment framework proposed by the present invention greatly reduces the cost of data annotation compared to other solutions that rely on large-scale labeled data. In practical applications, data annotation work often requires a lot of manpower, material resources and time, and the quality of annotation is also affected by factors such as the professional level of the annotation personnel. The present invention can achieve effective training and optimization of LLM by relying only on a small amount of labeled data, thereby significantly reducing the resource investment in data collection and annotation. This not only makes the training process more economical and efficient, but also reduces the dependence on professional annotation teams, providing more users and enterprises with limited resources with the possibility of applying LLM, and further promoting the popularization and development of LLM technology.

[0049] Secondly, the framework of the invention is easy to train during the training process. The number of network parameters involved in data annotation is small, and the resources required for training are also reduced accordingly. In the prior art, some complex model training often requires a large amount of computing resources and storage resources, which places high demands on hardware equipment and also increases energy consumption and equipment costs. The present invention makes the training process simpler and faster by streamlining network parameters, reducing the consumption of hardware resources. This not only improves training efficiency, but also reduces energy consumption and equipment costs, conforms to the current development trend of green energy conservation, and has good environmental and social benefits.

[0050] Furthermore, the present invention introduces a lightweight network that learns on limited labeled data and is trained through a learning-to-rank framework, so that it can identify the most honest answer among the candidate answers generated by the LLM. This innovative design not only significantly reduces the demand for labeled data and effectively alleviates the problem of data scarcity, but also demonstrates strong performance advantages in practical applications. Based on limited data, the lightweight network can still maintain good learning effects, accurately evaluate the honesty of candidate answers, and improve the output quality of the model. In addition, the introduction of the lightweight network also makes the entire system more flexible and efficient, and can quickly adapt to different application scenarios and changes in demand, providing users with a more personalized and precise service experience.

[0051] In response to the problem of lack of reference answers, the present invention combines the scores predicted by the lightweight network with the prediction probabilities of the LLM to generate pseudo labels. This mechanism has important significance and advantages. It extends honest alignment to unlabeled data, greatly reducing the reliance on manual labeling. In practical applications, unlabeled data often accounts for the majority of the data, and the present invention can make full use of this data, further improving the training effect of the model. By generating pseudo labels, the model can remain honest in the absence of clear reference answers, thereby improving the generalization ability and adaptability of the model. This mechanism also provides the possibility for self-learning and self-optimization of the model, so that the model can continue to improve and improve in practical applications, and better meet the growing needs of users.

[0052] Finally, the present invention uses a synthetic dataset to fine-tune the basic LLM, combining the specialized scoring capabilities of the lightweight network with the generalization capabilities of the LLM, avoiding dependence on a specific model architecture and creating a unified training process that can adapt to multiple architectures. This mechanism not only improves the versatility and flexibility of the model, but also provides broad space for further optimization and improvement of the model. By combining the specialized scoring capabilities of the lightweight network with the generalization capabilities of the LLM, the model can maintain its honesty while also possessing stronger generalization capabilities and adaptability, and can better cope with various complex application scenarios and changes in demand. In addition, this mechanism also provides new ideas and methods for subsequent research and development, promoting the further development and innovation of LLM technology.

[0053] In summary, through the above-mentioned mechanism, this invention enables efficient LLM training and optimization without relying on large-scale annotated data, providing an innovative and efficient technical solution for honest alignment of language models. This solution not only has significant technical advantages but also demonstrates wide applicability and positive effects in practical applications, providing strong support for the intelligent development of various industries. It has important practical significance and broad application prospects.

[0054] The core innovation of this invention lies in the addition of a lightweight network to effectively reduce data annotation costs. The design concept and functional positioning of this lightweight network are to assist the LLM in self-annotating data, thereby significantly expanding the amount of data available for model training based on limited annotated data. This design cleverly addresses the problem of data scarcity, enabling the model to be fully trained and optimized even with limited data resources.

[0055] The challenge of this invention lies in how to enhance the honesty of the LLM using a small amount of data. To address this challenge, a close collaborative mechanism is established between the lightweight network and the LLM. The lightweight network evaluates the honesty of the answers generated by the LLM, generating reliable annotation information for the unlabeled data. This process not only fully utilizes the LLM's inherent probabilistic assessment capabilities but also combines the lightweight network's quantitative honesty assessment, enabling the model to learn more honest answering strategies even with limited data.

[0056] It should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A training method that requires a small amount of labeled data to improve the honesty of a large model, characterized by: The specific steps include: Step 1: Use a predetermined amount of labeled data to fine-tune the LLM and standardize the output format of the LLM. Step 2: Collect the output of the fine-tuned LLM on the labeled data, as well as the hidden state corresponding to each response; Step 3: Label each collected response using the honesty index to construct preference data; Step 4: Use the preference data to train a lightweight network. Train a lightweight network that takes the hidden state as input and estimates the honesty score of the corresponding response. Step 5: For each unlabeled question, select the answer with the highest prediction score of the lightweight network and the probability weighted by the LLM as the labeled data; Step 6: Use the labeled data to train the LLM. Through training, the LLM learning can meet the honesty requirement.

2. The training method for improving the honesty of a large model requiring a small amount of labeled data as claimed in claim 1, characterized in that: In step 1, supervised fine-tuning of the LLM using a predetermined amount of labeled data and standardizing the output format of the LLM include: Step 1.1, set the format of the annotation data to ,in, For the problem, For reply; Step 1.2: Select a pre-trained model and introduce the labeled data into the pre-trained model; In step 1.3, for questions that the model can answer correctly, use the model's response directly. For questions that the model cannot answer correctly, add the string "I am not sure." to the end of the original model's response.

3. The training method for improving the honesty of a large model requiring a small amount of labeled data as claimed in claim 1, characterized in that: Step 2: Collect the output of the fine-tuned LLM on the labeled data, and the specific steps for the hidden state corresponding to each response include: Step 2.1: Input the question to the model, and the model generates multiple responses through beam search. In step 2.2, collect the hidden state of the last token of each reply after each transformer block in the large model.

4. The training method for improving the honesty of a large model requiring a small amount of labeled data as claimed in claim 3, characterized in that: In step 3, each collected response is labeled with an honesty indicator. The specific steps for constructing preference data include: Step 3.1: Establish a labeling strategy and set different weights based on whether the model's response is correct and whether it contains uncertainty statements; Step 3.2: Label each reply with an honesty indicator based on the labeling strategy; In step 3.3, data format optimization is performed, different outputs corresponding to the same input are grouped into comparison groups, and preference data is constructed.

5. The training method for improving the honesty of a large model requiring a small amount of labeled data as claimed in claim 1, characterized in that: The specific steps of step 4, using the preference data to train the lightweight network, include: Step 4.1, for the annotation score All answers to each question are normalized using softmax, and the normalized annotation scores are The calculation formula is shown in formula (1): （1）； in, n The number of answers to each question ranges from 10 to 40; Step 4.2, calculate the normalized score Prediction score with lightweight network The cross entropy loss between , the calculation formula of the loss function is shown in formula (2): （2）； In step 4.3, train different lightweight networks for the hidden states of each transformer block in the transformer model, and test the accuracy of the lightweight networks trained at different layers using the validation set. In step 4.4, the lightweight network with the highest accuracy is selected and the hidden state of the corresponding layer of the lightweight network is collected.

6. The training method for improving the honesty of a large model requiring a small amount of labeled data as claimed in claim 1, characterized in that: The specific steps of step 5, for each unlabeled question, selecting the answer with the highest prediction score of the lightweight network and the highest probability weight of the LLM as the labeled data include: Step 5.1: Input the question into the LLM and generate multiple responses through beam search, each response corresponding to a probability value; In step 5.2, the hidden state of the specific layer corresponding to the last token of each reply is input into the lightweight network to generate a prediction score; The specific layer refers to the output layer corresponding to the hidden state used by the lightweight network for input training; In step 5.3, the answer with the highest linear weighting of the lightweight network's prediction score and the LLM's probability is selected as the labeled data.

7. A training system for improving the honesty of a large model with a small amount of labeled data, used to implement the training method for improving the honesty of a large model with a small amount of labeled data as described in any one of claims 1-6, characterized in that: include: The LLM module is used to generate multiple answers for unlabeled data and output the probability value and hidden state information corresponding to each answer; Based on the built-in probability model of the LLM module, a specific probability value is assigned to each generated answer, and the probability value reflects the model's self-assessment of the credibility of the answer; A lightweight network is used to assess the honesty score of each answer. By learning and analyzing the hidden state information obtained from the LLM module, it identifies features and patterns related to honesty in the answers, providing additional quantitative indicators for the honesty of the answers, allowing the model to judge the reliability of the answers more comprehensively and objectively.

8. The training system for improving the honesty of a large model using a small amount of labeled data as claimed in claim 7, characterized in that: The lightweight network adopts a multi-layer MLP structure, where each layer of MLP is connected sequentially. The MLP consists of an input layer, a hidden layer, and an output layer, and the neurons between layers are fully connected. It learns on limited labeled data and is trained through a learning-to-rank framework to identify the most honest answer among the candidate answers generated by LLM.

9. The training system for improving the honesty of a large model using a small amount of labeled data as claimed in claim 7, characterized in that: The probability evaluation value output by the LLM module is combined with the honesty evaluation of the lightweight network. A comprehensive score is obtained by weighted calculation of the probability and honesty score, and then the answer with the highest comprehensive score is selected as the labeled data.

Images