Model fine tuning method and device for cross-inquiry big language model reply reliability evaluation
By acquiring sample question-answer pairs and using a reliability assessment model for cross-question comparison and score aggregation, this study addresses the problem that weak language models struggle to assess the reliability of responses from large language models. It achieves greater accuracy and efficiency in assessing the reliability of responses from large language models, thereby improving the efficiency and local deployment capabilities of weak language models.
Patent Information
- Application Number
- CN202411353270.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-01-10
AI Technical Summary
In existing technologies, weak language models are difficult to effectively evaluate the reliability of responses from large language models, and the high computational overhead and closed-source nature of large language models limit their use in local environments.
By acquiring a sample question-answer pair dataset, the target question-answer pairs are evaluated using a reliability assessment model. Cross-question comparison and score aggregation are performed, and meta-ranking techniques are used to fine-tune the reliability assessment model, thereby enhancing the reliability assessment capability of the large language model.
It achieves accuracy and efficiency in the reliability assessment of responses from large language models, avoids the interference of overconfidence and few-sample context learning, and enhances the advantages of weak language models in terms of efficiency and local deployment.
Smart Images
Figure CN119293170A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of language model technology, and in particular to a model fine-tuning method and device for evaluating the reliability of responses of a large language model across queries. Background Art
[0002] Stronger large language models have high potential for assessing answer reliability. However, these models are usually very large, and their use often brings high computational overhead and economic costs. In addition, most of these models are closed source, which limits their use in local environments. In contrast, weaker models are usually more efficient and have local deployment advantages.
[0003] Therefore, how to make the weak model effectively evaluate the response reliability of the large language model and realize the advantages of the weak model in efficiency and local deployment is a technical problem that technicians in this field need to solve. Summary of the invention
[0004] The present invention provides a model fine-tuning method and device for evaluating the reliability of cross-query large language model responses, so as to solve the defect in the prior art that weak language models are difficult to effectively evaluate the reliability of large language model responses, and realize effective evaluation of the reliability of large language model responses.
[0005] The present invention provides a model fine-tuning method for evaluating the reliability of a large language model response across queries, comprising the following steps: S1. Obtain a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; S2. Evaluate the reliability of the response of the target question-answer pair by using the reliability evaluation model to obtain an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; S3, evaluating the reliability of responses of each sample question and answer pair in the sample question and answer pair data set by using the reliability evaluation model, and obtaining a sample response reliability evaluation result of each sample question and answer pair; S4, based on the response reliability label information of each of the sample question-answer pairs, comparing the target response reliability initial evaluation result with the sample response reliability evaluation result of each of the sample question-answer pairs; S5. Aggregate the comparison results to obtain a response reliability evaluation score for the target question-answer pair; S6. Fine-tune the parameters of the reliability assessment model, and repeat S2-S5 using the reliability assessment model after parameter fine-tuning until the obtained reliability assessment score of the response of the target question and answer pair reaches a preset indicator.
[0006] According to the model fine-tuning method for evaluating the reliability of responses of a large language model across queries provided by the present invention, the step of obtaining a sample question-answer pair dataset specifically includes: Obtaining multiple reasoning questions of the target question and answer in the same field and in different fields; Obtaining answers to the reasoning questions through a language model, wherein the reasoning questions and the corresponding answers form sample question-answer pairs; Annotating the response reliability of each of the sample question-answer pairs to form response reliability label information of each of the sample question-answer pairs; The sample question-answer pairs are randomly matched in pairs to form a sample question-answer pair data set.
[0007] According to the model fine-tuning method for cross-query large language model response reliability evaluation provided by the present invention, the step of comparing the target response reliability initial evaluation result with the sample response reliability evaluation result of each sample question-answer pair specifically includes: Assume the target question-answer pair is , the i-th sample question-answer pair is , the comparison result of the target response reliability initial evaluation result and the sample response reliability evaluation result of the i-th sample question-answer pair is:
[0008] in, , respectively, indicate that the target pair is inferior to, equal to, or better than the i-th reference sample in reliability evaluation, and N is the total number of samples.
[0009] According to the model fine-tuning method for cross-query large language model response reliability evaluation provided by the present invention, the steps of aggregating the comparison results to obtain the response reliability evaluation score of the target question-answer pair specifically include: Based on the comparison result of the target response reliability initial evaluation result and the sample response reliability evaluation result of the ith sample question and answer pair, the ith reliability evaluation score of the target question and answer pair and the ith sample question and answer pair is obtained. : ; in, represents the symbolic function, is a hyperparameter used to adjust the weight of each sample pair; represents the reliability of the sample response of the i-th sample question-answer pair under independent evaluation, when When , it means that the reliability of the sample response of the i-th sample question and answer pair is high. When , it indicates that its reliability is low; Add up the reliability evaluation scores to get the response reliability evaluation score of the target question-answer pair. :
[0010] The reliability evaluation score of the response based on the target question-answer pair , and obtain the response reliability of the target question-answer pair : ; in, If the sign of is positive, it means the response is reliable; If the sign of is negative, it means the response is unreliable.
[0011] According to the model fine-tuning method for cross-query large language model response reliability assessment provided by the present invention, the step of fine-tuning the parameters of the reliability assessment model specifically includes: The initial learning rate, batch size and fine-tuning steps are set, and the parameters of the reliability assessment model are fine-tuned using the AdamW optimizer.
[0012] According to the model fine-tuning method for reliability evaluation of cross-query large language model responses provided by the present invention, the reliability evaluation result includes performance indicators and efficiency indicators; The performance index is characterized by the accuracy of the reliability evaluation model in recovering errors; and the efficiency index is characterized by the reasoning time of the reliability evaluation model.
[0013] The present invention also provides a model fine-tuning device for evaluating the reliability of responses of a large language model across queries, comprising the following modules: An acquisition module, configured to acquire a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; An initial reliability evaluation module is used to evaluate the reliability of the response of the target question-answer pair through a reliability evaluation model to obtain an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; A sample response reliability evaluation module is used to evaluate the response reliability of each sample question and answer pair in the sample question and answer pair data set through the reliability evaluation model to obtain a sample response reliability evaluation result of each sample question and answer pair; A comparison module, configured to compare the target response reliability initial evaluation result with the sample response reliability evaluation result of each of the sample question-answer pairs based on the response reliability label information of each of the sample question-answer pairs; An aggregation module, used to aggregate the comparison results to obtain a response reliability evaluation score of the target question-answer pair; A fine-tuning module is used to fine-tune the parameters of the reliability assessment model, and to re-call the reliability initial assessment module, the sample response reliability assessment module, the comparison module, and the aggregation module until the response reliability assessment score of the target question-answer pair reaches a preset index. The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a model fine-tuning method for reliability assessment of cross-query large language model responses as described in any of the above is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a model fine-tuning method for evaluating the reliability of responses of a large language model across queries as described in any one of the above.
[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned model fine-tuning methods for evaluating the reliability of large language model responses across queries.
[0016] The model fine-tuning method and device for evaluating the reliability of responses of a large language model across queries provided by the present invention obtain a sample question and answer pair data set; evaluate the reliability of responses of a target question and answer pair through a reliability evaluation model to obtain an initial evaluation result of the reliability of the target response; evaluate each sample question and answer pair in the sample question and answer pair data set through the reliability evaluation model to obtain a sample response reliability evaluation result of each sample question and answer pair; compare the initial evaluation result of the target response reliability with the sample response reliability evaluation result of each sample question and answer pair based on the response reliability label information of each sample question and answer pair; aggregate each comparison result to obtain a response reliability evaluation score of the target question and answer pair; fine-tune the parameters of the reliability evaluation model, and use the reliability evaluation model after parameter fine-tuning to repeatedly perform reliability evaluation, reliability evaluation result comparison and comparison result aggregation until the response reliability evaluation score of the target question and answer pair reaches a preset indicator. The target query-response binary pair is compared with multiple reference samples across queries; a set of pre-scored query-response pairs is used as a reference, and for any given query-response pair to be judged, the binary pair is compared based on the large language model to see whether it is more reliable than each reference sample, and then a voting mechanism is used to aggregate the comparison results and make a final reliability judgment; compared with traditional technologies, the present invention avoids the problem of overconfidence in the language model's judgment of the responses generated by the large language model, as well as the interference problem of the example order in few-sample context learning, and based on the weaker large language model, the reliability of the large language model's responses can be effectively evaluated in reasoning tasks, giving full play to the advantages of the weak language model in efficiency and local deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is a schematic diagram comparing the process of the meta-sorting method provided by an embodiment of the present invention with other existing methods.
[0019] Figure 2 It is a flow chart of a model fine-tuning method for evaluating the reliability of responses of a large language model across queries provided by the present invention.
[0020] Figure 3 It is a structural schematic diagram of a model fine-tuning device for evaluating the reliability of responses of a large language model across queries provided by the present invention.
[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] The present invention will be described in detail below in conjunction with specific implementation modes.
[0024] In some specific embodiments of the present invention, Figure 1 As shown, this solution provides a model fine-tuning method for evaluating the reliability of responses of a large language model across queries, and the method includes the following steps: Step S1: Obtain a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; Step S2: Evaluate the reliability of the response of the target question-answer pair by using the reliability evaluation model to obtain an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; Step S3: Evaluate each sample question and answer pair in the sample question and answer pair data set using the reliability evaluation model to obtain a sample response reliability evaluation result of each sample question and answer pair; Step S4: based on the response reliability label information of each of the sample question-answer pairs, compare the target response reliability initial evaluation result with the sample response reliability evaluation result of each of the sample question-answer pairs; Step S5: Aggregate the comparison results to obtain a response reliability evaluation score for the target question-answer pair; Step S6: fine-tune the parameters of the reliability assessment model, and repeat steps S2-S5 using the reliability assessment model after parameter fine-tuning until the response reliability assessment score of the target question and answer pair reaches a preset indicator.
[0025] It should be noted that existing large language models (LLMs) perform well in various tasks. However, they still have reliability issues. For example, these models often generate seemingly reasonable but actually wrong responses, a phenomenon known as "hallucination". Stronger large language models have high potential in evaluating the reliability of responses. For example, strong models at the level of GPT-4-turbo can effectively evaluate the quality of large language model responses by learning from the context of a small number of examples. However, these models are usually very large, and their use often brings high computational overhead and economic costs. In addition, most of these models are closed source, limiting their use in local environments.
[0026] Therefore, in order to improve the reliability of large language models (LLMs) in multi-query responses, this solution proposes a model fine-tuning method for evaluating the reliability of large language model responses across queries. Based on the fine-tuning technology of meta ranking, the language model is fine-tuned, and then the reliability of the large language model responses corresponding to different queries is effectively judged. This method evaluates and enhances the reliability judgment ability of the large language model for a given query-response pair through cross-query comparison and score aggregation, and makes it more accurate and efficient in evaluating the reliability of responses by fine-tuning the model based on meta ranking technology.
[0027] Specifically, meta-ranking is a large-scale response evaluation method based on cross-query comparison and score aggregation ranking. The specific scheme is as follows Figure 2As shown in the figure, unlike the traditional method of letting the large language model directly evaluate its own responses based on contextual learning, meta-ranking compares the target query-response binary pair with multiple reference samples across queries; meta-ranking uses a set of pre-scored query-response pairs as references. For any given query-response pair to be judged, it compares whether the binary pair is more reliable than each reference sample based on the large language model, and then uses a voting mechanism, i.e., an aggregation method, to aggregate these comparison results to obtain the final reliability judgment. In theory, this method avoids the problem of overconfidence in the language model's judgment of the responses generated by the large language model, as well as the interference problem of example order in few-shot contextual learning. The results of the verification experiment show that the meta-ranking method, based on a weaker large language model, can effectively evaluate the reliability of the large language model's responses in reasoning tasks, and is even stronger than the existing methods in GPT-3.5-turbo.
[0028] In some embodiments of the present invention, the step of obtaining a sample question-answer pair dataset specifically includes: Obtaining multiple reasoning questions of the target question and answer in the same field and in different fields; Obtaining answers to the reasoning questions through a language model, wherein the reasoning questions and the corresponding answers form sample question-answer pairs; Annotating the response reliability of each of the sample question-answer pairs to form response reliability label information of each of the sample question-answer pairs; The sample question-answer pairs are randomly matched in pairs to form a sample question-answer pair data set.
[0029] Specifically, this embodiment provides an implementation method for obtaining a sample question and answer pair dataset by obtaining multiple reasoning questions in the same field and different fields of the target question and answer pair, and constructing a cross-query sample question and answer pair dataset based on the sample question and answer pairs and the response reliability label information of the sample question and answer pairs.
[0030] In a possible embodiment, in order to perform model fine-tuning, a set of high-quality query-response datasets are first collected and prepared. These datasets include MMLU datasets and CMMLU datasets: MMLU dataset: Serves as an English benchmark and includes English reasoning problems from various domains.
[0031] CMMLU dataset: used as a Chinese benchmark, including Chinese reasoning problems from various domains.
[0032] In a possible embodiment, each dataset may contain multiple-choice questions and their corresponding labeled samples, with each subject having five labeled samples in the development set.
[0033] In a possible embodiment, the LLaMA-2 model can be used to generate responses to the above reasoning questions, and the correctness of the responses can be determined by true labels; all query-response pairs are randomly matched in pairs to complete data collection.
[0034] In some embodiments of the present invention, the step of comparing the target response reliability initial evaluation result with the sample response reliability evaluation result of each sample question-answer pair specifically includes: Assume the target question-answer pair is , the i-th sample question-answer pair is , the comparison result of the target response reliability initial evaluation result and the sample response reliability evaluation result of the i-th sample question-answer pair is: (1); in, , respectively, indicate that the target pair is inferior to, equal to, or better than the i-th reference sample in reliability evaluation, and N is the total number of samples.
[0035] Specifically, this embodiment provides an implementation method for comparing the response reliability evaluation results of the target question and answer pair and the sample question and answer pair. By comparing the response reliability of the target question and answer pair with each sample question and answer pair one by one, the comparison result of the response reliability of the target question and answer pair with each sample question and answer pair is obtained, thereby realizing cross-query comparison of the target question and answer pair.
[0036] In a possible embodiment, the comparison result is represented by three numerical values: -1, 0, and 1. The above three numerical values can intuitively show the superiority or inferiority of the response reliability evaluation result of the target question and answer pair compared with the response reliability evaluation result of the sample question and answer pair.
[0037] In some embodiments of the present invention, the step of aggregating the comparison results to obtain the response reliability evaluation score of the target question-answer pair specifically includes: Based on the comparison result of the target response reliability initial evaluation result and the sample response reliability evaluation result of the ith sample question and answer pair, the ith reliability evaluation score of the target question and answer pair and the ith sample question and answer pair is obtained. : (2); in, represents the symbolic function, is a hyperparameter used to adjust the weight; represents the reliability of the sample response of the i-th sample question-answer pair under independent evaluation, when When , it means that the reliability of the sample response of the i-th sample question and answer pair is high. When , it indicates that its reliability is low; Add up the reliability evaluation scores to get the response reliability evaluation score of the target question-answer pair. : (3); The reliability evaluation score of the response based on the target question-answer pair , and obtain the response reliability of the target question-answer pair : (4); in, If the sign of is positive, it means the response is reliable; If the sign of is negative, it means the response is unreliable.
[0038] Specifically, this embodiment provides an implementation method for aggregating the comparison results, which is essentially to adopt a voting method, and regard the comparison result of the response reliability of the target question and answer pair with that of each sample question and answer pair as a voting result. By aggregating all the voting results, we can obtain the response reliability score of the target question and answer pair relative to the cross-query sample question and answer pairs in different fields, where a positive score indicates that the response of the target question and answer pair is reliable, and vice versa, if the score is negative, it indicates that the response of the target question and answer pair is unreliable.
[0039] In a possible embodiment, by setting the hyperparameters The value of is used to adjust the voting weight and ensure the stability of the reliability evaluation model.
[0040] In some embodiments of the present invention, the step of fine-tuning the parameters of the reliability assessment model specifically includes: The initial learning rate, batch size and fine-tuning steps are set, and the parameters of the reliability assessment model are fine-tuned using the AdamW optimizer.
[0041] Specifically, this embodiment provides an implementation method for fine-tuning the parameters of the reliability assessment model, and the parameters of the reliability assessment model are fine-tuned by an optimizer.
[0042] In a possible embodiment, during the fine-tuning process, specific settings may be made for different models.
[0043] For example, the selected reliability assessment models include basic models such as LLaMA-2, ChatGLM-2, OpenChat-3.5, GPT-3.5-turbo and Phi-2 models.
[0044] On this basis, for the selection of hyperparameters, different Values, such as , , to adjust the voting weight.
[0045] Furthermore, the fine-tuning process includes using the AdamW optimizer, with the initial learning rate set to 1e-5 and the batch size set to 16, and fine-tuning for 5 epochs on each model to ensure the best performance with limited computing resources.
[0046] In some embodiments of the present invention, the reliability evaluation result includes a performance index and an efficiency index; The performance index is characterized by the accuracy of the reliability evaluation model in recovering errors; and the efficiency index is characterized by the reasoning time of the reliability evaluation model.
[0047] Specifically, this embodiment provides an implementation method of a reliability evaluation result, which quantitatively characterizes the reliability evaluation result of the reliability evaluation model through performance indicators and efficiency indicators.
[0048] In a possible embodiment, the accuracy of the reliability assessment model's response error and the reasoning time of the reliability assessment model are used as the main evaluation indicators.
[0049] In some embodiments of the present invention, in order to verify the effectiveness of the fine-tuning method based on the meta-ranking technology, a verification experiment is designed, which specifically includes the following aspects: First, the dataset: Specifically, the MMLU and CMMLU datasets are used.
[0050] Second, evaluation indicators: Specifically, the accuracy of finding the model reply error is used as the main performance metric, and the inference time (seconds) is used as the efficiency metric.
[0051] Thirdly, experimental implementation: Specifically, meta-ranking is implemented through prompts on LLaMA-2, ChatGLM-2, OpenChat-3.5, GPT-3.5-turbo, and Phi-2 models, and the models are fine-tuned. In particular, the fine-tuned JudgeLM-7B-v1 model is tested to understand the impact of fine-tuning on meta-ranking evaluation.
[0052] Fourth, the baseline method involved in the comparison: Specifically, we compare with few-shot context learning methods, including Unsure Choice, NumSemSets, Entropy, and P(True).
[0053] Fifth, experimental results and analysis: Specifically, the fine-tuned Phi-2 model significantly outperforms most baseline methods in finding errors, and even reaches the performance of the GPT-4-turbo method in some cases. At the same time, the fine-tuning method shows a significant advantage in efficiency, and can achieve high-precision evaluation using only a small number of reference samples.
[0054] The above experimental results show that the fine-tuning method based on meta-ranking technology performs well on all tested large language models.
[0055] In some embodiments of the present invention, a model fine-tuning device for cross-query large language model response reliability assessment is provided. The model fine-tuning device for cross-query large language model response reliability assessment provided by the present invention is described below. The model fine-tuning device for cross-query large language model response reliability assessment described below and the model fine-tuning method for cross-query large language model response reliability assessment described above can be referenced to each other.
[0056] Figure 3 is a schematic diagram of the structure of a model fine-tuning device for evaluating the reliability of a large language model response across queries provided by the present invention. Figure 3 As shown, the device includes the following modules: An acquisition module 31 is used to acquire a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; The reliability initial evaluation module 32 is used to evaluate the reliability of the response of the target question-answer pair through the reliability evaluation model to obtain the target response reliability initial evaluation result, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; A sample response reliability evaluation module 33 is used to evaluate the response reliability of each sample question and answer pair in the sample question and answer pair data set by using the reliability evaluation model to obtain a sample response reliability evaluation result of each sample question and answer pair; A comparison module 34, configured to compare the target response reliability initial evaluation result with the sample response reliability evaluation result of each of the sample question-answer pairs based on the response reliability label information of each of the sample question-answer pairs; Aggregation module 35, used to aggregate the comparison results to obtain a response reliability evaluation score of the target question-answer pair; The fine-tuning module 36 is used to fine-tune the parameters of the reliability assessment model, and re-call the reliability initial assessment module, the sample response reliability assessment module, the comparison module and the aggregation module until the response reliability assessment score of the target question and answer pair reaches a preset indicator.
[0057] The model fine-tuning device for cross-query large language model response reliability assessment provided in the embodiment of the present invention has similar implementation principles and beneficial effects to the implementation principles and beneficial effects of the model fine-tuning method for cross-query large language model response reliability assessment shown in the above embodiment. Please refer to the implementation principles and beneficial effects of the model fine-tuning method for cross-query large language model response reliability assessment shown in the above embodiment, and no further details will be given here.
[0058] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a model fine-tuning method for evaluating the reliability of responses of a large language model across queries, the method comprising: S1, obtaining a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; S2, evaluating the reliability of the response of the target question-answer pair through a reliability evaluation model, and obtaining an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; S3, evaluating each sample in the sample question-answer pair dataset through the reliability evaluation model S4: Evaluate the response reliability of the question and answer pairs to obtain the sample response reliability evaluation results of each sample question and answer pair; S5: Aggregate the comparison results to obtain the response reliability evaluation score of the target question and answer pair; S6: Fine-tune the parameters of the reliability evaluation model, and repeat S2-S5 using the reliability evaluation model after parameter fine-tuning until the response reliability evaluation score of the target question and answer pair reaches the preset index.
[0059] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0060] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model fine-tuning method for evaluating the reliability of cross-query large language model responses provided by the above methods, and the method includes: S1, obtaining a sample question and answer pair dataset, wherein the sample question and answer pair dataset involves multiple fields, wherein each field includes at least one sample question and answer pair; S2, evaluating the response reliability of the target question and answer pair through a reliability evaluation model to obtain an initial evaluation result of the target response reliability, wherein the target question and answer pair is evaluated by the large language model to be reliable. Model generation; S3, evaluate the reply reliability of each sample question and answer pair in the sample question and answer pair data set through the reliability evaluation model, and obtain the sample reply reliability evaluation result of each sample question and answer pair; S4, based on the reply reliability label information of each sample question and answer pair, compare the target reply reliability initial evaluation result with the sample reply reliability evaluation result of each sample question and answer pair; S5, aggregate the comparison results to obtain the reply reliability evaluation score of the target question and answer pair; S6, fine-tune the parameters of the reliability evaluation model, and repeat S2-S5 using the reliability evaluation model after parameter fine-tuning until the reply reliability evaluation score of the target question and answer pair reaches the preset indicator.
[0061] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the model fine-tuning method for evaluating the reliability of responses of a large language model across queries provided by the above methods, the method comprising: S1, obtaining a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; S2, evaluating the reliability of responses of a target question-answer pair through a reliability evaluation model, and obtaining an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by a large language model to be evaluated for reliability; S3, evaluating the reliability of the response of a target question-answer pair through a reliability evaluation model, and obtaining an initial evaluation result of the reliability of the target response. The estimation model evaluates the response reliability of each sample question and answer pair in the sample question and answer pair data set to obtain the sample response reliability evaluation result of each sample question and answer pair; S4, based on the response reliability label information of each sample question and answer pair, the target response reliability initial evaluation result is compared with the sample response reliability evaluation result of each sample question and answer pair; S5, aggregates the comparison results to obtain the response reliability evaluation score of the target question and answer pair; S6, fine-tunes the parameters of the reliability evaluation model, and repeats S2-S5 using the reliability evaluation model after parameter fine-tuning until the response reliability evaluation score of the target question and answer pair reaches the preset indicator.
[0062] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0063] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A model fine-tuning method for evaluating the reliability of large language model responses across queries, characterized in that: include: S1. Obtain a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; S2. Evaluate the reliability of the response of the target question-answer pair by using the reliability evaluation model to obtain an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; S3, evaluating the reliability of responses of each sample question and answer pair in the sample question and answer pair data set by using the reliability evaluation model, and obtaining a sample response reliability evaluation result of each sample question and answer pair; S4, based on the response reliability label information of each of the sample question-answer pairs, comparing the target response reliability initial evaluation result with the sample response reliability evaluation result of each of the sample question-answer pairs; S5. Aggregate the comparison results to obtain a response reliability evaluation score for the target question-answer pair; S6. Fine-tune the parameters of the reliability assessment model, and repeat S2-S5 using the reliability assessment model after parameter fine-tuning until the obtained reliability assessment score of the response of the target question and answer pair reaches a preset indicator.
2. The model fine-tuning method for evaluating the reliability of a large language model response across queries according to claim 1, characterized in that: The steps to obtain the sample question-answer pair dataset include: Obtaining multiple reasoning questions of the target question and answer in the same field and in different fields; Obtaining answers to the reasoning questions through a language model, wherein the reasoning questions and the corresponding answers form sample question-answer pairs; Annotating the response reliability of each of the sample question-answer pairs to form response reliability label information of each of the sample question-answer pairs; The sample question-answer pairs are randomly matched in pairs to form a sample question-answer pair data set.
3. The model fine-tuning method for evaluating the reliability of a large language model response across queries according to claim 1, characterized in that: The step of comparing the target response reliability initial evaluation result with the sample response reliability evaluation result of each sample question-answer pair specifically comprises: Let the target question-answer pair be , the i-th sample question-answer pair is , the comparison result of the target response reliability initial evaluation result and the sample response reliability evaluation result of the i-th sample question-answer pair is: ； in, , respectively, indicate that the target pair is inferior to, equal to, or better than the i-th reference sample in reliability evaluation, and N is the total number of samples.
4. The model fine-tuning method for evaluating the reliability of a large language model response across queries according to claim 3, characterized in that: The step of aggregating the comparison results to obtain the response reliability evaluation score of the target question-answer pair specifically includes: Based on the comparison result of the target response reliability initial evaluation result and the sample response reliability evaluation result of the ith sample question and answer pair, the ith reliability evaluation score of the target question and answer pair and the ith sample question and answer pair is obtained. : ； in, represents the symbolic function, is a hyperparameter used to adjust the weight of each sample pair; , represents the reliability of the sample response of the i-th sample question-answer pair under independent evaluation, when When , it means that the reliability of the sample response of the i-th sample question and answer pair is high. When , it indicates that its reliability is low; Add up the reliability evaluation scores to get the response reliability evaluation score of the target question-answer pair. : The reliability evaluation score of the response based on the target question-answer pair , and obtain the response reliability of the target question-answer pair : ； in, If the sign of is positive, it means the response is reliable; If the sign of is negative, it means the response is unreliable.
5. The model fine-tuning method for evaluating the reliability of a large language model response across queries according to claim 4, characterized in that: The steps for fine-tuning the parameters of the reliability assessment model include: The initial learning rate, batch size and fine-tuning steps are set, and the parameters of the reliability assessment model are fine-tuned using the AdamW optimizer.
6. The model fine-tuning method for evaluating the reliability of a large language model response across queries according to any one of claims 1 to 5, characterized in that: The reliability evaluation results include performance indicators and efficiency indicators; The performance index is characterized by the accuracy of the reliability evaluation model in recovering errors; and the efficiency index is characterized by the reasoning time of the reliability evaluation model.
7. A model fine-tuning device for evaluating the reliability of a large language model response across queries, characterized in that: include: An acquisition module, configured to acquire a sample question-answer pair dataset, wherein the sample question-answer pair dataset involves multiple fields, wherein each field includes at least one sample question-answer pair; An initial reliability evaluation module is used to evaluate the reliability of the response of the target question-answer pair through a reliability evaluation model to obtain an initial evaluation result of the reliability of the target response, wherein the target question-answer pair is generated by the large language model to be evaluated for reliability; A sample response reliability evaluation module is used to evaluate the response reliability of each sample question and answer pair in the sample question and answer pair data set through the reliability evaluation model to obtain a sample response reliability evaluation result of each sample question and answer pair; A comparison module, configured to compare the target response reliability initial evaluation result with the sample response reliability evaluation result of each of the sample question-answer pairs based on the response reliability label information of each of the sample question-answer pairs; An aggregation module, used to aggregate the comparison results to obtain a response reliability evaluation score of the target question-answer pair; The fine-tuning module is used to fine-tune the parameters of the reliability assessment model, and re-call the reliability initial assessment module, the sample response reliability assessment module, the comparison module and the aggregation module until the response reliability assessment score of the target question and answer pair reaches a preset indicator.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the model fine-tuning method for cross-query large language model response reliability evaluation is implemented as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model fine-tuning method for evaluating the reliability of cross-query large language model responses is implemented as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model fine-tuning method for evaluating the reliability of cross-query large language model responses is implemented as described in any one of claims 1 to 6.