Method and system for optimizing answer loyalty of large language model based on multi-dimensional reward function

Abstract

The invention discloses a method and system for optimizing answer loyalty of a large language model based on a multi-dimensional reward function, and relates to the technical field of artificial intelligence, and the method comprises the steps: employing the large language model to generate a long text answer and a short text answer of a question based on context; a short text accurate reward function is adopted to calculate a first reward score of the short text answer; calculating a second reward score of the long text answer by adopting a long text proxy reward function; calculating third reward scores of the long text answer and the short text answer by adopting a format reward function; adding the first reward function, the second reward function and the third reward function to obtain a final reward function; and based on the final reward function, guiding the large language model to generate an answer which is faithful to the context and meets the requirements of a structured format, so that the problems that in the prior art, data security isolation and privacy protection cannot be realized, and the security and compliance use of the data in the whole process cannot be guaranteed are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for optimizing the fidelity of responses from large language models based on a multi-dimensional reward function. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, Large-Scale Language Models (LLMs) have achieved remarkable results in the field of natural language processing. Leveraging their massive parameter scale and pre-training on vast amounts of text data, LLMs demonstrate powerful language understanding and generation capabilities, revolutionizing the field of text generation. In practical applications, LLMs can generate logically coherent, grammatically correct, and semantically rich text responses based on the input context, demonstrating outstanding performance in tasks such as document question answering, text summarization, and content creation assistance. They have been widely adopted in various industries including education, healthcare, finance, and law.

[0003] In education, large-scale language models can assist teachers in generating teaching materials and answering student questions, supporting personalized learning. In healthcare, they can help doctors analyze medical records and provide diagnostic suggestions. In the financial industry, they are used for risk assessment report writing and responding to customer inquiries. Taking text summarization as an example, large-scale language models can quickly extract the core points of long documents, greatly improving information retrieval efficiency. In document question answering tasks, they can accurately understand the user's question intent, extract relevant information from complex documents, and organize it into a reasonable answer.

[0004] However, large-scale language models have revealed a serious problem in application: the generated content sometimes deviates from the original input context, producing unfaithful, even fabricated or false, information—a phenomenon known as "content pseudo-authenticity." For example, in generating legal document summaries, the model may misinterpret the meaning of legal provisions, generating summaries that do not match the original text, misleading legal professionals. In the medical field, providing diagnostic advice based on false information could delay patients' treatment and cause serious consequences. This "content pseudo-authenticity" problem greatly reduces the reliability and credibility of the model-generated content, severely restricting the in-depth application of large-scale language models in key areas where information accuracy is extremely important.

[0005] Therefore, there is an urgent need for a method that can optimize the fidelity of responses from large language models based on a multi-dimensional reward function. Summary of the Invention

[0006] In view of this, the present invention proposes a method and system for optimizing the fidelity of responses from large language models based on a multi-dimensional reward function, which can achieve data security isolation and privacy protection, and ensure the security and compliant use of data throughout the entire process.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for optimizing the fidelity of responses in large language models based on a multi-dimensional reward function includes:

[0009] A large language model is used to generate long and short text answers to questions based on context.

[0010] The first reward score for the short text response is calculated using a short text precision reward function.

[0011] The second reward score for the long text response is calculated using a long text proxy reward function;

[0012] The third reward score for the long text response and the short text response is calculated using a format reward function;

[0013] The first reward function, the second reward function, and the third reward function are added together to obtain the final reward function;

[0014] The final reward function guides the large language model to generate answers that are both faithful to the context and conform to the requirements of structured format.

[0015] Based on the above technical solution, the present invention can be further improved as follows:

[0016] Optionally, the step of calculating the first reward score of the short text answer using the short text precision reward function includes:

[0017] Determine whether the short text answer is completely consistent with the standard answer. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0018] Optionally, the step of calculating the second reward score of the long text answer using a long text proxy reward function includes:

[0019] Determine whether the long text response can assist the large language model in generating a correct short text response. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0020] Optionally, the calculation of the third reward score for the long text response and the short text response using a format reward function includes:

[0021] Determine whether the long text answer and the short text answer conform to the preset structured format. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0022] A system for optimizing the fidelity of responses from large language models based on a multi-dimensional reward function includes:

[0023] The answer generation module is used to generate long and short text answers to questions based on context using a large language model.

[0024] The first calculation module is used to calculate the first reward score of the short text answer using a short text precision reward function;

[0025] The second calculation module is used to calculate the second reward score of the long text answer using a long text proxy reward function;

[0026] The third calculation module is used to calculate the third reward score of the long text answer and the short text answer using a format reward function;

[0027] The fourth calculation module is used to add the first reward function, the second reward function, and the third reward function to obtain the final reward function;

[0028] The guidance module is used to guide the large language model to generate answers that are both faithful to the context and conform to the structured format requirements, based on the final reward function.

[0029] Optionally, the first calculation module is further configured to:

[0030] Determine whether the short text answer is completely consistent with the standard answer. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0031] Optionally, the second calculation module is further configured to:

[0032] Determine whether the long text response can assist the large language model in generating a correct short text response. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0033] Optionally, the third calculation module is further configured to:

[0034] Determine whether the long text answer and the short text answer conform to the preset structured format. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0035] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method described herein.

[0036] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method when executed by a processor.

[0037] The present invention has the following advantages:

[0038] This invention presents a method for optimizing the fidelity of responses in large language models based on a multi-dimensional reward function. It employs a precise reward function for short texts to ensure accurate conclusions, a proxy reward function for long texts to verify the rationality of reasoning, and avoids the problem of "content falsification." A format reward function standardizes the output structure, improving the readability and interpretability of the responses. Combining these three elements forms the final reward function, guiding the model to generate responses that are both faithful to the context and conform to the required structured format. This approach overcomes the limitations of specific tasks, is applicable to various scenarios, and effectively improves the reliability and credibility of the model's output, providing strong support for the application of large language models in fields such as medicine and law where high information accuracy is required. Attached Figure Description

[0039] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein:

[0040] Figure 1 This is a flowchart illustrating the method for optimizing the fidelity of responses from a large language model based on a multi-dimensional reward function, as described in this invention.

[0041] Figure 2 This is a schematic diagram of the main components of the system for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] It should be noted that, where there is no conflict, the embodiments and features of the present invention can be combined with each other. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0046] Figure 1 This is a flowchart illustrating the method for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function in an embodiment of the present invention. Figure 1 As shown, the method for optimizing the fidelity of responses of a large language model based on a multi-dimensional reward function provided in this embodiment of the invention includes the following steps S101 to S106.

[0047] S101 uses a large language model to generate long and short text answers to questions based on context.

[0048] The system prompt guides the large language model to sequentially generate reasoning processes, long text answers containing specific and complete sentences, and concise short text answers summarizing the information in just a few words. For example, in response to the contextual question "What is the country of origin of Super Mario?", the model can generate the corresponding long text answer "Super Mario originated in Japan" and the concise short text answer "Japan". In this way, this application can simultaneously optimize both long and short text answers and assign differentiated reward scores to different types of text. The system prompt can also trigger zero-shot chain-of-thought reasoning in the policy model and gradually improve the reward optimization effect as training progresses.

[0049] The correlation constraint model between long and short texts generates content that is faithful to the context.

[0050] S102, the first reward score for the short text answer is calculated using the short text precise reward function.

[0051] The accuracy of short text responses is measured using Exact Match (EM). The system determines whether the short text response is completely identical to the standard answer. If it is, a bonus score of 1 is awarded; otherwise, a bonus score of 0 is awarded.

[0052] This short text precision reward function ensures that the generated short text answers correctly address the question based on the context, thereby improving the fidelity of LLMs in short text answers.

[0053] S103, the second reward score for the long text answer is calculated using the long text proxy reward function.

[0054] The fidelity of the text response can be measured by its ability to assist LLMs in generating correct short text responses. For each generated long text response, we replace it with the input context and provide it as new input to the LLMs to check whether the model can generate a correct short text response based on the long text. If the generated long text response enables the LLMs to generate a correct short text response, it indicates that the long text response is faithful to the context, contains complete and understandable sentences, and correctly answers the question. Therefore, we determine whether the long text response can assist the large language model in generating a correct short text response; if yes, a reward score of 1 is assigned; otherwise, a reward score of 0 is assigned.

[0055] S104 uses a formatted reward function to calculate the third reward score for long text responses and short text responses.

[0056] Determine whether the long text answer and the short text answer conform to the preset structured format. If yes, assign a reward score of 1; otherwise, assign a reward score of 0.

[0057] S105, add the first reward function, the second reward function and the third reward function to obtain the final reward function.

[0058] S106, based on the final reward function, guides the large language model to generate answers that are both faithful to the context and conform to the requirements of structured format.

[0059] During model training, the final reward function can be used with different reinforcement learning algorithms to improve the final performance of the model, such as the rule-based reinforcement learning algorithm GRPO.

[0060] Figure 2 This diagram illustrates the main components of the system for optimizing the fidelity of responses from a large language model based on a multi-dimensional reward function, as described in this embodiment of the invention. Figure 2 As shown, the system 1 for optimizing the fidelity of large language model responses based on a multi-dimensional reward function provided in this embodiment of the invention includes a response generation module 10, a first calculation module 20, a second calculation module 30, a third calculation module 40, a fourth calculation module 50, and a guidance module 60.

[0061] The answer generation module 10 is used to generate long and short text answers to questions based on context using a large language model.

[0062] The first calculation module 20 is used to calculate the first reward score of the short text answer using a short text precision reward function;

[0063] The second calculation module 30 is used to calculate the second reward score of the long text answer using a long text proxy reward function;

[0064] The third calculation module 40 is used to calculate the third reward score of the long text answer and the short text answer using a format reward function;

[0065] The fourth calculation module 50 is used to add the first reward function, the second reward function and the third reward function to obtain the final reward function;

[0066] The guidance module 60 is used to guide the large language model to generate an answer that is both faithful to the context and conforms to the structured format requirements based on the final reward function.

[0067] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 70 includes: a processor 701, a memory 702, and a bus 703;

[0068] The processor 701 and the memory 702 communicate with each other via the bus 703.

[0069] The processor 701 is used to call program instructions in the memory 702 to execute the methods provided in the above-described method embodiments, and to execute the methods provided in the embodiments of the present invention.

[0070] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions, which cause a computer to execute the method provided in this embodiment of the invention.

[0071] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0072] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function, characterized in that, The method comprises the following steps: generating a long text answer and a short text answer to a question based on a context by using a large language model; calculating a first reward score of the short text answer by using a short text precision reward function; calculating a second reward score of the long text answer by using a long text proxy reward function; calculating a third reward score of the long text answer and the short text answer by using a format reward function; adding the first reward function, the second reward function and the third reward function to obtain a final reward function; guiding the large language model to generate an answer which is loyal to the context and meets the requirements of a structured format based on the final reward function.

2. The method for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function according to claim 1, characterized in that, The step of calculating the first reward score of the short text answer by using the short text precision reward function comprises the following steps: determining whether the short text answer is completely consistent with a standard answer, if yes, assigning a reward score of 1, and if no, assigning a reward score of 0.

3. The method for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function according to claim 1, characterized in that, The step of calculating the second reward score of the long text answer by using the long text proxy reward function comprises the following steps: determining whether the long text answer can assist the large language model to generate a correct short text answer, if yes, assigning a reward score of 1, and if no, assigning a reward score of 0.

4. The method for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function according to claim 1, characterized in that, The step of calculating the third reward score of the long text answer and the short text answer by using the format reward function comprises the following steps: determining whether the long text answer and the short text answer meet a preset structured format, if yes, assigning a reward score of 1, and if no, assigning a reward score of 0.

5. A system for optimizing the fidelity of answers of a large language model based on a multi-dimensional reward function, characterized in that, The method comprises the following steps: generating a long text answer and a short text answer to a question based on a context by using a large language model; calculating a first reward score of the short text answer by using a short text precision reward function; calculating a second reward score of the long text answer by using a long text proxy reward function; calculating a third reward score of the long text answer and the short text answer by using a format reward function; adding the first reward function, the second reward function and the third reward function to obtain a final reward function; guiding the large language model to generate an answer which is loyal to the context and meets the requirements of a structured format based on the final reward function.

6. The system for optimizing the answer fidelity of a large language model based on a multi-dimension reward function according to claim 5, wherein, The first calculating module is further used for: determining whether the short text answer is completely consistent with a standard answer, if yes, assigning a reward score of 1, and if no, assigning a reward score of 0.

7. The system for optimizing the answer fidelity of a large language model based on a multi-dimension reward function according to claim 5, wherein, The second calculating module is further used for: determining whether the long text answer can assist the large language model to generate a correct short text answer, if yes, assigning a reward score of 1, and if no, assigning a reward score of 0. 8.The system for optimizing the answer fidelity of a large language model based on a multi-dimensional reward function according to claim 5, wherein, The third calculating module is further used for: determining whether the long text answer and the short text answer meet a preset structured format, if yes, assigning a reward score of 1, and if no, assigning a reward score of 0.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1 to 4.

10. A non-transitory computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 4.

Images