Large language model optimization and interaction method and device, electronic equipment and storage medium

Abstract

The invention provides a big language model optimization and interaction method and device, electronic equipment and a storage medium, and the optimization method comprises the steps: inputting a sample prompt into a big language model to obtain an answer outputted by the big language model, the answer comprising an assertion and a reference mark of the assertion; based on the assertion, the reference mark of the assertion and the reference file corresponding to the reference mark, generating an assertion reward and a reference reward; on the basis of the assertion rewards and assertion masks of the answers, parameter iteration is carried out on the large language model, and the assertion masks are obtained by masking the parts, except assertions, in the answers; and based on the reference rewards and reference masks of the answers, parameter iteration is performed on the large language model, and the reference masks are obtained by masking the parts, except the reference marks, in the answers. According to the method, the device, the electronic equipment and the storage medium provided by the invention, the differentiated rewards can be accurately distributed to each subtask, so that the reference credibility of the large language model is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a large language model optimization and interaction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] For a large language model (LLM), in a retrieval-augmented generation (RAG) scenario and other scenarios in which the model needs to generate answers based on external knowledge sources, whether the model can provide accurate and traceable references directly relates to the credibility and practical value of its output.

[0003] In order to improve the referencing ability of the large language model, the model output is usually regarded as a whole in the related art, and a single reward signal is formed by evaluating the model output as a whole, which is used to optimize the large language model in the RAG scenario. However, the guidance of this single reward in reinforcement learning is not clear, resulting in poor optimization effect for the referencing ability of the large language model. SUMMARY

[0004] The present application provides a large language model optimization and interaction method, device, electronic equipment and storage medium to solve the defect of insufficient optimization of the referencing ability of the large language model in the related art.

[0005] The present application provides a large language model optimization method, comprising: inputting a sample prompt into a large language model to obtain an answer output by the large language model, the answer comprising an assertion and a reference mark of the assertion; generating an assertion reward and a reference reward based on the assertion, the reference mark of the assertion, and a reference file corresponding to the reference mark; iterating parameters of the large language model based on the assertion reward and an assertion mask of the answer, the assertion mask being obtained by masking a part of the answer other than the assertion; and iterating parameters of the large language model based on the reference reward and a reference mask of the answer, the reference mask being obtained by masking a part of the answer other than the reference mark.

[0006] According to the large language model optimization method provided by the present application, the generation of the assertion reward comprises: determining a minimum provable set covering the assertion from a reference set of the reference file, and performing redundancy scoring on the reference set based on the minimum provable set to obtain an evidence coverage score; determining the assertion reward based on the evidence coverage score.

[0007] According to the large language model optimization method provided by the present invention, determining the minimum provable set covering the assertions from the reference set of the reference documents includes: Based on the coverage score of each reference file for each assertion in the reference set, and the similarity between any two reference files in the reference set, the minimum provable set covering the assertion is determined from the reference set, with the similarity between reference files to be included in the minimum provable set as a penalty.

[0008] According to the large language model optimization method provided by the present invention, the step of generating the assertion reward further includes: The assertion is evaluated for answer quality to obtain an answer quality score; Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; The determination of the assertion reward based on the evidence coverage score includes: The assertion reward is determined based on the evidence coverage score, the answer quality score, and / or the evidence consistency score.

[0009] According to the large language model optimization method provided by the present invention, the step of generating the citation reward includes: Based on the reference files corresponding to the citation tags of the assertions and the set of sample reference files corresponding to the sample prompts, a citation quality assessment is performed to obtain a citation quality score. Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The citation reward is determined based on at least one of the citation quality score, the evidence consistency score, and the document reliability score.

[0010] According to the large language model optimization method provided by the present invention, the answer further includes a retrieval instruction for retrieving the answer; The method further includes: Based on the information retrieved using the large language model to output the answer, the retrieval reward is determined. Based on the retrieval reward and the retrieval mask of the answer, the large language model is subjected to parameter iteration, wherein the retrieval mask is obtained by masking the part of the answer other than the retrieval instruction.

[0011] According to the large language model optimization method provided by the present invention, the step of determining the retrieval reward based on the information retrieved by the large language model for outputting the answer includes: The retrieval cost is determined based on at least one of the following factors: the number of times the large language model is used to retrieve the answer, lexical consumption, and latency. Based on the coverage of the assertion of the files obtained in each round of the search for the answer using the large language model, the reward is determined and the search stops. The retrieval reward is determined based on the retrieval cost and the immediate stop reward.

[0012] According to the large language model optimization method provided by the present invention, determining the retrieval reward based on the retrieval cost and the immediate stopping reward includes: Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The retrieval reward is determined based on the retrieval cost, the immediate stop reward, and the document reliability score.

[0013] The present invention also provides an interaction method, comprising: Get user prompts; Input the user prompt into the interaction model to obtain the interaction response output by the interaction model; The interaction model is obtained based on the large language model optimization method described above.

[0014] The present invention also provides a large language model optimization device, comprising: An input unit is used to input sample prompts into a large language model and obtain the answer output by the large language model, wherein the answer includes assertions and reference tags of the assertions; The reward generation unit is used to generate assertion rewards and reference rewards based on the assertion, the reference mark of the assertion, and the reference file corresponding to the reference mark; An optimization unit is configured to perform parameter iteration on the large language model based on the assertion reward and the assertion mask of the answer, wherein the assertion mask is obtained by masking the part of the answer excluding the assertions; and to perform parameter iteration on the large language model based on the reference reward and the reference mask of the answer, wherein the reference mask is obtained by masking the part of the answer excluding the reference markers.

[0015] The present invention also provides an interactive device, comprising: The acquisition unit is used to acquire user prompts; An interaction unit is used to input the user prompt into the interaction model and obtain the interaction response output by the interaction model. The interaction model is obtained based on the large language model optimization method described above.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the large language model optimization methods or interaction methods described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large language model optimization method or interaction method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the large language model optimization methods or interaction methods described above.

[0019] The large language model optimization and interaction method, apparatus, electronic device, and storage medium provided by this invention generate assertion rewards and citation rewards for two sub-tasks: generating assertions and generating citation tags, respectively, based on assertions, citation tags, and corresponding reference files, thus achieving reward differentiation. Based on the assertion rewards and assertion masks, the large language model's parameters are iterated; additionally, based on the citation rewards and citation masks, the parameters of the large language model are iterated, enabling the differentiated rewards to be accurately allocated to the corresponding sub-tasks for optimization, thereby effectively improving the citation credibility of the large language model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the large language model optimization method provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the interaction method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of the large language model optimization device provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the interactive device provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] For large language models (LLMs), in scenarios such as retrieval enhancement and RAG generation where the model needs to generate answers based on external knowledge sources, the model's ability to provide accurate and traceable citations directly affects the credibility and practical value of its output.

[0028] Currently, there are three main types of solutions for optimizing citation capabilities in large language models: One approach focuses on improving citation quality without relying on manual annotation or expensive proprietary model training, such as the SelfCite method. SelfCite optimizes citation quality by introducing context ablation and combines it with SimPO preference optimization for model learning. However, the off-policy learning mechanism in SelfCite makes the large language model susceptible to data distribution shifts, leading to performance instability under different data environments. Furthermore, SelfCite's reward signal is entirely derived from the large language model's own probability distribution. If the model misunderstands a particular context, this inherently erroneous probability distribution will directly generate misleading rewards, incorrectly guiding the model's optimization direction. Moreover, each citation quality evaluation requires multiple forward propagations, resulting in significant computational overhead when processing long documents, especially during the initial optimization phase of the large language model, which requires substantial inference resources and severely limits its scalability and deployment efficiency.

[0029] Another approach focuses on directly improving the model's search capabilities and answer generation through end-to-end reinforcement training. For example, methods like Search-R1, R1-Searcher, and ZEROSEARCH all employ end-to-end reinforcement training, enabling multiple rounds of interaction between the large language model and the retrieval engine to directly output the final answer, thereby enhancing the large language model's ability to utilize externally retrieved information. However, the training data for this type of approach often leans towards objective questions, performing poorly on tasks requiring complex reasoning, subjective judgment, or open-ended answers. More importantly, after retrieving information, this approach lacks an optimization mechanism for information citation itself; that is, it focuses on how to find the answer rather than how to reliably output the source of the answer's citation.

[0030] In addition, another class of methods attempts to improve the reliability of large language models through refined reward design. For example, there are two-stage reinforcement learning methods that directly quantify multiple high-level objectives as reward signals, aiming to make large language models exhibit higher credibility in RAG scenarios. Although this approach aims to provide more comprehensive incentives, its drawback lies in the lack of differentiation in the feedback reward signals. Since the quality of the answer depends simultaneously on the correctness of the answer itself and the reliability of the citation, this merged single reward signal is difficult to achieve refined optimization of large language models, thus limiting the optimization effect of large language models.

[0031] In summary, the advantages of reinforcement learning for large language models in RAG scenarios are not clearly defined, which directly limits the optimization of the citation capabilities of large language models. To address the above issues, this invention provides a method for optimizing large language models. Figure 1 This is a flowchart illustrating the large language model optimization method provided by the present invention, as follows: Figure 1 As shown, the method includes: Step 110: Input the sample prompt into the large language model to obtain the answer output by the large language model. The answer includes assertions and reference tags of the assertions.

[0032] Specifically, the sample prompt is the prompt used as input to the large language model during optimization. The sample prompt may include a user query, which is either a question the user expects the large language model to answer or a task the user expects the large language model to complete; the sample prompt may also include a user query and model instructions, whereby the model instructions are used to instruct the large language model on how to process the user query and the content retrieved based on the user query. This embodiment of the invention does not specifically limit this.

[0033] By inputting sample prompts into the large language model, the model can perform at least one round of retrieval based on the sample prompts and generate an answer based on the retrieved content. Furthermore, after inputting the sample prompts, the large language model can consider the content to be retrieved based on the prompts and invoke search engine tools to perform the retrieval. Based on the retrieved content, it determines whether to directly output the answer or proceed to the next round of retrieval. This process is repeated one or more times to obtain the answer output by the large language model.

[0034] For example, the process by which a large language model goes through multiple rounds of thinking and searching to output an answer can be represented in the following form: [ "query", " <think> thought process< / think> <search> ["Search content 1", "Search content 2", "Search content 3"]< / search> ", " <information> The search engine returned the following results: {"Search content 1": "[1] xxxx\n\n[2]xxxx\", "Search content 2": "[3] xxxx\", "Search content 3": "[4]xxxx\"}< / information> ", " <think> thought process< / think> <search> ["Search content 1"]}< / search> ", " <information> The search engine tool returned the following results: {"Search content 1": "[5] xxxx\n\n[6]xxxx\n\n[7] xxxx\n\n[8]xxxx\"}< / information> ", " <think> thought process< / think> <search> ["Search content 1", "Search content 2"]}< / search> ", " <information> The search engine tool returned the following results: {"Search content 1": "[9] xxxx\n\n

[10] xxxx", "Search content 2": "

[11] xxxx"}< / information> ", " <think> thought process< / think> <answer> The given answer sentence 1 is [n][k]. The given answer sentence 2 is [k][j].< / answer> " ] Among them, “query” refers to the sample prompt, “search content” is the content that needs to be searched, which is the result of the large language model’s thinking, and “"search content 1": "[1] xxxx\n\n[2]xxxx\"” is the content that the large language model obtains by calling the search engine tool for “search content 1”. <answer> The given answer sentence 1 is [n][k]. The given answer sentence 2 is [k][j].< / answer> "This is the answer output by the large language model."

[0035] Here, the answer output by the large language model may include assertions and assertion citations; or, the answer output by the large language model may include assertions, assertion citations, and retrieval instructions for retrieving the output answer.

[0036] In this context, an assertion refers to a statement, declaration, or opinion. For example, an assertion could be the one described in the example above. <answer> The given answer sentence 1 is [n][k]. The given answer sentence 2 is [k][j].< / answer>The answer includes phrases like "Given answer sentence 1" and "Given answer sentence 2". For each assertion, a citation marker is present in the answer. This citation marker identifies the basis for the assertion, essentially supporting evidence. It can also be referred to as a citation superscript. An assertion can have one or more citation markers, each corresponding to a reference document. Reference documents are files retrieved by the large language model based on the sample prompts, such as papers, reports, manuals, and technical documents.

[0037] Specifically, the answer output by the large language model can include one or more assertions. For each assertion, there are one or more reference markers, and each reference marker corresponds to a reference file. For example, the reference markers for the assertion "the given answer sentence 1" can be [n] and [k], where [n] represents the nth reference file and [k] represents the kth reference file.

[0038] Step 120: Based on the assertion, the reference marker of the assertion, and the reference file corresponding to the reference marker, generate the assertion reward and the reference reward.

[0039] Specifically, for a sample prompt, when evaluating the quality of the answer corresponding to that sample prompt, the evaluation can be conducted from the perspective of assertion and citation, thereby obtaining assertion rewards and citation rewards respectively.

[0040] When evaluating assertions, the evaluation can consider the assertion itself, its citation markers, and the reference documents corresponding to those citation markers. For example, the quality of an assertion's generation can be assessed based on its accuracy, completeness, logical coherence, and readability. Alternatively, it can be determined whether the reference documents support the assertion, based on the assertion and its citation markers. Furthermore, in cases where an assertion has multiple citation markers, it can be determined whether the reference documents cited by each marker point to the same conclusion or contain contradictions; this embodiment of the invention does not specifically limit this. The resulting assertion reward can specifically guide the large language model towards the goal of optimizing assertion quality from the perspective of assertions.

[0041] When evaluating citations, the assertion, its citation marker, and the corresponding reference document can all be considered. For example, the assertion and the reference document corresponding to its citation marker can be compared with the reference document corresponding to the same assertion pre-marked in the sample prompt to determine whether the retrieval and citation of the reference document are reasonable. Alternatively, based on the assertion and the reference document corresponding to its citation marker, it can be determined whether the reference document supports the assertion. Furthermore, based on the publication source and publication time of the reference document, its authority and timeliness can be measured; this embodiment of the invention does not specifically limit these aspects. The resulting citation reward can specifically guide the large language model towards the goal of optimizing citation quality from the perspective of citations.

[0042] It is understood that the generation of assertion rewards and the generation of reference rewards can be performed simultaneously or sequentially, and this embodiment of the invention does not specifically limit this.

[0043] Step 130: Based on the assertion reward and the assertion mask of the answer, perform parameter iteration on the large language model, wherein the assertion mask is obtained by masking the part of the answer excluding the assertions; and based on the citation reward and the citation mask of the answer, perform parameter iteration on the large language model, wherein the citation mask is obtained by masking the part of the answer excluding the citation markers.

[0044] Specifically, the assertion reward and citation reward obtained in step 120 differentiate the reward signal. To apply the differentiated reward signal to the parameter iteration of a large language model, the output answer of the large language model can be masked according to functional fragments. Specifically, assertions and citations can be treated as different functional fragments and masked separately in the answer. This decomposes the output of the large language model into functionally independent parts. Specifically, a dedicated policy head and parameter subspace / expert hybrid model (MoE) are designed within the large language model to be responsible for the generation of different functional fragments.

[0045] In this embodiment of the invention, assertions are masked as functional fragments in the answer, that is, other tokens in the answer besides assertions are masked, thereby obtaining an assertion mask. It is understood that the assertion mask retains the tokens representing assertions in the answer while masking non-assertive tokens.

[0046] Furthermore, by masking citation markers as functional segments within the answer, that is, by masking other tokens in the answer besides citation markers, a citation mask can be obtained. Understandably, the citation mask preserves the tokens representing citation markers in the answer while masking the tokens that are not citation markers.

[0047] It is understandable that a part related to assertion generation can be separated in the large language model. Specifically, this could include a policy head designed for assertion generation, and at least one of a parameter subspace / expert hybrid model designed for assertion generation. Additionally, a part related to citation tag generation can be separated in the large language model. Specifically, this could include a policy head designed for citation tag generation, and at least one of a parameter subspace / expert hybrid model designed for citation tag generation. Furthermore, a router can be introduced into the large language model. During the inference process of the large language model, the router can decide whether to activate the part related to assertion generation or the part related to citation tag generation to perform the generation action, thereby achieving the output of the answer. Here, the granularity of the router's decision can be at the word level or the sentence level.

[0048] Building upon this foundation, when iteratively optimizing a large language model using reinforcement learning, assertion masks can be used to precisely allocate assertion rewards to the subtasks of assertion generation within the large language model. This guides the large language model to iterate its parameters based on assertion rewards, thereby optimizing its assertion generation capabilities. In this process, the application of assertion masks ensures that assertion rewards are only allocated to the tokens representing assertions in the answer, without affecting non-assertion tokens. Therefore, during backpropagation, the gradient based on assertion rewards flows back to the large language model only through the tokens representing assertions. Updating the parameters of the large language model based on these rewards only improves its ability to generate assertions, without affecting its ability to generate other tokens.

[0049] Furthermore, when iteratively optimizing a large language model using reinforcement learning, a citation mask can be used to precisely allocate the citation reward obtained from citations to the subtask of generating citation tags. This guides the large language model to iterate its parameters based on the citation reward, thereby optimizing its citation generation capability. In this process, the application of the citation mask ensures that the citation reward is only allocated to the lexical units representing citation tags in the answer, without affecting non-citation lexical units. Therefore, during backpropagation, the gradient based on the citation reward only flows back to the large language model through the lexical units representing citation tags. Updating the parameters of the large language model based on the citation reward only improves its ability to generate citation tags, without affecting its ability to generate other lexical units.

[0050] In this way, the learning objectives and gradient directions of the two subtasks of generating assertions and generating citation tags are independent in the large language model. This ensures that the optimization directions of generating assertions and generating citation tags do not interfere with each other, avoids gradient conflicts that may be caused by a unified gradient, and reduces the risk of overfitting.

[0051] In the method provided in this embodiment of the invention, assertion rewards and citation rewards are generated for the two subtasks of generating assertions and generating citation tags, respectively, based on assertions, citation tags, and reference files corresponding to the citation tags, thus achieving reward differentiation. Based on the assertion rewards and assertion masks, parameters of the large language model are iterated. Additionally, based on the citation rewards and citation masks, parameters of the large language model are iterated, enabling the differentiated rewards to be accurately allocated to the corresponding subtasks for optimization, thereby effectively improving the citation credibility of the large language model.

[0052] Based on the above embodiments, the step of generating the assertion reward includes: Determine the minimum verifiable set covering the assertion from the reference set of the reference documents, and perform a redundancy score on the reference set based on the minimum verifiable set to obtain the evidence coverage score; The assertion reward is determined based on the evidence coverage score.

[0053] Specifically, the evidence coverage score reflects how well the references cited in the assertion cover the assertion when used as evidence, and the redundancy of the references used in the assertion, provided the evidence supports the assertion. Ideally, the references cited in the assertion should serve as evidence for the assertion, completely cover the assertion's statement, and there should be no redundant references cited in the assertion. Therefore, the higher the degree of coverage of the assertion by the references, the higher the evidence coverage score; the lower the coverage, the lower the evidence coverage score. Furthermore, the lower the redundancy of the references, the higher the evidence coverage score; and the higher the redundancy, the lower the evidence coverage score.

[0054] Evidence coverage score can be obtained by redundancy scoring of the citation set based on the minimum provable set of covering assertions. Here, all references cited in the answer can be considered as a set, specifically denoted as the citation set. From this citation set, a subset can be determined that is the smallest subset that guarantees evidence coverage for all assertions in the answer, supported by this subset; this subset is denoted as the minimum provable set. Given this minimum provable set, the evidence coverage score can be obtained by assessing the redundancy of the citation set compared to the minimum provable set.

[0055] Based on this, assertion rewards can be determined based on evidence coverage scores. For example, evidence coverage scores can be directly used as assertion rewards, or evidence coverage scores can be transformed into assertion rewards through rules, or evidence coverage scores can be combined with other evaluation scores for assertion quality to obtain assertion rewards. This embodiment of the invention does not make specific limitations on this.

[0056] Based on any of the above embodiments, in the step of generating the assertion reward, determining the minimum provable set covering the assertion from the reference set of the reference document includes: Based on the coverage score of each reference file for each assertion in the reference set, and the similarity between any two reference files in the reference set, the minimum provable set covering the assertion is determined from the reference set, with the similarity between reference files to be included in the minimum provable set as a penalty.

[0057] Specifically, to evaluate the evidence coverage score, it is necessary to determine the minimum provable set and thereby assess the redundancy of the citation set. To determine the minimum provable set, a combination of an approximate greedy approach and redundancy suppression can be used.

[0058] First, the coverage score of each reference file in the reference set for each assertion can be calculated. Based on this, a reference file can be selected to be placed into the minimum provable set first. Here, the reference file placed into the minimum provable set first can be the reference file with the highest coverage score for each assertion.

[0059] After placing the reference documents into the minimum verifiable set, the coverage score of the entire minimum verifiable set for each assertion can be updated based on the coverage score of each reference document included in the minimum verifiable set for each assertion. After obtaining the coverage score of the entire minimum verifiable set for each assertion, the coverage score of each assertion is compared with a coverage score threshold. For assertions whose coverage scores exceed the coverage score threshold, it is confirmed that the assertion has been fully covered, and the coverage scores of each reference document for that assertion will no longer be considered in subsequent processes.

[0060] Following this, from each remaining reference file in the reference set, one reference file is selected and placed into the minimum verifiable set. When selecting a reference file for the minimum verifiable set, not only is the coverage score of the reference file for each assertion not yet fully covered considered, but also the similarity between the reference file and reference files already placed in the minimum verifiable set. Specifically, the coverage score of the reference file for each assertion not yet fully covered can be used as a reward, and the similarity between the reference file and reference files already placed in the minimum verifiable set can be used as a penalty. This achieves coverage of all forgings with as few reference files as possible while minimizing the similarity of reference files in the minimum verifiable set, thus achieving redundancy suppression for the minimum verifiable set.

[0061] The above process can be repeated until all assertions are covered by the minimum provable set, thus obtaining the minimum provable set.

[0062] For example, suppose the set of assertions is The reference collection is We can find the smallest provable set. , making All were Coverage and coverage division Coverage threshold.

[0063] The method provided in this embodiment of the invention uses the similarity between reference documents placed into the minimum verifiable set as a penalty term to achieve redundancy suppression in the process of determining the minimum verifiable set, thereby ensuring the reliability of the minimum verifiable set and thus ensuring the reliability of the evidence coverage score determined therefrom.

[0064] Furthermore, updating and iterating the large language model based on this can guide the large language model to prioritize the main evidence with the highest coverage score when sorting reference documents, and then arrange the evidence as supplementary or counterexamples, so that the output reference documents can both cover the assertions and are not redundant.

[0065] Based on any of the above embodiments, in the assertion reward generation step, the step of performing redundancy scoring on the reference set based on the minimum provable set to obtain an evidence coverage score includes: Based on the minimum provable set, the reference set is evaluated for redundancy, and the redundancy score of the reference set is obtained. The evidence coverage score is determined based on the number of reference documents contained in the citation set and / or the coverage of the citation set to the assertion, as well as the redundancy score.

[0066] Specifically, when measuring evidence coverage scores, the redundancy of the citation set can be evaluated not only based on the minimum provable set, but also by considering the number of references included in the citation set and / or the coverage of the citation set to the assertion. Understandably, the more references included in the citation set, the higher the cost of generating citations and the lower the evidence coverage score; conversely, the higher the coverage of the citation set to the assertion, the better the citation set supports the viewpoint expressed by the assertion, and the higher the evidence coverage score.

[0067] In the method provided in this embodiment of the invention, by introducing an evidence coverage score, the large language model can be guided to update and iterate in a direction where the reference document both covers assertions and is not redundant.

[0068] Based on any of the above embodiments, the step of generating the assertion reward further includes: The assertion is evaluated for answer quality to obtain an answer quality score; Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; The determination of the assertion reward based on the evidence coverage score includes: The assertion reward is determined based on the evidence coverage score, the answer quality score, and / or the evidence consistency score.

[0069] Specifically, when determining the corresponding assertion reward for assertions in the answer, in addition to applying the evidence coverage score, the answer quality score and / or evidence consistency score can also be applied. For example, the assertion reward can be obtained by adding the answer quality score, evidence consistency score, and evidence coverage score together, or by weighted summation of the answer quality score, evidence consistency score, and evidence coverage score. This embodiment of the invention does not specifically limit this approach.

[0070] The answer quality score reflects the quality of the assertions. It is understood that a higher answer quality score indicates a better assertion. Specifically, the answer quality score can be obtained by evaluating the quality of all assertions in the answer. For example, it can be obtained by inputting all assertions into a pre-trained answer quality evaluation model, or by scoring each assertion separately based on aspects such as accuracy, completeness, logic, and readability, and then combining these scores to determine the answer quality score. This embodiment of the invention does not impose specific limitations on this method.

[0071] The evidence consistency score reflects whether the evidence cited by each assertion is consistent with the viewpoint expressed by the assertion, and whether the viewpoints expressed by multiple pieces of evidence cited by each assertion are consistent. Understandably, a higher evidence consistency score indicates more reliable assertions generated based on reference documents. Furthermore, by adding the evidence consistency score to the assertion reward, iterating parameters of the large language model based on the assertion reward can correct the reference illusion problem of the large language model and prevent the large language model from tampering with reference document details when generating assertions.

[0072] Evidence consistency score can be obtained by performing implication and / or contradiction discrimination on the assertions and the references corresponding to the assertion's citation marks, as well as contradiction discrimination on the references corresponding to different citation marks of the assertions.

[0073] The implication determination of assertions and their references specifically involves judging whether the references reflect the content expressed by the assertion; that is, whether the assertion is implied within the references. Understandably, if the assertion is implied within the references, it means the references can serve as the basis for the assertion, and the viewpoint expressed by the assertion is consistent with the references. If the assertion is not implied within the references, it means the references cannot serve as the basis for the assertion, and the viewpoint expressed by the assertion differs from the references; this could be due to an error in assertion generation or an incorrect citation of the references. Here, implication determination can be implemented using Natural Language Inference (NLI).

[0074] The process of contradiction detection for assertions and their references involves determining whether the viewpoint expressed by the assertion contradicts that of the references, i.e., whether there is a contradiction between the assertion and the references. It can be understood that if there is no contradiction between the assertion and the references, it means that there is no logical contradiction in citing the references for the assertion; conversely, if there is a contradiction, it means that there is a logical contradiction in citing the references for the assertion, the references cannot support the viewpoint of the assertion, and may even become evidence proving that the assertion is wrong or that the large language model suffers from citation illusion. Here, contradiction detection can be achieved through cross-validation, which can also be implemented based on NLI (Non-Limited Language Interface).

[0075] The assertion is determined by checking for contradictions in the references corresponding to different citation markers. Specifically, when an assertion cites multiple references, contradiction checks are performed between different references supporting the same assertion. Contradiction checks between different references supporting the same assertion specifically determine whether their viewpoints contradict each other; that is, whether there are contradictions between different references supporting the same assertion. It's understandable that if contradictions exist between references supporting the same assertion, it means that some of these references cannot be used as the basis for the assertion. The assertion did not fully consider the viewpoints of each reference when citing them, which could be due to an error in the application of references to the assertion, or an error in the assertion generated by applying the references.

[0076] Therefore, by combining the results of the implication and / or contradiction discrimination of the above assertions and the references corresponding to the citation marks of the assertions, as well as the results of the contradiction discrimination of the references corresponding to different citation marks of the assertions, the evidence consistency score of the assertions can be determined.

[0077] Based on this, the assertion reward can be determined based on at least one of the answer quality score, evidence consistency score, and evidence coverage score. The resulting assertion reward not only considers the assertion itself, but also the influence of the cited references on its own judgment. Based on this, parameter iteration of the large language model can optimize the assertion generation quality of the large language model.

[0078] Based on any of the above embodiments, the step of generating the citation reward includes: Based on the reference files corresponding to the citation tags of the assertions and the set of sample reference files corresponding to the sample prompts, a citation quality assessment is performed to obtain a citation quality score. Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The citation reward is determined based on at least one of the citation quality score, the evidence consistency score, and the document reliability score.

[0079] Specifically, the citation reward can be determined based on at least one of the citation quality score, evidence consistency score, and document reliability score. For example, the citation reward can be obtained by adding the citation quality score, evidence consistency score, and document reliability score together, or by weighted summation of the citation quality score, evidence consistency score, and document reliability score. This embodiment of the invention does not impose any specific limitations on this.

[0080] The citation quality score reflects the quality of the references cited in the assertion. Understandably, the higher the match between the assertion and the references, and the more logically sound the argument derived from the references, the higher the citation quality score.

[0081] Here, the sample prompts are pre-annotated data. For each sample prompt, sample answers and the corresponding set of reference documents can be pre-annotated, denoted here as the sample reference document set. Therefore, when measuring the citation quality score, at least one of precision, recall, and F1 score can be determined between the cited reference document and the sample reference document set. This determines whether the citation of the reference document is reasonable, thus obtaining the citation quality score. It is understood that the citation quality score can be the aforementioned precision, recall, or F1 score, or it can be a score obtained by weighted summation or other integration of at least two of the aforementioned precision, recall, and F1 score. This embodiment of the invention does not specifically limit this.

[0082] The evidence consistency score reflects whether the evidence cited by each assertion is consistent with the view expressed by the assertion, and whether the view expressed by multiple pieces of evidence cited by each assertion is consistent. The evidence consistency score is obtained in the same way as in the above embodiment for calculating the assertion reward, and will not be repeated here.

[0083] Document reliability score reflects the reliability of a referenced document. It can be determined based on the document's publication source and / or publication date. The authority of the publication source itself reflects the document's reliability. The authority of the publication source can be comprehensively scored based on factors such as its ranking, citation frequency, peer review intensity, domain reputation, and institutional endorsement. Generally, a more authoritative source results in a higher reliability score. Furthermore, the publication date reflects the timeliness of the viewpoints expressed, especially in rapidly changing fields like finance. The older the publication date, the more likely the viewpoints have been updated, leading to a lower reliability score.

[0084] Furthermore, the document reliability score can also consider the ranking of the reference document itself in the answer generation based on sample prompts. This ranking is typically the retrieval ranking, which can be reflected in the ranking of the relevance between all retrieved documents and the search content after a round of retrieval by the large language model. It is understood that the retrieval ranking can be quantified as a score; the higher the score, the higher the probability of being cited when generating answers. The publication source and / or publication time of the reference document can be combined with the retrieval ranking score of the reference document to obtain the document reliability score; this embodiment of the invention does not specifically limit this.

[0085] For example, suppose the source authority coefficient determined based on the publication sources of the referenced documents is . The time decay factor determined based on the publication date of the reference document is: , ,in The time difference between the publication date and the present. These are the weighting coefficients, and The setting can be determined based on the field of the reference document; for example, a larger setting can be used for fields where information changes rapidly. Based on this, the score for file retrieval and ranking can be determined using the following formula. : Among them, the BM25 score is the result of the BM25 (Best Match 25) search ranking algorithm for files.d The score. Vector similarity refers to the score of the files. d Vector similarity between the retrieved file and other retrieved files. and Preset weights.

[0086] Based on this, the citation reward can be determined based on at least one of the citation quality score, evidence consistency score, and document reliability score. The citation reward obtained here not only takes into account the situation of the reference document itself, but also the relationship between the reference document and the statement that cites the reference document. Based on this, parameter iteration of the large language model can optimize the citation tag generation quality of the large language model.

[0087] Based on any of the above embodiments, the answer further includes a retrieval instruction for retrieving the answer; Large language model optimization methods also include: Based on the information retrieved using the large language model to output the answer, the retrieval reward is determined. Based on the retrieval reward and the retrieval mask of the answer, the large language model is subjected to parameter iteration, wherein the retrieval mask is obtained by masking the part of the answer other than the retrieval instruction.

[0088] Specifically, the answer output by the large language model may include not only assertions and citation tags, but also retrieval instructions applied to retrieve the output assertions and citation tags.

[0089] In optimizing large language models for RAG scenarios, in addition to optimizing the assertions and citation tags in the explicitly output answers of the large language model, optimization can also be achieved by focusing on the retrieval tasks performed during the reasoning process of the large language model.

[0090] Understandably, during the process of a large language model engaging in multiple rounds of thinking and searching to output an answer, the model's thinking and searching processes are also recorded, only hidden during interaction. Therefore, in the process of optimizing a large language model, the records of its thinking and searching can be obtained, thus providing information on the retrieval process it undertakes to output an answer. This allows for the evaluation of the large language model's retrieval process, leading to a retrieval reward. This reward can then be used to specifically guide the large language model towards the goal of optimizing its retrieval process.

[0091] When evaluating the retrieval process, information about the retrieval process used to output the answer can be referenced from the large language model. This information may include at least one of the following: the number of retrievals performed, the lexical consumption incurred by each retrieval, and the latency introduced by the retrieval. The more retrievals performed, the more resources are consumed and the longer the retrieval takes. This retrieval information can all be considered as the cost of the retrieval and added as a negative value to the retrieval reward.

[0092] Accordingly, the retrieval information output by the large language model can be masked, thus decomposing the retrieval process into a functionally independent part. This corresponds to designing a dedicated strategy head and parameter subspace / expert hybrid model within the large language model to handle the retrieval instructions. Here, the retrieval instructions can be… <search> ["Search content 1", "Search content 2", "Search content 3"]< / search> In the form of "".

[0093] In this embodiment of the invention, the retrieval instruction is masked as a functional fragment in the answer; that is, other tokens in the answer besides the retrieval instruction are masked, thereby obtaining the retrieval mask. It is understood that the retrieval mask retains the tokens representing the retrieval instruction in the answer and masks the tokens that are not retrieval instructions. The large language model is divided into retrieval-related parts, which may specifically include at least one of a strategy head designed for retrieval and a parameter subspace / expert hybrid model designed for retrieval. Based on this, the router can decide during the inference process of the large language model whether to activate the parts related to assertion generation, the parts related to reference tag generation to perform generation actions, or activate the retrieval-related parts to perform indexing actions, thereby achieving the output of the answer.

[0094] Building upon this foundation, when iteratively optimizing a large language model using reinforcement learning, a retrieval mask can be used to precisely allocate the retrieval reward obtained from the retrieved information to the sub-tasks of the large language model's retrieval process. This guides the large language model to iterate its parameters based on the retrieval reward, thereby optimizing its retrieval capabilities. In this process, the application of the retrieval mask ensures that the retrieval reward is only allocated to the tokens representing the retrieval instruction in the answer, without affecting tokens that are not retrieval instructions. Therefore, during backpropagation, the gradient based on the retrieval reward flows back to the large language model only through the tokens representing the retrieval instruction. Updating the parameters of the large language model based on the retrieval reward only improves its ability to generate retrieval instructions, without affecting its ability to generate other tokens.

[0095] Therefore, in the large language model, the assertion generation subtask, the reference tag generation subtask, and the retrieval subtask each iterate parameters independently. The learning objectives and gradient directions of the three subtasks of assertion generation, reference tag generation, and retrieval are independent of each other, so that the optimization directions of assertion generation, reference tag generation, and retrieval are not interfered with each other, avoiding gradient conflicts that may be caused by a unified gradient and reducing the risk of overfitting.

[0096] In the method provided in this embodiment of the invention, a retrieval reward is generated based on the information retrieved for the output answer using a large language model. This, along with assertion rewards and citation rewards, achieves reward differentiation. Consequently, the parameters of the large language model are iterated based on assertion rewards and assertion masks, citation rewards and citation masks, and retrieval rewards and retrieval masks, respectively. This allows the differentiated rewards to be accurately allocated to the corresponding sub-tasks for optimization, thereby effectively improving the citation credibility of the large language model.

[0097] Based on any of the above embodiments, determining the retrieval reward based on the information retrieved by the large language model for outputting the answer includes: The retrieval cost is determined based on at least one of the following factors: the number of times the large language model is used to retrieve the answer, lexical consumption, and latency. Based on the coverage of the assertion of the files obtained in each round of the search for the answer using the large language model, the reward is determined and the search stops. The retrieval reward is determined based on the retrieval cost and the immediate stop reward.

[0098] Specifically, the information retrieved by the large language model for the output answer may include at least one of the following: the number of searches performed for the output answer, lexical consumption, and latency. It may also include the files obtained for each round of retrieval, i.e., the search results. Among these, lexical consumption refers to the lexical units consumed in generating the search command, and latency is the delay time caused by performing the search.

[0099] Based on the above information, the search reward can be determined from two aspects.

[0100] One aspect is cost measurement, which can be based on at least one of the following: the number of searches, token consumption, and latency. Understandably, the more searches, the greater the token consumption, and the longer the latency, the higher the search cost; conversely, the fewer searches, the lower the token consumption, and the shorter the latency, the lower the search cost.

[0101] Another aspect is stopping upon sufficient evidence. Specifically, this involves calculating the coverage of the files retrieved in each round of retrieval to the final output assertions. If only the files retrieved after the final round achieve a coverage of all assertions that meets a preset threshold, while none of the files retrieved in previous rounds do, then the large language model is considered to have stopped further retrieval after finding sufficient evidence and instead outputs the answer, thus achieving stopping upon sufficient evidence. A stopping reward can be given to the large language model in this case. Conversely, if, before the final round, the coverage of all files retrieved in any round already meets the preset threshold, meaning sufficient evidence was found before the final round, then the large language model is deemed not to have achieved stopping upon sufficient evidence, and no stopping reward is given.

[0102] Based on this, the above two aspects can be combined to determine the search reward, namely, the search cost and the immediate stop reward. For example, the search cost can be subtracted from the immediate stop reward to obtain the search reward.

[0103] In the method provided in this embodiment of the invention, by combining retrieval cost and immediate cessation reward, the large language model can be updated and iterated in the direction of stopping upon sufficient evidence, thereby optimizing the retrieval process of the large language model, reducing the retrieval consumption of the large language model, and avoiding redundant retrieval processes.

[0104] The optimized large language model can use the remaining retrieval budget for targeted supplementary retrieval when the assertion coverage is low or the evidence consistency is poor during the retrieval process. When the assertion coverage and evidence consistency meet the requirements, that is, when sufficient evidence is retrieved, the retrieval can be stopped.

[0105] Based on any of the above embodiments, determining the retrieval reward based on the retrieval cost and the immediate stop reward includes: Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The retrieval reward is determined based on the retrieval cost, the immediate stop reward, and the document reliability score.

[0106] Specifically, when measuring the retrieval reward of a large language model, in addition to considering retrieval cost and immediate stop reward, a reliability assessment can be performed on the reference documents used to output the final answer, and the resulting document reliability score can be used to determine the retrieval reward.

[0107] Here, the file reliability score reflects the reliability of the reference file itself. The method for obtaining the file reliability score is the same as that used in the above embodiments for calculating the citation reward, and will not be repeated here.

[0108] Therefore, the retrieval reward can be obtained by weighting and summing the retrieval cost, the immediate stop reward, and the document reliability score.

[0109] In the method provided in this embodiment of the invention, adding a document reliability score to the retrieval reward can guide the large language model to prioritize more authoritative and recently published documents during the parameter iteration process of the large language model. This avoids documents published on "old news" or "weak websites" becoming the main evidence for the answer output, reduces the risk of outdated applications, and improves the reliability of reference documents.

[0110] Based on any of the above embodiments, this invention provides a method for optimizing a large language model, comprising the following steps: First, the large language model is fine-tuned using cold-start data, enabling it to learn the format of multi-turn calls to search engine tools and the fixed format of outputting citation tags for answers.

[0111] Subsequently, reinforcement learning is applied to the large language model. During the reinforcement learning process for the large language model, the following steps are executed round by round: The sample prompts are input into the large language model. The large language model processes the sample prompts through multiple rounds of thinking and searching, and then outputs the answer. The output answer from the large language model, along with the information retrieved by the large language model to produce the output answer, is obtained.

[0112] Based on the answer output by the large language model, and the information retrieved by the large language model for the output answer, the following rewards are calculated: Answer quality score This is used to reflect the quality of assertions and can be obtained by evaluating the quality of assertions in the answer.

[0113] Citation quality score This reflects the citation quality of the references cited by the assertions. It can be obtained by calculating the precision, recall, and F1 score of each reference cited by the assertion relative to the sample set of references. It can be the sentence-level average of the above values.

[0114] Consistency score of evidence This reflects whether the evidence cited in each assertion is consistent with the viewpoint expressed by the assertion, and whether the viewpoints expressed by multiple pieces of evidence cited by each assertion are consistent. It can be obtained by performing an implication / contradiction assessment on the assertions in the answer and the references they cite.

[0115] Evidence Coverage Score This reflects the coverage of references cited in an assertion as evidence, and the redundancy of references used in the assertion, provided the evidence supports the assertion. Assertion and reference coverage can be modeled as costly set coverage, thereby reducing the number of references and the redundancy of similar references while encouraging coverage to meet a preset threshold.

[0116] Document reliability score Prior scores can be determined based on the publication source and / or publication date of the referenced documents; for example, prior scores can be determined based on the publication source. Determining time decay based on publication time By combining these two factors, a weighted reward is given to highly authoritative and timely reference documents, while a penalty is given to low-authority and outdated reference documents.

[0117] Search score Search cost is determined by the number of searches, term consumption, and latency. And a reward will be given for stopping the operation immediately if sufficient evidence is available.

[0118] Points will be rejected if there is no evidence. If the evidence obtained from the search is insufficient to support the assertion, a compliant refusal to answer or an uncertainty statement such as "cannot be determined / more evidence is needed" will be generated and a positive reward will be given.

[0119] Based on this, the assertion reward, citation reward, and retrieval reward are calculated separately: Among them, assertion reward Through the Weighted average; Reference Rewards Through the Weighted average; Search rewards Through the The weighted average is obtained.

[0120] The loss function applied when iterating parameters for a large language model. It can be represented as: in, For KL divergence loss, To monitor learning loss, and Preset weights.

[0121] When iterating parameters for a large language model, the loss... Assertion rewards This applies only to lexical units representing assertions in large language models; loss. Citation rewards This applies only to lexical units representing citation markers in large language models; loss. Search rewards This only applies to lexical units representing retrieval commands in large language models.

[0122] Furthermore, in the application of search rewards During parameter iteration for large language models, on-policy stabilization and convergence can be achieved by incorporating mini-batch offline replay to reuse high-value retrieval trajectories. Here, the on-policy strategy can be implemented using GRPO (Generalized Reinforcement Policy Optimization) or PPO (Proximal Policy Optimization).

[0123] In the method provided in this embodiment of the invention, the rewards are differentiated by generating assertion rewards, citation rewards, and retrieval rewards respectively. As a result, the parameters of the large language model are iterated based on the assertion rewards, citation rewards, and retrieval rewards respectively, so that the differentiated rewards can be accurately allocated to the corresponding sub-tasks for optimization, thereby effectively improving the citation credibility of the large language model.

[0124] Furthermore, in the method provided in this embodiment of the invention, a large language model is trained based on a multi-dimensional reward system that includes answer quality score, citation quality score, evidence consistency score, evidence coverage score, document reliability score, retrieval score, and rejection score for lack of evidence. This is combined with differentiated routing, as well as evidence consistency score, evidence coverage score based on the minimum verifiable set, document reliability score based on timeliness and authority, and retrieval score involving retrieval cost. This guides the large language model to achieve highly credible question answering with fewer citations, verifiability, low conflict, and controllable cost.

[0125] Figure 2 This is a flowchart illustrating the interaction method provided by the present invention. For example... Figure 2 As shown, the method includes: Step 210: Obtain the user prompt.

[0126] Step 220: Input the user prompt into the interaction model to obtain the interaction response output by the interaction model; the interaction model is obtained based on the large language model optimization method described in the above embodiments.

[0127] Specifically, user prompts are prompts given to the user input interaction model during the human-computer interaction process. User prompts can be user queries, questions that the user expects the interaction model to answer, or tasks that the user expects the interaction model to complete.

[0128] After obtaining the user prompt, it can be input into the interaction model. This interaction model is the large language model optimized using the large language model optimization method provided in the above embodiments. Under the optimization of the large language model optimization method provided in the above embodiments, the interaction model not only has the ability to output answers including assertion citation tags and their references, but also ensures that the references can serve as the basis for the assertions, that there are no contradictions or conflicts between the assertions and the references, and that the interaction model, during the thinking, retrieval, and output process, can minimize redundant retrieval times and redundant references, and control retrieval costs.

[0129] After the user prompt is input into the interaction model, the interaction model can respond to the prompt, thereby obtaining and outputting the interactive response. This interactive response can be an answer containing assertions, reference tags to the assertions, and the corresponding reference documents.

[0130] In the method provided in this embodiment of the invention, the interaction model obtained by the large language model optimization method is used for human-computer interaction, which can achieve high-reliability interaction with fewer references, verifiability, low conflict and controllable cost, and effectively reduce interaction cost while ensuring the interaction experience.

[0131] The large language model optimization device and interaction device provided by the present invention are described below. The large language model optimization device and interaction device described below can be referred to in correspondence with the large language model optimization method and interaction method described above.

[0132] Figure 3 This is a schematic diagram of the structure of the large language model optimization device provided by the present invention. Figure 3 As shown, the device includes: Input unit 310 is used to input sample prompts into a large language model and obtain the answer output by the large language model, the answer including assertions and reference tags of the assertions; The reward generation unit 320 is used to generate an assertion reward and a reference reward based on the assertion, the reference mark of the assertion, and the reference file corresponding to the reference mark. The optimization unit 330 is configured to perform parameter iteration on the large language model based on the assertion reward and the assertion mask of the answer, wherein the assertion mask is obtained by masking the part of the answer excluding the assertions; and to perform parameter iteration on the large language model based on the reference reward and the reference mask of the answer, wherein the reference mask is obtained by masking the part of the answer excluding the reference markers.

[0133] In the apparatus provided in this embodiment of the invention, assertion rewards and citation rewards are generated for the two subtasks of generating assertions and generating citation tags, respectively, based on assertions, citation tags, and reference files corresponding to the citation tags, thus achieving reward differentiation. Based on the assertion rewards and assertion masks, parameters of the large language model are iterated, and further based on the citation rewards and citation masks, the differentiated rewards can be accurately allocated to the corresponding subtasks for optimization, thereby effectively improving the citation credibility of the large language model.

[0134] Based on any of the above embodiments, the reward generation unit is specifically used for: Determine the minimum verifiable set covering the assertion from the reference set of the reference documents, and perform a redundancy score on the reference set based on the minimum verifiable set to obtain the evidence coverage score; The assertion reward is determined based on the evidence coverage score.

[0135] Based on any of the above embodiments, the reward generation unit is specifically used for: Based on the coverage score of each reference file for each assertion in the reference set, and the similarity between any two reference files in the reference set, the minimum provable set covering the assertion is determined from the reference set, with the similarity between reference files to be included in the minimum provable set as a penalty.

[0136] Based on any of the above embodiments, the reward generation unit is further configured to: The assertion is evaluated for answer quality to obtain an answer quality score; Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; The assertion reward is determined based on the evidence coverage score, the answer quality score, and / or the evidence consistency score.

[0137] Based on any of the above embodiments, the reward generation unit is specifically used for: Based on the reference files corresponding to the citation tags of the assertions and the set of sample reference files corresponding to the sample prompts, a citation quality assessment is performed to obtain a citation quality score. Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The citation reward is determined based on at least one of the citation quality score, the evidence consistency score, and the document reliability score.

[0138] Based on any of the above embodiments, the answer further includes a retrieval instruction for retrieving the answer; the reward generation unit is further configured to: Based on the information retrieved using the large language model to output the answer, the retrieval reward is determined. Based on the retrieval reward and the retrieval mask of the answer, the large language model is subjected to parameter iteration, wherein the retrieval mask is obtained by masking the part of the answer other than the retrieval instruction.

[0139] Based on any of the above embodiments, the reward generation unit is specifically used for: The retrieval cost is determined based on at least one of the following factors: the number of times the large language model is used to retrieve the answer, lexical consumption, and latency. Based on the coverage of the assertion of the files obtained in each round of the search for the answer using the large language model, the reward is determined and the search stops. The retrieval reward is determined based on the retrieval cost and the immediate stop reward.

[0140] Based on any of the above embodiments, the reward generation unit is specifically used for: Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The retrieval reward is determined based on the retrieval cost, the immediate stop reward, and the document reliability score.

[0141] Figure 4 This is a schematic diagram of the structure of the interactive device provided by the present invention. Figure 4 As shown, the device includes: Acquisition unit 410 is used to acquire user prompts; The interaction unit 420 is used to input the user prompt into the interaction model and obtain the interaction response output by the interaction model; The interaction model is obtained based on the large language model optimization method provided in the above embodiments.

[0142] In the device provided in the embodiments of the present invention, the interaction model obtained by the large language model optimization method is used for human-computer interaction, which can achieve high-reliability interaction with fewer references, verifiability, low conflict and controllable cost, and effectively reduce interaction cost while ensuring the interaction experience.

[0143] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a large language model optimization method, which includes: Input the sample prompt into the large language model to obtain the answer output by the large language model. The answer includes assertions and reference tags for the assertions. Based on the assertion, the reference marker of the assertion, and the reference file corresponding to the reference marker, an assertion reward and a reference reward are generated; Based on the assertion reward and the assertion mask of the answer, the large language model is subjected to parameter iteration, wherein the assertion mask is obtained by masking the part of the answer excluding the assertion; and based on the citation reward and the citation mask of the answer, the large language model is subjected to parameter iteration, wherein the citation mask is obtained by masking the part of the answer excluding the citation marker.

[0144] Furthermore, the processor 510 can invoke logical instructions in the memory 530 to execute an interactive method, which includes: Get user prompts; Input the user prompt into the interaction model to obtain the interaction response output by the interaction model; The interaction model is obtained based on the large language model optimization method.

[0145] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the large language model optimization method provided by the above methods, the method comprising: Input the sample prompt into the large language model to obtain the answer output by the large language model. The answer includes assertions and reference tags for the assertions. Based on the assertion, the reference marker of the assertion, and the reference file corresponding to the reference marker, an assertion reward and a reference reward are generated; Based on the assertion reward and the assertion mask of the answer, the large language model is subjected to parameter iteration, wherein the assertion mask is obtained by masking the part of the answer excluding the assertion; and based on the citation reward and the citation mask of the answer, the large language model is subjected to parameter iteration, wherein the citation mask is obtained by masking the part of the answer excluding the citation marker.

[0147] The computer can also execute the interaction methods provided by the above methods, which include: Get user prompts; Input the user prompt into the interaction model to obtain the interaction response output by the interaction model; The interaction model is obtained based on the large language model optimization method.

[0148] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large language model optimization method provided by the methods described above, the method comprising: Input the sample prompt into the large language model to obtain the answer output by the large language model. The answer includes assertions and reference tags for the assertions. Based on the assertion, the reference marker of the assertion, and the reference file corresponding to the reference marker, an assertion reward and a reference reward are generated; Based on the assertion reward and the assertion mask of the answer, the large language model is subjected to parameter iteration, wherein the assertion mask is obtained by masking the part of the answer excluding the assertion; and based on the citation reward and the citation mask of the answer, the large language model is subjected to parameter iteration, wherein the citation mask is obtained by masking the part of the answer excluding the citation marker.

[0149] When executed by a processor, the computer program also implements interactive methods for performing the methods described above, including: Get user prompts; Input the user prompt into the interaction model to obtain the interaction response output by the interaction model; The interaction model is obtained based on the large language model optimization method.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing a large language model, characterized in that, include: Input the sample prompt into the large language model to obtain the answer output by the large language model. The answer includes assertions and reference tags for the assertions. Based on the assertion, the reference marker of the assertion, and the reference file corresponding to the reference marker, an assertion reward and a reference reward are generated; Based on the assertion reward and the assertion mask of the answer, the parameters of the large language model are iterated. The assertion mask is obtained by masking the part of the answer other than the assertion. Furthermore, based on the reference reward and the reference mask of the answer, the large language model is subjected to parameter iteration, wherein the reference mask is obtained by masking the part of the answer excluding the reference marker.

2. The large language model optimization method according to claim 1, characterized in that, The steps for generating the assertion reward include: Determine the minimum verifiable set covering the assertion from the reference set of the reference documents, and perform a redundancy score on the reference set based on the minimum verifiable set to obtain the evidence coverage score; The assertion reward is determined based on the evidence coverage score.

3. The large language model optimization method according to claim 2, characterized in that, Determining the minimum provable set covering the assertion from the reference set of the reference document includes: Based on the coverage score of each reference file for each assertion in the reference set, and the similarity between any two reference files in the reference set, the minimum provable set covering the assertion is determined from the reference set, with the similarity between reference files to be included in the minimum provable set as a penalty.

4. The large language model optimization method according to claim 2, characterized in that, The steps for generating the assertion reward also include: The assertion is evaluated for answer quality to obtain an answer quality score; Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; The determination of the assertion reward based on the evidence coverage score includes: The assertion reward is determined based on the evidence coverage score, the answer quality score, and / or the evidence consistency score.

5. The large language model optimization method according to claim 1, characterized in that, The steps for generating the reference reward include: Based on the reference files corresponding to the citation tags of the assertions and the set of sample reference files corresponding to the sample prompts, a citation quality assessment is performed to obtain a citation quality score. Implication and / or contradiction determination are performed on the assertions and the references corresponding to the citations of the assertions, and contradiction determination is performed on the references corresponding to different citations of the assertions to obtain the evidence consistency score; Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The citation reward is determined based on at least one of the citation quality score, the evidence consistency score, and the document reliability score.

6. The method for optimizing a large language model according to any one of claims 1 to 5, characterized in that, The answer also includes search instructions for retrieving the answer; The method further includes: Based on the information retrieved using the large language model to output the answer, the retrieval reward is determined. Based on the retrieval reward and the retrieval mask of the answer, the large language model is subjected to parameter iteration, wherein the retrieval mask is obtained by masking the part of the answer other than the retrieval instruction.

7. The large language model optimization method according to claim 6, characterized in that, The process of determining the retrieval reward based on information retrieved from the large language model to output the answer includes: The retrieval cost is determined based on at least one of the following factors: the number of times the large language model is used to retrieve the answer, lexical consumption, and latency. Based on the coverage of the assertion of the files obtained in each round of the search for the answer using the large language model, the reward is determined and the search stops. The retrieval reward is determined based on the retrieval cost and the immediate stop reward.

8. The large language model optimization method according to claim 7, characterized in that, The determination of the retrieval reward based on the retrieval cost and the immediate stop reward includes: Based on the publication source and / or publication time of the reference document, a document reliability assessment is performed to obtain a document reliability score; The retrieval reward is determined based on the retrieval cost, the immediate stop reward, and the document reliability score.

9. An interaction method, characterized in that, include: Get user prompts; Input the user prompt into the interaction model to obtain the interaction response output by the interaction model; The interaction model is obtained based on the large language model optimization method as described in any one of claims 1 to 8.

10. A large language model optimization device, characterized in that, include: An input unit is used to input sample prompts into a large language model and obtain the answer output by the large language model, wherein the answer includes assertions and reference tags of the assertions; The reward generation unit is used to generate assertion rewards and reference rewards based on the assertion, the reference mark of the assertion, and the reference file corresponding to the reference mark; An optimization unit is used to perform parameter iteration on the large language model based on the assertion reward and the assertion mask of the answer, wherein the assertion mask is obtained by masking the part of the answer excluding the assertions; Furthermore, based on the reference reward and the reference mask of the answer, the large language model is subjected to parameter iteration, wherein the reference mask is obtained by masking the part of the answer excluding the reference marker.

11. An interactive device, characterized in that, include: The acquisition unit is used to acquire user prompts; An interaction unit is used to input the user prompt into the interaction model and obtain the interaction response output by the interaction model. The interaction model is obtained based on the large language model optimization method as described in any one of claims 1 to 8.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the large language model optimization method as described in any one of claims 1 to 8, or the interaction method as described in claim 9.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the large language model optimization method as described in any one of claims 1 to 8, or the interaction method as described in claim 9.

Images