Answer tracing method and device, equipment and storage medium
By semantic grading of answers output by large language models, answers are solved and the problem of difficult to trace is achieved, and the answers are accurately traced and high credibility is achieved, which is suitable for application scenarios that require high credibility.
Patent Information
- Application Number
- CN202411620795.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-03-07
AI Technical Summary
The answers generated by large language models are difficult to trace, limiting their use in application scenarios that require high credibility and verifiability.
By obtaining the answers output by the knowledge retrieval system to the user's questions, the semantic hierarchical prompt words are constructed based on the preset semantic hierarchical limited text, and the answers are semantic hierarchical, and traceability information is obtained.
It realizes accurate traceability of answers, improves the credibility and verifiability of answers, and is suitable for application scenarios that require high credibility.
Smart Images
Figure CN119577088A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for answer tracing. Background Art
[0002] With the rapid development of artificial intelligence technology, large language models (such as GPT-4) are increasingly used in various knowledge base question-answering systems. These models can generate high-quality natural language answers, greatly improving the user experience. However, since the answers generated by large language models are based on their training data and algorithmic reasoning, it is often difficult for users to trace the specific sources of these answers. This traceability problem limits the use of large language models in certain application scenarios that require high credibility and verifiability. Therefore, how to trace the answers output by large language models is a technical problem that needs to be solved urgently by technicians in this field. Summary of the invention
[0003] In view of this, the present disclosure proposes an answer tracing method, device, equipment and storage medium, which can achieve accurate tracing of answers.
[0004] According to a first aspect of the present disclosure, a method for tracing an answer is provided, comprising:
[0005] Obtain answers output by the knowledge retrieval system to user questions;
[0006] Based on the answer and the preset semantically graded limiting text, constructing the semantically graded prompt words of the answer;
[0007] Based on the semantic grading prompt words, semantically grading the answer to obtain a semantic grading result of the answer;
[0008] Based on the semantic classification result, the source information of the answer is obtained.
[0009] In a possible implementation, when the answer is semantically graded based on the semantic graded prompt words to obtain the semantic graded result of the answer, it is implemented based on a large language model.
[0010] In one possible implementation, the semantic classification result of the answer includes at least one of a first-level semantic classification result, a second-level semantic classification result and a third-level semantic classification result, wherein the first-level semantic classification result is a semantic title in the answer, the second-level semantic classification result is an explanation text of the first-level semantic classification result, and the third-level semantic classification result is an explanation text of the second-level semantic classification result.
[0011] In a possible implementation, when obtaining the answer output by the knowledge retrieval system to the user's question, it also includes:
[0012] Determining whether the length of the answer exceeds a preset length threshold;
[0013] If the length threshold is exceeded, extracting a summary of the answer;
[0014] Based on the summary, the answer is traced back to its source.
[0015] In a possible implementation, when extracting the summary of the answer, it includes:
[0016] Based on the answer and the preset summary extraction limiting text, construct a summary extraction prompt word of the answer;
[0017] A summary of the answer is extracted based on the summary extraction prompt word.
[0018] According to a second aspect of the present disclosure, there is provided a device for tracing the source of an answer, comprising:
[0019] The answer acquisition module is used to obtain the answers output by the knowledge retrieval system to the user's questions;
[0020] A semantically graded prompt word construction module, used to construct the semantically graded prompt words of the answer based on the answer and a preset semantically graded limiting text;
[0021] A semantic grading module, used to semantically grade the answer based on the semantic grading prompt words to obtain a semantic grading result of the answer;
[0022] The tracing module is used to obtain the tracing information of the answer based on the semantic classification result.
[0023] In a possible implementation manner, the device further includes:
[0024] A summary extraction module, used to determine whether the length of the answer exceeds a preset length threshold, and if it exceeds the length threshold, extract a summary of the answer;
[0025] The tracing module is further used to trace the answer based on the summary.
[0026] In a possible implementation, the summary extraction module includes:
[0027] A summary extraction prompt word construction unit, used to construct a summary extraction prompt word of the answer based on the answer and a preset summary extraction limiting text;
[0028] A summary extraction unit is used to extract a summary of the answer based on the summary extraction prompt word.
[0029] According to a third aspect of the present disclosure, there is provided an answer tracing device, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to execute the method described in the first aspect of the present disclosure.
[0030] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of the present disclosure.
[0031] The present disclosure provides a method, device, equipment and storage medium for tracing the source of an answer, the method comprising: obtaining an answer output by a knowledge retrieval system for a user's question; constructing a semantic grading prompt word for the answer based on the answer, a preset semantic grading limited text and a grading result output format; semantically grading the answer based on the semantic grading prompt word to obtain a semantic grading result of the answer; and obtaining tracing information of the answer based on the semantic grading result. The method disclosed in the present disclosure can achieve accurate tracing of the source of the answer.
[0032] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
[0034] Figure 1 A flowchart of a method for tracing the source of an answer according to an embodiment of the present disclosure is shown;
[0035] Figure 2 A schematic diagram of a semantic classification result according to an embodiment of the present disclosure is shown;
[0036] Figure 3 A flowchart showing an example of a method for tracing the source of an answer according to an embodiment of the present disclosure;
[0037] Figure 4 A schematic block diagram of a device for tracing the source of an answer according to an embodiment of the present disclosure is shown;
[0038] Figure 5 A schematic block diagram of a reply tracing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0039] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0040] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0041] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0042] <Method Example>
[0043] Figure 1 FIG. 1 is a flowchart of a method for tracing the source of an answer according to an embodiment of the present disclosure. Figure 1 As shown, the method includes steps S1100-S1400.
[0044] S1100, obtain the answer output by the knowledge retrieval system to the user's question. Specifically, when the user uses the knowledge retrieval system to search, he needs to input a question to the knowledge retrieval system first. The knowledge retrieval system will search in the database associated with the knowledge retrieval system according to the question input by the user, and give a corresponding answer according to the search result. After the knowledge retrieval system gives the corresponding answer, the system executing the disclosed method can obtain the answer output by the knowledge retrieval system to the user's question. Among them, the system executing the disclosed method can be the same system as the knowledge retrieval system, or it can be a different system, which is not specifically limited here. In the case that the system executing the disclosed method and the knowledge retrieval system are the same system, the knowledge retrieval system is responsible for both answering the questions and tracing the answers.
[0045] S1200, constructing semantically graded prompt words for the answer based on the read answer and the preset semantically graded limiting text.
[0046] First of all, it should be noted that a semantic grading restriction text is pre-set in the system executing the method disclosed herein, and the semantic grading restriction text is used to instruct the large language model to semantically grade the answer, instruct the large language model to output the semantic grading result in a specified format, and clarify the maximum number of levels in the semantic grading result. The large language model in the present disclosure can be a natural language processing model such as GPT-4.
[0047] In a specific example, the semantically graded definition text may be as follows:
[0048] Please act as a semantic segmentation expert, divide the following input sentence into several semantic points according to semantics, and output it in Markdown format. The output in Markdown format includes the first-level semantic classification results, the second-level semantic classification results, and the third-level semantic classification results. Among them, the first-level semantic classification result is the semantic title in the answer, marked with "#semantic title". The second-level semantic classification result is the explanatory text of the first-level semantic classification result, recorded at the bottom of the corresponding first-level semantic classification result, and marked with "##explanatory text". The third-level semantic classification result is the explanatory text of the second-level semantic classification result, recorded at the bottom of the corresponding second-level semantic classification result, and marked with "###explanatory text". There should be no more than three levels.
[0049] After reading the answer, the preset semantic classification limiting text is obtained, and the semantic classification limiting text and the answer are filled into the corresponding positions in the first prompt word frame to obtain the semantic classification prompt word for the answer. The first prompt word frame can be as follows:
[0050] ”'Semantic classification qualified text
[0051] Input sentence:
[0052] answer"'
[0053] After obtaining the semantic classification prompt words for the answer, step S1300 may be executed to semantically classify the answer based on the semantic classification prompt words of the answer to obtain a semantic classification result of the answer.
[0054] In a possible implementation, when semantically grading answers based on the semantic grading prompt words of the answers and obtaining the semantic grading results of the answers, it is implemented based on a large language model. Specifically, the semantic grading prompt words of the answers are input into a pre-trained large language model, and the large language model can output the semantic grading results of the answers according to the semantic grading prompts.
[0055] When the semantic classification limiting text in the semantic classification prompt words is as shown in the example above, the semantic classification results output by the large language model will include at least one of the first-level semantic classification results, the second-level semantic classification results and the third-level semantic classification results, wherein the first-level semantic classification results are the semantic titles in the answers, the second-level semantic classification results are the explanation texts of the first-level semantic classification results, and the third-level semantic classification results are the explanation texts of the second-level semantic classification results. In a specific example, the semantic classification results for the answers are as follows: Figure 2 shown.
[0056] After obtaining the semantic classification result of the answer, step S1400 can be executed to obtain the source information of the answer based on the semantic classification result. The specific source tracing steps are as follows:
[0057] First, traverse each first-level semantic classification result.
[0058] Second, for the current first-level semantic classification results traversed, the minimum semantic classification result in the current first-level semantic classification results is searched, and the text matching the minimum semantic classification result is searched in the database associated with the knowledge retrieval system as the traceability information of the current first-level semantic classification results. After the traversal is completed, the traceability information of each first-level semantic classification result is obtained.
[0059] It should be noted here that if there is only a second-level semantic classification result below the current first-level semantic classification result, the second-level semantic classification result is the minimum semantic classification result in the current first-level semantic classification result. If there are both a second-level semantic classification result and a third-level semantic classification result below the current first-level semantic classification result, the third-level semantic classification result is the minimum semantic classification result in the current first-level semantic classification result. The minimum semantic classification result can be one or more, and is not specifically limited here.
[0060] For each minimum semantic hierarchy structure, a database associated with the knowledge retrieval system is searched for text that matches the minimum semantic hierarchy result, and the text that matches the minimum semantic hierarchy result is used as the traceability information of the current first-level semantic hierarchy result. When searching for text that matches the minimum semantic hierarchy result in a database associated with the knowledge retrieval system, the following steps may be included:
[0061] First, the similarity between the minimum semantic classification result and each text in the database is calculated, and the similarity matrix of the minimum semantic classification result is constructed based on the calculated similarities. Specifically, the similarity between the minimum semantic classification result and each text in the database is calculated in turn to obtain the similarity corresponding to each text. The similarity corresponding to each text is recorded as S1, S2, S3, ..., S n , where n is the total number of texts in the database. The similarity matrix S of the minimum semantic classification result can be obtained by arranging the similarities corresponding to each text in order, S = [S1, S2, S3, ..., S n ]. When calculating the similarity, a cosine similarity algorithm, a Euclidean distance algorithm, or a Manhattan distance algorithm may be used, which is not specifically limited here.
[0062] Secondly, obtain the number of times each text in the database has appeared in the traceability information, and build the occurrence frequency matrix of the minimum semantic classification result based on the calculated number of times. According to the order of calculating similarity, count the number of times each text in the database has appeared in the traceability information, and record the number of times each text has appeared in the traceability information as f1, f2, f3, ..., f n, the inverse of the number of times each text has appeared in the traceability information is arranged in order to obtain the appearance frequency matrix F of the minimum semantic classification result, F = [1 / f1, 1 / f2, 1 / f3, ..., 1 / f n ].
[0063] It should be noted here that since there will be multiple first-level semantic segmentation results in one answer, when tracing the answer, each first-level semantic classification result will be traced in turn to obtain the tracing information corresponding to each first-level semantic classification result. After obtaining the tracing information of each first-level semantic classification result, the number of occurrences of the text corresponding to the tracing information will be increased by 1. In this way, the number of times each text in the data has appeared in the tracing information can be counted in time.
[0064] Finally, based on the similarity matrix and the frequency matrix, the text that matches the minimum semantic classification result is found in the database associated with the knowledge retrieval system. Specifically, first, based on the similarity matrix S and the frequency matrix F, the final similarity calculation result S′ is calculated, and the calculation formula of S′ is: S′=S·FT. Then, the maximum value is screened out from the final similarity calculation result S′, and the text corresponding to the screened maximum value is used as the text that matches the minimum semantic classification result. In this embodiment, the final similarity between the text in the database and the minimum semantic classification result is equal to the similarity corresponding to the text multiplied by the inverse of the number of times it has appeared in the traceability information, which makes the more times the text appears in the traceability information, the smaller its corresponding final similarity, so that in the subsequent traceability process, there are more opportunities to match other texts in the database, and ultimately more articles in the database appear in the traceability information of the answer.
[0065] The text that matches the minimum semantic classification result is used as the traceability information of the current first-level semantic classification result. The matching text can be an article or a specific section of the article, which is not specifically limited here.
[0066] After the traversal is completed, the traceability information of each first-level semantic classification result can be obtained.
[0067] Fourth, based on the traceability information of each first-level semantic classification result, construct the traceability information of the answer. Specifically, the traceability information of each first-level semantic classification result is annotated to the corresponding position of each first-level semantic classification result, and the semantic classification result annotated with the traceability information of each first-level semantic classification result is used as the traceability information of the answer.
[0068] In a possible implementation, when obtaining the answer output by the knowledge retrieval system to the user's question, the following steps are also included:
[0069] First, determine whether the length of the answer exceeds a preset length threshold. Specifically, the length threshold is the maximum character length that the large language model can handle.
[0070] Second, when the length threshold is exceeded, extract the summary of the answer. Specifically, when extracting the summary of the answer, the following steps may be included:
[0071] First, based on the answer and the preset summary extraction limiting text, construct the summary extraction prompt words of the answer. It should be noted here that the system executing the disclosed method is pre-set with summary extraction limiting text, which is used to instruct the large language model to extract the summary of the answer and instruct the large language model to extract the summary according to the specified summary extraction rules. In a specific example, the summary extraction limiting text can be as follows:
[0072] You are an expert in abstract extraction. Please summarize the following text into a brief summary of no more than X words, and make sure that the extracted content retains the original intent and important information.
[0073] After reading the answer, the preset summary extraction limiting text is obtained, and the summary extraction limiting text and the answer are filled into the corresponding positions in the second prompt word frame to obtain the summary extraction prompt word for the answer. The second prompt word frame can be as follows:
[0074] ”'Abstract Extract limited text. Please summarize the following answers:
[0075]
{answer}
[0076] ”'
[0077] Secondly, after obtaining the summary extraction prompt words for the answer, the summary extraction prompt words of the answer are output to the pre-trained large language model, and the large language model can output the summary of the answer.
[0078] After the large language model outputs the summary of the answer, the answer will be traced back based on the summary. The specific tracing process is described above and will not be repeated here.
[0079] In order to clearly illustrate the answer tracing method disclosed in the present invention, the following is combined with Figure 3 The answer tracing method is explained again. The specific steps are as follows:
[0080] S1, obtain the answer output by the knowledge retrieval system to the user's question.
[0081] S2, determine whether the length of the answer exceeds a preset length threshold, if it exceeds the length threshold, execute step S3, if it does not exceed the length threshold, execute step S5.
[0082] S3, based on the answer and the preset summary extraction limiting text, construct the summary extraction prompt words of the answer.
[0083] S4, extracting prompt words from the summary of the answer and outputting them to the pre-trained large language model, which then outputs a summary of the answer, and uses the summary of the answer as the final answer to be traced.
[0084] S5, constructing semantically graded prompt words for the answer based on the read answer and the preset semantically graded limiting text.
[0085] S6, inputting the semantic classification prompt words of the answer into a pre-trained large language model, and outputting the semantic classification result of the answer through the large language model.
[0086] S7, based on the semantic classification results output by the large language model, obtain the traceability information of the answer.
[0087] The specific implementation steps in this example are as mentioned above and will not be repeated here.
[0088] The present disclosure provides a method for tracing the source of an answer, including: obtaining an answer output by a knowledge retrieval system for a user's question; constructing a semantic grading prompt word for the answer based on the answer, a preset semantic grading limiting text, and a grading result output format; semantically grading the answer based on the semantic grading prompt word to obtain a semantic grading result of the answer; and obtaining tracing information of the answer based on the semantic grading result. The method disclosed in the present disclosure can achieve accurate tracing of the source of the answer.
[0089] <Device Example>
[0090] Figure 4 FIG. 2 is a schematic block diagram of a device for tracing the source of an answer according to an embodiment of the present disclosure. Figure 4 As shown, the device 100 includes:
[0091] The answer acquisition module 110 is used to obtain the answer output by the knowledge retrieval system to the user's question;
[0092] A semantically graded prompt word construction module 120 is used to construct semantically graded prompt words of the answer based on the answer and a preset semantically graded limited text;
[0093] A semantic classification module 130 is used to perform semantic classification on the answers based on the semantic classification prompt words to obtain a semantic classification result of the answers;
[0094] The tracing module 140 is used to obtain tracing information of the answer based on the semantic classification result.
[0095] In a possible implementation, the device further includes:
[0096] A summary extraction module is used to determine whether the length of the answer exceeds a preset length threshold, and if it exceeds the length threshold, extract a summary of the answer;
[0097] The tracing module is also used to trace the answer based on the summary.
[0098] In a possible implementation, the summary extraction module includes:
[0099] A summary extraction prompt word construction unit, used to construct a summary extraction prompt word of the answer based on the answer and a preset summary extraction limiting text;
[0100] The summary extraction unit is used to extract the summary of the answer based on the summary extraction prompt word.
[0101] <Equipment Embodiment>
[0102] Figure 5 FIG. 2 shows a schematic block diagram of a device for tracing the source of an answer according to an embodiment of the present disclosure. Figure 5 As shown, the answer tracing device 200 includes: a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the above-mentioned answer tracing methods when executing the executable instructions.
[0103] Here, it should be noted that the number of processors 210 can be one or more. At the same time, in the answer tracing device 200 of the embodiment of the present disclosure, an input device 230 and an output device 240 may also be included. Among them, the processor 210, the memory 220, the input device 230 and the output device 240 may be connected through a bus or in other ways, which are not specifically limited here.
[0104] The memory 220 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and various modules, such as the program or module corresponding to the answer tracing method of the embodiment of the present disclosure. The processor 210 executes various functional applications and data processing of the answer tracing device 200 by running the software programs or modules stored in the memory 220.
[0105] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include display devices such as display screens.
[0106] <Storage Medium Embodiment>
[0107] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is also provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 210, any of the above-mentioned answer tracing methods is implemented.
[0108] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for tracing the source of an answer, characterized in that: include: Obtain answers output by the knowledge retrieval system to user questions; Based on the answer and the preset semantically graded limiting text, constructing the semantically graded prompt words of the answer; Based on the semantic grading prompt words, semantically grading the answer to obtain a semantic grading result of the answer; Based on the semantic classification result, the source information of the answer is obtained.
2. The method according to claim 1, characterized in that When the answer is semantically graded based on the semantic graded prompt words to obtain the semantic graded result of the answer, it is implemented based on a large language model.
3. The method according to claim 1, characterized in that The semantic classification results of the answer include at least one of the first-level semantic classification results, the second-level semantic classification results and the third-level semantic classification results, wherein the first-level semantic classification result is the semantic title in the answer, the second-level semantic classification result is the explanation text of the first-level semantic classification result, and the third-level semantic classification result is the explanation text of the second-level semantic classification result.
4. The method according to claim 1, characterized in that When obtaining the answers output by the knowledge retrieval system to user questions, it also includes: Determining whether the length of the answer exceeds a preset length threshold; If the length threshold is exceeded, extracting a summary of the answer; Based on the summary, the answer is traced back to its source.
5. The method according to claim 4, characterized in that When extracting a summary of said answer, include: Based on the answer and the preset summary extraction limiting text, construct a summary extraction prompt word of the answer; A summary of the answer is extracted based on the summary extraction prompt word.
6. A device for tracing the source of an answer, characterized in that: include: The answer acquisition module is used to obtain the answers output by the knowledge retrieval system to the user's questions; A semantically graded prompt word construction module, used to construct the semantically graded prompt words of the answer based on the answer and a preset semantically graded limiting text; A semantic grading module, used to semantically grade the answer based on the semantic grading prompt words to obtain a semantic grading result of the answer; The tracing module is used to obtain the tracing information of the answer based on the semantic classification result.
7. The device according to claim 6, characterized in that Also includes: A summary extraction module, used to determine whether the length of the answer exceeds a preset length threshold, and if it exceeds the length threshold, extract a summary of the answer; The tracing module is further used to trace the answer based on the summary.
8. The device according to claim 7, characterized in that The summary extraction module comprises: A summary extraction prompt word construction unit, used to construct a summary extraction prompt word of the answer based on the answer and a preset summary extraction limiting text; A summary extraction unit is used to extract a summary of the answer based on the summary extraction prompt word.
9. A device for tracing the source of an answer, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 5 when executing the executable instructions.
10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.