Enhanced question answering method and electronic device based on large language model

Abstract

The present application relates to an enhanced question-answering method and electronic device based on a large language model, the method comprising: obtaining an input query, generating a first content according to the input query and a knowledge database; further, dividing the first content into a plurality of first content blocks, and searching a knowledge database based on the first content block for each first content block to obtain a text block corresponding to the first content block; determining a first content block that satisfies fact verification, and generating a second content through a large language model based on the first content block that satisfies fact verification and the corresponding text block; determining a correlation verification result between the second content and the input query, and when the correlation verification result satisfies a preset relationship, using the second content as an answer to the input query. The present method can improve the accuracy and reliability of generated text facts.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an enhanced question-answering method and electronic device based on a large language model. Background Art

[0002] In the question-and-answer scenario, the question text can be input in the form of a conversation. The input question text will be provided to the AI ​​question-and-answer model as a prompt. The AI ​​question-and-answer model can generate the reply content corresponding to the current question text based on the prompt.

[0003] In the field of artificial intelligence technology, AI question-answering models are deep learning models trained based on massive amounts of text data. In question-answering scenarios, the trained AI question-answering model retrieves relevant information from a large number of documents through retrieval and generation steps, and outputs predicted answers based on this information. The accuracy of knowledge retrieval in this method is relatively low, and the generated text contains inaccurate or erroneous factual statements, which is not conducive to improving the accuracy of the output answers. Summary of the invention

[0004] Based on this, it is necessary to provide a method that can improve the accuracy and reliability of generated text facts in response to the above technical problems.

[0005] In a first aspect, the present application provides an enhanced question answering method based on a large language model, comprising:

[0006] Get input query;

[0007] generating first content according to the input query and the knowledge database;

[0008] Splitting the first content into a plurality of first content blocks; searching a knowledge database based on each first content block to obtain a text block corresponding to the first content block;

[0009] Determine a first content block that satisfies fact verification, and generate second content through a large language model based on the first content block that satisfies fact verification and a corresponding text block;

[0010] A correlation verification result between the second content and the input query is determined, and when the correlation verification result satisfies a preset relationship, the second content is used as an answer to the input query.

[0011] In one embodiment, splitting the first content into a plurality of first content blocks includes:

[0012] Based on the first content, determining a split example;

[0013] The split examples are input as prompt words into the large language model, and the large language model splits the first content into multiple first content blocks according to the prompt words; wherein the first content blocks are independent sentences, and the number of the first content blocks is less than the number threshold.

[0014] In one embodiment, the knowledge database includes a document library and a vector library, the document library includes a plurality of text blocks, and the text blocks are vectorized by an embedding model to obtain a vector database;

[0015] For each first content block, searching a knowledge database based on the first content block to obtain a text block corresponding to the first content block includes:

[0016] For each first content block, similarity calculation is performed on the first content block with the document library and the vector library respectively, and N text blocks are determined based on the results of the similarity calculation, and the N text blocks are used as text blocks of the first content block.

[0017] In one embodiment, determining a first content segment satisfying fact verification includes:

[0018] For each first content block, the first content block and the corresponding N text blocks are combined into a group;

[0019] For each combination, the large language model is used to determine whether the search content in the combination supports the verification content by means of prompt words; wherein the search content is the N text blocks in the combination, and the verification content is the first content block in the combination;

[0020] For each combination, if the retrieved content supports the verification content, it is determined that the first content block in the combination satisfies the fact verification.

[0021] In one embodiment, it is characterized in that, for each first content block, similarity calculation is performed on the first content block with the document library and the vector library respectively, N text blocks are determined based on the results of the similarity calculation, and the N text blocks are used as text blocks of the first content block, including:

[0022] Segmenting the first content block and the text blocks in the document library;

[0023] Calculate the similarity between the first content block after word segmentation and the text block after word segmentation to obtain the relevance score between the first content block and each text block;

[0024] Vectorizing the first content block, performing similarity calculation between the vectorized first content block and the text block vectors in the vector library, and obtaining a correlation score between the first content block and each text block vector;

[0025] The relevance scores between the first content block and each text block and the relevance scores between the first content block and each text block vector are weightedly combined to obtain the scores of the first content block and each text block; wherein the scores are sorted from large to small, and the text blocks with the top N scores are selected as the text blocks of the first content block.

[0026] In one embodiment, determining a verification result of the relevance of the second content to the input query includes:

[0027] The pre-trained StructBert model is used to determine the semantic relevance between the input query and the second content to obtain a relevance verification result; wherein the relevance verification result includes implication, contradiction or neutral relationship. If the relevance verification result is an implication relationship, the relevance verification satisfies the set conditions.

[0028] In one embodiment, the enhanced question-answering method further includes:

[0029] If the relevance verification result does not satisfy the preset relationship, the large language model enhances the input query based on the input query and the query example to obtain a second enhanced query;

[0030] The large language model is used to regenerate second content according to the second enhanced query.

[0031] In one embodiment, the enhanced question-answering method further includes:

[0032] The large language model enhances the input query based on the input query and the query example to obtain a second enhanced query, including: the large language model modifies the input query based on the input query and the query example, and uses the modified input query as the second enhanced query;

[0033] The method also includes outputting a fallback response if the number of times the second content is regenerated according to the second enhanced query using the large language model is greater than a number threshold.

[0034] In one embodiment, the knowledge database includes a document library and a vector library, the document library includes a plurality of text blocks, and the text blocks are vectorized by an embedding model to obtain a vector database; generating first content according to an input query and the knowledge database includes: enhancing the input query according to the input query and the knowledge database to obtain a first enhanced query, and generating first content according to the first enhanced query using a large language model;

[0035] Among them, according to the input query and the knowledge database, the input query is enhanced to obtain a first enhanced query, including: using a hybrid retrieval method for the input query, weighted combination of the full-text retrieval relevance score and the vector retrieval relevance score to obtain a hybrid score of the input query and the text block, and using M text blocks with the highest hybrid score as the first enhanced query.

[0036] In one embodiment, the full-text search relevance score is calculated as follows:

[0037]

[0038]

[0039] Among them, the BM25 algorithm is used for calculation. is the number of terms in the query, is the i-th term in the query, is the inverse block frequency, N is the total number of blocks, Contains terms The number of text blocks, It is a term The number of occurrences in text block D, is the length of the text block D, is the average of all text blocks, and b are tuning parameters.

[0040] In a second aspect, in order to solve the above technical problems, the present application also provides an electronic device, the electronic device comprising:

[0041] A memory is used to store a computer program; a processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of any one of the above-mentioned enhanced question-answering methods based on a large language model are implemented.

[0042] The above-mentioned enhanced question-answering method and electronic device based on a large language model can obtain the first content based on the initial input query and the knowledge database using the large language model, and split the obtained first content into multiple first content blocks, and retrieve the knowledge database based on the first content blocks to obtain the corresponding text blocks. Further, the large language model is input based on the first content blocks verified by facts and the corresponding text blocks to obtain the second content, so as to ensure that the first content blocks used to generate the second content are consistent with the facts and are reliable enough, so that the generated second content is accurate and reliable enough. Further, the second content needs to meet the relevance verification with the initial input query before it can be output. In the process of generating answers based on the large language model, that is, the process from inputting the query to the final inputting the content is continuously improved to improve the accuracy and reliability of the generated text facts. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is an application environment diagram of an enhanced question answering method based on a large language model in one embodiment;

[0044] Figure 2is a first flow chart of an enhanced question-answering method based on a large language model in one embodiment;

[0045] Figure 3 is a schematic diagram of a process of dividing first content into blocks in one embodiment;

[0046] Figure 4 A schematic diagram of a process of performing fact verification on a first content block and a text block in one embodiment;

[0047] Figure 5 A schematic diagram of a process for verifying first content segmentation and text block segmentation in one embodiment;

[0048] Figure 6 A schematic diagram of a flow chart of how to regenerate the second content if fact verification is not satisfied in one embodiment;

[0049] Figure 7 A schematic diagram of a process of modifying an input query and regenerating second content in one embodiment;

[0050] Figure 8 is a second flow chart of an enhanced question-answering method based on a large language model in one embodiment;

[0051] Fig. 9 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] like Figure 1The application environment shown in the figure includes a terminal 102 and a server 104. The enhanced question-answering method based on a large language model provided in the embodiment of the present application is applied to the server 104. Among them, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The user can input a specific query through the terminal 102, and then the terminal 102 uploads the input query to the server 104. The server 104 determines the answer based on the input query and can send the answer to the terminal 102. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0054] In one embodiment, Figure 2 As shown, an enhanced question answering method based on a large language model is provided, comprising the following steps:

[0055] Step 201, obtaining an input query;

[0056] Among them, the user can ask questions through the terminal (which can be a smart phone, laptop or smart watch, etc.). That is, the user inputs a specific query through the terminal. The specific query input can be: "Help me check the birth information of someone A." Furthermore, the user can ask questions or inquire through the terminal through text input or voice input. It should be noted that the server obtains the user's input query from the terminal, and the format of the obtained input query is not limited to text format, but can also be a vector format. If the server obtains the input query in vector format, specifically, the input query collected by the terminal can be in text or voice format, and the text or voice format is converted to vector format after collection and sent to the server; it can also be that the text or voice format is directly sent to the server after collection, and the server receives the input query and performs vector format conversion.

[0057] Step 202, generating first content according to the input query and the knowledge database;

[0058] Among them, the knowledge database contains a large amount of document information. The input query can be compared with the document information in the database, or the input query can be vectorized and compared with the vector information in the knowledge database. The input query can be input into the large language model, and the first content can be generated by searching in the large language model. The large language model can be an artificial intelligence model trained by deep learning technology, which can understand and generate natural language. The model can learn language patterns by analyzing large amounts of text data, and then apply these patterns to understand the input questions and generate answers. This process includes parsing the intent of the question, retrieving information, generating text, and optimizing to ensure accuracy and relevance.

[0059] Step 203: split the first content into a plurality of first content blocks; for each first content block, search the knowledge database based on the first content block to obtain a text block corresponding to the first content block.

[0060] Specifically, the generated first content can be split into facts through a combination of syntactic analysis and a mature deep learning model to obtain multiple first content blocks. Among them, the first content block can be a text fragment or a factual statement. For example, the input query in the above example is "Help me find the birth information of someone A." You will get an answer, which is the first content. For example, the first content is "Someone A was born in C in year B." By combining traditional syntactic analysis and deep learning models to split the above answer, you can get "Someone A was born in year B." and "Someone A was born in C." Two factual statements, which are the first content blocks. Furthermore, these independent statements are searched in the knowledge database to obtain multiple text blocks with strong correlation corresponding to each independent statement.

[0061] Step 204, determining a first content block that satisfies fact verification, and generating second content through a large language model based on the first content block that satisfies fact verification and a corresponding text block;

[0062] Among them, the first content block can be judged by the AI ​​model, specifically to perform fact verification. For example, the AFC model can be used to verify the content block to determine whether the retrieved content supports the corresponding fact statement. If the retrieved content supports the verified content, the fact verification is satisfied. The first content block that passes the fact verification and the corresponding text block are generated through the large language model for the second content, thereby further improving the reliability of the generated second content.

[0063] Step 205: determine the correlation verification result between the second content and the input query, and when the correlation verification result satisfies a preset relationship, use the second content as an answer to the input query.

[0064] Among them, considering that some of the generated second content may have textual similarities with the input query, but for the semantics of the sentence, they are sometimes opposite or unrelated. Therefore, correlation verification is required, and Pearson sparsity can be used to measure the correlation between two quantities. When the second content is semantically related to the input query, it can be output; otherwise, it cannot be output.

[0065] In the enhanced question-answering method based on a large language model provided in the above-mentioned embodiment, the first content is obtained based on the input query using the large language model, and the obtained first content is split into multiple first content blocks, and the knowledge database is retrieved based on the first content blocks to obtain corresponding text blocks. Further, the large language model is input based on the first content blocks verified by facts and the corresponding text blocks to obtain the second content, so as to ensure that the first content blocks used to generate the second content are consistent with the facts and are reliable enough, and then the generated second content is accurate and reliable enough. Further, the second content needs to meet the relevance verification with the initial input query before it can be output. Furthermore, in the process of generating answers based on the large language model, that is, from the input query to the final input content, this embodiment is continuously improved and enhanced to improve the accuracy and reliability of the generated text facts.

[0066] In one embodiment, Figure 3 As shown, splitting the first content into multiple first content blocks includes the following steps:

[0067] Step 301, determining a splitting example based on the first content;

[0068] It should be noted that the splitting example is used to illustrate the splitting of an entire content into several blocks, so as to guide the large language model to split the first content. It should be noted that in different usage environments, there are differences in the splitting results required for different first contents. Furthermore, when using the large language model to split the first content in a specific usage environment, it is necessary to determine the splitting example corresponding to the first content in the usage environment. Specifically, the splitting example can be manually determined based on the first content, or a plurality of splitting example templates can be set, and the server side selects a suitable splitting example from the templates of the plurality of splitting examples according to the first content.

[0069] Step 302: Input the split examples as prompt words into the large language model, and the large language model splits the first content into multiple first content blocks according to the prompt words; wherein the first content blocks are independent sentences, and the number of the first content blocks is less than the quantity threshold.

[0070] Specifically, the first content is the initial answer generated by searching the database through the input query, which may contain the correct answer but still needs further verification to obtain a more accurate answer. Therefore, the split example can be used as a prompt word to guide the large language model to split the first content into multiple first content blocks. The first content blocks after splitting are independent sentences, and the number of blocks needs to be limited, which shall not exceed the set number threshold. If the number of blocks generated by splitting is too large, it may produce fragmented sentences and cause sentence ambiguity.

[0071] In this embodiment, by splitting the first content into multiple first content text blocks, the process of processing the first content can be made more efficient, and for complex inquiries, the block division can simplify the inquiry so that each block can be processed independently, reducing complexity. Furthermore, the content after the block division can be processed in parallel, which significantly improves efficiency. It should be noted that block division can reduce the difficulty of query processing, but excessive block division will result in incomplete block semantics, which will have a counterproductive effect. Therefore, setting a threshold limit on the number of blocks also lays a guarantee for the processing of the first content.

[0072] In one embodiment, the knowledge database includes a document library and a vector library. The document library includes multiple text blocks. The text blocks are vectorized through an embedded model to obtain a vector database. Specifically, the database consists of a document library and a vector library. The text blocks of the document library can be extracted by identifying the content of each element in the existing knowledge base document through layout analysis, OCR technology, table recognition and other technologies, and then restored to the reading order of the interface through a layout restoration algorithm. The vector database semantically segments the text blocks in the document library on the basis of the document library, vectorizes the text blocks through an embedded model, and stores the vectorized text blocks in the vector database.

[0073] Among them, in the above step 203: for each first content block, searching the knowledge database based on the first content block to obtain a text block corresponding to the first content block, including: for each first content block, performing similarity calculation on the first content block with the document library and the vector library respectively, determining N text blocks based on the results of the similarity calculation, and using the N text blocks as the text blocks of the first content block.

[0074] Among them, each content block must perform similarity calculations with the document library and the vector library respectively. Specifically, for each first content block, perform similarity calculations with the text block in the document library to obtain the similarity between the first content block and the text block in the document library. The first content block is vectorized through an embedded model, and the similarity is calculated with the vectorized form of the text block in the vector library to obtain the similarity between the first content block and the vector library. The similarity calculation results of the document library and the vector library are merged to obtain the final similarity result. All similarity results are sorted from high to low, and the top N text blocks in similarity ranking are selected as the text blocks corresponding to the first content.

[0075] In the above embodiment, by performing similarity calculation on the first content block in the knowledge database to obtain N corresponding text blocks ranked high in similarity to the first content block, the intention of the input query can be understood more accurately. By performing similarity calculation on the input query and the knowledge database, the text blocks with strong similarity can be determined more quickly and accurately as the text blocks corresponding to each first content block, thereby improving the efficiency of the question-answering method.

[0076] In one embodiment, Figure 4 As shown, in the above step 204: determining the first content segment satisfying fact verification includes the following steps:

[0077] Step 401: for each first content block, combine the first content block and the corresponding N text blocks into a group.

[0078] Specifically, for example, if the first content block is A, select N text blocks with the highest similarity ranking, A1, A2, A3, A4, .... An. That is, the first content block A and the corresponding N text blocks A1 to An are combined into a combination. This combination method can be used for subsequent analysis, comparison or other processing.

[0079] Step 402: For each combination, the big data model is allowed to determine whether the search content in the combination supports the verification content by means of prompt words; the search content is the N text blocks in the combination, and the verification content is the first content block in the combination.

[0080] Step 403: For each combination, if the retrieved content supports the verification content, it is determined that the first content block in the combination satisfies the fact verification.

[0081] Specifically, the first content block and the N text blocks with the highest similarity ranking corresponding to the first content block are taken as a combination to determine whether they meet the fact verification. Further, the fact verification can be to let the large language model judge whether the content of the retrieved text block supports the corresponding first content block by means of prompt words for each first content block and the corresponding text block retrieved. The prompt words are as follows:

[0082] {As a knowledgeable and honest chatbot that always tells the truth, your task is to determine whether the "verification content" is supported based on the "retrieval content". If the result is "yes", return the "verification content" and "retrieval content"; if the result is "no", do not return.}

[0083] Among them, "retrieval content" refers to the text block retrieved from the knowledge base, and "verification content" refers to the first content block that needs to be verified.

[0084] In this embodiment, after the first content block and the corresponding N text blocks ranked high in similarity are combined, the large language model is used to perform fact verification on the first content block in the combination using the N text blocks in the combination. If the fact verification is not satisfied, the combination including the first content block and the text block cannot be returned. Thus, the first content block that passes the verification is the correct content, thereby reducing a portion of erroneous content that does not meet the fact verification, and significantly improving the accuracy.

[0085] In one embodiment, Figure 5 As shown, for each first content block, similarity calculation is performed between the first content block and the document library and the vector library, and N text blocks are determined based on the result of the similarity calculation, and the N text blocks are used as text blocks of the first content block, including the following steps:

[0086] Step 501: Segment the first content block and the text blocks in the document library.

[0087] Specifically, this step is to segment the first content block and the text block in the document library, wherein the first content block and the text block in the document library exist in the form of independent sentences. That is, segmenting the independent sentence will result in multiple words. For example, "Eating more apples is good for health." After segmentation, it will be ["more", "eat", "apple", "beneficial", "for", "health"] multiple words.

[0088] Step 502: Calculate the similarity between the first content block after word segmentation and the text block after word segmentation to obtain the correlation score between the first content block and each text block.

[0089] Among them, this step is to obtain the first content block and the text block word segmentation in the document library, and then calculate the correlation between each word in the first content block and each word in the text block in the document library, and obtain the similarity of the text block corresponding to each word in the first content block, and then perform weighted summation on the similarity of the text block corresponding to each word in the first content block to obtain the correlation score between the first content block and the document.

[0090] Assume that there is the following first content block and several text blocks in the document library:

[0091] The first content block: "Benefits of a healthy diet". Text blocks in the document library: Text block 1: "A balanced diet is good for health", Text block 2: "The importance of exercise and a healthy diet", Text block 3: "A healthy lifestyle includes diet and exercise", Text block 4: "Benefits and methods of a healthy diet".

[0092] First, the first content block and each text block in the document library are segmented: the segmentation result of the first content block: ["health", "diet", "benefits"].

[0093] Word segmentation results of the document library text block: Text block 1: ["balanced", "diet", "healthy", "beneficial"],

[0094] Text block 2: ["exercise", "and", "health", "diet", "importance"],

[0095] Text block 3: ["health", "lifestyle", "includes", "diet", "and", "exercise"],

[0096] Text block 4: [“health”, “diet”, “benefits”, “and”, “methods”].

[0097] Next, the similarity between each word in the first content block and the word set of each text block in the document library is calculated. Here, Jaccard similarity is used as a calculation method, which measures the ratio of the size of the intersection of two sets to the size of the union.

[0098] Similarity calculation:

[0099] For text block 1:

[0100] The Jaccard similarity between “health” and [“balanced”, “diet”, “healthy”, “beneficial”] is 1 / 4 (because “healthy” appears once, and there are 4 different words in total);

[0101] The Jaccard similarity between “diet” and [“balanced”, “diet”, “healthy”, “beneficial”] is 1 / 4;

[0102] The Jaccard similarity between “benefit” and [“balanced”, “diet”, “healthy”, “beneficial”] is 1 / 4 (because “beneficial” and “benefit” have similar meanings, the similarity can be considered to be 1);

[0103] Overall similarity: (1 / 4+1 / 4+1 / 4) / 3=3 / 12=1 / 4.

[0104] For text block 2:

[0105] The Jaccard similarity between “health” and [“exercise”, “and”, “health”, “diet”, “importance”] is 1 / 5;

[0106] The Jaccard similarity between “diet” and [“exercise”, “and”, “health”, “diet”, “importance”] is 1 / 5;

[0107] The Jaccard similarity between “benefit” and [“exercise”, “and”, “health”, “diet”, “importance”] is 0 (because “benefit” does not appear directly in text block 2);

[0108] Overall similarity: (1 / 5+1 / 5+0) / 3=2 / 15.

[0109] For text block 3:

[0110] The Jaccard similarity between “health” and [“health”, “lifestyle”, “includes”, “diet”, “and”, “exercise”] is 1 / 6;

[0111] The Jaccard similarity between “diet” and [“health”, “lifestyle”, “includes”, “diet”, “and”, “exercise”] is 1 / 6;

[0112] The Jaccard similarity between “benefit” and [“health”, “lifestyle”, “includes”, “diet”, “and”, “exercise”] is 0 (because “benefit” does not appear directly in text block 3);

[0113] Overall similarity: (1 / 6+1 / 6+0) / 3=2 / 18=1 / 9.

[0114] For text block 4:

[0115] The Jaccard similarity between “health” and [“health”, “diet”, “benefits”, “and”, “methods”] is 1 / 5;

[0116] The Jaccard similarity between “diet” and [“health”, “diet”, “benefits”, “and”, “methods”] is 1 / 5;

[0117] The Jaccard similarity between “benefit” and [“health”, “diet”, “benefit”, “and”, “method”] is 1 / 5;

[0118] Overall similarity: (1 / 5+1 / 5+1 / 5) / 3=1 / 5.

[0119] Finally, the similarity scores between the first content block and each text block in the document library are obtained, and based on these scores, it is possible to determine which text blocks are most relevant to the first content block.

[0120] Step 503: vectorize the first content block, calculate the similarity between the vectorized first content block and the text block vectors in the vector library, and obtain the correlation score between the first content block and each text block vector.

[0121] Specifically, after the first content block is vectorized by the embedding model, the similarity score between the first content block and the text block vector is obtained by calculating the inner product of the vectorized first content block and the text block vector in the vector library. The vector inner product can measure the similarity between two vectors. The larger the value, the more similar the two vectors are. In text processing, it usually means that the relevance of the text content is higher.

[0122] According to the first content block in the above embodiment and some text blocks in the document library, they have been converted into vector form by TF-IDF method.

[0123] The first content block: "Benefits of a healthy diet". Text blocks in the document library: Text block 1: "A balanced diet is good for health"; Text block 2: "The importance of exercise and a healthy diet"; Text block 3: "A healthy lifestyle includes diet and exercise"; Text block 4: "Benefits and methods of a healthy diet".

[0124] Assume that after vectorization, the following vector representation is obtained:

[0125] First content block vector: Q=[0.5,0.3,0.2].

[0126] Vectors of text blocks in the document library:

[0127] Text block 1 vector: D1=[0.4,0.6,0.1];

[0128] Text block 2 vector: D2=[0.1,0.2,0.8];

[0129] Text block 3 vector: D3=[0.6,0.3,0.2];

[0130] Text block 4 vector: D4=[0.4,0.6,0.7].

[0131] Compute the inner product of vectors:

[0132] The vector inner product calculation formula is:

[0133] Inner product = Q1*D11+Q2*D12+Q3*D13.

[0134] Specifically, the calculation results are: the inner product of the first content block and text block 1: 0.5*0.4+0.3*0.6+0.2*0.1=0.2+0.18+0.02=0.4; the inner product of the first content block and text block 2: 0.5*0.1+0.3*0.2+0.2*0.8=0.05+0.06+0.16=0.27; the inner product of the first content block and text block 3: 0.5*0.6+0.3*0.3+0.2*0.2=0.3+0.09+0.04=0.43; the inner product of the first content block and text block 4: 0.5*0.4+0.3*0.6+0.2*0.2=0.2+0.18+0.04=0.44.

[0135] Output relevance score:

[0136] Based on the vector inner product calculation results, the following relevance scores can be obtained:

[0137] The relevance score of text block 1 to the first content block is 0.4;

[0138] The relevance score of text block 2 to the first content block is 0.27;

[0139] The relevance score of text block 3 and the first content block of the query is 0.43;

[0140] The relevance score of text block 4 to the first content block of the search is 0.44;

[0141] Finally, the similarity scores between the first content block and the text block vectors in the vector library are obtained. And based on these scores, it can be determined which text block is most relevant to the first content block.

[0142] Step 504: Perform a weighted combination of the correlation scores between the first content block and each text block and the correlation scores between the first content block and each text block vector to obtain the scores of the first content block and each text block; wherein, the scores are sorted from high to low, and the text blocks with the top N scores are selected as the text blocks of the first content block.

[0143] Specifically, this step is to weight the relevance score between the first content block and the text block in the text library and the similarity score generated by the inner product calculation between the vectorized first content block and the text block vector in the vector library to obtain the score of the first content text block and each block, sort the scores of each first content text block and its corresponding blocks from high to low, and select the top N text blocks as the text blocks of the first content block. According to the above embodiment, the document library weight is set to 0.4 and the vector library weight is set to 0.6. To weight the combination of the two relevance scores, the following formula can be used:

[0144] Final relevance score = document library relevance score * document library weight + vector library relevance score * vector library weight.

[0145] The weights given are 0.4 for the document library and 0.6 for the vector library. These weights will be used to calculate the final relevance score for each text block. The final relevance score for each text block is calculated.

[0146] Text Block 1:

[0147] Document library relevance score: 0.25;

[0148] Vector library relevance score: 0.4;

[0149] Final correlation score: 0.25*0.4+0.4*0.6=0.1+0.24=0.34.

[0150] Text Block 2:

[0151] Document library relevance score: 0.2;

[0152] Vector library relevance score: 0.27;

[0153] Final correlation score: 0.2*0.4+0.27*0.6=0.08+0.162=0.242.

[0154] Text Block 3:

[0155] Document library relevance score: 0.111;

[0156] Vector library relevance score: 0.43;

[0157] Final correlation score: 0.111*0.4+0.43*0.6=0.0444+0.258=0.3024.

[0158] Text Block 4:

[0159] Document library relevance score: 0.2;

[0160] Vector library relevance score: 0.44;

[0161] Final correlation score: 0.2*0.4+0.44*0.6=0.08+0.264=0.344.

[0162] Final relevance score results:

[0163] Text block 1: 0.34;

[0164] Text block 2: 0.242;

[0165] Text block 3: 0.3024;

[0166] Text block 4: 0.344;

[0167] The final text block similarity after weighted combination is sorted from high to low, and the order is: text block 4, text block 1, text block 3, text block 2. If the top three text blocks are taken as the text blocks of the first content block in this embodiment, text block 4, text block 1, and text block 3 are taken as the text blocks of the first content block.

[0168] In this embodiment, the similarity score is obtained by calculating the similarity between the first content block and the text block in the document library, and the similarity score is obtained by calculating the similarity between the first content block and the text block in the vector library. Different similarity calculation methods may perform differently under different types of data. If the similarity calculated by the document library alone or the similarity calculated by the vector library alone may result in a large error in the final similarity result. Therefore, it is necessary to weight the two methods to obtain a more comprehensive and accurate similarity score.

[0169] In one embodiment, determining a verification result of the relevance of the second content to the input query includes:

[0170] The pre-trained StructBert model is used to determine the semantic relevance between the input query and the second content to obtain a relevance verification result; wherein the relevance verification result includes implication, contradiction or neutral relationship. If the relevance verification result is an implication relationship, the relevance verification satisfies the set conditions.

[0171] Specifically, in the fact verification process, if only similarity is used to measure, it is impossible to distinguish whether the second content is relevant to the input query. There may be a high similarity, but the second content and the input query are semantically opposite or irrelevant. For example, the question: "What is your favorite sport?" The answer: "My favorite color is red." At this time, most of the text between the second content and the input query is relevant, but the second content is meaningless to the input query. Therefore, this method uses the pre-trained StructBert model to determine the semantic relevance between the input query and the generated second content. The two contain three types of relationships: implication, contradiction or neutrality. If the second content and the input query are in an implication relationship, the relevance verification relationship is satisfied; if the relationship between the second content and the input query is neutral or contradictory, the relevance verification relationship cannot be satisfied. Only when the two satisfy the implication relationship can they be directly output as the answer to the question.

[0172] In this embodiment, the model accurately reduces a portion of the answers that are only textually similar to the original question, effectively avoiding the situation where the generated answers are wrong or irrelevant to the question asked.

[0173] In one embodiment, Figure 6 As shown, the method for enhancing question answering includes:

[0174] Step 601: If the relevance verification result does not satisfy the preset relationship, the large language model enhances the input query based on the input query and the query example to obtain a second enhanced query;

[0175] Specifically, if the generated second content does not satisfy the relevance verification relationship, it cannot be directly output as the answer to the question, and then after a mindless repeated search based on the initial input query, a reliable and accurate answer cannot be obtained. In this embodiment, when the generated second content does not satisfy the relevance verification relationship, the input query is enhanced to obtain a second enhanced query. For example, a strongly related input query, i.e., a second enhanced query, can be generated based on the input query and the query example using a large language model. For example, the input query can be expanded, wherein the expansion can make the input query more complete and detailed, so as to better understand the background, scope and details of the query, and the query can be enriched by adding more information, examples, context or relevant knowledge. For example, the second enhanced query generated by expanding the input query can be "What kind of fruit is an apple?" The expanded input query helps to provide a more comprehensive perspective, so that the generated second content can better grasp the core of the problem, thereby obtaining a deeper and more comprehensive answer.

[0176] Step 602: Regenerate the second content according to the second enhanced query using the large language model.

[0177] Furthermore, the second enhanced query obtained through the large language model and the query example uses the large language model to perform a series of retrieval steps based on the strongly related input query to regenerate the second content. For example, if the original query is expanded, if the original input query is "What is an apple?", the generated answer may only be "Apple is a common fruit." The answer generated for the enhanced second enhanced query "What kind of fruit is an apple?" will introduce the specific results such as the nutrition, growth environment, shelf life, and benefits of apples to the human body.

[0178] In this embodiment, sometimes the generated answer does not meet the relevance verification, that is, the answer is irrelevant to the question semantics or is opposite. At this time, even if the same input query is used again to generate an answer through the large language model, it still cannot meet the relevance verification. Therefore, the large language model is used to enhance the input query based on the input query and the query example to generate a reinforced query. Enhancing the input query can improve the understanding ability of the question-answering method based on the large language model for different types of input queries, thereby improving the relevance and accuracy of the generated answers.

[0179] In one embodiment, the large language model enhances the input query based on the input query and the query example to obtain a second enhanced query for the enhanced question answering method, including:

[0180] The large language model modifies the input query based on the input query and the query example, and uses the modified input query as the second enhanced query.

[0181] Specifically, when generating the second content, if the relevance verification relationship is not met, it cannot be directly output as the answer to the question. At this time, after mindlessly repeating the search for the initial input query, it is still impossible to get an answer that satisfies the semantic relevance. In this embodiment, a large language model can be used to modify the input query based on the input query and the query example. Among them, the input query is modified to correct the inaccurate, unclear and irrelevant content in the query. Ensure the accuracy and answerability of the query. Use the modified input query as the second query. For example, the input query is "How to make tomatoes and eggs?" The second content obtained may be a vague tutorial on how to make tomatoes and eggs, and it cannot give the user a clear answer. After modifying the input inquiry, the second enhanced inquiry is obtained, "How to make a dish involving tomatoes and eggs?" The second content obtained may be "The ingredients are tomatoes and eggs, so you can make a dish of scrambled eggs with tomatoes. First, beat the eggs and add an appropriate amount of salt. Heat the oil in a pan, pour in the egg liquid, stir-fry quickly until half-cooked and then serve. Next, sauté the minced garlic, add the diced tomatoes, stir-fry until the juice comes out, then pour the scrambled eggs back into the pan, stir-fry evenly, and finally add salt and sugar to taste." The modified input inquiry is clearer and more accurate, which helps to avoid misunderstandings and confusion, and ensures that the correct second content can be provided for the core of the inquiry.

[0182] If the number of times the second content is regenerated according to the second enhanced query using the large language model is greater than a number threshold, a fallback response is output.

[0183] It should be noted that the purpose of the user's question is to obtain an answer. If the regenerated second content does not meet the relevance verification, the input query will be repeatedly modified and new second content will be generated in a continuous cycle. In the end, no definite second content will be generated, that is, the final answer will not be output. Therefore, it is necessary to set a threshold limit on the number of times the second content is regenerated. If the number of times the second content is repeatedly generated reaches the threshold, the second content will not be generated in a cycle again. At this time, a fallback answer is output as the final answer. Among them, the fallback answer can be a format statement, and its content can be pre-set, such as "This question is beyond the scope of ability and cannot be answered", and this answer can be output as a fallback answer.

[0184] In this embodiment, the input query is modified to regenerate the second content. The specific process is as follows: Figure 7 As shown, the following steps are included:

[0185] Step 701: The correlation verification result does not meet the preset condition;

[0186] Step 702: The large language model modifies the input query based on the input query and the query example;

[0187] Step 703: Determine whether the number of times the second content is regenerated exceeds a number threshold;

[0188] Step 704: If yes, output a fallback answer;

[0189] Step 705: If not, use the modified input query as the second enhanced query;

[0190] Step 706: Regenerate the second content according to the second enhanced query using the large language model;

[0191] Step 707: Determine whether the second content and the input query satisfy the relevance verification; if so, execute step 708, if not, return to step 702;

[0192] Step 708: The generated second content is used as the output answer.

[0193] In this embodiment, the re-modification of the input query can improve the relevance between the input query and the regenerated second content, that is, the generated second content can be more relevant to the semantics of the input query, greatly avoiding the situation where the question and the answer are irrelevant. At the same time, in the process of re-modifying the input query to regenerate the second content for multiple times, the relevance between the modified input query and the regenerated second content will also be continuously improved. It should be noted that the number of times the input query is modified to regenerate the second content using the large language model should be limited by a number threshold to prevent the cycle of modifying the query and generating new answers from being constantly in a state where the final answer cannot be generated.

[0194] In one embodiment, the input query is enhanced according to the input query and the knowledge database to obtain a first enhanced query, and the first content is generated according to the first enhanced query using the large language model;

[0195] Among them, according to the input query and the knowledge database, the input query is enhanced to obtain a first enhanced query, including: using a hybrid retrieval method for the input query, weighted combination of the full-text retrieval relevance score and the vector retrieval relevance score to obtain a hybrid score of the input query and the text block, and using M text blocks with the highest hybrid score as the first enhanced query.

[0196] In one embodiment, the full-text search relevance score is calculated as follows:

[0197]

[0198]

[0199] The BM25 algorithm is used for calculation, n is the number of terms in the query, and q is i is the i-th term in the query, IDF(q i) is the inverse block frequency, N is the total number of blocks, n(q i ) is the term containing the term q i The number of text blocks, f(q i ,D) is the term q i The number of times it appears in text block D, len(D) is the length of text block D, avg_len is the average number and height of all text blocks, k i and b are adjustment parameters. It should be noted that the inverse text block frequency is used to measure the importance of a word in a text block, and the IDF value will decrease as the frequency of a word in a text block increases.

[0200] In this embodiment, the BM25 algorithm is used to balance the influence of long text blocks and short text blocks in the relevance score by considering the word frequency and the inverse text block frequency, and combining the text block length information, which helps to avoid the long text block from getting an inappropriately high score due to the high word frequency, or the short text block from being underestimated due to the low word frequency. In addition, the parameters ki and b in the BM25 algorithm are both adjustable parameters, which means that the algorithm is allowed to be optimized according to different data sets and application scenarios, where ki controls the importance of word frequency and b controls the degree of normalization of the text block. Furthermore, the BM25 algorithm can be combined with other retrieval methods such as vector retrieval to achieve hybrid retrieval to further improve search performance. For example, by combining BM25 and vector retrieval, the literal similarity of BM25 and the semantic similarity of vector retrieval can be used simultaneously to achieve more accurate retrieval results. Furthermore, the BM25 algorithm can adapt to different data distributions and text block features, which enables it to provide stable performance in a variety of document libraries.

[0201] In one embodiment, Figure 8 As shown in FIG. 1 , the enhanced question answering method based on the large language model specifically includes the following steps:

[0202] Step 801: Obtain input query;

[0203] Step 802: obtaining a first enhanced query by searching a knowledge database;

[0204] Step 803: Generate first content from the first enhanced query using the large language model;

[0205] Step 804: using the large language model to split the first content into a plurality of first content blocks, wherein the number of the first content blocks is less than a quantity threshold;

[0206] Step 805: searching the knowledge database and performing similarity calculation to determine the N text blocks corresponding to the first content block;

[0207] Step 806: fact verification;

[0208] Step 807: Generate second content through a large language model using the first content blocks verified by facts and the corresponding text blocks;

[0209] Step 808: semantic relevance verification;

[0210] Step 809: if the relevance verification is satisfied, the second content is output as the answer;

[0211] Step 810: if the relevance verification is not satisfied, a second enhanced query is generated by rewriting the input query;

[0212] Step 811: determine whether the number of times the second content is regenerated is greater than a number threshold; if it is greater than the number threshold, execute step 812; if it is less than the number threshold, return to step 802;

[0213] Step 812: Output a fallback response.

[0214] In this embodiment, in the process of generating answers based on the large language model, that is, the process from inputting the query to the final inputting of the content is continuously improved and enhanced to improve the accuracy and reliability of the generated text facts.

[0215] Based on the same inventive concept, the present invention also provides an electronic device such as Fig. 9 As shown, a computer program is stored on the electronic device, and the steps of implementing the method of replying when the computer program is executed include:

[0216] Memory for storing computer programs;

[0217] The processor is used for a computer program, and when executing the computer program stored in the memory, it can implement the above-mentioned method for generating fact verification based on retrieval enhancement.

[0218] Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium. The non-volatile storage medium stores an operating system and a computer program, and the memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The I / O interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for verifying the fact generated based on retrieval enhancement is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0219] The above is a detailed introduction to a method for generating fact verification based on retrieval enhancement and an electronic device provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the devices, equipment and readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0220] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An enhanced question answering method based on a large language model, characterized in that: The method comprises: Get input query; Generating first content according to the input query and the knowledge database; the first content is a primary answer generated by searching the knowledge database through the input query; Splitting the first content into a plurality of first content blocks; for each of the first content blocks, searching the knowledge database based on the first content block to obtain a text block corresponding to the first content block; wherein each of the first content blocks corresponds to N text blocks, and splitting the first content into a plurality of first content blocks comprises: splitting the first content into a plurality of first content blocks using a large language model, wherein the first content blocks are independent sentences; Determine a first content block that satisfies fact verification, and generate second content through the large language model based on the first content block that satisfies fact verification and a corresponding text block; Determine a correlation verification result between the second content and the input query, and if the correlation verification result satisfies a preset relationship, use the second content as an answer to the input query; Wherein, determining the first content block that satisfies fact verification includes: For each of the first content blocks, the first content block and the corresponding N text blocks are combined into a group; For each combination, the large language model is allowed to determine whether the search content in the combination supports the verification content by means of prompt words; wherein the search content is the N text blocks in the combination, and the verification content is the first content block in the combination; For each combination, if the search content supports the verification content, it is determined that the first content block in the combination satisfies the fact verification.

2. The enhanced question answering method based on a large language model according to claim 1, characterized in that: Splitting the first content into a plurality of first content blocks includes: Based on the first content, determining a splitting example; The split example is input as a prompt word into the large language model, and the large language model splits the first content into multiple first content blocks according to the prompt word; wherein the first content blocks are independent sentences, and the number of the first content blocks is less than a quantity threshold.

3. The enhanced question answering method based on a large language model according to claim 1, characterized in that: The knowledge database includes a document library and a vector library. The document library includes a plurality of text blocks. The text blocks are vectorized by embedding models to obtain a vector database. For each of the first content blocks, searching the knowledge database based on the first content block to obtain a text block corresponding to the first content block includes: For each of the first content blocks, similarity calculations are performed on the first content block with the document library and the vector library respectively, and N text blocks are determined based on the results of the similarity calculations, and the N text blocks are used as text blocks of the first content block.

4. The enhanced question answering method based on a large language model according to claim 3, characterized in that: For each of the first content blocks, respectively, similarity calculation is performed on the first content block with the document library and the vector library, and N text blocks are determined based on the results of the similarity calculation, and the N text blocks are used as text blocks of the first content block, including: Segmenting the first content block and the text block in the document library; Calculate the similarity between the first content block after word segmentation and the text block after word segmentation to obtain a relevance score between the first content block and each text block; Vectorizing the first content block, performing similarity calculation between the vectorized first content block and the text block vectors in the vector library, and obtaining a correlation score between the first content block and each text block vector; The relevance scores between the first content block and each text block and the relevance scores between the first content block and each text block vector are weightedly combined to obtain the scores of the first content block and each text block; wherein the scores are sorted from large to small, and the text blocks with the top N scores are selected as the text blocks of the first content block.

5. The enhanced question answering method based on a large language model according to claim 1, characterized in that: The method further comprises: Determining a correlation verification result between the second content and the input query includes: A pre-trained StructBert model is used to determine the semantic relevance between the input query and the second content to obtain a relevance verification result; wherein the relevance verification result includes an implication, contradiction or neutral relationship. If the relevance verification result is an implication relationship, then the relevance verification meets the set conditions.

6. The enhanced question answering method based on a large language model according to claim 1, characterized in that: The method further comprises: If the relevance verification result does not satisfy the preset relationship, the large language model enhances the input query based on the input query and the query example to obtain a second enhanced query; The large language model is used to regenerate second content according to the second enhanced query.

7. The enhanced question answering method based on a large language model according to claim 6, characterized in that: The large language model enhances the input query based on the input query and the query example to obtain a second enhanced query, including: the large language model modifies the input query based on the input query and the query example, and uses the modified input query as the second enhanced query; The method also includes outputting a fallback response if the number of times the second content is regenerated according to the second enhanced query using the large language model is greater than a number threshold.

8. The enhanced question answering method based on a large language model according to any one of claims 1 to 7, characterized in that: The knowledge database includes a document library and a vector library. The document library includes a plurality of text blocks. The text blocks are vectorized by embedding models to obtain a vector database. Generating first content according to the input query and the knowledge database includes: enhancing the input query according to the input query and the knowledge database to obtain a first enhanced query, and generating first content according to the first enhanced query using a large language model; Among them, according to the input query and the knowledge database, the input query is enhanced to obtain a first enhanced query, including: using a hybrid retrieval method for the input query, weighted combination of the full-text retrieval relevance score and the vector retrieval relevance score to obtain a hybrid score of the input query and the text block, and using M text blocks with top hybrid scores as the first enhanced query.

9. The enhanced question answering method based on a large language model according to claim 8, characterized in that: The calculation method of the full-text search relevance score is as follows: ， ， Among them, the BM25 algorithm is used for calculation. is the number of terms in the query, is the i-th term in the query, is the inverse block frequency, N is the total number of blocks, Contains terms The number of text blocks, It is a term The number of occurrences in text block D, is the length of the text block D, is the average of all text blocks, and b are tuning parameters.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the enhanced question answering method based on a large language model as described in any one of claims 1 to 9 are implemented.

Images