Large language model adaptive retrieval enhancement method, system and equipment

Abstract

The invention discloses a large language model adaptive retrieval enhancement method, system and device, and belongs to the technical field of artificial intelligence, the method comprises the following steps: splitting a user question into n sub-query statements, forming n vector indexes, searching in a knowledge base, and forming n retrieval results; inputting the n sub-query statements into a first large language model to obtain a first answer result; performing relevancy calculation on the n retrieval results and the first answer result one by one, and performing sorting according to calculation results; inputting the user question into a second large language model to obtain a second answer result; the ranked first K retrieval results and the second answer result are fused to generate a final output answer; according to the method, the answer accuracy of the method for combining the large language model and the retrieval knowledge base can be effectively improved, and the answer quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a large language model adaptive retrieval enhancement method, system and device. Background Art

[0002] Currently, large language models have the problem of generating hallucinations, that is, generating false information. Retrieval technology can provide more accurate answers by referencing information in the knowledge base. In general, if Figure 1 As shown in the figure, according to the user's question, the search indiscriminately accesses the vector library to obtain the context, and splices the search results with the output results of the large language model to form the final answer. This conventional method may introduce context irrelevant to the topic, which in turn leads to low-quality text generation content. The large language model does not perform adaptive training on the knowledge base to ensure that the generated results are consistent with the context semantics. At the same time, the search context may introduce conflicting views. Ultimately, the answer content is conflicting and the quality is low.

[0003] Therefore, we consider sorting the search results and integrating them with the output of the large language model to avoid conflicting content, low-quality content, and context-irrelevant content in the answers obtained, effectively improving the accuracy of the answers combined with the large language model and the retrieval knowledge base, and improving the quality of the answers. Summary of the invention

[0004] In view of this, in order to solve the problems existing in the prior art, the present invention provides a large language model adaptive retrieval enhancement method, system and device, which are convenient for effectively improving the answer accuracy of the method combining a large language model with a retrieval knowledge base and improving the answer quality.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a large language model adaptive retrieval enhancement method, the method comprising the following steps:

[0007] S1. Split the user's question into n sub-query statements to form n vector indexes, search in the knowledge base, and form n retrieval results;

[0008] S2, inputting the n sub-query statements into the first language model to obtain a first answer result;

[0009] S3, calculating the relevance of the n search results and the first answer result one by one, and sorting them according to the calculation results;

[0010] S4, inputting the user's question into the second largest language model to obtain a second answer result;

[0011] S5. Merge the first K sorted search results with the second answer result to generate a final output answer.

[0012] In an optional implementation, in step S2 and step S4, the first largest language model and the second largest language model are neural network models with a parameter magnitude exceeding 10 billion, and have the capabilities of text understanding, text generation, and question-answering dialogue.

[0013] In an optional implementation, in step S3, the relevance is calculated using a semantic similarity formula:

[0014]

[0015] Among them, cos(θ) represents the relevance, A and B represent the high-dimensional semantic vectors of the search result and the first answer result sentence respectively, i represents the i-th dimension, and n represents the total number of dimensions of the two high-dimensional semantic vectors A or B.

[0016] In an optional implementation, in step S3, the correlation is calculated using a neural network model, and the method includes:

[0017] First, the dot product operation is used to construct a matching matrix between the search results and the first answer result sentence; then the features in the matrix are extracted based on two-dimensional convolution and pooling operations, and finally the correlation between the two is calculated based on a fully connected network.

[0018] In an optional embodiment, in step S5, the first K search results sorted in step S3 are merged with the second answer result in step S4 by setting sentence concatenation or by recurrent neural network encoding and decoding.

[0019] In an optional embodiment, the loss function of the training sample output sequence of the recurrent neural network is:

[0020]

[0021] Among them, P represents probability, x1,…,x T is the input sequence, y1,…,y T' is the output sequence of the training sample; C is the output result of the recurrent neural network encoder. For each step t' in the decoding stage, the decoder output is y t' .

[0022] In a second aspect, the present invention further provides a large language model adaptive retrieval enhancement system, which is applied to the above-mentioned large language model adaptive retrieval enhancement method to implement large language model adaptive retrieval enhancement and improve the accuracy and quality of answers. The system includes:

[0023] The retrieval module is used to split the user's question into n sub-query statements, form n vector indexes, search in the knowledge base, and form n retrieval results;

[0024] An answer module, used for inputting the n sub-query statements into the first language model to obtain a first answer result, and also used for inputting the user's question into the second language model to obtain a second answer result;

[0025] A calculation module, used for calculating the relevance between the n search results and the first answer result one by one, and sorting them according to the calculation results;

[0026] The fusion module is used to fuse the first K sorted search results with the second answer result to generate a final output answer.

[0027] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned large language model adaptive retrieval enhancement method.

[0028] In a fourth aspect, the present invention also provides a storage medium having stored thereon one or more programs readable by a computing device, wherein the one or more programs include instructions which, when executed by the computing device, enable the computing device to perform a large language model adaptive retrieval enhancement method as described above.

[0029] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0030] The present invention provides a large language model adaptive retrieval enhancement method, system and device. The retrieval technology and the large language model are used in combination. The retrieval provides a textbook for the large model, which can query specific information, integrate the output of the large language model, and jointly generate the final answer. The present invention uses the large language model to dynamically determine the retrieval timing, sort the retrieval results, and integrate them with the output of the large language model, so that the answers obtained can avoid the appearance of conflicting content, low-quality content, and context-irrelevant content, effectively improving the answer accuracy of the method combining the large language model with the retrieval knowledge base, and improving the answer quality.

[0031] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0032] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0035] Figure 1 A schematic diagram of a conventional question-and-answer method provided in an embodiment of the present invention.

[0036] Figure 2 A schematic flow chart of a large language model adaptive retrieval enhancement method provided in an embodiment of the present invention.

[0037] Figure 3 A schematic diagram of a calculation process for performing relevance evaluation using a neural network model provided in an embodiment of the present invention.

[0038] Figure 4 A schematic diagram of the process of completing output of neural network encoding and decoding provided by an embodiment of the present invention.

[0039] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0041] In the description of the present invention, it should be noted that: in some processes described in the specification and drawings of this application, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear in this document or may be performed in parallel. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0042] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Embodiment 1:

[0044] See also Figure 2 As shown, an embodiment of the present invention provides a large language model adaptive retrieval enhancement method, the method comprising the following steps:

[0045] S1. Split the user's question into n sub-query statements to form n vector indexes, search in the knowledge base, and form n retrieval results;

[0046] S2. Input the n sub-query statements into the first language model to obtain the first answer result;

[0047] S3, calculating the relevance of the n search results and the first answer result in step S2 one by one, and sorting them according to the calculation results;

[0048] S4, inputting the user's question into the second largest language model to obtain a second answer result;

[0049] S5. Merge the first K sorted search results with the second answer result to generate the final output answer.

[0050] Combine the following Figure 2-Figure 4 As shown, the specific implementation of the method of the present invention and the working principle involved are described in detail:

[0051] Currently, large language models have the problem of generating hallucinations, that is, generating false information. Retrieval technology can provide more accurate answers by referencing information in the knowledge base. The two are used together, and retrieval provides a textbook for large models, which can query specific information and integrate the output of large language models to jointly generate the final answer.

[0052] The present invention utilizes a large language model to dynamically determine the timing of retrieval, and sorts and merges the retrieval results, which are then merged with the output of the large model. The obtained answers avoid the appearance of conflicting content, low-quality content, and context-irrelevant content, effectively improving the answer accuracy of the method combining the large model with the retrieval knowledge base, and improving the answer quality.

[0053] In the embodiment of the present invention, see Figure 2As shown: Among them, the input is a user question, generally a text question sentence. The first large language model and the second large language model are the same large language model or different large language models, both of which refer to neural network models with a parameter magnitude of more than 10 billion (a large model is a model with a parameter magnitude of more than billions, such as GPT3.5, ChatGLM-6B and other models), which have the ability to understand text, generate text, and conduct question-answering dialogues. A knowledge base refers to a database of text categories in a specific field or industry. The large language model and the knowledge base are independent modules and do not have a direct relationship. Through some operations, the answers to the user's questions are returned in a comprehensive manner, also in text form. The specific implementation details are as follows:

[0054] Step 1: Split the user's question into n sub-query statements to form n vector indexes, search in the knowledge base, and form retrieval results;

[0055] Step 2: Input the above n sub-query statements into the large language model to obtain the answer result;

[0056] Step 3: Calculate the relevance of the n search results in step 1 and the answer results in step 2 one by one, and sort them. The higher the relevance, the higher the ranking. The relevance can be calculated by the semantic similarity formula or evaluated by the neural network model;

[0057] The correlation calculation method is as follows:

[0058] In one embodiment, the representation vectors of the two answer result (search result and answer result) sentences are first extracted by a deep neural network model, and then the semantic similarity is calculated by the following similarity calculation formula:

[0059]

[0060] Among them, cos(θ) represents semantic similarity, A and B represent high-dimensional semantic vectors of two answer result sentences, i represents the i-th dimension, and n represents the total number of dimensions of the two high-dimensional vectors A or B.

[0061] In another embodiment, see Figure 3 As shown in the figure, the calculation process of similarity evaluation through the neural network model is as follows:

[0062] First, the dot product operation is used to construct a matching matrix between the two sentences. For two sentences of length M and N, the similarity matrix between their words is M*N, and each element in it is the similarity between words.

[0063] Then, the features in the matrix are extracted based on two-dimensional convolution and pooling operations, and finally the similarity between sentences is calculated based on the fully connected network.

[0064] Step 4: Input the user's question into the large language model and get the result t;

[0065] Step 5: sort the first K search results r1, r2, ..., r K , and the result t in step 4 are fused to generate the final output answer. In this step, the fusion can be completed by setting sentence concatenation or by encoding and decoding the neural network model.

[0066] Among them, the statement concatenation form is as follows:

[0067] {“recommended content” t, “knowledge base” 1.r1, 2.r2, …, kr K}.

[0068] The process of neural network encoding and decoding to complete the output is as follows:

[0069] For t, r1, r2, ..., r K Encode each sentence and get the sentence vector x1, x2, x3, ..., x K+1 ;

[0070] The recurrent neural network encoder is used to model the relationship between words in a sentence in a high-dimensional space to obtain a global word vector representation C; finally, at the decoder end, according to the maximum likelihood estimation criterion, the feature vectors of the input items are gradually used to generate output items to obtain the final output result. Specifically, the conditional probability of the output sequence based on the input sequence is maximized according to the maximum likelihood estimation:

[0071] The input sequence is x1,…,x T ;

[0072]

[0073] Among them, P represents the probability, C is the output result of the recurrent neural network encoder, which is y1,…,y T' is the output sequence of the training sample. For each step t' in the decoding stage, the decoder output is y t' .

[0074] And get the loss function of the output sequence:

[0075]

[0076] In model training, the parameters of the neural network are obtained by minimizing the above loss function. Finally, the encoding-decoding process is performed to obtain the output sentence.

[0077] From the description of the above embodiments, those skilled in the art can know that: the present invention provides a large language model adaptive retrieval enhancement method, which utilizes a large language model to dynamically determine the retrieval timing, and sorts and merges the retrieval results, and merges them with the output of the large model. The obtained answer avoids the appearance of conflicting content, low-quality content, and context-irrelevant content, effectively improving the answer accuracy of the method combining the large model with the retrieval knowledge base, and improving the answer quality.

[0078] Embodiment 2:

[0079] The present invention also provides a large language model adaptive retrieval enhancement system, which is applied to a large language model adaptive retrieval enhancement method described in the above embodiment 1, effectively improving the answer accuracy of the method combining the large model with the retrieval knowledge base and improving the answer quality. The system includes:

[0080] The retrieval module is used to split the user's question into n sub-query statements, form n vector indexes, search in the knowledge base, and form n retrieval results;

[0081] An answer module, used to input n sub-query statements into the first language model to obtain a first answer result, and also used to input the user's question into the second language model to obtain a second answer result;

[0082] A calculation module, used to calculate the relevance of the n search results and the first answer result one by one, and sort them according to the calculation results;

[0083] The fusion module is used to fuse the top K sorted retrieval results with the second answer result to generate the final output answer.

[0084] The large language model adaptive retrieval enhancement system provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment, which will not be repeated here.

[0085] Example 3

[0086] Reference Figure 5 As shown, an embodiment of the present invention further provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, and the processor executes the computer program to implement a large language model adaptive retrieval enhancement method in the above-mentioned embodiment 1.

[0087] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 11, and calling data stored in the memory 11.

[0088] Embodiment 4:

[0089] An embodiment of the present invention also provides a storage medium on which one or more programs readable by a computing device are stored. The one or more programs include instructions. When the instructions are executed by the computing device, the computing device executes a large language model adaptive retrieval enhancement method in Example 1.

[0090] Examples of computer-readable storage media include: read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, non-volatile memory, CD-ROM, DVD-ROM, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage system, optical data storage system, hard disk, solid-state disk, and any other system configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent system, a server, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0091] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0093] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0094] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model adaptive retrieval enhancement method, characterized in that: The method comprises the following steps: S1. Split the user's question into n sub-query statements to form n vector indexes, search in the knowledge base, and form n retrieval results; S2, inputting the n sub-query statements into the first language model to obtain a first answer result; S3, calculating the relevance of the n search results and the first answer result one by one, and sorting them according to the calculation results; S4, inputting the user's question into the second largest language model to obtain a second answer result; S5. Merge the first K sorted search results with the second answer result to generate a final output answer.

2. A large language model adaptive retrieval enhancement method according to claim 1, characterized in that: In step S2 and step S4, the first largest language model and the second largest language model are neural network models with a parameter magnitude exceeding 10 billion, and have the capabilities of text understanding, text generation, and question-answering dialogue.

3. The large language model adaptive retrieval enhancement method according to claim 1, characterized in that: In step S3, the relevance is calculated using the semantic similarity formula: Among them, cos(θ) represents the relevance, A and B represent the high-dimensional semantic vectors of the search result and the first answer result sentence respectively, i represents the i-th dimension, and n represents the total number of dimensions of the two high-dimensional semantic vectors A or B.

4. The large language model adaptive retrieval enhancement method according to claim 3, characterized in that: In step S3, the correlation is calculated using a neural network model, and the method includes: First, the dot product operation is used to construct a matching matrix between the search results and the first answer result sentence; then the features in the matrix are extracted based on two-dimensional convolution and pooling operations, and finally the correlation between the two is calculated based on a fully connected network.

5. The large language model adaptive retrieval enhancement method according to claim 1, characterized in that: In step S5, the first K search results sorted in step S3 are merged with the second answer result in step S4 by setting sentence concatenation or by recurrent neural network encoding and decoding.

6. A large language model adaptive retrieval enhancement method according to claim 5, characterized in that: The loss function of the training sample output sequence of the recurrent neural network is: Among them, P represents probability, x1,…,x T is the input sequence, y1,…,y T' is the output sequence of the training sample; C is the output result of the recurrent neural network encoder. For each step t' in the decoding stage, the decoder output is y t' .

7. A large language model adaptive retrieval enhancement system, characterized in that: When applied, a large language model adaptive retrieval enhancement method as described in any one of claims 1 to 6 is executed to improve the accuracy and quality of answers. The system includes: The retrieval module is used to split the user's question into n sub-query statements, form n vector indexes, search in the knowledge base, and form n retrieval results; An answer module, used for inputting the n sub-query statements into the first language model to obtain a first answer result, and also used for inputting the user's question into the second language model to obtain a second answer result; A calculation module, used for calculating the relevance between the n search results and the first answer result one by one, and sorting them according to the calculation results; The fusion module is used to fuse the first K sorted search results with the second answer result to generate a final output answer.

8. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement a large language model adaptive retrieval enhancement method as described in any one of claims 1 to 6.

9. A storage medium having stored thereon one or more programs readable by a computing device, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform a large language model adaptive retrieval enhancement method according to any one of claims 1 to 6.

Images