Large language model reasoning system and method of self-adaptive input window in RAG scene

Abstract

The invention relates to a large language model reasoning system and method of a self-adaptive input window in an RAG scene. The system comprises a user agent part used for obtaining a query request of a user; the vector database VDB part is connected with the user agent part and the large language model part, and the vector database VDB part is used for obtaining large language model input window parameters according to the large language model part, receiving a query request of a user, querying candidate text block groups from a local knowledge base, and sending the candidate text block groups to the user agent part; filling the query request of the user and the candidate text block group into a prompt template to obtain a prompt template containing a context; and the large language model part is used for performing content reasoning on the prompt template containing the context by adopting a corresponding large language model and returning a reasoning result to the user agent part. Compared with the prior art, the method has the advantages that a large window of a large language model is fully utilized, and more local knowledge is conveyed and reasoned.

Description

Technical Field

[0001] The present invention relates to the field of model reasoning technology, and in particular to a large language model reasoning system and method for adaptive input window in a RAG scenario. Background Art

[0002] The large language model question and answer application scenario RAG (Retrieval-augmented Generation) based on the local knowledge base is a very important and typical application scenario of the large language model. It can be used in corporate customer service, public utility window consulting services, anti-fraud consulting and other occasions.

[0003] The implementation of RAG is usually based on the vector database VDB in front of the large language model question-answering interface. The user's query text Query first passes through VDB, and after a similarity query of VDB, an approximate result text block Chunk is obtained. Then, it is filled into the large model prompt template PT together with the Query as parameters to form the final large model prompt template Prompt, which is sent to the large language model LLM.

[0004] However, the current RAG implementation still has big problems in terms of accuracy in certain specific fields and types of local documents. This problem is more prominent when some topics are scattered or the same topic spans a large amount of text. This is mainly because when RAG was proposed, the input window of LLM was very small, 2K. RAG in VDB can usually only take the most similar text blocks as candidates, and cannot fully utilize the fast progress of LLM. It cannot be used flexibly in the case of large input windows. Summary of the invention

[0005] The purpose of the present invention is to provide a large language model reasoning system and method for adaptive input window in RAG scenario to improve reasoning flexibility.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A large language model inference system with adaptive input window in RAG scenario, including user agent part, vector database VDB part and large language model part.

[0008] User agent part: used to obtain the user's query request;

[0009] Vector database VDB part: connected to the user agent part and the large language model part, the vector database VDB part is used to obtain the large language model input window parameters according to the large language model part, receive the user's query request, and query the candidate text block group from the local knowledge base, and then fill the user's query request and the candidate text block group into the prompt template to obtain the prompt template containing the context;

[0010] Large language model part: used to use the corresponding large language model to perform content reasoning on the prompt template containing the context, and return the reasoning result to the user agent part.

[0011] Furthermore, the vector database VDB part includes:

[0012] Query request receiving module: used for receiving the query request of the user;

[0013] Local knowledge base module: used to acquire local knowledge and build a local knowledge base based on the local knowledge;

[0014] Window adaptation module: used for obtaining the input window parameters of the currently served large language model, and querying the candidate text block group from the local knowledge base according to the input window parameters of the large language model, the query request of the user and the size of the prompt template;

[0015] Prompt template filling module: used to fill the user's query request and candidate text block groups into the prompt template to obtain a prompt template containing context.

[0016] Furthermore, the window adaptation module includes:

[0017] Data acquisition submodule: used to obtain the metadata of the large language model in the large language model part through the API, and obtain the large language model input window parameters of the current service;

[0018] Similarity calculation submodule: used for searching for relevant text blocks in the local knowledge base according to the similarity based on the query request of the user;

[0019] Sorting submodule: used to sort the related text blocks from high to low according to similarity to form a related text block list;

[0020] Candidate text block calculation submodule: used to perform calculations based on the user's query request, large language model input window parameters and the size of the prompt template to select candidate text blocks from the relevant text block list to form a candidate text block group.

[0021] Furthermore, the calculation expression of the candidate text block calculation submodule is:

[0022] k*Chunk_size+PT_size+Query_size

[0023] = <LIW_size<=

[0024] (k+1)*Chunk_size+PT_size+Query_size

[0025] Where k is the number of candidate text blocks, Chunk_size is the size parameter of the text block, Query_size is the size parameter of the query request, LIW_size is the input window size parameter of the large language model, and PT_size is the size parameter of the prompt template.

[0026] Furthermore, the large language model part includes:

[0027] Large language model file module: used to store multiple large language models and corresponding metadata;

[0028] Reasoning facility module: used to receive the prompt template containing context sent by the vector database VDB part, use the corresponding large language model to perform content reasoning, and return the reasoning results to the user agent part.

[0029] The present invention also provides a large language model reasoning method with an adaptive input window in a RAG scenario, comprising the following steps:

[0030] Get the user's query text;

[0031] Obtaining a large language model input window parameter, and in combination with the user's query request, querying a candidate text block group from a local knowledge base, and then filling the user's query request and the candidate text block group into a prompt template to obtain a prompt template containing a context;

[0032] The corresponding large language model is used to perform content reasoning on the prompt template containing the context, and the reasoning result is returned.

[0033] Furthermore, the step of obtaining a prompt template containing the context includes:

[0034] Querying candidate text block groups from the local knowledge base according to large language model input window parameters, a user's query request, and a size of a prompt template;

[0035] The user's query request and the candidate text block group are filled into the prompt template to obtain a prompt template containing context.

[0036] Furthermore, the step of searching for candidate text block groups includes:

[0037] Based on the query request of the user, searching for relevant text blocks in the local knowledge base according to similarity;

[0038] Sort the related text blocks according to similarity from high to low to form a related text block list;

[0039] A calculation is performed based on the query request of the user, the parameters of the large language model input window and the size of the prompt template to select candidate text blocks from the relevant text block list to form a candidate text block group.

[0040] Furthermore, the calculation expression is:

[0041] k*Chunk_size+PT_size+Query_size

[0042] = <LIW_size<=

[0043] (k+1)*Chunk_size+PT_size+Query_size

[0044] Where k is the number of candidate text blocks, Chunk_size is the size parameter of the text block, Query_size is the size parameter of the query request, LIW_size is the input window size parameter of the large language model, and PT_size is the size parameter of the prompt template.

[0045] Furthermore, the step of obtaining the large language model input window parameters includes:

[0046] Get the metadata of the large language model and obtain the large language model input window parameters of the current service.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The present invention considers the input window size of the large language model in the vector database VDB part, queries the candidate text block group from the local knowledge base through its size parameter and the user's query request, and fills it into the prompt template with the user's query request to use the large language model for reasoning. By dynamically applying the input window size instead of being limited to a fixed size, the flexibility of large language model reasoning can be improved.

[0049] (2) The window adaptation module of the present invention dynamically obtains the input window parameters of the large model and uses the parameters to flexibly adapt the combination of related text blocks searched out. The combination of related Chunks is sorted according to similarity and combined with adjacent text blocks so that the total size of the candidate text block combination is closest to the current large language model input window size, so as to make full use of the large window of the model and transmit as much local knowledge-related text as possible to the large model for reasoning, thereby providing more accurate and appropriate answer content. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0052] This embodiment provides a large language model reasoning system with adaptive input window in RAG scenario. The system proposes to flexibly adapt the combination of relevant text blocks (Chunk) searched out by the vector database (VDB) according to the input window size of the large language model (LLM). This Chunk text block combination is sorted according to similarity and combined with adjacent text blocks, so that the total size of the Chunk text block combination is closest to the current LLM input window size, thereby giving the LLM the most relevant text, and thus giving more accurate and appropriate answer content. Figure 1 As shown, the system is mainly implemented by a user agent part 1, a vector database VDB part 2, and a large language model part 3.

[0053] The user agent part 1 is mainly a user agent, such as a browser or a large language model client.

[0054] The vector database VDB part 2 is placed before the large language model part 3. The text content of the local knowledge base should be stored in advance and tokenized and indexed. The vector database VDB part is used to obtain the large language model input window parameters according to the large language model part 3, receive the user's query request, and query the candidate text block group from the local knowledge base, and then fill the user's query request and the candidate text block group into the prompt template to obtain a prompt template containing context.

[0055] The large language model part 3 mainly includes reasoning facilities and corresponding one or more large language model data files, which are used to use the corresponding large language model to perform content reasoning on the prompt template (large model prompt) containing the context, and return the reasoning result to the user agent part 1.

[0056] The innovation of this technical solution is concentrated in the vector database VDB part 2, as shown in the attached Figure 1 As shown in , the vector database VDB part 2 includes a query request receiving module 21, a local knowledge base module 22, a window adaptation module 23 and a prompt template filling module 24. The query request receiving module 21 is used to receive the query request of the user; the local knowledge base module 22 is used to obtain local knowledge and build a local knowledge base based on the local knowledge; the window adaptation module 23 is used to obtain the input window parameters of the large language model of the current service, and query the candidate text block group from the local knowledge base according to the large language model input window parameters, the user's query request and the size of the prompt template; the prompt template filling module 24 is used to fill the user's query request and the candidate text block group into the prompt template to obtain a prompt template containing context.

[0057] The window adaptation module 23 includes a data acquisition submodule 231, a similarity calculation submodule 232, a sorting submodule 233 and a candidate text block calculation submodule 234. The data acquisition submodule 231 is used to obtain the metadata of the large language model in the large language model part 3 through the API to obtain the large language model input window parameters of the current service; the similarity calculation submodule 232: is used to query the relevant text blocks in the local knowledge base according to the similarity based on the query request of the user; the sorting submodule 233: is used to sort the relevant text blocks from high to low according to the similarity to form a relevant text block list; the candidate text block calculation submodule 234: is used to calculate based on the query request of the user, the large language model input window parameters and the size of the prompt template to select candidate text blocks from the relevant text block list to form a candidate text block group.

[0058] Specifically, before each session starts, the window adaptation module 23 first obtains the metadata of the currently serving large language model from the large language model part 3, especially the input window LIW parameter. Based on this parameter, for each question asked by the user, when VDB obtains a list of text chunks sorted from high to low according to similarity, it no longer selects only the highest text chunk as the context of the prompt template PT, but selects a group of chunks from high to low according to priority, so that the size of this group of chunks is accumulated, plus the PT size and the query size, which is closest to the size of LIW, that is:

[0059] k*Chunk_size+PT_size+Query_size

[0060] = <LIW_size<=

[0061] (k+1)*Chunk_size+PT_size+Query_size

[0062] The large language model part 3 includes a large language model file module 31 and an inference facility module 32. The large language model file module 31 is used to store multiple large language models and corresponding metadata; the inference facility module 32 is used to receive the prompt template containing the context sent by the vector database VDB part 2, and use the corresponding large language model to perform content inference, and return the inference result to the user agent part 1.

[0063] Through the above three parts, the execution steps of the system are:

[0064] 1. The local knowledge document is divided into text segments according to the preset block size (e.g., 512 bytes) and overlap (20 bytes);

[0065] 2. After the text fragment is embedded in the pre-trained embedding model, it is indexed as in the vector database VDB part 2;

[0066] 3. Before the session starts, the window adaptation module 23 in the vector database VDB part 2 obtains the metadata of the large language model through the API, and obtains the input window parameters of the large language model currently being served, such as the 2K window of Xingchen 7B, or the 4K window of ChatGLM3-7B, or the 32K window of QWen14B, taking LIW size = 32K as an example.

[0067] 4. The user sends a query request to the large language model through the proxy.

[0068] 5. The vector database VDB part 2 is placed before the large language model part 3. After receiving the query, it compares the embedded vectors and indexes the corresponding vector result list based on the similarity result. Then, based on the table item identifier related to the vector result list, it queries the local knowledge base for the related text block Chunk list. The Chunk list is sorted from high to low according to the similarity.

[0069] 6. The window adaptation module 23 calculates the candidate Chunk group according to the LIW size using the following formula as the local knowledge context Context of the prompt template PT.

[0070] k*Chunk_size+PT_size+Query_size

[0071] = <LIW_size<=

[0072] (k+1)*Chunk_size+PT_size+Query_size

[0073] 7. The vector database VDB part 2 fills the candidate Chunk group and Query into PT as the final LLM prompt template Prompt, and sends the prompt template Prompt to the reasoning facility module 32 of the large language model part 3.

[0074] 8. The large language model infers the result based on the sufficient content and returns it to the user agent part 1, and the process ends.

[0075] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0076] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A large language model reasoning system with adaptive input window in RAG scenario, characterized in that: It includes a user agent part (1), a vector database VDB part (2) and a large language model part (3). User agent part (1): used to obtain the user's query request; Vector database VDB part (2): connected to the user agent part (1) and the large language model part (3), the vector database VDB part (2) is used to obtain the large language model input window parameters according to the large language model part (3), receive the user's query request, and query the candidate text block group from the local knowledge base, and then fill the user's query request and the candidate text block group into the prompt template to obtain the prompt template containing the context; The large language model part (3) is used to use the corresponding large language model to perform content reasoning on the prompt template containing the context, and return the reasoning result to the user agent part (1).

2. According to claim 1, a large language model reasoning system with adaptive input window in RAG scenario, characterized in that: The vector database VDB part (2) comprises: A query request receiving module (21): used for receiving a query request from the user; Local knowledge base module (22): used to obtain local knowledge and build a local knowledge base based on the local knowledge; A window adaptation module (23): used to obtain the large language model input window parameters of the current service, and query the candidate text block group from the local knowledge base according to the large language model input window parameters, the user's query request and the size of the prompt template; Prompt template filling module (24): used to fill the user's query request and candidate text block groups into the prompt template to obtain a prompt template containing context.

3. According to claim 2, a large language model reasoning system with adaptive input window in RAG scenario is characterized in that: The window adaptation module (23) comprises: Data acquisition submodule (231): used to acquire metadata of the large language model in the large language model part (3) through an API, and obtain input window parameters of the large language model currently being served; Similarity calculation submodule (232): used for searching for relevant text blocks in the local knowledge base according to the similarity based on the query request of the user; Sorting submodule (233): used to sort the related text blocks from high to low according to similarity to form a related text block list; The candidate text block calculation submodule (234) is used to perform calculations based on the user's query request, the large language model input window parameters and the size of the prompt template to select candidate text blocks from the relevant text block list to form a candidate text block group.

4. According to claim 3, a large language model reasoning system with adaptive input window in RAG scenario is characterized in that: The calculation expression of the candidate text block calculation submodule (234) is: k*Chunk_size+PT_size+Query_size = <LIW_size<= (k+1)*Chunk_size+PT_size+Query_size Where k is the number of candidate text blocks, Chunk_size is the size parameter of the text block, Query_size is the size parameter of the query request, LIW_size is the input window size parameter of the large language model, and PT_size is the size parameter of the prompt template.

5. According to claim 1, a large language model reasoning system with adaptive input window in RAG scenario, characterized in that: The large language model part (3) includes: Large language model file module (31): used to store multiple large language models and corresponding metadata; Reasoning facility module (32): used to receive the prompt template containing context sent by the vector database VDB part (2), and use the corresponding large language model to perform content reasoning, and return the reasoning result to the user agent part (1).

6. A large language model reasoning method with adaptive input window in RAG scenario, characterized in that: The following steps are involved: Get the user's query text; Obtaining a large language model input window parameter, and in combination with the user's query request, querying a candidate text block group from a local knowledge base, and then filling the user's query request and the candidate text block group into a prompt template to obtain a prompt template containing a context; The corresponding large language model is used to perform content reasoning on the prompt template containing the context, and the reasoning result is returned.

7. The large language model inference method with adaptive input window in RAG scenario according to claim 6 is characterized in that: The step of obtaining a prompt template containing context includes: Querying candidate text block groups from the local knowledge base according to large language model input window parameters, a user's query request, and a size of a prompt template; The user's query request and the candidate text block group are filled into the prompt template to obtain a prompt template containing context.

8. The large language model inference method with adaptive input window in RAG scenario according to claim 7, characterized in that: The step of searching for candidate text block groups comprises: Based on the query request of the user, searching for relevant text blocks in the local knowledge base according to similarity; Sort the related text blocks according to similarity from high to low to form a related text block list; A calculation is performed based on the query request of the user, the parameters of the large language model input window and the size of the prompt template to select candidate text blocks from the relevant text block list to form a candidate text block group.

9. The large language model inference method with adaptive input window in RAG scenario according to claim 8, characterized in that: The calculation expression is: k*Chunk_size+PT_size+Query_size = <LIW_size<= (k+1)*Chunk_size+PT_size+Query_size Where k is the number of candidate text blocks, Chunk_size is the size parameter of the text block, Query_size is the size parameter of the query request, LIW_size is the input window size parameter of the large language model, and PT_size is the size parameter of the prompt template.

10. The large language model inference method with adaptive input window in RAG scenario according to claim 6, characterized in that: The step of obtaining the large language model input window parameters includes: Get the metadata of the large language model and obtain the large language model input window parameters of the current service.

Images