LLM-based intelligent data retrieval method

Abstract

The invention discloses an intelligent data retrieval method based on LLM, and relates to the field of application of large models. Comprising the steps of 1, respectively constructing a text knowledge base and a data resource knowledge base for guiding the analysis direction of a large model, reducing the answer range of the large model and guiding the large model to generate answers, and 2, performing context association on incomplete semantics of a question by using a large model LLM according to the question input by a user to obtain a user question with context information, 3, performing intention recognition and task analysis on the user problem with the context information by using the large model, and determining information required to be queried by the user; 4, performing semantic similarity retrieval in the data resource knowledge base according to a task of large model analysis; and 5, post-processing the retrieved data resources, performing streaming output on a processed result, and performing front-end display.

Description

Technical Field

[0001] The invention discloses an intelligent data retrieval method based on LLM, and relates to the application field of large models. Background Art

[0002] Currently, it is very difficult to find relevant data resources when facing massive data. For some searches with unclear intentions and tasks, using existing search engines will lead to unsatisfactory search results and low search efficiency, resulting in poor user search experience and failure to obtain effective information. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides an intelligent data retrieval method based on LLM, which realizes efficient, fast and accurate retrieval functions in massive unstructured data and presents the structure to users in real time.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides an intelligent data retrieval method based on LLM, comprising:

[0006] Step 1: Build a text knowledge base and a data resource knowledge base respectively to guide the analysis direction of the big model, narrow the scope of the big model's answers, and guide the big model to generate answers.

[0007] Step 2: Based on the question input by the user, use the large model LLM to contextualize the incomplete semantics of the question and obtain the user question with context information.

[0008] Step 3: Use the big model to identify the intent and analyze the tasks of user questions with contextual information to clarify the information the user needs to query;

[0009] Step 4: Based on the task parsed by the large model, perform semantic similarity retrieval in the data resource knowledge base;

[0010] Step 5: Post-process the retrieved data resources, stream the processed results, and display them on the front end.

[0011] Furthermore, step 1 of the LLM-based intelligent data retrieval method includes:

[0012] Step 11: Use the BGE model to vectorize the knowledge in the text: clean, integrate, and slice the knowledge, use the BGE model to generate a 1024-dimensional real vector, and use Emb text represents a text knowledge base,

[0013] Step 12: Use the BGE model to vectorize the data in the text. First, clean and concatenate the data, use the BGE model to generate a 1024-dimensional real vector, and use Emb data Represents a data repository.

[0014] Furthermore, in step 2 of the LLM-based intelligent data retrieval method, the large model LLM is used to contextualize the incomplete semantics of the question, including: when contextualizing the incomplete semantics of the question, a query statement with context information is generated, and for the historical question Q history , the current user input question Q i ,Use large models for contextual association:

[0015]

[0016] Where Q user is a user question with context information, LLM(Q i ) indicates the result of semantic integrity analysis of user query using large model.

[0017] Furthermore, step 3 of the LLM-based intelligent data retrieval method specifically includes:

[0018] Step 31: First, Embed in the Knowledge Base text Retrieve similar knowledge from the user question Q user As input, the BGE model is used to calculate Q user Vectorize and generate query vectors And in the text knowledge base Emb text Search and The most similar vector, as Q user Knowledge

[0019] Step 32: Build the prompt project: Use the user question Q user as well as Build prompt,

[0020] Input prompt into the large model LLM and guide the large model to use the following formula:

[0021]

[0022] Conduct Q user Intent recognition, and based on the intent recognition user Perform task analysis, which includes multi-task analysis and single-task analysis, and finally obtain TASK Quser (task1,task2,…,task n) represents all possible subtasks after parsing.

[0023] Furthermore, in step 4 of the intelligent data retrieval method based on LLM, the subtask TASK Quser (task1,task2,…,task n ), respectively in the data resource knowledge base Emb data Perform semantic similarity retrieval in and obtain the retrieved data resource information.

[0024] The present invention also provides an intelligent data retrieval device based on LLM, comprising a knowledge base management module, a problem association module, a task analysis module, a retrieval module and an output display module.

[0025] The knowledge base management module constructs a text knowledge base and a data resource knowledge base respectively, which are used to guide the analysis direction of the big model, narrow the scope of the big model's answers, and guide the big model to generate answers.

[0026] The question association module uses the large model LLM to contextualize the incomplete semantics of the question based on the question input by the user, and obtains the user question with context information.

[0027] The task analysis module uses a large model to identify the intent of user questions with contextual information and analyze tasks to clarify the information that users need to query;

[0028] The retrieval module performs semantic similarity retrieval in the data resource knowledge base according to the task of large model analysis;

[0029] The output display module post-processes the retrieved data resources, streams the processed results, and displays them on the front end.

[0030] Furthermore, the knowledge base management module of the LLM-based intelligent data retrieval device includes:

[0031] Step 11: Use the BGE model to vectorize the knowledge in the text: clean, integrate, and slice the knowledge, use the BGE model to generate a 1024-dimensional real vector, and use Emb text represents a text knowledge base,

[0032] Step 12: Use the BGE model to vectorize the data in the text. First, clean and concatenate the data, use the BGE model to generate a 1024-dimensional real vector, and use Emb data Represents a data repository.

[0033] Furthermore, the question association module of the LLM-based intelligent data retrieval device uses the large model LLM to contextualize the incomplete semantics of the question, including: when contextualizing the incomplete semantics of the question, generating a query sentence with context information, for the historical question Q history , the current user input question Q i ,Use large models for contextual association:

[0034]

[0035] Where Q user is a user question with context information, LLM(Q i ) indicates the result of semantic integrity analysis of user query using large model.

[0036] Furthermore, the task parsing module of the LLM-based intelligent data retrieval device specifically includes:

[0037] Step 31: First, Embed in the Knowledge Base text Retrieve similar knowledge from the user question Q user As input, the BGE model is used to calculate Q user Vectorize and generate query vectors And in the text knowledge base Emb text Search and The most similar vector, as Q user Knowledge

[0038] Step 32: Build the prompt project: Use the user question Q user as well as Build prompt,

[0039] Input prompt into the large model LLM and guide the large model to use the following formula:

[0040]

[0041] Conduct Q user Intent recognition, and based on the intent recognition user Perform task analysis, which includes multi-task analysis and single-task analysis. Finally, we get Represents all possible subtasks after parsing.

[0042] Furthermore, the retrieval module of the intelligent data retrieval device based on LLM can be used to analyze the subtasks according to the large model. In the data resource knowledge base Emb dataPerform semantic similarity retrieval in and obtain the retrieved data resource information.

[0043] The benefits of the method of the present invention are:

[0044] (1) It can receive diverse user inputs, perform context association, intent recognition, and task analysis, and understand user demand information more accurately and quickly, with high interpretability and generalizability;

[0045] (2) When faced with massive amounts of data, it can provide search results quickly and accurately, with high efficiency and stability;

[0046] (3) Changed the inherent data resource retrieval model, making the retrieval more intelligent and rational.

[0047] It can also greatly improve the efficiency and accuracy of data resource retrieval in massive data, and can automatically and intelligently generate more professional query statements based on the diverse and colloquial data input by users, achieving end-to-end query results with high interpretability, generalization and professionalism. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0049] Large language models (LLMs) have stronger learning capabilities, better generalization capabilities, and more efficient computing capabilities. They can be applied to natural language processing fields, such as machine translation, speech recognition, text generation, etc.; they can also be applied to image processing fields, such as image classification, image recognition, object detection, etc.; they can also be applied to recommendation systems, such as movie recommendations, product recommendations, music recommendations, etc.; they can also be applied to the game field. Large models are accelerating the development of artificial intelligence by relying on their strong generalization, practicality, innovation, and continuous self-learning, evolution, and intelligent capabilities.

[0050] Semantic similarity is a technology based on natural language processing and machine learning, which is used to calculate the similarity of the semantic relationship between two texts. Semantic similarity models, such as BGE and M3E, use graph embedding technology to convert text data into graph structures, and generate semantic vectors by learning graph structures. This can better capture the complex relationship between texts and improve the accuracy and richness of semantic representation. BGE also has the ability of cross-language mapping and can establish effective semantic connections between Chinese and English. In this way, texts in different languages ​​can be associated and compared through BGE, providing better support for tasks such as cross-language information retrieval and machine translation. Advantages of the BGE model: It has rich semantic information, efficient cross-language mapping, scalability and flexibility, and can be applied in cross-language information retrieval, machine translation, multilingual text classification and clustering, sentiment analysis and opinion mining.

[0051] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0052] Example 1

[0053] The present invention provides an intelligent data retrieval method based on LLM, comprising:

[0054] Step 1: Build a text knowledge base and a data resource knowledge base respectively to guide the analysis direction of the big model, narrow the scope of the big model's answers, and guide the big model to generate answers.

[0055] This may further include:

[0056] Step 11: Use the BGE model to vectorize the knowledge in the text: clean, integrate, and slice the knowledge, use the BGE model to generate a 1024-dimensional real vector, and use Emb text Represents the text knowledge base and saves the vector. Assume that for the text text, BEG is used for vectorization, where Emb text Representing a text knowledge base:

[0057] Emb text =BGE(text)

[0058] Step 12: Use the BGE model to vectorize the data in the text. First, clean and concatenate the data, use the BGE model to generate a 1024-dimensional real vector, and use Emb data Represents the data knowledge base. And save the vectors. Assume that for the data resource data, use BGE for vectorization, where Emb data Represents a data knowledge base:

[0059] Emb data =BGE(data)

[0060] Step 2: Based on the question input by the user, use the large model LLM to contextualize the incomplete semantics of the question and obtain the user question with context information.

[0061] The large model LLM is used to contextualize the incomplete semantics of the question, including: generating a query statement with context information when contextualizing the incomplete semantics of the question. history , the current user input question Q i ,Use large models for contextual association:

[0062]

[0063] Where Q user is a user question with context information, LLM(Q i ) indicates the result of semantic integrity analysis of user query using large model.

[0064] Step 3: Use the large model to identify the intent and analyze the tasks of user questions with contextual information to clarify the information the user needs to query.

[0065] Specifically, it may include:

[0066] Step 31: First, Embed in the Knowledge Base text Retrieve similar knowledge from the user question Q user As input, the BGE model is used to calculate Q user Vectorize and generate query vectors And in the text knowledge base Emb text Search and The most similar vector, as Q user Knowledge

[0067] Step 32: Build the prompt project: Use the user question Q user as well as Build prompt,

[0068] Input prompt into the large model LLM and guide the large model to use the following formula:

[0069]

[0070] Conduct Q user Intent recognition, and based on the intent recognition userPerform task analysis, which includes multi-task analysis, single-task analysis, etc., and finally obtain Represents all possible subtasks after parsing.

[0071] Step 4: Based on the task parsed by the large model, perform semantic similarity retrieval in the data resource knowledge base.

[0072] Among them, the subtasks that can be analyzed according to the large model In the data resource knowledge base Emb data Perform semantic similarity retrieval in and obtain the retrieved data resource information as follows:

[0073]

[0074] It is the data resource information retrieved last.

[0075] Step 5: Post-process the retrieved data resources, stream the processed results, and display them on the front end. Functions such as hyperlinks can be provided to directly jump to the data resource platform. Indicates processing of data resources:

[0076]

[0077] Including data cleaning, integration, adding hyperlinks, streaming returns, etc.

[0078] Example 2

[0079] The present invention also provides an intelligent data retrieval device based on LLM, comprising a knowledge base management module, a problem association module, a task analysis module, a retrieval module and an output display module.

[0080] The knowledge base management module constructs a text knowledge base and a data resource knowledge base respectively, which are used to guide the analysis direction of the big model, narrow the scope of the big model's answers, and guide the big model to generate answers.

[0081] The question association module uses the large model LLM to contextualize the incomplete semantics of the question based on the question input by the user, and obtains the user question with context information.

[0082] The task analysis module uses a large model to identify the intent of user questions with contextual information and analyze tasks to clarify the information that users need to query;

[0083] The retrieval module performs semantic similarity retrieval in the data resource knowledge base according to the task of large model analysis;

[0084] The output display module post-processes the retrieved data resources, streams the processed results, and displays them on the front end.

[0085] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0086] Likewise, the device of the present invention is beneficial in that:

[0087] (1) It can receive diverse user inputs, perform context association, intent recognition, and task analysis, and understand user demand information more accurately and quickly, with high interpretability and generalizability;

[0088] (2) When faced with massive amounts of data, it can provide search results quickly and accurately, with high efficiency and stability;

[0089] (3) Changed the inherent data resource retrieval model, making the retrieval more intelligent and rational.

[0090] It can also greatly improve the efficiency and accuracy of data resource retrieval in massive data, and can automatically and intelligently generate more professional query statements based on the diverse and colloquial data input by users, achieving end-to-end query results with high interpretability, generalization and professionalism.

[0091] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0092] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. An intelligent data retrieval method based on LLM, characterized by include: Step 1: Build a text knowledge base and a data resource knowledge base respectively to guide the analysis direction of the big model, narrow the scope of the big model's answers, and guide the big model to generate answers. Step 2: Based on the question input by the user, use the large model LLM to contextualize the incomplete semantics of the question and obtain the user question with context information. Step 3: Use the big model to identify the intent and analyze the tasks of user questions with contextual information to clarify the information the user needs to query; Step 4: Based on the task parsed by the large model, perform semantic similarity retrieval in the data resource knowledge base; Step 5: Post-process the retrieved data resources, stream the processed results, and display them on the front end.

2. The intelligent data retrieval method based on LLM according to claim 1 is characterized by: Step 1 includes: Step 11: Use the BGE model to vectorize the knowledge in the text: clean, integrate, and slice the knowledge, use the BGE model to generate a 1024-dimensional real vector, and use Emb text represents a text knowledge base, Step 12: Use the BGE model to vectorize the data in the text. First, clean and concatenate the data, use the BGE model to generate a 1024-dimensional real vector, and use Emb data Represents a data repository.

3. The intelligent data retrieval method based on LLM according to claim 1 is characterized in that In step 2, the large model LLM is used to contextualize the incomplete semantics of the question, including: when contextualizing the incomplete semantics of the question, a query statement with context information is generated. For the historical question Q history , the current user input question Q i ,Use large models for contextual association: Where Q user is a user question with context information, LLM(Q i ) indicates the result of semantic integrity analysis of user query using large model.

4. The intelligent data retrieval method based on LLM according to claim 1 is characterized by: Step 3 specifically includes: Step 31: First, Embed in the Knowledge Base text Retrieve similar knowledge from the user question Q user As input, the BGE model is used to calculate Q user Vectorize and generate query vector Emb Quser , and in the text knowledge base Emb text Search and Embed Quser The most similar vector, as Q user Knowledge Quser , Step 32: Build the prompt project: Use the user question Q user and Knowledge Quser Build prompt, Input prompt into the large model LLM and guide the large model to use the following formula: TASK Quser (task1,task2,…,task n )= {LLMpromptKnowledgeQuser,Quser, task analysisLLMpromptKnowledgeQuser,Quser, intention recognition Conduct Q user Intent recognition, and based on the intent recognition user Perform task analysis, which includes multi-task analysis and single-task analysis, and finally obtain TASK Quser (task1,task2,…,task n ) represents all possible subtasks after parsing.

5. The intelligent data retrieval method based on LLM according to claim 1 is characterized in that the subtask TASK Quser (task1,task2,…,task n ), respectively in the data resource knowledge base Emb data Perform semantic similarity retrieval in and obtain the retrieved data resource information.

6. An intelligent data retrieval device based on LLM, characterized in that It includes knowledge base management module, problem association module, task analysis module, retrieval module and output display module. The knowledge base management module constructs a text knowledge base and a data resource knowledge base respectively, which are used to guide the analysis direction of the large model, narrow the scope of the large model's answers, and guide the large model to generate answers. The question association module uses the large model LLM to contextualize the incomplete semantics of the question based on the question input by the user, and obtains the user question with context information. The task analysis module uses a large model to identify the intent of user questions with contextual information and analyze tasks to clarify the information that users need to query; The retrieval module performs semantic similarity retrieval in the data resource knowledge base according to the task parsed by the large model; The output display module post-processes the retrieved data resources, streams the processed results, and displays them on the front end.

7. The intelligent data retrieval device based on LLM according to claim 6, characterized in that Knowledge base management module, including: Step 11: Use the BGE model to vectorize the knowledge in the text: clean, integrate, and slice the knowledge, use the BGE model to generate a 1024-dimensional real vector, and use Emb text represents a text knowledge base, Step 12: Use the BGE model to vectorize the data in the text. First, clean and concatenate the data, use the BGE model to generate a 1024-dimensional real vector, and use Emb data Represents a data repository.

8. The intelligent data retrieval device based on LLM according to claim 6, characterized in that the question The association module uses the large model LLM to contextualize the incomplete semantics of the question, including: generating a query statement with context information when contextualizing the incomplete semantics of the question, and for the historical question Q history , the current user input question Q i ,Use large models for contextual association: Where Q user is a user question with context information, LLM(Q i ) indicates the result of semantic integrity analysis of user query using large model.

9. The intelligent data retrieval device based on LLM according to claim 6, characterized in that Task parsing module, the specific execution includes: Step 31: First, Embed in the Knowledge Base text Retrieve similar knowledge from the user question Q user As input, the BGE model is used to calculate Q user Vectorize and generate query vectors And in the text knowledge base Emb text Search and The most similar vector, as Q user Knowledge Step 32: Build the prompt project: Use the user question Q user and Knowledge Quser Build prompt, Input prompt into the large model LLM and guide the large model to use the following formula: TASK Quser (task1,task2,…,task n )= {LLMpromptKnowledgeQuser,Quser, task analysisLLMpromptKnowledgeQuser,Quser, intention recognition Conduct Q user Intent recognition, and based on the intent recognition user Perform task analysis, which includes multi-task analysis and single-task analysis, and finally obtain TASK Quser (task1,task2,…,task n ) represents all possible subtasks after parsing.

10. The intelligent data retrieval device based on LLM according to claim 6, characterized in that The retrieval module analyzes the subtasks according to the large model Quser (task1,task2,…,task n ), respectively in the data resource knowledge base Emb data Perform semantic similarity retrieval in and obtain the retrieved data resource information.

Images