Large model information retrieval and generation method based on data lake mass documents

Abstract

The invention discloses a large model information retrieval and generation method based on a large batch of documents in a data lake. The method comprises the steps that a user uploads files to the data lake, a system prepares to receive a large amount of unstructured data, and the system captures file changes by monitoring file adding or modifying events in the data lake; triggering a file analysis and segmentation process, and performing text slicing and vectorization processing on document contents in the file for subsequent retrieval; a user inputs a query question to the system, and the system generates and returns an answer to the user. According to the method, the data processing capability, the retrieval precision, the professional knowledge application capability and the user interaction experience are improved, and more efficient and intelligent document processing and question and answer services are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of large model information retrieval and generation, and in particular to a method for large model information retrieval and generation based on a large number of documents in a data lake. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a natural language processing (NLP) model architecture that combines retrieval and generation capabilities. The main goal of RAG is to enhance the performance of the model in dealing with open-domain problems by combining pre-trained generation models (such as GPT) with retrieval systems (such as BM25 or Dense Vector-based retrieval models).

[0003] The following problems exist in various current RAG systems, such as Ragflow and Dify:

[0004] Poor support for large-volume file parsing: Enterprises usually store a large amount of unstructured data, including various documents and files. The RAG system needs to solve the problem of how to efficiently parse and manage these large-volume files to ensure that the document content can be processed and analyzed in a timely and accurate manner;

[0005] Inaccurate document content matching: In massive document data, how to achieve accurate content matching is a key challenge. The RAG system needs to be able to accurately find the most relevant documents to the user's query from a large amount of data, ensuring the accuracy and relevance of the retrieval results, thereby improving the user experience;

[0006] Poor understanding of industry knowledge: In a specific industry, the accuracy of answering questions depends on a deep understanding of industry knowledge. The RAG system needs to improve the accuracy of understanding and answering field-specific questions by building and utilizing industry-specific knowledge graphs. This involves how to effectively integrate and apply knowledge graphs to enhance the professionalism of the system;

[0007] Poor user-friendly Q&A experience: The Q&A function based on systems such as Ragflow and Dify lacks a natural interactive experience and cannot fully understand user intent. The RAG system needs to improve the system's interactive capabilities to make it more humanized, understand user intent more naturally and intelligently, provide corresponding answers, and improve user satisfaction and usage experience. Summary of the invention

[0008] In view of this, the present invention provides a method for large-model information retrieval and generation based on a large number of documents in a data lake.

[0009] The present invention discloses a method for large-model information retrieval and generation based on a large number of documents in a data lake, which includes:

[0010] Users upload files to the data lake, and the system prepares to receive a large amount of unstructured data. The system captures file changes by monitoring new or modified file events in the data lake; triggers the file parsing and segmentation process, and performs text slicing and vectorization processing on the document content in the file for subsequent retrieval; users input query questions into the system, and the system generates and returns answers to the users.

[0011] Furthermore, the user uploads the file to the data lake, including:

[0012] Build a distributed data lake and configure it to support multi-format file upload and automatic classification storage; use event-driven architecture to manage the flow of uploaded files to ensure that large quantities of documents can enter the system and be managed.

[0013] Furthermore, monitoring the events of adding or modifying files in the data lake includes:

[0014] When users update files in the system, the system listener automatically recognizes the file updates and initiates the parsing and indexing update process, ensuring that the content in the document library is always up to date.

[0015] Furthermore, the trigger file parsing and segmentation process performs text slicing and vectorization processing on the document content in the file, including:

[0016] Use natural language processing technology to parse documents, extract structured and unstructured data, perform text slicing and vectorization, and convert the parsed text into vectors to support subsequent hybrid search.

[0017] Furthermore, before the user inputs a query question to the system, the system also includes:

[0018] Store the parsed text and vector data in a hybrid index search engine, build an index library that supports hybrid keyword and vector search, and optimize index parameters to ensure that relevant documents can be returned during queries.

[0019] Furthermore, the user inputs a query question to the system, and the system generates and returns an answer to the user, including:

[0020] The user enters a query into the system, and the system searches for similar questions through the intent index library and uses a large language model to identify the user's intent.

[0021] According to the result of user intention recognition, the corresponding RAG strategy is selected to perform knowledge retrieval and answer generation; wherein the RAG strategy includes simple answers, complex dialogues, and advanced queries combined with knowledge graphs;

[0022] The large language model combines the user's question with the retrieved document content, generates an answer and returns it to the user, completing the entire question-and-answer process.

[0023] Furthermore, the system searches for similar questions through the intent index library and calls a large language model to identify user intent, including:

[0024] Identify the user's query intent through the preset intent index library and large language model, and find similar questions or historical queries.

[0025] Furthermore, the large language model combines the user's question with the retrieved document content, generates an answer and returns it to the user, including:

[0026] Based on user interaction and feedback, a large language model is used to re-rank the search results so that users can discover the content they care about most through result sorting or result presentation.

[0027] Furthermore, when the user continuously inputs multiple questions, context information is maintained in the user's continuous questioning so that the system can match the current question with the user's historical queries and answers.

[0028] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0029] 1. The present invention can significantly improve the accuracy of document information retrieval and user question-and-answer experience; it improves data processing capabilities, retrieval accuracy, professional knowledge application capabilities and user interaction experience, and realizes more efficient and intelligent document processing and question-and-answer services.

[0030] 2. Improved efficiency: Efficiently manage and process large amounts of unstructured data through distributed data lakes and automated parsing tools.

[0031] 3. Improved content retrieval accuracy: A hybrid index search engine is used to improve retrieval precision and relevance, thereby ensuring that accurate answers are provided quickly.

[0032] 4. Improved answer accuracy and professionalism: Dynamically select appropriate RAG strategies based on user intent to achieve smarter answer generation and better user experience. Combined with industry knowledge graphs, the accuracy and professionalism of the system's answers in specific fields can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0034] Figure 1 The present invention is a flowchart of a method for retrieving and generating large-model information based on a large number of documents in a data lake according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0036] See also Figure 1 The present invention provides an embodiment of a method for large-model information retrieval and generation based on a large number of documents in a data lake, which includes:

[0037] Users upload files to the data lake: Users upload files to the data lake, and the system is ready to receive a large amount of unstructured data;

[0038] Monitoring file change events: The system monitors new or modified file events in the data lake to capture file changes in a timely manner;

[0039] Document parsing and segmentation: triggers the document parsing and segmentation process, slices and vectorizes the document content to prepare for subsequent retrieval;

[0040] Storage index construction: Store the generated text data and vector data into the search engine repository to build a hybrid index to support accurate content retrieval;

[0041] User asks a query: The user enters a query and the system prepares to answer the query;

[0042] Intent identification and similar question retrieval: The system searches for similar questions through the intent index library and calls the large language model (LLM) to identify user intent;

[0043] Select and execute RAG strategies: According to the user's intention, select the appropriate RAG strategy (such as simple answer, retrieval + generation, or strategy combined with knowledge graph) to perform knowledge retrieval and answer generation;

[0044] Generate and return answers: Submit user questions and retrieved content to a large language model, generate accurate answers and return them to the user, completing the entire question-answering process.

[0045] About the integration of data lake and intelligent retrieval generation: The innovative integration of data lake technology and RAG (Retrieval-Augmented Generation) realizes efficient management and intelligent retrieval generation of large-scale unstructured data. This integration not only improves data processing efficiency, but also expands the application scope of RAG technology.

[0046] About hybrid index search optimization: The use of a hybrid index search engine that supports both keywords and vectors not only improves the retrieval effect of traditional text matching, but also achieves more accurate semantic search through vectorization technology, enhancing the accuracy of content retrieval and user experience.

[0047] About dynamic RAG strategy selection mechanism: The introduction of multiple user intent recognition strategies and dynamic RAG strategy selection mechanism can flexibly adjust the retrieval and generation methods according to real-time user needs, thereby providing more intelligent and personalized answers.

[0048] About industry knowledge graph integration: Integrate industry knowledge graphs to enhance the system's knowledge base and achieve higher professionalism and accuracy in answering questions in specific fields. This approach enables the system to provide more authoritative answers and improves its applicability in professional scenarios.

[0049] The present invention achieves timely and accurate analysis of large quantities of unstructured files through the following processing:

[0050] Data lake infrastructure construction: Use distributed data lake technology to achieve large-scale document storage and management. Through event-driven architecture, automatically monitor file upload, modification and other events to ensure that data changes can be captured in a timely manner.

[0051] Document parsing and segmentation: Introducing advanced natural language processing (NLP) technology, including OCR (optical character recognition), text extraction, entity recognition and other tools, to parse document content into structured data. Slice and vectorize the parsed text to improve retrieval and analysis efficiency.

[0052] Distributed computing and parallel processing: Use distributed computing frameworks (such as Apache Spark) to parallelize parsing tasks to meet the needs of efficient parsing of large batches of files and ensure the timeliness and processing power of the system.

[0053] Listener module and automatic triggering: Develop a listener module to monitor the document status in the data lake in real time and automatically trigger the file parsing and index update process to ensure the real-time nature of the data.

[0054] Error handling and fault tolerance mechanism: Design a complete error detection and retry mechanism to ensure that when errors occur during the parsing process, they can be effectively handled and re-parsed to ensure the accuracy of data processing.

[0055] The present invention accurately matches the parsed document content through the following processing:

[0056] Optimization of hybrid index search engines: A hybrid search engine based on keywords and vectors is used to vectorize text content to support semantic-level search. The accuracy of matching is improved by combining traditional retrieval models such as BM25 with deep learning models such as Dense Vector.

[0057] Dynamic retrieval strategy based on user intent: Introduce a user intent recognition module to understand the real intent behind the user query through a large language model (LLM) and dynamically select the most suitable retrieval strategy, such as simple retrieval, deep generation, or advanced retrieval combined with knowledge graphs.

[0058] Multi-round dialogue context understanding: Supports multi-round dialogue queries, maintains context information during the user's continuous questioning, and enables the system to more accurately match the current question based on the user's historical queries and answers.

[0059] Re-ranking strategy for search results: Based on user interaction and feedback, use machine learning models to re-rank search results to improve the ranking and presentation of content that users care about most, thereby enhancing user experience.

[0060] The working principle of the present invention mainly includes the following steps:

[0061] 1. Build data lake infrastructure: Build a data lake cluster to support efficient processing of large-scale data files;

[0062] 2. Build a listener / updater module: Develop a listener to monitor file changes, parse files and store the results in the index library;

[0063] 3. Build a hybrid index search engine: deploy a search engine and optimize the index and search experience to support multiple query methods;

[0064] 4. Build RAG strategy selection and execution module: Based on user intent recognition, design and implement various RAG strategy processing logics;

[0065] 5. Integrated Large Language Model (LLM) caller: Integrate multiple LLM models and optimize their calling performance to quickly generate answers.

[0066] In order to specifically demonstrate the implementation mode of the present invention, the following is a detailed implementation case:

[0067] S1. The user uploads the file to the data lake;

[0068] Build a distributed data lake that supports multi-format file uploads and automatic classification storage. Use event-driven architecture to manage the flow of uploaded files, ensuring that large quantities of documents can enter the system smoothly and be effectively managed.

[0069] Implementation case: In the document management system of a large enterprise, employees can upload contracts, reports, project files and other documents in batches to the data lake, and the system automatically classifies and stores them by file type or department.

[0070] S2, monitor file change events;

[0071] Develop a listener module to continuously monitor the file status changes (such as addition or modification) in the data lake. Once a document upload or modification event is detected, the subsequent data processing flow, such as file parsing and index building, is immediately triggered.

[0072] Implementation case: When employees update contracts or add new report files in the enterprise system, the system listener automatically recognizes these changes and initiates the parsing and indexing update process, ensuring that the content in the document library is always up to date.

[0073] S3, file parsing and segmentation;

[0074] Use NLP (natural language processing) technology to parse documents, extract structured and unstructured data, and perform text slicing and vectorization. Convert the parsed text into vectors to support subsequent efficient hybrid search.

[0075] Implementation case: In the enterprise's archive management system, newly uploaded project reports and technical documents are parsed using NLP technology to extract key information (such as project stages and technical points) and generate vector representations to facilitate subsequent intelligent retrieval.

[0076] S4, storage index construction;

[0077] Store the parsed text and vector data in a hybrid index search engine to build an index library that supports hybrid keyword and vector searches. Optimize index parameters to ensure that relevant documents can be returned quickly and accurately when querying.

[0078] Implementation case: When enterprise employees query historical project data, the system can quickly retrieve relevant project reports and technical documents through keyword or semantic search, supporting efficient enterprise decision-making.

[0079] S5. The user asks a query;

[0080] Provide a query interface for enterprise users, support text input and voice input, and allow users to raise various document-related queries. The system needs to be able to understand natural language and quickly respond to user query needs.

[0081] Implementation case: In an enterprise’s internal knowledge base system, employees can type or voice inquiries for contract details or technical documents for a specific project, and the system will immediately understand and process the query.

[0082] S6, intent recognition and similar question retrieval;

[0083] Develop an intent recognition module to identify the user's query intent through a preset intent index library and a large language model (LLM), find similar questions or historical queries, and ensure the accuracy and relevance of the response.

[0084] Implementation case: The system can identify the query intent and retrieve similar issues or archived project documents and solutions to quickly provide reference information.

[0085] S7, select and execute RAG strategy;

[0086] Based on the results of user intent recognition, the appropriate RAG strategy is dynamically selected, such as template matching, simple dialogue, document retrieval, document retrieval plus knowledge graph strategy, etc., to generate the answer that best meets the user's needs.

[0087] Implementation case: The system can use the "document retrieval + generation" strategy to retrieve relevant information from the data lake documents and use LLM to generate detailed compliance explanations.

[0088] S8. Generate and return a response.

[0089] The LLM generation model is used to combine the user's question with the retrieved document content to generate an answer, optimize the accuracy and naturalness of the answer, and then return the answer to the user. At the same time, the system needs to continuously learn from user feedback and improve the quality of the answer.

[0090] Implementation example: The system generates natural and fluent answers through LLM and provides specific guidance in combination with the latest company policy documents.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for large-scale model information retrieval and generation based on a large number of documents in a data lake, characterized in that: include: Users upload files to the data lake, and the system prepares to receive a large amount of unstructured data. The system captures file changes by monitoring new or modified file events in the data lake; triggers the file parsing and segmentation process, and performs text slicing and vectorization processing on the document content in the file for subsequent retrieval; users input query questions into the system, and the system generates and returns answers to the users.

2. The method according to claim 1, characterized in that The user uploads a file to the data lake, including: Build a distributed data lake and configure it to support multi-format file upload and automatic classification storage; use event-driven architecture to manage the flow of uploaded files to ensure that large quantities of documents can enter the system and be managed.

3. The method according to claim 1, characterized in that The monitoring of new or modified file events in the data lake includes: When users update files in the system, the system listener automatically recognizes the file updates and initiates the parsing and indexing update process, ensuring that the content in the document library is always up to date.

4. The method according to claim 1, characterized in that: The trigger file parsing and segmentation process performs text slicing and vectorization processing on the document content in the file, including: Use natural language processing technology to parse documents, extract structured and unstructured data, perform text slicing and vectorization, and convert the parsed text into vectors to support subsequent hybrid search.

5. The method according to claim 1, characterized in that Before the user inputs a query question to the system, the system also includes: Store the parsed text and vector data in a hybrid index search engine, build an index library that supports hybrid keyword and vector search, and optimize index parameters to ensure that relevant documents can be returned during queries.

6. The method according to claim 1, characterized in that The user inputs a query question to the system, and the system generates and returns an answer to the user, including: The user enters a query into the system, and the system searches for similar questions through the intent index library and uses a large language model to identify the user's intent. According to the result of user intention recognition, the corresponding RAG strategy is selected to perform knowledge retrieval and answer generation; wherein the RAG strategy includes simple answers, complex dialogues, and advanced queries combined with knowledge graphs; The large language model combines the user's question with the retrieved document content, generates an answer and returns it to the user, completing the entire question-and-answer process.

7. The method according to claim 6, characterized in that The system searches for similar questions through the intent index library and uses a large language model to identify user intent, including: Identify the user's query intent through the preset intent index library and large language model, and find similar questions or historical queries.

8. The method according to claim 6, characterized in that The large language model combines the user's question with the retrieved document content to generate an answer and return it to the user, including: Based on user interaction and feedback, a large language model is used to re-rank the search results so that users can discover the content they care about most through result sorting or result presentation.

9. The method according to claim 1, characterized in that: When a user enters multiple questions in succession, context information is maintained during the user's successive questions so that the system can match the current question with the user's historical queries and answers.

Images