Multi-document retrieval method and system based on AI-agent

Abstract

The invention provides an AI-agent-based multi-document retrieval method and system, and the method comprises the steps: carrying out the preloading according to the format of a document, thereby achieving the intelligent partitioning of a multi-source heterogeneous document, i.e., decomposing a large text into smaller and easier-to-manage blocks; the method comprises the following steps of: performing fine tuning by using vertical domain data on the basis of general BAAI Genral Embedding, and completing the conversion from a text to a vector; carrying out preliminary retrieval on the vectorized data by adopting a mixed retrieval and reordering two-segment method; the intelligent agent technology is used for achieving self-reflection learning retrieval and adjustment and optimization, and retrieval results are adjusted and optimized through a reflection mechanism. According to the method, intelligent identification segmentation and vectorization can be carried out on the electronic archives in different formats, a multi-dimensional mixed retrieval result is optimized by using a reflection mechanism, the correlation between texts and themes is fully considered, and the method has wider applicability, higher robustness and controllability.

Description

Technical Field

[0001] The present invention belongs to the field of archive management, and in particular relates to a multi-document retrieval method and system based on AI-agent. Background Art

[0002] With the rapid development of artificial intelligence and information technology, multi-document retrieval has become increasingly important in the fields of electronic archive management, such as academic research, business intelligence analysis, and legal document query. Traditional text retrieval technology often relies on keyword matching, which has great limitations when dealing with complex, multi-language, and multi-format documents. The emergence of AI-agent technology, semantic vectors, and intelligent text processing technologies has provided a new way to solve this problem.

[0003] At present, the main challenges facing the field of multi-document retrieval include: how to intelligently process multi-source heterogeneous documents, how to improve the accuracy of cross-language retrieval, and how to optimize the relevance ranking of retrieval results. Existing technologies can often only effectively process documents in specific formats or languages, lacking versatility and robustness.

[0004] In view of the current situation, it is necessary to introduce intelligent text processing and intelligent agent technology to carry out multi-document retrieval and optimization. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method that can reduce manual intervention and improve the accuracy and efficiency of multi-document retrieval in view of the shortcomings of the prior art.

[0006] The present invention specifically provides a multi-document retrieval method based on AI-agent, comprising the following steps:

[0007] Step 1, intelligent text segmentation: pre-process multi-source heterogeneous documents, pre-load documents and analyze their formats, identify the content of the documents, and automatically decompose large texts into small text blocks through an intelligent segmenter based on natural language processing technology;

[0008] Step 2, multilingual semantic vector conversion: convert text content into vector representation to facilitate efficient semantic retrieval. In order to improve the performance of the model in a specific field, the general model is fine-tuned using data from vertical domains. This step ensures that the generated semantic vector is closer to the domain-specific semantics.

[0009] Step 3, index hybrid search and filtering: A two-stage method of multiple retrieval techniques and re-ranking is used to conduct preliminary retrieval of the vectorized text. This process considers the multi-dimensional correlation between the text and the retrieval topic, conducts a comprehensive evaluation of the final retrieval results, and improves the retrieval efficiency.

[0010] Step 4: Agent autonomous tuning: Use intelligent agent technology to achieve self-reflective learning retrieval and tuning.

[0011] In step 1, a document is preloaded using a document loader, wherein the document loader includes a PDF Loader loader, a Markdown loader, and a Word loader, and the content of the document includes text, a title, a table, a table title, and a reference;

[0012] In the present invention, the PDF Loader can adopt a fast and feature-rich PDF rendering engine developed by Jiangsu Yongshanqiao Archives Management Service Co., Ltd., which provides many advanced features such as table analysis and image processing, allowing users to read, split, merge and convert PDFs.

[0013] The Markdown loader and Word loader are unstructured text loaders implemented using LangChain's UnstructuredLoader for Markdown formatted text and unstructured text, providing a "load" method for loading data as a document from a configured source. They also optionally implement "delayed loading" for delaying the loading of data into memory.

[0014] The intelligent segmenter includes a recursive character text segmenter, a text semantic segmenter and a proposition segmenter. The intelligent segmenter divides the text into small blocks according to a certain strategy so as to adapt to the context window of the subsequent module and improve the accuracy of knowledge retrieval.

[0015] The recursive character text segmenter pre-segments the text according to the priority of the separators by specifying a block length and a set of delimiters, splitting the text into block documents that are smaller than the original document (the length of the block documents is generally less than 100 characters), then merging the block documents that do not exceed the specified block length, and recursively splitting the text that exceeds the specified block length until the desired block size is obtained. In the end, the text sizes of these blocks are not exactly the same, but their lengths will be close to the specified block length; the recursive character text segmenter is parameterized by a character list and tries to split on these characters in sequence until the block is small enough. The default character list is ["\n\n","\n","",""]. ​​The effect of this is to keep all paragraphs (then sentences, then words) together for as long as possible, because these usually appear to be the most semantically relevant parts of the text.

[0016] The text semantic segmenter segments text by identifying breakpoints, which are determined in the following way: if the distance between the embedding vectors of two consecutive paragraphs exceeds a threshold (usually 0.8), which may indicate that they are very different in semantics, then a breakpoint is set between the two consecutive paragraphs; once the breakpoints are determined, the text can be segmented into independent blocks based on these breakpoints. Each block is relatively independent in semantics, but may still be related to the previous or next block, which is very useful for identifying coherent and related blocks of information in a larger text.

[0017] The proposition segmenter is a retrieval unit called proposition, which generates propositions by feeding a text into a large model such as chatgpt, and then uses the proposition as one of the mixed sources to participate in the recall during the retrieval stage.

[0018] The PDF Loader is used to perform the following steps:

[0019] Step 1-1, read the PDF file in pages;

[0020] Step 1-2, detect and perform OCR (Optical Character Recognition) on the text and text box, wherein the OCR recognition is trained and fine-tuned based on the PaddleOCR model, and the text box coordinates and recognition results are returned;

[0021] Steps 1-3: Analyze the layout of the document image and identify different types of areas, such as tables, titles, paragraphs, etc. This step follows the idea of ​​target recognition to analyze the document layout. Relying on the massive data accumulated in the field of archive management, a multi-scenario, high-precision layout analysis model is trained to identify the layout of the document.

[0022] Step 1-4, parse the document content and process the identified area. This step is mainly for processing the table data contained in the text, the purpose is to format the identified table data into plain text; specifically, it includes the following steps:

[0023] Step 1-4-1, in order to improve the accuracy of the question-answering system's understanding and response to table data, the LORE table structure recognition model is used. The LORE (Logical Location Regression Network for Table Structure Recognition) table structure recognition model is used to identify elements in the table area. The core of the LORE model is the logical location regression network. The LORE table structure recognition model combines the spatial position and logical position of table cells for end-to-end modeling and prediction. The LORE table structure recognition model uses a key point segmentation network based on a convolutional neural network CNN to extract visual features, and predicts the spatial position and logical position of cells through two regression heads; finally, the LORE table structure recognition model accurately restores the row and column structure of the entire table and outputs a machine-parseable representation;

[0024] Step 1-4-2, output the cell content and the cell's spatial position and logical position;

[0025] Steps 1-5 integrate the results of layout analysis and OCR recognition, introduce a layout recovery process, and convert the document into markdown (a lightweight markup language) for output, which specifically includes the following steps:

[0026] Step 1-5-1, group the tables, titles, and paragraphs output by the layout analysis;

[0027] Step 1-5-2, merge the texts in the group according to their spatial and logical positions;

[0028] Step 1-5-3, convert the merged text into markdown according to the type. For example, the title corresponds to the titles of different levels in markdown.

[0029] In step 1-2, the OCR recognition includes the following steps:

[0030] Step 1-2-1, perform text detection on the document and obtain the text box coordinates;

[0031] Step 1-2-2, rotate and crop each text box;

[0032] Step 1-2-3, perform text recognition on the cropped document;

[0033] Step 1-2-4, filter out the recognition results whose confidence level is lower than the threshold (generally 0.5).

[0034] Step 1 includes the following steps:

[0035] Step a1, reading the document;

[0036] Step a2, automatically selecting a document loader for preloading according to the document format;

[0037] Step a3, the intelligent segmenter segments the loaded document to obtain segmented documents;

[0038] Step a4: adding context information to the block documents.

[0039] Step 2 includes the following steps:

[0040] Step 2-1, based on the BGE (BAAI General Embedding) model, take the block document as input and generate the corresponding semantic vector. The BGE model adopts the instruction fine-tuning method, which allows the model to be fine-tuned according to different task requirements, including the following steps:

[0041] Step 2-1-1, generate corpus, the fine-tuning dataset format is [query, positive sample set, negative sample set], where query is the query text, the positive sample set represents the text related to the query, and the negative sample set represents the text irrelevant to the query;

[0042] Step 2-1-2, set fine-tuning parameters, for example, the learning rate is 1e -5 , the number of training times is 5, and the maximum length of text is 1024;

[0043] Step 2-1-3, fine-tune the Embeding model to treat the semantic relevance task as a binary classification task, use the cross entropy loss function as the loss function, use back propagation to update the model parameters, and output the fine-tuned model; the cross entropy loss function is:

[0044]

[0045] Among them, H(p,q) is the cross entropy between probability distribution q and probability distribution p, p(x i ) represents the probability value of the i-th sample under the probability distribution, q(x i ) represents the probability value of the i-th sample under probability distribution q, and n is the total number of samples;

[0046] Step 2-2, store the semantic vector and the original text into the vector library.

[0047] Step 3 includes the following steps:

[0048] Step 3-1, use Jieba as a word segmenter to segment the block documents and remove stop words;

[0049] Step 3-2, build the BM25 retriever. The core idea of ​​the BM25 retriever is based on term frequency (TF) and inverse document frequency (IDF), and introduces document length information to calculate the correlation between document D and query Q. The formula is:

[0050]

[0051] Among them, Score(D,Q) is the relevance score between document D and query Q; q i is the i-th word in the query; f(q i ,D) is q i The frequency in document D; IDF(q i ) is q i is the inverse document frequency; |D| is the length of document D; avgdl is the average length of all documents; k1 and b are constants, usually k1 is between 1.2 and 2, and b is 0.75;

[0052] Step 3-3, construct a vector retriever, which is a tool for efficiently finding and retrieving data points in high-dimensional space, and is mainly used to process and search data represented in vector form. In the present invention, IVF_FlAT (Inverted File with Flat Quantization) is used as a vector retriever for data indexing and retrieval. The steps of IVF_FlAT index construction include:

[0053] Step 3-3-1, the vectors in the vector library are clustered into two or more clusters, each cluster contains a group of similar vectors;

[0054] Step 3-3-2, create an inverted file to save the identifier of each cluster and the corresponding vector list;

[0055] Step 3-3-3, when searching, first find the cluster with the highest similarity to the query vector, then use the similarity to further search within the cluster with the highest similarity to find the final nearest neighbor;

[0056] Step 3-4, assigning weights to the vector searcher and the BM25 searcher. In the present invention, the weights of the vector searcher and the BM25 searcher are set to 0.6 and 0.4 respectively. In the present invention, the weight of the vector searcher is better, which means that the score of the vector searcher is considered more important in the final ranking;

[0057] Step 3-5, construct a fusion index searcher, which includes a vector searcher and a BM25 searcher, that is, the contents retrieved by the vector search and the BM25 search need to be fused; set search parameters, which include the number of searched data and the search method, and the search method includes hybrid search and vector search;

[0058] Step 3-6, retrieve the user input, and output the block document information with the highest score in the vector searcher and BM25 searcher based on the fusion ranking algorithm of reciprocal rank fusion RRF (reciprocal rank fusion);

[0059] The reciprocal rank fusion (RRF) is a method that combines multiple result sets with different relevance indicators into a single result set. RRF does not need to be tuned, and different relevance indicators do not have to be correlated with each other to obtain high-quality results. The advantage of this method is that it does not use relevance scores, but only relies on ranking calculations. The formula for the reciprocal rank fusion RRF is:

[0060]

[0061] in, Represents a collection of documents. Represents document d in the document collection R is the ranking of the searcher. In the present invention, the set R includes the vector searcher and the BM25 searcher. k is a constant, usually 60. r(d) is the ranking of document d in the rth searcher.

[0062] Step 3-7, constructing a tuple of <user input, fragment information> based on the retrieved fragment information and the user input;

[0063] Step 3-8: Based on the BGE Rerank model, the relationship between the two tuples is scored for similarity, and the text fragments are sorted in reverse order according to the similarity scores, thereby completing the sorting of the text fragments;

[0064] Step 3-9, according to the search parameters (such as the number of search data), the sorted text fragments are evaluated, a specified number of text fragments are obtained, and the final search results are output.

[0065] Step 4 includes the following steps:

[0066] Step 4-1, query conversion of user input, specifically includes the following steps:

[0067] Step 4-1-1, query rewriting: Use the big model's prompt word engineering to perform a query rewriting process on the user input to generate a broader and more general query for the original query. For example, for the user input "Zhang San's joining department", the output of this step is "Which department did Zhang San join?";

[0068] Step 4-1-2, query decomposition: Use the prompt word engineering of the large model to perform query decomposition on the user input, decomposing the original query into simpler sub-queries to retrieve information more comprehensively. For example, for the user input "What are the departments of Zhang San and Li Si", the output of this step is "Zhang San's department, Li Si's department";

[0069] Step 4-1-3, merge the original query, query rewriting, and query decomposition results to output the final query;

[0070] Step 4-2, set the query parameters, use the output of 4-1 as the query input, execute step 3 index hybrid search and filtering, and obtain the block document d and context information;

[0071] Step 4-3, in order to improve the retrieval accuracy, the context information of the retrieved block document d is echoed, and by expanding the context information around the block document d, richer context information can be provided, thereby improving the accuracy of the retrieval result;

[0072] Step 4-4, performing a correlation check, specifically includes the following steps:

[0073] Step 4-4-1, based on question x, the block document d retrieved in step 4-2 and the context information, determine whether document d provides useful information for solving problem x, and output relevant or irrelevant; wherein the determination method is: take document d as known information input into the big model, and let the big model output whether document d is helpful for solving problem x, and return the specified label, where relevant means helpful and irrelevant means unhelpful;

[0074] Step 4-4-2, retain the document d corresponding to the relevant and record it as the block document d';

[0075] Step 4-5, perform support check, specifically including the following steps:

[0076] Step 4-5-1, based on question x, block document d', and answer y generated by the large model, determine whether answer y is supported by block document d', and output fully supported or no support; the large model generates answers by taking document d' as context information, and having the large model answer question x, with the result recorded as y. The large model outputs whether answer y is based on document d', and returns the specified label, fully supported means based on, and no support means not based on;

[0077] Step 4-5-2, keep the document d corresponding to fully supported and record it as d";

[0078] Step 4-6, determine whether d" is empty, if it is empty, return to execute step 4-1; if it is not empty, output the final result d".

[0079] The present invention also provides a multi-document retrieval system based on AI-agent, including an intelligent text segmentation module, a multi-language semantic vector conversion module, an index hybrid search and filtering module and an agent autonomous tuning module;

[0080] The intelligent text segmentation module is used to pre-process multi-source heterogeneous documents by pre-loading documents and analyzing the formats to identify the content of the documents. Based on natural language processing technology, the intelligent segmenter automatically breaks down large texts into smaller, more manageable text blocks. The module can recognize electronic files in different formats, such as PDF, Word, Markdown, etc., and intelligently segment them for subsequent processing.

[0081] The multilingual semantic vector conversion module is used to convert text content into vector representations to facilitate efficient semantic retrieval. In order to improve the performance of the model in a specific field, the general model is fine-tuned using data from vertical domains. This step ensures that the generated semantic vectors are closer to the domain-specific semantics. This module supports text conversion in multiple languages, ensuring that documents in different languages ​​can be processed and providing wider applicability. Finally, the text is converted into a fixed-length vector for subsequent retrieval and analysis.

[0082] The index hybrid search and filtering module uses a two-stage method of multiple retrieval techniques and re-ranking to perform preliminary retrieval on the vectorized text. This process considers the multi-dimensional correlation between the text and the retrieval topic, comprehensively evaluates the final retrieval results, and improves the retrieval efficiency.

[0083] The agent self-tuning module uses intelligent agent technology to achieve self-reflective learning retrieval and tuning; by rephrasing the query in various ways, the relevance, context and comprehensiveness of the retrieved information can be significantly improved; by analyzing historical retrieval results, the intelligent agent can identify deficiencies in the retrieval process and make self-adjustments; based on real-time feedback, the intelligent agent can dynamically adjust the retrieval and filtering strategies. This module not only focuses on the relevance of the text, but also considers multi-dimensional factors such as topic, context, user preferences, etc., to provide more comprehensive and accurate retrieval results.

[0084] The present invention has the following beneficial effects:

[0085] (1) Intelligent segmentation and recognition: Through intelligent text segmentation technology, electronic files in different formats can be efficiently and accurately segmented, improving the flexibility and convenience of document processing.

[0086] (2) Multilingual compatibility: The multilingual semantic vector conversion module can process documents in multiple languages. By fine-tuning the general embedding model, the adaptability of the retrieval method to different language environments is enhanced.

[0087] (3) Efficient retrieval and filtering: The index hybrid search and filtering module adopts a two-stage retrieval method to improve retrieval efficiency. It fully considers the relevance of text and topic, and optimizes the multi-dimensional hybrid retrieval results to improve the retrieval quality, providing users with more accurate information retrieval services.

[0088] (4) Self-learning and tuning: The self-tuning module of the AI-agent can self-learn and optimize the retrieval process through a reflection mechanism, thus enhancing the adaptability and robustness of the system.

[0089] (5) Improve work efficiency: Overall, the present invention greatly improves the work efficiency of document retrieval and management through automated and intelligent processing procedures, reducing the necessity of manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0091] Figure 1 It is a structural composition diagram of the system of the present invention.

[0092] Figure 2 It is the work flow chart of the system of the present invention. DETAILED DESCRIPTION

[0093] In the embodiment of the present invention, a multi-document retrieval method based on AI-agent is specifically provided. Figure 1 As shown, it includes intelligent text segmentation, multi-language semantic vector conversion, index hybrid search and filtering, and agent autonomous tuning.

[0094] The intelligent text segmentation is a pre-processing of multi-source heterogeneous documents. By analyzing its format and pre-loading the document, the module uses natural language processing technology to automatically break down long text into smaller and more manageable text blocks through an intelligent segmenter. It can recognize and process electronic files in different formats, such as PDF, Word, and Markdown, and realize intelligent segmentation for subsequent processing.

[0095] The multilingual semantic vector conversion is to convert text information into vector form to achieve fast and efficient semantic search. It supports fine-tuning the general model using data from specific industries. This fine-tuning process ensures that the output semantic vector is more in line with the semantic requirements of specific fields. In addition, the module has multilingual processing capabilities and can handle documents in various languages, thereby achieving its wide range of applications.

[0096] The index hybrid search and filtering includes: hybrid retrieval and re-ranking.

[0097] The hybrid search can quickly and accurately search for text fragments related to user input. It uses a search technology that combines two search algorithms to improve the relevance of search results. It can combine keywords and semantics for simultaneous retrieval. This hybrid search method can improve the accuracy and comprehensiveness of the search while ensuring search efficiency.

[0098] The reranking is used to semantically rerank the output of the previous step. The reranking model improves the result of semantic ranking by reranking the candidate document list according to the semantic matching degree of the user question. The principle is to calculate the relevance score between the user question and each given candidate document, and return a document list sorted from high to low by relevance.

[0099] The agent self-tuning introduces a self-reflective retrieval strategy to improve the quality of retrieval, including its factual accuracy, without compromising its generality through on-demand retrieval and self-reflection. Specifically, given an input prompt, it transforms the user query to make it clearer and more specific. This part evaluates the relevance of the user input to the retrieval fragment and then generates the corresponding task output. Then, it generates a critical mark based on it to criticize its own output and select the best output in terms of factuality and overall quality.

[0100] like Figure 2 The flowchart of the present embodiment is shown, which specifically includes the following steps:

[0101] Step a1, read the electronic file and upload the electronic document sample to be retrieved, for example, taking the excerpt of "Yongshanqiao Employee Handbook.pdf", the labor contract-Zhang San.pdf, and the labor contract-Li Si.pdf documents as examples, the samples all contain elements of tables, titles, and paragraphs.

[0102] Step a2, automatically select a loader for preloading according to the document format. For the document "Yongshanqiao Employee Handbook.pdf", a PDF Loader named "LlamaParseLoader" will be used to load the document and identify the tables, titles, and paragraphs therein. In order to restore the results in the retrieval stage, the loader needs to convert the recognized text into markdown format.

[0103] Step a3, splitting and dividing the loaded document, including the following steps:

[0104] Step a3-1, select a suitable text segmenter according to the configuration. For example, the selected tokenizer is RecursiveCharacterMarkdownSpliter, which will segment the loaded document according to the markdown text format.

[0105] Step a3-2, set the relevant parameters for file processing, including the character list, the maximum length of a single-segment document, and the overlapping length of adjacent blocks. For example, set the character list to ["\n\n","\n","",""], the maximum length of a single-segment document to 1024, and the overlapping length of adjacent blocks to 512.

[0106] Step a3-3, execute the block logic, and the processed results are shown in Table 1 below.

[0107] Table 1

[0108]

[0109] It can be seen that the current document is divided into two block documents, and Markdown is used as the output format of document parsing. The recognition result is consistent with the content of the original document.

[0110] Step a4, adding context information to the text blocks, specifically includes the following steps:

[0111] Step a4-1, according to the context window size of 2, select the current block and the previous and next block information.

[0112] Step a4-2, merging the selected information and storing it as the context metadata information of the current block, in order to reduce the semantic loss caused by the block.

[0113] Step a5: Based on the BGE model, the block documents are taken as input to generate corresponding semantic vectors.

[0114] Step a6: Use the vector library to store the generated semantic vector.

[0115] Step a7, use Jieba as a word segmenter to segment the segmented documents and remove stop words; for example, for "inappropriate appearance or inappropriate work clothes", the result of Jieba's word segmentation is: ['appearance', 'appearance', 'inappropriate', 'or', 'work clothes', 'wear', 'inappropriate', 'of'], and the output after removing stop words is: ['appearance', 'appearance', 'inappropriate', 'work clothes', 'wear', 'inappropriate'].

[0116] Step a8, taking the word segmentation result as input, build the BM25 retriever.

[0117] Step a9, taking the semantic vector stored in the vector library as input, constructing a vector retriever.

[0118] Step a10, assigning weights to the vector retriever and the BM25 retriever. In the present invention, the weights of the vector retriever and the BM25 retriever are set to 0.6 and 0.4 respectively, which means that the score of the vector retriever is considered more important in the final ranking.

[0119] Step a11, construct a hybrid index retriever, set the retrieval mode to hybrid retrieval, and set the number of retrieval matches to 5.

[0120] Step a12, searching the user input, the search text is "verbal warning", this embodiment is based on the RRF fusion sorting algorithm, and outputs the highest-scoring fragment information in different search engines;

[0121] Step a13, constructing a tuple of <user input, fragment information> based on the retrieved fragment information and the user input;

[0122] Step a14, based on the BGE Rerank model, similarity scores are scored and sorted for the two-tuple relationships;

[0123] Step a15, according to the search parameters, the sorted text fragments are evaluated and the final search results are output as shown in Table 2 below.

[0124] Table 2

[0125]

[0126]

[0127] You can see that the results include the block information, as well as the matching score between the current block and the user's search information, and are sorted by similarity.

[0128] After the agent self-tuning, this example does not filter out any retrieved text, because the agent believes that this result can be used as the optimal result. For the user's search text "Zhang San's entry department", the search results of the hybrid search module are shown in Table 3 below.

[0129] Table 3

[0130]

[0131] It can be seen that the search results contain noise information of irrelevant searches of Li Si, so relevant tuning work is required, which includes the following steps:

[0132] Step a15, query conversion is performed on the user input, and the conversion result of "Zhang San's joining department" is: "Which department did Zhang San join?".

[0133] Step a16, decompose the original query into simpler sub-queries to retrieve information more comprehensively.

[0134] Step a17, merge the original query and the query conversion result, and output the final query, which is: "Zhang San's joining department", "Which department did Zhang San join?";

[0135] Step a18, set query parameters and execute the retrieval process of a11-a15;

[0136] Step a19, perform a relevance check, based on the question "Zhang San's employment department" and the retrieved documents, determine the relevance of the retrieved documents to the retrieval question. In the example, the relevance of Li Si's irrelevant search to the retrieval question is output as "irrelevant", so it is filtered out. The final retrieval results are shown in Table 4 below, and the correct results are obtained.

[0137] Table 4

[0138]

[0139] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a multi-document retrieval method based on an AI-agent provided by the present invention and some or all of the steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0140] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0141] The present invention provides a multi-document retrieval method and system based on AI-agent. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A multi-document retrieval method based on AI-agent, characterized in that: The steps include: Step 1, intelligent text segmentation: pre-process multi-source heterogeneous documents, pre-load documents and analyze their formats, identify the content of the documents, and automatically decompose large texts into small text blocks through an intelligent segmenter based on natural language processing technology; Step 2: Multilingual semantic vector conversion: convert the text content into vector representation; Step 3, index hybrid search and filtering: use multiple retrieval techniques and a two-stage method of re-ranking to perform preliminary retrieval on the vectorized text; Step 4: Agent autonomous tuning: Use intelligent agent technology to achieve self-reflective learning retrieval and tuning.

2. The method according to claim 1, characterized in that In step 1, a document is preloaded using a document loader, wherein the document loader includes a PDF Loader loader, a Markdown loader, and a Word loader, and the content of the document includes text, a title, a table, a table title, and a reference.

3. The method according to claim 2, characterized in that The intelligent segmenter includes a recursive character text segmenter, a text semantic segmenter and a proposition segmenter, and the intelligent segmenter divides the text into small blocks; The recursive character text segmenter pre-segments the text according to the priority order of the separators by specifying a block length and a set of separators, segmenting the text into block documents of smaller length than the original document, then merging the block documents that do not exceed the specified block length, and recursively splitting the text that exceeds the specified block length until the desired block size is obtained; The text semantic segmenter segments the text by identifying breakpoints, and the breakpoints are determined in the following way: if the distance between the embedding vectors of two consecutive paragraphs exceeds a threshold, a breakpoint is set at the position between the two consecutive paragraphs; The proposition segmenter is a retrieval unit called proposition, which generates propositions by feeding a text into a large model, and then in the retrieval stage, the proposition is used as a mixed source to participate in the recall.

4. The method according to claim 3, characterized in that The PDF Loader is used to perform the following steps: Step 1-1, read the PDF file in pages; Step 1-2: Detect and perform OCR recognition on the text and text box. The OCR recognition is trained and fine-tuned based on the PaddleOCR model, and the text box coordinates and recognition results are returned. Step 1-3, analyze the layout of the document image and identify different types of areas; Steps 1-4, parsing the document content and processing the identified area, specifically include the following steps: Step 1-4-1, use the LORE table structure recognition model to identify elements in the table area. The LORE table structure recognition model combines the spatial position and logical position of table cells for end-to-end modeling and prediction. The LORE table structure recognition model uses a key point segmentation network based on a convolutional neural network to extract visual features, and predicts the spatial position and logical position of cells through two regression heads respectively; finally, the LORE table structure recognition model restores the row and column structure of the entire table and outputs a machine-parseable representation; Step 1-4-2, output the cell content and the cell's spatial position and logical position; Steps 1-5 integrate the results of layout analysis and OCR recognition, introduce a layout recovery process, and convert the document into markdown format for output, which includes the following steps: Step 1-5-1, group the tables, titles, and paragraphs output by the layout analysis; Step 1-5-2, merge the texts in the group according to their spatial and logical positions; Step 1-5-3, convert the merged text into markdown according to the type.

5. The method according to claim 4, characterized in that In step 1-2, the OCR recognition includes the following steps: Step 1-2-1, perform text detection on the document and obtain the text box coordinates; Step 1-2-2, rotate and crop each text box; Step 1-2-3, perform text recognition on the cropped document; Step 1-2-4, filter out the recognition results whose confidence level is lower than the threshold.

6. The method according to claim 5, characterized in that Step 1 includes the following steps: Step a1, reading the document; Step a2, automatically selecting a document loader for preloading according to the document format; Step a3, the intelligent segmenter segments the loaded document to obtain segmented documents; Step a4: adding context information to the block documents.

7. The method according to claim 6, characterized in that Step 2 includes the following steps: Step 2-1, based on the BGE model, take the block document as input and generate the corresponding semantic vector. The BGE model adopts the instruction fine-tuning method, which includes the following steps: Step 2-1-1, generate corpus, the fine-tuning dataset format is [query, positive sample set, negative sample set], where query is the query text, the positive sample set represents the text related to the query, and the negative sample set represents the text irrelevant to the query; Step 2-1-2, set fine-tuning parameters; Step 2-1-3, fine-tune the Embeding model to treat the semantic relevance task as a binary classification task, use the cross entropy loss function as the loss function, use back propagation to update the model parameters, and output the fine-tuned model; the cross entropy loss function is: Among them, H(p,q) is the cross entropy between probability distribution q and probability distribution p, p(x i ) represents the probability value of the i-th sample under the probability distribution, q(x i ) represents the probability value of the i-th sample under probability distribution q, and n is the total number of samples; Step 2-2, store the semantic vector and the original text into the vector library.

8. The method according to claim 7, characterized in that Step 3 includes the following steps: Step 3-1, use Jieba as a word segmenter to segment the block documents and remove stop words; Step 3-2, build the BM25 retriever. The BM25 retriever is based on word frequency and inverse document frequency, and introduces document length information to calculate the correlation between document D and query Q. The formula is: Among them, Score(D,Q) is the relevance score between document D and query Q; q i is the i-th word in the query; f(q i ,D) is q i The frequency in document D; IDF(q i ) is q i is the inverse document frequency; |D| is the length of document D; avgdl is the average length of all documents; k1 and b are constants; Step 3-3, constructing a vector retriever: Use IVF_FlAT as a vector retriever for data indexing and retrieval. The steps of IVF_FlAT index construction include: Step 3-3-1, the vectors in the vector library are clustered into two or more clusters, each cluster contains a group of similar vectors; Step 3-3-2, create an inverted file to save the identifier of each cluster and the corresponding vector list; Step 3-3-3, when searching, first find the cluster with the highest similarity to the query vector, then use the similarity to further search within the cluster with the highest similarity to find the final nearest neighbor; Step 3-4, assign weights to the vector retriever and BM25 retriever; Step 3-5, construct a fusion index searcher, which includes a vector searcher and a BM25 searcher, and combines the contents retrieved by the vector search and the BM25 search; set search parameters, which include the number of searched data and the search method, and the search method includes hybrid search and vector search; Step 3-6, retrieve the user input, and output the block document information with the highest score in the vector searcher and BM25 searcher based on the fusion sorting algorithm of inverse sorting and RRF; The reciprocal sorting fusion RRF formula is: in, Represents a collection of documents. Represents document d in the document collection R is the ranking of the retriever, which includes the vector retriever and the BM25 retriever, k is a constant, and r(d) is the ranking of document d in the rth retriever; Step 3-7, constructing a tuple of <user input, fragment information> based on the retrieved fragment information and the user input; Step 3-8: Based on the BGE Rerank model, the relationship between the two tuples is scored for similarity, and the text fragments are sorted in reverse order according to the similarity scores, thereby completing the sorting of the text fragments; Step 3-9, according to the search parameters, the sorted text fragments are evaluated, a specified number of text fragments are obtained, and the final search results are output.

9. The method according to claim 8, characterized in that Step 4 includes the following steps: Step 4-1, query conversion of user input, specifically includes the following steps: Step 4-1-1, query rewriting: Use the big model’s prompt word engineering to perform query rewriting on user input; Step 4-1-2, query decomposition: Use the prompt word engineering of the large model to perform query decomposition on the user input and decompose the original query into sub-queries; Step 4-1-3, merge the original query, query rewriting, and query decomposition results to output the final query; Step 4-2, set the query parameters, use the output of 4-1 as the query input, execute step 3 index hybrid search and filtering, and obtain the block document d and context information; Step 4-3, echoing the context information of the retrieved block document d; Step 4-4, performing a correlation check, specifically includes the following steps: Step 4-4-1, based on question x, the block document d retrieved in step 4-2 and the context information, determine whether document d provides useful information for solving problem x, and output relevant or irrelevant; wherein the determination method is: take document d as known information input into the big model, and let the big model output whether document d is helpful for solving problem x, and return the specified label, where relevant means helpful and irrelevant means unhelpful; Step 4-4-2, retain the document d corresponding to the relevant and record it as the block document d'; Step 4-5, perform support check, specifically including the following steps: Step 4-5-1, based on question x, block document d', and answer y generated by the large model, determine whether answer y is supported by block document d', and output fully supported or no support; the large model generates answers by taking document d' as context information, and having the large model answer question x, with the result recorded as y. The large model outputs whether answer y is based on document d', and returns the specified label, fully supported means based on, and no support means not based on; Step 4-5-2, keep the document d corresponding to fully supported and record it as d"; Step 4-6, determine whether d" is empty, if it is empty, return to execute step 4-1; if it is not empty, output the final result d".

10. A multi-document retrieval system based on an AI-agent implemented by the method according to any one of claims 1 to 9, characterized in that: It includes intelligent text segmentation module, multi-language semantic vector conversion module, index hybrid search and filtering module and agent autonomous tuning module; The intelligent text segmentation module is used to pre-process multi-source heterogeneous documents by pre-loading documents and analyzing the formats, identifying the contents of the documents, and automatically decomposing large texts into small text blocks through an intelligent segmenter based on natural language processing technology; The multilingual semantic vector conversion module is used to convert text content into vector representation; The index hybrid search and filtering module uses multiple retrieval techniques and a re-ranked two-stage method to perform preliminary retrieval on the vectorized text; The agent autonomous tuning module uses intelligent agent technology to achieve self-reflective learning retrieval and tuning.

Images