Method for improving recall rate of large model retrieval enhancement system based on knowledge density

Abstract

The invention provides a method for improving the recall rate of a large model retrieval enhancement system based on knowledge density. The method aims at solving the challenges of a traditional retrieval enhancement system in the aspects of document truncation, context integration and the like. The method comprises the following steps: firstly, intelligently segmenting a document into text blocks with medium lengths through an intelligent document segmentation technology, marking whether the text blocks and the text blocks can form a context or not, and further segmenting the text blocks into shortest text blocks; secondly, performing text vectorization processing by applying an embedded model, performing dense and sparse vector calculation on a text, and storing a result and a relationship between text blocks into a database; in addition, through preliminary mixed retrieval and score calculation based on knowledge density, most relevant medium-length text blocks are recalled accurately, a long document is formed, and background knowledge is formed. According to the method, accurate related contents can be provided, context information can be integrated, the recall rate of a large model enhanced retrieval system can be remarkably improved, and good balance is achieved between complexity and performance.

Description

Technical Field

[0001] The present invention relates to information retrieval technology, and in particular to a method for improving the recall rate of a large model retrieval enhancement system (RAG) based on knowledge density. Background Art

[0002] With the rapid development of information technology, especially the integration of artificial intelligence and big data technology, information retrieval systems have become an important tool for acquiring, processing and analyzing information. However, traditional RAG systems have gradually exposed some limitations when dealing with growing amounts of data and complex queries. For example, the document truncation problem: When faced with long documents, the system truncates the content due to technical limitations, resulting in the loss of key information and affecting the comprehensiveness and accuracy of the information. Context integration loss: When existing systems integrate information from different sources, they often find it difficult to maintain contextual coherence, resulting in semantic gaps in retrieval results. Summary of the invention

[0003] Purpose of the invention: In view of the above-mentioned problems existing in the existing RAG technology, the purpose of the present invention is to propose a method for improving the recall rate of a large model retrieval enhancement system based on knowledge density, which can provide accurate and relevant content and integrate contextual information to achieve information retrieval with a high recall rate.

[0004] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution: a method for improving the recall rate of a large model retrieval enhancement system based on knowledge density, comprising the following steps:

[0005] Divide the cleaned text according to semantic content, construct medium-length text blocks within a preset first length range, and set whether the tag mark can form a context with the following text;

[0006] The medium-length text block is further divided into a plurality of shortest text blocks within a preset second length range, and information of the source medium-length text block is recorded;

[0007] The embedding model is used to calculate the vectors of the shortest text blocks, and these vectors as well as the information of the shortest text blocks and the medium-length text blocks of the source are stored in the database;

[0008] Preliminarily recalling a preset first number of shortest text blocks related to the query from a database using a hybrid retrieval technique according to the query text;

[0009] Use the re-ranking model to calculate the similarity scores of the medium-length text blocks from the shortest text blocks in the initial recall, and calculate the final score by using the effective knowledge density as the weight;

[0010] The final scores are used for sorting, and the long document recall consisting of the second largest number of medium-length text blocks and their contexts is taken as the background knowledge transmitted to the large model.

[0011] Preferably, the method for cleaning the text includes:

[0012] Convert the received document into Markdown format;

[0013] According to the characteristics of the text, use the regularization format to filter out paragraphs that start with the specified text, and remove the header, footer, table of contents, and references based on the Markdown format;

[0014] Combine texts under the same topic into one paragraph.

[0015] Preferably, the following method is used to set labels for medium-length text blocks spanning different topics to indicate whether they can form a context with the following text: construct a role: "Expert in judging whether Chinese and English sentences can form a context" and an instruction: "Please check whether the following two sentences can form a context. If they can, return "yes", otherwise return "no". Please strictly ensure that only "yes" or "no" is returned", and use a large model to determine whether two sentences spanning different topics can form a context. If so, they are placed in the same medium-sized text block. If not, a new text block is used to store the text of the next topic, and the label "whether it can form a context with the following text" is used to record it.

[0016] Preferably, an embedding model is used to calculate dense and sparse vectors for the segmented shortest text blocks.

[0017] Preferably, the method of using the re-ranking model to calculate the similarity scores of the medium-length text blocks from the shortest text blocks initially recalled, and calculating the final scores by using the effective knowledge density as a weight, comprises:

[0018] By recalling the IDs of the preset first number of source medium-length text blocks, the number of times each different ID appears is calculated F i ;

[0019] Use the re-ranking model to calculate the similarity score S between the medium-length text block of the shortest text source in the preliminary recall and the query text i ;

[0020] Determine the effective knowledge density weight of a medium-length text block by L i / L max Calculate, where L i is the length of the medium-length text block, L max is the maximum length of all medium-length text blocks;

[0021] The final score for each medium-length text block is calculated using the following formula: i :

[0022]

[0023] Where α is the preset proportionality factor.

[0024] Preferably, the method of forming a long document of the selected medium-length text blocks and their contexts is as follows: first, a preset second number of medium-length text blocks with the top scores are selected according to the calculated final scores, and then, based on their label records of "whether they can form a context with the following text", the longest length text block within a preset third length range covering the context is constructed.

[0025] Preferably, the query text and the recalled background knowledge are transmitted to the large model by constructing instructions in a thought chain manner.

[0026] Preferably, the medium-length text block does not exceed 300 words, and the shortest text block does not exceed 150 words.

[0027] A computer system includes a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor. When the computer program / instruction is executed by the processor, the steps of the method for enhancing the recall rate of a large model retrieval system based on knowledge density are implemented.

[0028] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the method for enhancing the recall rate of a large model retrieval system based on knowledge density.

[0029] Beneficial effects: The present invention aims at improving the recall rate of the retrieval system in scenarios of large-scale data retrieval and complex query processing. To achieve this goal while optimizing resource allocation, advanced semantic segmentation and large language model technologies are used for information processing and retrieval enhancement to ensure the accuracy of retrieval results and the coherence of the context. First, the document is preprocessed through intelligent segmentation technology, and then the document content is converted into vector form using an embedding model and stored in a database. Further, through hybrid retrieval and a scoring strategy based on effective knowledge density, the most relevant documents to the query are accurately recalled. For the recalled documents, a large model question-and-answer strategy driven by a thinking chain is used to construct documents and query texts into instructions to guide a large language model for in-depth analysis and answers. Compared with the prior art, the present invention has the following advantages:

[0030] The method proposed in the present invention achieves a significant improvement in the recall rate and accuracy of the large model retrieval enhancement system through intelligent document segmentation, text recall strategy based on effective knowledge density, and large model question-answering strategy driven by thought chain. It optimizes the structured expression of documents through intelligent segmentation technology based on semantic segmentation, and solves the document truncation problem existing in the traditional large model retrieval enhancement system by ending with natural text end symbols such as period; while ensuring that the most relevant and accurate information is recalled first by using the text recall strategy based on effective knowledge density, the longest length text block within the preset third length range covering the context is constructed by using the label record of "whether it can form a context with the following text". Ensure that the context is not lost; at the same time, the analysis and answer quality of the large model is improved through the question-answering strategy driven by thought chain.

[0031] In summary, in the large model enhanced retrieval system, the present invention performs well in maintaining the coherence and consistency of retrieval results through intelligent segmentation based on semantic segmentation and recall strategy based on effective knowledge density, providing users with an efficient and accurate information retrieval solution, and at the same time providing new ideas and methods for the development of information retrieval technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a diagram of the overall steps of the method according to the embodiment of the present invention.

[0033] Figure 2 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0035] like Figure 1As shown, an embodiment of the present invention discloses a method for improving the recall rate of a large model retrieval enhancement system based on knowledge density, and the main steps include: dividing the cleaned text according to semantic content, constructing medium-length text blocks in a preset first length range, and setting whether the label mark can form a context with the following text; dividing the medium-length text block into multiple shortest text blocks in a preset second length range again, and recording the source medium-length text block information; using an embedding model to calculate vectors for the divided shortest text blocks, and storing these vectors and the shortest text blocks and the source medium-length text block information in a database; using a hybrid retrieval technology to preliminarily recall a preset first number of shortest text blocks related to the query from the database according to the query text; using a re-ranking model to calculate the similarity score of the medium-length text blocks from the source of the preliminarily recalled shortest text blocks, and calculating the final score by using the effective knowledge density as the weight; using the final score for sorting, taking the long document recall consisting of the preset second number of medium-length text blocks ranked first and their context as the background knowledge transmitted to the large model.

[0036] The present invention optimizes and enhances the information retrieval process through a series of innovative technical means. In specific implementation, first, advanced text recognition technology is used to convert documents of different formats into text formats that are easy to analyze. Then, irrelevant information is removed through data cleaning to ensure the purity of the text data. Then, the text is divided into multiple medium-length text blocks and their shortest text blocks according to semantic content using intelligent segmentation technology, and dense and sparse vector calculations are performed through an embedded model to convert the text content into vector form and store it in a database. Furthermore, combined with hybrid retrieval technology and a re-ranking model, texts related to the query are preliminarily recalled from the database, and a recall strategy based on knowledge density is used to optimize the order of documents. In addition, the analysis and answering quality of large models is improved through context integration technology and a question-and-answer strategy driven by a thinking chain.

[0037] The specific implementation steps of the embodiment of the present invention are further described in detail below.

[0038] Step (1), receiving and processing documents from different sources, including but not limited to PDF or other electronic formats, and using advanced text recognition technology such as Doc2X to convert these documents into Markdown format that is easy to analyze and process.

[0039] Step (2) is to perform a thorough data cleaning on the text converted by the text recognition technology, and remove all irrelevant information, including format tags, headers and footers, directories, etc., through a series of preset rules to ensure the purity of the text data. Specifically, the steps include:

[0040] In step (2.1), we first use the regularization format to filter out paragraphs that begin with specific text, such as captions, email addresses, and table titles, based on the characteristics of the text, and then remove the table of contents and references based on the Markdown format.

[0041] Step (2.2), based on the above cleaned text, use the characteristics of Markdown text to combine the text under the same topic into one paragraph.

[0042] Step (3) uses semantic analysis technology to divide the cleaned text into multiple medium-length text blocks and the shortest text block according to the semantic content, and sets whether the tag mark can form a context with the following text, which specifically includes the following steps:

[0043] Step (3.1), since the text contains both Chinese and English, the langid library is used to determine the language type of the text before segmentation.

[0044] Step (3.2), based on different languages, use spacy to load models of different languages ​​to segment the paragraph into multiple sentences based on semantics.

[0045] Step (3.3), based on the above multiple sentences, use tiktoken to call the tokenizer of GPT-3.5 to calculate the number of tokens in the sentence.

[0046] Step (3.4) constructs a medium-length text block of no more than 300 tokens, and constructs a label of "whether it can form a context with the following text" based on the same topic.

[0047] Step (3.5), for the case where a medium-sized text block may span different topics, use the constructed role: "Expert in judging whether Chinese and English sentences can form a context" and the instruction: "Please judge whether the following two sentences can form a context. If they can form a context, return "yes", otherwise return "no". Please strictly ensure that only "yes" or "no" is returned.", and use Qwen1.5-14B-Chat to judge whether two sentences spanning different topics can form a context. If they can, they are placed in the same medium-sized text block. If not, a new text block is used to store the text of the next topic, and the label "whether it can form a context with the following text" is used to record it.

[0048] Step (4), the segmented medium-length text block is further segmented into the shortest text block of no more than 150 words, and the ID of the medium-length text block from which it comes is recorded.

[0049] Step (5), using the embedding model to calculate dense and sparse vectors for the generated shortest text block, and storing these vectors, the shortest text block and the source medium-length text block ID in the database, including the following specific steps:

[0050] Step (5.1) uses embedding models such as BGE-M3 to calculate dense and sparse vectors for the shortest text block.

[0051] Step (5.2), store the dense and sparse vectors, the shortest length text block and the source medium length text block ID in the milvus database.

[0052] Step (6), based on the query text, using hybrid retrieval technology to preliminarily recall the shortest text block related to the query from the database, specifically includes the following steps:

[0053] In step (6.1), sparse and dense vectors are calculated for the query using an embedding model such as BGE-M3.

[0054] In step (6.2), the inner product is used to calculate the similarity between the query vector and the text vector in the database, and the 40 most similar shortest text blocks and their source medium text block IDs are recalled.

[0055]

[0056] Where A and B are the query vector and the text vector in the database respectively. i and B i are the i-th elements of vectors A and B respectively.

[0057] Step (7) uses a reranking model such as BGE-reranker-v2-M3 to calculate the similarity score of the medium-length document of the shortest text source initially recalled, and calculates the final score using the effective knowledge density as the weight, which specifically includes the following steps:

[0058] Step (7.1), calculate the number of times each different ID appears in the text block IDs of the 40 sources recalled F i ;

[0059]

[0060] Step (7.2) uses the re-ranking model to calculate the similarity score S for the medium-length text of the shortest text source in the initial recall i ;

[0061] S i =reranker(query,med_doc i )

[0062] where med_doc i It represents the medium-length document with ID i, and query represents the query text.

[0063] Step (7.3) determines the effective knowledge density weight of the medium-length text, through L i / L max Calculate, where L i is the length of the medium document, L max The maximum length among all medium-length documents is;

[0064] Step (7.4) According to the above steps, F i , S i and the weight L i / L max , the following formula is used to calculate the final score of each medium-length text, ensuring that the score can comprehensively reflect the relevance, similarity, and knowledge density of the text:

[0065]

[0066] Where α is a preset proportionality factor, which can be set to 0.25.

[0067] Step (8) uses the final scores to sort and takes the top five medium-length text blocks and their contexts as the long document recall as background knowledge, which specifically includes the following steps:

[0068] Step (8.1), after obtaining the final score, use the score to sort the relevant medium-length documents to obtain the top five medium-length text blocks.

[0069] Step (8.2), for each medium-length text block, use the label of "whether it can form a context with the following text" to construct the longest text block covering the upper and lower parts.

[0070] Step (9), transmitting the query text and the recalled background knowledge and the instructions constructed by the thought chain method to the big model;

[0071] Step (10), after receiving the instruction, the large model uses the ability of natural language processing based on the recalled relevant background knowledge to analyze the questions posed by the query text and give accurate responses to complete the entire retrieval enhancement process.

[0072] exist Figure 2In the paper, a flow chart of the method for improving the recall rate of the large model retrieval enhancement system based on knowledge density retrieval is described. First, the document and dense and sparse vectors are stored in the database using document intelligent segmentation technology; secondly, the dense and sparse vectors of the query are calculated using the embedding model and the recall strategy based on effective knowledge density is applied to recall relevant background knowledge; finally, the large model question-answering strategy driven by the thinking chain is used to complete the query response.

[0073] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the present invention proposes a method for improving the recall rate of a large model retrieval enhancement system based on knowledge density retrieval, which can significantly improve the recall rate of a large model enhanced retrieval system and achieve a good balance between complexity and performance.

[0074] A computer system disclosed in an embodiment of the present invention includes a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor. When the computer program / instruction is executed by the processor, the steps of the method for enhancing the recall rate of a large model retrieval system based on knowledge density in the aforementioned embodiment are implemented.

[0075] A computer program product disclosed in an embodiment of the present invention includes a computer program / instruction, which, when executed by a processor, implements the steps of the method for enhancing the recall rate of a large model retrieval system based on knowledge density improvement in the aforementioned embodiment.

[0076] The contents not described in detail in the present invention belong to the prior art known to professionals in the field.

Claims

1. A method to improve the recall rate of large model retrieval system based on knowledge density, It is characterized in that The following steps are involved: Divide the cleaned text according to semantic content, construct medium-length text blocks within a preset first length range, and set whether the tag mark can form a context with the following text; The medium-length text block is further divided into a plurality of shortest text blocks within a preset second length range, and information of the source medium-length text block is recorded; The embedding model is used to calculate the vectors of the shortest text blocks, and these vectors as well as the information of the shortest text blocks and the medium-length text blocks of the source are stored in the database; Preliminarily recalling a preset first number of shortest text blocks related to the query from a database using a hybrid retrieval technique according to the query text; Use the re-ranking model to calculate the similarity scores of the medium-length text blocks from the shortest text blocks in the initial recall, and calculate the final score by using the effective knowledge density as the weight; The final scores are used for sorting, and the long document recall consisting of the second largest number of medium-length text blocks and their contexts is taken as the background knowledge transmitted to the large model.

2. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that Methods for cleaning text include: Convert the received document into Markdown format; According to the characteristics of the text, use the regularization format to filter out paragraphs that start with the specified text, and remove the header, footer, table of contents, and references based on the Markdown format; Combine texts under the same topic into one paragraph.

3. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that The following method is used to set labels for medium-length text blocks spanning different topics to indicate whether they can form a context with the following text: Construct a role: "an expert in judging whether Chinese and English sentences can form a context" and instructions: "Please check whether the following two sentences can form a context. If they can, return "yes", otherwise return "no". Please strictly ensure that only "yes" or "no" is returned", and use a large model to determine whether two sentences spanning different topics can form a context. If so, they are placed in the same medium-sized text block. If not, use a new text block to store the text of the next topic and record it with the label "whether it can form a context with the following text".

4. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that The embedding model is used to calculate dense and sparse vectors for the segmented shortest text blocks.

5. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that The re-ranking model is used to calculate the similarity scores of the medium-length text blocks from the shortest text blocks initially recalled, and the final scores are calculated using the effective knowledge density as the weight, including: By recalling the IDs of the preset first number of source medium-length text blocks, the number of times each different ID appears is calculated. i ; Use the re-ranking model to calculate the similarity score S between the medium-length text block of the shortest text source in the preliminary recall and the query text i ; Determine the effective knowledge density weight of a medium-length text block by L i / L max Calculate, where L i is the length of the medium-length text block, L max is the maximum length of all medium-length text blocks; The final score for each medium-length text block is calculated using the following formula: i : Where α is the preset proportionality coefficient.

6. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that The method of forming a long document of the selected medium-length text blocks and their contexts is as follows: first, a preset second number of medium-length text blocks with the highest scores are selected according to the calculated final scores, and then, based on their label records of "whether they can form a context with the following text", the longest text block within a preset third length range covering the context is constructed.

7. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that The query text and the recalled background knowledge are constructed through thought chaining to transmit instructions to the big model.

8. The method for improving the recall rate of a large model retrieval enhancement system based on knowledge density according to claim 1, It is characterized in that The medium-length text block does not exceed 300 words, and the shortest text block does not exceed 150 words.

9. A computer system comprising a memory, a processor and a computer program / instructions stored in the memory and executable on the processor, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method for improving the recall rate of a large model retrieval system based on knowledge density according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the method for improving the recall rate of a large model retrieval system based on knowledge density according to any one of claims 1 to 7 are implemented.

Images