Multi-knowledge granularity text retrieval method and device for RAG

Abstract

The invention provides a multi-knowledge-granularity text retrieval method and device for RAG, and the method comprises the steps: sorting an original text into a JSON file, carrying out the recursive character and semantic dicing processing, generating a target JSON file, and storing the target JSON file in a BM25 database and a vector database; when a user queries, preprocessing to generate a keyword list and a query embedded vector; searching for a target text in the BM25 database, and calculating a score of correlation with the keyword to determine a first to-be-selected text; determining a second to-be-selected text according to the query embedding vector; and performing duplicate removal processing on all the first to-be-selected texts and the second to-be-selected texts, and determining and outputting a final recommended text by utilizing a sorting model. By considering the problem of different search granularities, the keyword granularity is solved by adopting a BM25 algorithm. According to the semantic retrieval strategy combining sentence granularity and segment granularity, challenges of different text information lengths and different retrieval purposes are effectively handled, and therefore the advantages of traditional retrieval and semantic retrieval are fully utilized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a text retrieval method and device for RAG with multiple knowledge granularities. Background Art

[0002] With the rapid development of information technology, the number of documents is huge and diverse, and traditional retrieval methods can no longer meet the needs of efficient and accurate information acquisition. This has promoted the development of text retrieval technology and has become an important part of information management. Text retrieval uses computers to match words in natural language to achieve rapid search of documents.

[0003] Currently, text retrieval systems use a variety of advanced algorithms and models, such as keyword matching, Boolean retrieval, vector space models, and deep learning technology. Although these methods have made some progress, they still have limitations. For example, keyword-based retrieval often relies on the accuracy of user input, which may cause relevant information to be missed. Different retrieval methods also set different search granularity when processing information: Boolean retrieval may be too strict, while loose keyword matching may return a large number of irrelevant results, making it difficult for users to find the information they need. In addition, the relevance-based ranking algorithms used by many retrieval systems may not fully reflect the true value of the results in specific fields or user preferences.

[0004] Therefore, improving the accuracy and flexibility of text retrieval and enhancing user experience are issues that need to be addressed urgently. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a text retrieval method and device with multiple knowledge granularities for RAG, so as to solve the problems of low accuracy and poor flexibility existing in the current text retrieval system.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of the present invention discloses a text retrieval method for RAG with multiple knowledge granularities, the method comprising:

[0008] Get the original text, and organize it into a structured format to get a JSON file;

[0009] Recursively perform character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generate a target JSON file based on the text blocks;

[0010] All target JSON files are stored in the BM25 database, and the JSON files after recursive character segmentation are stored in the vector database;

[0011] When receiving a query statement input by a user, preprocessing the query statement to obtain a keyword list and a query embedding vector corresponding to the query statement;

[0012] For each keyword in the keyword list, search for all target texts containing the keyword in the BM25 database, and calculate the relevance score between each target text and the keyword;

[0013] Sort all the relevance scores from large to small, and determine the target texts corresponding to the first n relevance scores as the first candidate texts;

[0014] Calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database respectively;

[0015] Determine the second candidate text according to all cosine similarities;

[0016] All first candidate texts and all second candidate texts are deduplicated, and a final recommended text is determined using a ranking model, and the final recommended text is output.

[0017] Preferably, the original text is obtained and structured to obtain a JSON file, including:

[0018] Get the original text;

[0019] Segmenting the original text according to each paragraph in the original text to obtain a plurality of paragraphs;

[0020] Creating a position index for each of the paragraphs according to the chapter sequence position to which each of the paragraphs belongs;

[0021] A JSON file is generated based on each of the paragraphs and its corresponding position index.

[0022] Preferably, recursively performing character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generating a target JSON file based on the text blocks, comprises:

[0023] Recursively perform character segmentation processing on the JSON data in the JSON file to obtain a plurality of preliminary text blocks, and create a first sub-index of each preliminary text block in the paragraph to which it belongs;

[0024] For each of the preliminary text blocks, creating a copy of the preliminary text block and identifying semantic units in the copy;

[0025] According to the semantic units, the copy is segmented based on a preset segmentation strategy to obtain semantic text blocks, and a second sub-index of each semantic text block in the paragraph to which it belongs is created;

[0026] A target JSON file is generated based on each of the preliminary text blocks and its corresponding first sub-index, and a target JSON file is generated based on each of the semantic text blocks and its corresponding second sub-index.

[0027] Preferably, storing all target JSON files into the BM25 database, and storing the JSON files after recursive character segmentation into the vector database, comprises:

[0028] For each of the target JSON files, preprocess the target JSON files, and calculate the word frequency of each word in the target JSON file and the frequency of occurrence of the word in all target JSON files to obtain an inverse document frequency;

[0029] Calculate the BM25 weight of the word based on the word frequency and the inverse document frequency;

[0030] Construct an inverted index list according to the BM25 weight of the word;

[0031] The preprocessed target JSON file and the corresponding inverted index list are stored in the BM25 database;

[0032] For each of the JSON files after the recursive character segmentation process, vectorize the JSON files after the recursive character segmentation process using a sentence embedding model to generate multiple high-dimensional vectors;

[0033] Identify each sentence corresponding to each of the high-dimensional vectors, generate a fine-grained vector and store it in a vector database;

[0034] The semantics corresponding to each of the high-dimensional vectors is identified, and each sentence corresponding to each of the high-dimensional vectors is clustered and grouped according to a preset clustering rule to obtain a plurality of coarse-grained category sets and store them in the vector database.

[0035] Preferably, when receiving a query statement input by a user, preprocessing the query statement to obtain a keyword list and a query embedding vector corresponding to the query statement includes:

[0036] When receiving a query statement input by a user, performing word segmentation and denoising processing on the query statement to obtain a keyword list corresponding to the query statement;

[0037] The query statement is vectorized using a sentence embedding model to obtain a query embedding vector corresponding to the query statement.

[0038] Preferably, the cosine similarity includes sentence-level cosine similarity and paragraph-level cosine similarity;

[0039] The step of determining the second text to be selected according to all cosine similarities includes:

[0040] Sort all sentence-level cosine similarities from largest to smallest;

[0041] Extract the first m sentence-level cosine similarities from all the sorted sentence-level cosine similarities, and determine the texts corresponding to the first m sentence-level cosine similarities as the sentence-level candidate texts;

[0042] Sort all paragraph-level cosine similarities from largest to smallest;

[0043] Extract the first m paragraph-level cosine similarities from all the sorted paragraph-level cosine similarities, and determine the texts corresponding to the first m paragraph-level cosine similarities as the paragraph-level candidate texts;

[0044] The sentence-level candidate text and the paragraph-level candidate text are determined as the second candidate text.

[0045] A second aspect of the present invention discloses a text retrieval device with multiple knowledge granularities for RAG, the device comprising:

[0046] The acquisition unit is used to acquire the original text and organize it into a structured form to obtain a JSON file;

[0047] A processing unit, configured to perform recursive character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generate a target JSON file based on the text blocks;

[0048] A storage unit, used for storing all target JSON files into the BM25 database, and storing the JSON files after recursive character segmentation processing into the vector database;

[0049] A preprocessing unit, configured to preprocess a query statement input by a user to obtain a keyword list and a query embedding vector corresponding to the query statement when receiving the query statement input by the user;

[0050] A search unit, configured to search, for each keyword in the keyword list, all target texts containing the keyword in the BM25 database, and calculate a relevance score between each target text and the keyword;

[0051] A first determination unit is used to sort all the relevance scores from large to small, and determine the target texts corresponding to the first n relevance scores as the first texts to be selected;

[0052] A calculation unit, used to calculate cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database respectively;

[0053] A second determination unit, used for determining a second text to be selected according to all cosine similarities;

[0054] The output unit is used to perform deduplication processing on all the first candidate texts and all the second candidate texts, determine the final recommended text by using the ranking model, and output the final recommended text.

[0055] Preferably, the acquisition unit includes:

[0056] The acquisition module is used to obtain the original text;

[0057] A segmentation module, used for segmenting the original text according to each paragraph in the original text to obtain multiple paragraphs;

[0058] A first creation module, used for creating a position index for each paragraph according to the position of the chapter sequence to which each paragraph belongs;

[0059] The first generating module is used to generate a JSON file based on each of the paragraphs and its corresponding position index.

[0060] Preferably, the processing unit comprises:

[0061] A first processing module is used to perform recursive character segmentation processing on the JSON data in the JSON file to obtain a plurality of preliminary text blocks, and to create a first sub-index of each preliminary text block in the paragraph to which it belongs;

[0062] A second creation module, for creating a copy of each of the preliminary text blocks, and identifying semantic units in the copy;

[0063] A block cutting module, used for cutting the copy into blocks according to the semantic units and based on a preset block cutting strategy to obtain semantic text blocks, and creating a second sub-index of each semantic text block in the paragraph to which it belongs;

[0064] The second generation module is used to generate a target JSON file based on each of the preliminary text blocks and its corresponding first sub-index, and to generate a target JSON file based on each of the semantic text blocks and its corresponding second sub-index.

[0065] Preferably, the storage unit comprises:

[0066] A first calculation module is used to preprocess each target JSON file, and calculate the word frequency of each word in the target JSON file and the frequency of occurrence of the word in all target JSON files to obtain an inverse document frequency;

[0067] A second calculation module, used for calculating the BM25 weight of the word based on the word frequency and the inverse document frequency;

[0068] A construction module, used to construct an inverted index list according to the BM25 weight of the word;

[0069] A first storage module, used to store the preprocessed target JSON file and the corresponding inverted index list into a BM25 database;

[0070] A second processing module is used to perform vectorization processing on each of the JSON files after the recursive character segmentation processing by using a sentence embedding model to generate multiple high-dimensional vectors;

[0071] A recognition module, used to recognize each sentence corresponding to each of the high-dimensional vectors, generate a fine-grained vector and store it in a vector database;

[0072] The second storage module is used to identify the semantics corresponding to each of the high-dimensional vectors, and cluster and group each sentence corresponding to each of the high-dimensional vectors according to a preset clustering rule to obtain multiple coarse-grained category sets and store them in the vector database.

[0073] Based on the above-mentioned embodiment of the present invention, a text retrieval method for RAG with multiple knowledge granularities is provided, the original text is organized into a JSON file, and recursive character and semantic segmentation processing is performed to generate a target JSON file and store it in the BM25 database and the vector database. When a user queries, the preprocessing generates a keyword list and a query embedding vector; the target text in the BM25 database is searched, and the relevance score with the keyword is calculated to determine the first candidate text; the second candidate text is determined according to the query embedding vector; all the first candidate texts and the second candidate texts are deduplicated, and the final recommended text is determined and output using the sorting model. By considering the problem of different search granularities, the BM25 algorithm is used to solve the keyword granularity. Combining the semantic retrieval strategy of sentence granularity and paragraph granularity, it effectively responds to the challenges of different lengths of text information and different retrieval purposes, thereby making full use of the advantages of traditional retrieval and semantic retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0075] Figure 1 A flowchart of a text retrieval method with multiple knowledge granularities for RAG provided by an embodiment of the present invention;

[0076] Figure 2 A flow chart for structuring the original text provided by an embodiment of the present invention;

[0077] Figure 3 A flowchart of recursive character segmentation and semantic segmentation of JSON data provided by an embodiment of the present invention;

[0078] Figure 4 A flowchart of a storage process provided by an embodiment of the present invention;

[0079] Figure 5 A schematic diagram of a search query process provided by an embodiment of the present invention;

[0080] Figure 6 A structural block diagram of a text retrieval device with multiple knowledge granularities for RAG provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0083] As can be seen from the background technology, keyword-based retrieval often relies on the accuracy of user input, which may cause relevant information to be missed. Different retrieval methods also set different search granularity when processing information: Boolean retrieval may be too strict, while loose keyword matching may return a large number of irrelevant results, making it difficult for users to find the information they need. In addition, the relevance-based ranking algorithms used by many retrieval systems may not fully reflect the true value of the results in specific fields or user preferences.

[0084] Therefore, an embodiment of the present invention provides a text retrieval method for RAG with multiple knowledge granularities. First, the original text is organized into a JSON file, and recursive character and semantic segmentation processing is performed to generate a target JSON file, which is stored in the BM25 database and the vector database. When a query statement input by a user is received, a keyword list and a query embedding vector are generated by preprocessing; the target text in the BM25 database is searched, and the correlation score with the keyword is calculated to determine it as the first candidate text; the second candidate text is determined according to the query embedding vector; all first candidate texts and all second candidate texts are deduplicated, and the final recommended text is determined and output using a ranking model. By considering the problem of different search granularities, the BM25 algorithm is used to solve the keyword granularity, and the semantic retrieval strategy of sentence granularity and paragraph granularity is combined to effectively cope with the challenges of different lengths of text information and different retrieval purposes, thereby making full use of the advantages of traditional retrieval and semantic retrieval. Finally, through the application of the hybrid retrieval method, the present invention effectively realizes the combination of keyword retrieval and semantic retrieval, and improves the overall system performance.

[0085] See also Figure 1 , showing a flowchart of a text retrieval method with multiple knowledge granularities for RAG provided by an embodiment of the present invention.

[0086] It should be noted that the multi-knowledge granularity text retrieval method for RAG includes three parts: text processing, text storage and retrieval query. The following is an explanation of these three parts:

[0087] Step S101: Obtain the original text and organize it into a structured form to obtain a JSON file.

[0088] In the specific implementation of step S101, the original text is obtained, which may be the original text input by the user, and the original text is structured and sorted to obtain a JSON file. The specific sorting process is as follows: Figure 2 As shown, Figure 2 Included:

[0089] Step S201: Obtain original text.

[0090] The text may be a document.

[0091] In the specific implementation of step S201, the original text to be processed is obtained.

[0092] Step S202: Segment the original text according to each paragraph in the original text to obtain a plurality of paragraphs.

[0093] In the specific implementation of step S202 , the original text is segmented according to the paragraph structure in the original text to obtain a plurality of paragraphs.

[0094] Step S203: creating a position index for each paragraph according to the chapter sequence position to which each paragraph belongs.

[0095] In the specific implementation of step S203 , a corresponding position index is created for each paragraph according to the chapter sequence position to which each paragraph belongs in the original text.

[0096] For example, the position index of the second paragraph of the first chapter in the original text is set to 1.2; the position index of the first paragraph of the third chapter in the original text is set to 3.1.

[0097] It is understandable that in normal writing, there are usually significant differences in semantics between chapters and paragraphs, and each paragraph has its own unique role. Therefore, by building an index for each paragraph, they can be distinguished from other paragraphs. In the subsequent sentence context recovery, only the relevant paragraphs are recovered instead of the adjacent paragraphs, which can effectively reduce the length of the recovery.

[0098] In some embodiments, if it is detected that some paragraphs in the original text have corresponding subheadings, the subheadings are used as metadata of the original text and bound to the corresponding paragraphs, so that a JSON file can be subsequently generated together with the paragraphs and their corresponding position indexes.

[0099] Step S204: Generate a JSON file based on each paragraph and its corresponding position index.

[0100] In the specific implementation of step S204, a JSON file is generated according to each paragraph and its corresponding position index.

[0101] Understandably, Figure 2 The content shown is the first part of the text retrieval method provided by the embodiment of the present invention: the content of text preprocessing in text processing. Preprocessing the original text can better store the text in the database. The content of preliminary segmentation and semantic segmentation in text processing is explained below in conjunction with the corresponding content of step S102.

[0102] Step S102: recursively perform character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generate a target JSON file based on the text blocks.

[0103] In the specific implementation of step S102, the JSON data in the JSON file is recursively segmented by characters and semantics to obtain a plurality of text blocks containing sub-indexes, wherein the text blocks include preliminary text blocks and semantic text blocks, and the target JSON file is generated according to the text blocks. The specific segmentation process is as follows: Figure 3 As shown, Figure 3 Included:

[0104] Step S301: recursively perform character segmentation processing on the JSON data in the JSON file to obtain multiple preliminary text blocks, and create a first sub-index of each preliminary text block in its own paragraph.

[0105] In the specific implementation process of step S301, the JSON data in the JSON file is recursively segmented according to a preset string length to obtain multiple preliminary text blocks; and according to the paragraph to which each preliminary text block belongs, the first sub-index of the preliminary text block in the paragraph to which it belongs is created.

[0106] For example, the JSON data in the JSON file is cut into blocks of 500 characters to obtain multiple preliminary text blocks, each of which has a length of 500 characters.

[0107] It can be understood that the purpose of performing recursive character segmentation is to divide the text into smaller units to facilitate subsequent recursive refinement.

[0108] For another example, if a preliminary text block belongs to the first position of the second paragraph of the first chapter in the text, the first sub-index is set to 1.2.1.

[0109] It should be noted that creating a first sub-index for each preliminary text block helps ensure that the context of the sentence can be effectively restored in subsequent retrieval queries.

[0110] It can be understood that the content shown in step S301 is the first part of the text retrieval method provided by the embodiment of the present invention: the content of the initial segmentation in text processing.

[0111] Step S302: For each preliminary text block, create a copy of the preliminary text block and identify semantic units in the copy.

[0112] In practical applications, it is usually judged whether the preliminary text block is longer than the user query. Specifically, if the effective information only accounts for one tenth or even one twentieth of the preliminary text block (500 words in length), the subsequent text vectorization process will further weaken the effective information, making it difficult to retrieve correctly, so further segmentation is required. At this time, semantic segmentation is used to segment each preliminary text block according to semantic similarity to ensure the semantic integrity of the segmentation.

[0113] In the specific implementation process of step S302, for each preliminary text block, a copy of the preliminary text block is created, and semantic units in the copy are identified, including identifying sentences, phrase structures, named entities (such as names of people, places, organizations, etc.) and topics in the copy.

[0114] It should be noted that the semantic segmentation of the preliminary text block is performed on a copy of the preliminary text block.

[0115] Step S303: According to the semantic units, the copy is segmented based on a preset segmentation strategy to obtain semantic text blocks, and a second sub-index of each semantic text block in the paragraph to which it belongs is created.

[0116] In the process of implementing step S303, the copy is segmented according to the semantic units based on a preset segmentation strategy (such as a specific segmentation rule, a machine learning strategy or a deep learning strategy) to obtain semantic text blocks, and according to the paragraph to which each semantic text block belongs, a second sub-index of the semantic text block in the paragraph to which it belongs is created.

[0117] It can be understood that creating a second sub-index for each semantic text block also helps to ensure that the context of the sentence can be effectively restored in subsequent retrieval queries.

[0118] Step S304: Generate a target JSON file based on each preliminary text block and its corresponding first sub-index, and generate a target JSON file based on each semantic text block and its corresponding second sub-index.

[0119] In the specific implementation of step S304, a target JSON file is generated based on each preliminary text block and a first sub-index corresponding to each preliminary text block, and a target JSON file is generated based on each semantic text block and a second sub-index corresponding to each semantic text block.

[0120] It is understandable that, in combination with step S102 and Figure 3The content shown in the figure has explained the first part of the text retrieval method provided by the embodiment of the present invention: the entire content of text processing has been explained. In the embodiment of the present invention, the method of combining recursive character segmentation and semantic segmentation is adopted to improve the efficiency and accuracy of text processing while maintaining the semantic integrity of the text. In addition, in the segmentation process, metadata is used to retain the position of the segment relative to the original text, so that the original data can be quickly located during the retrieval process.

[0121] The second part of the text retrieval method provided in the embodiment of the present invention: text storage is explained below.

[0122] It is understandable that BM25 is a retrieval algorithm based on word frequency and document frequency. Compared with semantic retrieval, it has advantages in efficiency and accuracy of keyword search. BM25 effectively captures the importance of words and improves retrieval accuracy by scoring them according to the frequency of their occurrence in documents and their weight in the entire document collection. However, BM25 also faces the problem of sparse data. Many words may only appear in a few documents, resulting in too low weights, which may lead to the loss of important information. In comparison, semantic vectors can better capture the semantic relationship between words, understand user query intent, and handle polysemous words and synonyms through machine learning and natural language processing technology, thereby improving the accuracy and comprehensiveness of retrieval.

[0123] Therefore, the embodiment of the present invention uses the BM25 database and the vector database for storage at the same time, which can combine the high efficiency of traditional word frequency retrieval with the deep understanding of semantic retrieval, and significantly improve the accuracy and relevance of the retrieval.

[0124] Step S103: All target JSON files are stored in the BM25 database, and the JSON files after the recursive character segmentation process are stored in the vector database.

[0125] In the specific implementation of step S103, all target JSON files are stored in the BM25 database, and the JSON files after the recursive character segmentation process are stored in the vector database.

[0126] Specific stored procedures such as Figure 4 As shown, Figure 4 Included:

[0127] Step S401: for each target JSON file, preprocess the target JSON file, and calculate the word frequency of each word in the target JSON file and the frequency of the word appearing in all target JSON files to obtain the inverse document frequency.

[0128] In the specific implementation process of step S401, for each target JSON file, the target JSON file is preprocessed (such as word segmentation, stop word removal, stem extraction, etc.), and the word frequency of each word in the preprocessed target JSON file and the frequency of each word appearing in all target JSON files are calculated to obtain the inverse document frequency (IDF).

[0129] It should be noted that the frequency of each word in the target JSON file after preprocessing is calculated according to formula (1).

[0130]

[0131] Among them, TF is the frequency of each word in the preprocessed target JSON file; d is the preprocessed target JSON file (i.e., text), t is the term in the query; f(t,d) is the frequency (TF) of term t in text d; |d| is the length of text d (number of words); avgdl is the average text length of the entire text set; k1 and b are adjustable parameters. For example, k1 = 1.2, b = 0.75.

[0132] It should be noted that k1 is used to control the degree of word frequency saturation, that is, as f(t,d) increases, the speed of weight increase gradually decreases. b is used to control the normalization of document length. b = 1 fully considers the document length, and b = 0 ignores the document length.

[0133] It can be understood that the calculation of the inverse document frequency is specifically performed according to formula (2).

[0134]

[0135] Among them, IDF(t) is the inverse document frequency; N is the total number of documents in the document set; n(t) is the number of documents containing the term t.

[0136] Step S402: Calculate the BM25 weight of the word based on the word frequency and inverse document frequency.

[0137] In the specific implementation of step S402, the BM25 weight of the word is calculated based on the word frequency and the inverse document frequency, as well as formula (3).

[0138] ω(t,d)=IDF(t)*TF(t,d)(3)

[0139] Among them, ω(t,d) is the BM25 weight of the word; IDF(t) is the inverse document frequency; TF(t,d) is the frequency of term t in text d.

[0140] Step S403: construct an inverted index list according to the BM25 weight of the word.

[0141] In the specific implementation of step S403, for each word in the preprocessed target JSON file, all paragraphs containing the word (paragraphs in the preprocessed target JSON file) are searched, and an inverted index list of the word is constructed based on all paragraphs and the BM25 weight of the word.

[0142] It should be noted that in the inverted index list of the word, the entry corresponding to each word contains the index information of the paragraph, that is, the identifier pointing to the specific paragraph position. This helps to quickly locate the relevant content and improve the retrieval speed.

[0143] It is understandable that each word in the inverted index list is accompanied by the word's BM25 weight, which reflects the importance of the word in a specific paragraph. The BM25 weight takes into account factors such as word frequency and document frequency, and can help the system evaluate the relevance of a paragraph to a query.

[0144] Step S404: Store the preprocessed target JSON file and the corresponding inverted index list into the BM25 database.

[0145] In the specific implementation of step S404, the preprocessed target JSON file and the inverted index list corresponding to the preprocessed target JSON file are stored in the BM25 database.

[0146] Step S405: for each JSON file after the recursive character segmentation process, use the sentence embedding model to perform vectorization processing on the JSON file after the recursive character segmentation process to generate multiple high-dimensional vectors.

[0147] In the specific implementation of step S405, for each JSON file after recursive character segmentation, that is, Figure 3 The preliminary text chunks in the sentence embedding model (such as BGE 1 Sentence embedding model) vectorizes the preliminary text blocks and generates multiple high-dimensional vectors.

[0148] Step S406: Identify each sentence corresponding to each high-dimensional vector, generate a fine-grained vector and store it in the vector database.

[0149] In the specific implementation process of step S406, each sentence corresponding to each high-dimensional vector is identified, and a fine-grained vector is generated for each sentence separately and stored in the vector database.

[0150] It is understandable that fine-grained dicing of high-dimensional vectors and then storing them in a vector database will help with subsequent applications in queries that require precise matching of specific semantics, and the fine-grained dicing storage method can provide more fine-grained semantic information.

[0151] Step S407: Identify the semantics corresponding to each high-dimensional vector, and cluster and group each sentence corresponding to each high-dimensional vector according to a preset clustering rule to obtain multiple coarse-grained category sets and store them in a vector database.

[0152] In the specific implementation of step S407, the semantics corresponding to each high-dimensional vector is identified, and several sentences corresponding to each high-dimensional vector are grouped into a larger coarse-grained category set according to a preset clustering rule, and then all the coarse-grained category sets are stored in the vector database.

[0153] In the specific application process, the corresponding semantics of each high-dimensional vector is identified, and several sentences are merged into a larger text block (i.e., a coarse-grained category set) according to the topic or context (or which sentences can be merged is determined based on criteria such as semantic similarity, thematic consistency, or chronological order), and then multiple text blocks are obtained.

[0154] For example, a series of related sentences can be grouped together to form a paragraph.

[0155] It is understandable that coarse-grained dicing of high-dimensional vectors and then storing them in a vector database can help with subsequent queries that require understanding of contextual semantics, and the coarse-grained dicing storage method can provide more complete semantic information.

[0156] So far, the vector database includes but is not limited to sentence-level embedding vectors and paragraph-level vectors.

[0157] It should be noted that, combined with Figure 4 The content shown in the figure has explained the entire content of the second part of the text retrieval method provided by the embodiment of the present invention: text storage. In the embodiment of the present invention, BM25 scores words according to their frequency of occurrence in the document and their weight in the entire document collection, thereby effectively capturing the importance of words and improving retrieval accuracy. By using two different granularities for document segmentation and storage in the vector database, the system can select the most suitable retrieval granularity according to the query requirements, thereby improving the accuracy and relevance of the retrieval.

[0158] The third part of the text retrieval method provided in the embodiment of the present invention: retrieval query is explained below.

[0159] Step S104: When a query statement input by a user is received, the query statement is preprocessed to obtain a keyword list and a query embedding vector corresponding to the query statement.

[0160] In the specific implementation of step S104, when a query statement input by a user is received, the query statement is segmented and denoised, and the processed query statement is sorted into a list to obtain a keyword list corresponding to the query statement; the query statement is vectorized using a sentence embedding model (such as a bge-sentence-embedding model) to obtain a query embedding vector (text embedding) corresponding to the query statement.

[0161] Combination Figure 5 The schematic diagram of the retrieval query process shown is that after the user enters the query statement, the query statement is segmented, stop word processing is performed, and vectorization processing is performed.

[0162] Step S105: For each keyword in the keyword list, search for all target texts containing the keyword in the BM25 database, and calculate the relevance score between each target text and the keyword.

[0163] In the specific implementation process of step S105, for each keyword in the keyword list, all target texts (such as corpora) containing the keyword are searched from the BM25 database, the BM25 weight is calculated for each keyword, and the relevance score of each target text and the keyword is calculated.

[0164] It should be noted that in the process of calculating the BM25 weight for each keyword, BM25 will comprehensively consider the number of times the keyword appears in the target text (word frequency) and the rarity of the keyword in the entire target text set (IDF), and calculate the contribution of the keyword to each text. The specific calculation is shown in formula (4).

[0165]

[0166] Among them, q is the query; d is the target text; t is the keyword; ω(t,d) is the BM25 weight.

[0167] Step S106: sort all the relevance scores from large to small, and determine the target texts corresponding to the first n relevance scores as the first texts to be selected.

[0168] In the specific implementation of step S106, all relevance scores are sorted in descending order, the first n relevance scores are extracted from all sorted relevance scores, and the target text corresponding to the first n relevance scores is determined as the first candidate text.

[0169] Step S107: Calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database respectively.

[0170] In the specific implementation of step S107, the sentence-level cosine similarity between the query embedding vector and the sentence-level embedding vector in the vector database is calculated, and the paragraph-level cosine similarity between the query embedding vector and the paragraph-level embedding vector in the vector database is calculated.

[0171] Step S108: Determine the second text to be selected according to all cosine similarities.

[0172] In the process of specifically implementing step S108, all sentence-level cosine similarities are sorted in descending order; the first m sentence-level cosine similarities are extracted from all the sorted sentence-level cosine similarities, and the texts corresponding to the first m sentence-level cosine similarities are determined as sentence-level candidate texts; all paragraph-level cosine similarities are sorted in descending order; the first m paragraph-level cosine similarities are extracted from all the sorted paragraph-level cosine similarities, and the texts corresponding to the first m paragraph-level cosine similarities are determined as paragraph-level candidate texts; the sentence-level candidate text and the paragraph-level candidate text are determined as the second candidate text.

[0173] In practical applications, for text selection at the sentence level, it is necessary to consider that a single sentence may not fully express its meaning, so it is necessary to consider the content before and after the sentence, as well as the position index of each sentence in the original document. The position index can be used to query through metadata (such as paragraph index) to extract relevant contextual content to restore the complete paragraph. This not only helps users better understand the sentence, but also provides a richer information background, thereby improving the readability and comprehension of the information.

[0174] It should be noted that the process of determining the first candidate text and the process of determining the second candidate text are parallel, and the execution order in the embodiment of the present invention is only used for illustration and does not limit the specific execution order.

[0175] Step S109: performing deduplication processing on all first candidate texts and all second candidate texts, and using the ranking model to determine the final recommended text, and outputting the final recommended text.

[0176] In the process of implementing step S109, all the first candidate texts and all the second candidate texts are deduplicated. That is to say, there will be some repeated retrieval results in the target texts corresponding to the first n relevance scores, the texts corresponding to the first m paragraph-level cosine similarities, and the texts corresponding to the first m sentence-level cosine similarities. After deduplication, a candidate set of recommended texts is obtained, and a ranking model (such as the open source model BGE-ReRanker-v2-MiniCPM) is used to further filter the candidate set of recommended texts, determine the final recommended text, and output the final recommended text.

[0177] It should be noted that the specific sorting process is: first, treat each document in the query and candidate document set as a text pair (query: candidate document); then, design a prompt word to ask the big model whether the "candidate document" in the text pair (query: candidate document) can be used as an answer to the "query", so that the big model only outputs "yes" or "no"; finally, combine the text pair and the prompt word, input it into the MiniCPM re-ranking model, and take the logit corresponding to the last token of the last layer as the ranking score to rank the candidate documents.

[0178] It is understandable that the results after deduplication are sent to the MiniCPM re-ranking model for ranking. The sorted results are arranged in descending order of ranking scores to ensure that the most relevant content is displayed to users first.

[0179] It should be noted that by combining similarity calculations at the document level and sentence level, multi-level search results are achieved to ensure the global relevance and local accuracy of information, thereby improving the comprehensiveness and accuracy of search results. In addition, the MiniCPM re-ranking model is used for intelligent sorting, making the relevance of search results more accurate, further improving user satisfaction. Overall, the search query part significantly improves the speed and accuracy of users obtaining the required information through multi-way retrieval of words, sentences, and paragraphs and large model re-ranking technology, greatly improving the search experience and system performance.

[0180] In an embodiment of the present invention, an efficient, accurate and flexible text file retrieval method is proposed, which can significantly improve the retrieval effect. By considering the problem of different search granularities, the method adopts the BM25 algorithm to solve the keyword granularity, and combines the semantic retrieval strategy of sentence granularity and paragraph granularity, effectively responding to the challenges of different lengths of text information and different retrieval purposes, thereby making full use of the advantages of traditional retrieval and semantic retrieval. In addition, based on the re-ranking method of the lightweight MiniCPM large model, the retrieval results can be intelligently sorted according to the search input, achieving a good sorting effect. As an end-to-end text retrieval tool, the system meets user needs with a high accuracy, especially when dealing with granularity mismatches in retrieval questions. At the same time, the ability of the large model to predict the next token further improves the effect after sorting. Finally, through the application of the hybrid retrieval method, the present invention effectively realizes the combination of keyword retrieval and semantic retrieval, and improves the overall system performance.

[0181] Corresponding to the text retrieval method with multiple knowledge granularities for RAG provided in the above embodiment of the present invention, see Figure 6, showing a structural block diagram of a multi-knowledge granularity text retrieval device for RAG provided by an embodiment of the present invention, the device includes: an acquisition unit 601, a processing unit 602, a storage unit 603, a preprocessing unit 604, a search unit 605, a first determination unit 606, a calculation unit 607, a second determination unit 608 and an output unit 609.

[0182] The acquisition unit 601 is used to acquire the original text and perform structured arrangement to obtain a JSON file.

[0183] The processing unit 602 is used to perform recursive character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain multiple text blocks containing sub-indexes, and generate a target JSON file based on the text blocks.

[0184] The storage unit 603 is used to store all target JSON files into the BM25 database, and store the JSON files after the recursive character segmentation process into the vector database.

[0185] The preprocessing unit 604 is used to preprocess the query statement when receiving the query statement input by the user, and obtain a keyword list and a query embedding vector corresponding to the query statement.

[0186] The preprocessing unit 604 is specifically used to perform word segmentation and denoising on the query statement when receiving the query statement input by the user, so as to obtain a keyword list corresponding to the query statement; and use the sentence embedding model to vectorize the query statement to obtain a query embedding vector corresponding to the query statement.

[0187] The search unit 605 is used to search for all target texts containing the keyword in the BM25 database for each keyword in the keyword list, and calculate the relevance score between each target text and the keyword.

[0188] The first determination unit 606 is used to sort all the relevance scores from large to small, and determine the target texts corresponding to the first n relevance scores as the first texts to be selected.

[0189] The calculation unit 607 is used to calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database.

[0190] The second determining unit 608 is configured to determine a second text to be selected according to all cosine similarities.

[0191] The output unit 609 is used to perform deduplication processing on all the first candidate texts and all the second candidate texts, determine the final recommended text by using the ranking model, and output the final recommended text.

[0192] In the embodiment of the present invention, as an end-to-end text retrieval tool, the system meets user needs with high accuracy, especially when dealing with retrieval problem granularity mismatch. At the same time, with the help of the ability of the large model to predict the next token, the effect after sorting is further improved. Finally, through the application of the hybrid retrieval method, the present invention effectively realizes the combination of keyword retrieval and semantic retrieval, and improves the overall system performance.

[0193] Combination Figure 6 The content shown, the acquisition unit 601, includes: an acquisition module, a segmentation module, a first creation module and a first generation module.

[0194] The acquisition module is used to obtain the original text.

[0195] The segmentation module is used to segment the original text according to each paragraph in the original text to obtain multiple paragraphs.

[0196] The first creation module is used to create a position index for each paragraph according to the chapter sequence position to which each paragraph belongs.

[0197] The first generation module is used to generate a JSON file based on each paragraph and its corresponding position index.

[0198] Combination Figure 6 The content shown, the processing unit 602, includes: a first processing module, a second creation module, a cutting module and a second generation module.

[0199] The first processing module is used to perform recursive character segmentation processing on the JSON data in the JSON file to obtain multiple preliminary text blocks, and to create a first sub-index of each preliminary text block in the paragraph to which it belongs.

[0200] The second creation module is used to create a copy of each preliminary text block and identify semantic units in the copy.

[0201] The chunking module is used to chunk the copy according to the semantic units and based on a preset chunking strategy to obtain semantic text chunks, and to create a second sub-index for each semantic text chunk in the paragraph to which it belongs.

[0202] The second generation module is used to generate a target JSON file based on each preliminary text block and its corresponding first sub-index, and to generate a target JSON file based on each semantic text block and its corresponding second sub-index.

[0203] Combination Figure 6 The content shown, the storage unit 603, includes: a first calculation module, a second calculation module, a construction module, a first storage module, a second processing module, an identification module and a second storage module.

[0204] The first calculation module is used to preprocess the target JSON file for each target JSON file, and calculate the word frequency of each word in the target JSON file and the frequency of the word appearing in all target JSON files to obtain the inverse document frequency.

[0205] The second calculation module is used to calculate the BM25 weight of the word based on the word frequency and the inverse document frequency.

[0206] The construction module is used to build an inverted index list based on the BM25 weight of the word.

[0207] The first storage module is used to store the preprocessed target JSON file and the corresponding inverted index list into the BM25 database.

[0208] The second processing module is used to vectorize each JSON file after the recursive character segmentation process by using a sentence embedding model to generate multiple high-dimensional vectors.

[0209] The recognition module is used to identify each sentence corresponding to each high-dimensional vector, generate fine-grained vectors and store them in the vector database.

[0210] The second storage module is used to identify the semantics corresponding to each high-dimensional vector, and cluster and group each sentence corresponding to each high-dimensional vector according to a preset clustering rule to obtain multiple coarse-grained category sets and store them in the vector database.

[0211] Combination Figure 6 The content shown, the cosine similarity includes sentence level cosine similarity and paragraph level cosine similarity; the second determination unit 608 includes:

[0212] The first sorting module is used to sort all sentence-level cosine similarities in descending order.

[0213] The first extraction module is used to extract the first m sentence-level cosine similarities from all the sorted sentence-level cosine similarities, and determine the texts corresponding to the first m sentence-level cosine similarities as the sentence-level candidate texts.

[0214] The second sorting module is used to sort all paragraph-level cosine similarities in descending order.

[0215] The second extraction module is used to extract the first m paragraph-level cosine similarities from all the sorted paragraph-level cosine similarities, and determine the texts corresponding to the first m paragraph-level cosine similarities as the paragraph-level candidate texts.

[0216] The determination module is used to determine the sentence-level candidate text and the paragraph-level candidate text as the second candidate text.

[0217] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0218] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0219] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text retrieval method for RAG with multiple knowledge granularities, characterized in that: The method comprises: Get the original text, and organize it into a structured format to get a JSON file; Recursively perform character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generate a target JSON file based on the text blocks; All target JSON files are stored in the BM25 database, and the JSON files after recursive character segmentation are stored in the vector database; When receiving a query statement input by a user, preprocessing the query statement to obtain a keyword list and a query embedding vector corresponding to the query statement; For each keyword in the keyword list, search for all target texts containing the keyword in the BM25 database, and calculate the relevance score between each target text and the keyword; Sort all the relevance scores from large to small, and determine the target texts corresponding to the first n relevance scores as the first candidate texts; Calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database respectively; Determine the second candidate text according to all cosine similarities; All first candidate texts and all second candidate texts are deduplicated, and a final recommended text is determined using a ranking model, and the final recommended text is output.

2. The method according to claim 1, characterized in that The original text is obtained and structured to obtain a JSON file, including: Get the original text; Segmenting the original text according to each paragraph in the original text to obtain a plurality of paragraphs; Creating a position index for each of the paragraphs according to the chapter sequence position to which each of the paragraphs belongs; A JSON file is generated based on each of the paragraphs and its corresponding position index.

3. The method according to claim 1, characterized in that The recursive character segmentation and semantic segmentation processing are performed on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and a target JSON file is generated based on the text blocks, including: Recursively perform character segmentation processing on the JSON data in the JSON file to obtain a plurality of preliminary text blocks, and create a first sub-index of each preliminary text block in the paragraph to which it belongs; For each of the preliminary text blocks, creating a copy of the preliminary text block and identifying semantic units in the copy; According to the semantic units, the copy is segmented based on a preset segmentation strategy to obtain semantic text blocks, and a second sub-index of each semantic text block in the paragraph to which it belongs is created; A target JSON file is generated based on each of the preliminary text blocks and its corresponding first sub-index, and a target JSON file is generated based on each of the semantic text blocks and its corresponding second sub-index.

4. The method according to claim 1, characterized in that: The storing of all target JSON files into the BM25 database, and the storing of the JSON files after recursive character segmentation into the vector database, include: For each of the target JSON files, preprocess the target JSON files, and calculate the word frequency of each word in the target JSON file and the frequency of occurrence of the word in all target JSON files to obtain an inverse document frequency; Calculate the BM25 weight of the word based on the word frequency and the inverse document frequency; Construct an inverted index list according to the BM25 weight of the word; The preprocessed target JSON file and the corresponding inverted index list are stored in the BM25 database; For each of the JSON files after the recursive character segmentation process, vectorize the JSON files after the recursive character segmentation process using a sentence embedding model to generate multiple high-dimensional vectors; Identify each sentence corresponding to each of the high-dimensional vectors, generate a fine-grained vector and store it in a vector database; The semantics corresponding to each of the high-dimensional vectors is identified, and each sentence corresponding to each of the high-dimensional vectors is clustered and grouped according to a preset clustering rule to obtain a plurality of coarse-grained category sets and store them in the vector database.

5. The method according to claim 1, characterized in that When receiving a query statement input by a user, preprocessing the query statement to obtain a keyword list and a query embedding vector corresponding to the query statement includes: When receiving a query statement input by a user, performing word segmentation and denoising processing on the query statement to obtain a keyword list corresponding to the query statement; The query statement is vectorized using a sentence embedding model to obtain a query embedding vector corresponding to the query statement.

6. The method according to claim 1, characterized in that The cosine similarity includes sentence-level cosine similarity and paragraph-level cosine similarity; The step of determining the second text to be selected according to all cosine similarities includes: Sort all sentence-level cosine similarities from largest to smallest; Extract the first m sentence-level cosine similarities from all the sorted sentence-level cosine similarities, and determine the texts corresponding to the first m sentence-level cosine similarities as the sentence-level candidate texts; Sort all paragraph-level cosine similarities from largest to smallest; Extract the first m paragraph-level cosine similarities from all the sorted paragraph-level cosine similarities, and determine the texts corresponding to the first m paragraph-level cosine similarities as the paragraph-level candidate texts; The sentence-level candidate text and the paragraph-level candidate text are determined as the second candidate text.

7. A text retrieval device with multiple knowledge granularities for RAG, characterized in that: The device comprises: The acquisition unit is used to acquire the original text and organize it into a structured form to obtain a JSON file; A processing unit, configured to perform recursive character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generate a target JSON file based on the text blocks; A storage unit, used for storing all target JSON files into the BM25 database, and storing the JSON files after recursive character segmentation processing into the vector database; A preprocessing unit, configured to preprocess a query statement input by a user to obtain a keyword list and a query embedding vector corresponding to the query statement when receiving the query statement input by the user; A search unit, configured to search, for each keyword in the keyword list, all target texts containing the keyword in the BM25 database, and calculate a relevance score between each target text and the keyword; A first determination unit is used to sort all the relevance scores from large to small, and determine the target texts corresponding to the first n relevance scores as the first texts to be selected; A calculation unit, used to calculate cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database respectively; A second determination unit, used for determining a second text to be selected according to all cosine similarities; The output unit is used to perform deduplication processing on all the first candidate texts and all the second candidate texts, determine the final recommended text by using the ranking model, and output the final recommended text.

8. The device according to claim 7, characterized in that The acquisition unit comprises: The acquisition module is used to obtain the original text; A segmentation module, used for segmenting the original text according to each paragraph in the original text to obtain multiple paragraphs; A first creation module, used for creating a position index for each paragraph according to the position of the chapter sequence to which each paragraph belongs; The first generating module is used to generate a JSON file based on each of the paragraphs and its corresponding position index.

9. The device according to claim 7, characterized in that The processing unit comprises: A first processing module is used to perform recursive character segmentation processing on the JSON data in the JSON file to obtain a plurality of preliminary text blocks, and to create a first sub-index of each preliminary text block in the paragraph to which it belongs; A second creation module, for creating a copy of each of the preliminary text blocks, and identifying semantic units in the copy; A block cutting module, used for cutting the copy into blocks according to the semantic units and based on a preset block cutting strategy to obtain semantic text blocks, and creating a second sub-index of each semantic text block in the paragraph to which it belongs; The second generation module is used to generate a target JSON file based on each of the preliminary text blocks and its corresponding first sub-index, and to generate a target JSON file based on each of the semantic text blocks and its corresponding second sub-index.

10. The device according to claim 7, characterized in that The storage unit comprises: A first calculation module is used to preprocess each target JSON file, and calculate the word frequency of each word in the target JSON file and the frequency of occurrence of the word in all target JSON files to obtain an inverse document frequency; A second calculation module, used for calculating the BM25 weight of the word based on the word frequency and the inverse document frequency; A construction module, used to construct an inverted index list according to the BM25 weight of the word; A first storage module, used to store the preprocessed target JSON file and the corresponding inverted index list into a BM25 database; A second processing module is used to perform vectorization processing on each of the JSON files after the recursive character segmentation processing by using a sentence embedding model to generate multiple high-dimensional vectors; A recognition module, used to recognize each sentence corresponding to each of the high-dimensional vectors, generate a fine-grained vector and store it in a vector database; The second storage module is used to identify the semantics corresponding to each of the high-dimensional vectors, and cluster and group each sentence corresponding to each of the high-dimensional vectors according to a preset clustering rule to obtain multiple coarse-grained category sets and store them in the vector database.

Images