A method and apparatus for multi-knowledge granularity text retrieval for RAG

Abstract

The application provides a multi-knowledge granularity text retrieval method and device for RAG, which integrates original text into a JSON file, and performs recursive character and semantic segmentation processing to generate a target JSON file, which is stored in a BM25 database and a vector database. When a user queries, a keyword list and a query embedding vector are generated by preprocessing; the target text in the BM25 database is searched, the relevance score with the keyword is calculated to determine the first selected text; the second selected text is determined according to the query embedding vector; all the first selected text and the second selected text are subjected to deduplication processing, the final recommended text is determined by using a sorting model, and is output. By considering the problem of different search granularities, the BM25 algorithm is used to solve the keyword granularity. The semantic retrieval strategy combining sentence granularity and paragraph granularity effectively copes with the challenge of different lengths of text information and different retrieval purposes, so that the advantages of traditional retrieval and semantic retrieval are fully utilized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a text retrieval method and apparatus for RAG with multiple knowledge granularities. Background Technology

[0002] With the rapid development of information technology, the sheer volume and diverse formats of documents have made traditional retrieval methods insufficient to meet the demands for efficient and accurate information acquisition. This has driven the development of text retrieval technology, making it an important component of information management. Text retrieval uses computers to match words in natural language, enabling rapid document searching.

[0003] Currently, text retrieval systems employ various advanced algorithms and models, such as keyword matching, Boolean search, vector space models, and deep learning techniques. While these methods have made progress, limitations remain. For example, keyword-based retrieval often relies on the accuracy of user input, potentially leading to the omission of relevant information. Different retrieval methods also set different search granularities when processing information: Boolean search may be too strict, while loose keyword matching may return a large number of irrelevant results, making it difficult for users to find the information they need. Furthermore, many retrieval systems use relevance-based ranking algorithms, which may not fully reflect the true value of the results in specific domains or under certain user preferences.

[0004] Therefore, improving the accuracy and flexibility of text retrieval and enhancing user experience are urgent issues that need to be addressed. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a text retrieval method and apparatus for RAG with multiple knowledge granularities, in order to solve the problems of low accuracy and poor flexibility in current text retrieval systems.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses a multi-knowledge granularity text retrieval method for RAG, the method comprising:

[0008] Obtain the raw text and structure it to obtain a JSON file;

[0009] The JSON data in the JSON file is recursively segmented into character blocks and semantic blocks to obtain multiple text blocks containing sub-indexes, and the target JSON file is generated based on the text blocks.

[0010] All target JSON files are stored in the BM25 database, and the JSON files after recursive character segmentation are stored in the vector database;

[0011] When a query statement is received from a user, the query statement is preprocessed to obtain a keyword list and query embedding vector corresponding to the query statement;

[0012] For each keyword in the keyword list, find all target texts in the BM25 database that contain the keyword, and calculate the relevance score between each target text and the keyword;

[0013] Sort all relevance scores from largest to smallest, and determine the target texts corresponding to the top n relevance scores as the first candidate texts;

[0014] Calculate the cosine similarity between the query embedding vector and the sentence-level and paragraph-level embedding vectors in the vector database, respectively.

[0015] The second candidate text is determined based on all cosine similarities;

[0016] All first candidate texts and all second candidate texts are deduplicated, and the final recommended text is determined using a ranking model. The final recommended text is then output.

[0017] Preferably, the step of obtaining the original text and performing structured processing to obtain a JSON file includes:

[0018] Get the original text;

[0019] The original text is divided into multiple paragraphs based on each paragraph in the original text;

[0020] Create a position index for each paragraph based on its position in the chapter sequence to which it belongs;

[0021] A JSON file is generated based on each paragraph and its corresponding position index.

[0022] Preferably, the recursive character segmentation and semantic segmentation of the JSON data in the JSON file to obtain multiple text blocks containing sub-indexes, and the generation of the target JSON file based on the text blocks, includes:

[0023] The JSON data in the JSON file is recursively segmented into blocks to obtain multiple preliminary text blocks, and a first sub-index is created for each preliminary text block within its respective paragraph.

[0024] For each of the initial text blocks, a copy of the initial text block is created, and semantic units in the copy are identified;

[0025] Based on the semantic unit, the copy is divided into blocks according to a preset slicing strategy to obtain semantic text blocks, and a second sub-index is created for each semantic text block in its respective paragraph;

[0026] A target JSON file is generated based on each of the initial text blocks and its corresponding first sub-index, and a target JSON file is generated based on each of the semantic text blocks and its corresponding second sub-index.

[0027] Preferably, storing all target JSON files in a BM25 database and storing the JSON files after recursive character segmentation in a vector database includes:

[0028] For each target JSON file, the target JSON file is preprocessed, and the word frequency of each word in the target JSON file and the frequency of the word in all target JSON files are calculated to obtain the inverse document frequency;

[0029] Calculate the BM25 weight of the word based on the word frequency and the inverse document frequency;

[0030] Construct an inverted index list based on the BM25 weights of the words;

[0031] The preprocessed target JSON file and the corresponding inverted index list are stored in the BM25 database;

[0032] For each JSON file after recursive character segmentation, a sentence embedding model is used to vectorize the JSON file after recursive character segmentation to generate multiple high-dimensional vectors.

[0033] Identify each sentence corresponding to each of the high-dimensional vectors, generate fine-grained vectors, and store them in a vector database;

[0034] The semantics corresponding to each high-dimensional vector are identified, and each sentence corresponding to each high-dimensional vector is clustered according to a preset clustering rule to obtain multiple coarse-grained category sets and store them in the vector database.

[0035] Preferably, when a user-input query is received, the query is preprocessed to obtain a keyword list and query embedding vector corresponding to the query, including:

[0036] When a query statement is received from a user, the query statement is segmented and denoised to obtain a list of keywords corresponding to the query statement.

[0037] The query statement is vectorized using a sentence embedding model to obtain the corresponding query embedding vector.

[0038] Preferably, the cosine similarity includes sentence-level cosine similarity and paragraph-level cosine similarity;

[0039] The step of determining the second candidate text based on all cosine similarities includes:

[0040] Sort all sentences by cosine similarity at all levels from largest to smallest.

[0041] Extract the top m sentence-level cosine similarity from all sorted sentence-level cosine similarity scores, and determine the text corresponding to the top m sentence-level cosine similarity scores as candidate texts at the sentence level.

[0042] Sort all paragraphs by cosine similarity of their hierarchical levels from largest to smallest.

[0043] Extract the top m paragraph cosine similarity scores from all sorted paragraph cosine similarity scores, and determine the texts corresponding to the top m paragraph cosine similarity scores as paragraph-level candidate texts.

[0044] The sentence-level candidate text and the paragraph-level candidate text are determined as the second candidate text.

[0045] A second aspect of this invention discloses a text retrieval device for RAG with multiple knowledge granularities, the device comprising:

[0046] The acquisition unit is used to acquire the raw text, perform structured processing, and obtain a JSON file;

[0047] The processing unit is used to perform recursive character segmentation and semantic segmentation on the JSON data in the JSON file to obtain multiple text blocks containing sub-indexes, and generate the target JSON file based on the text blocks;

[0048] A storage unit is used to store all target JSON files into a BM25 database, and to store the JSON files after recursive character segmentation into a vector database;

[0049] The preprocessing unit is used to preprocess the query statement when it receives the query statement input by the user, and obtain the keyword list and query embedding vector corresponding to the query statement.

[0050] The search unit is used to search for all target texts in the BM25 database that contain each keyword in the keyword list, and to calculate the relevance score between each target text and the keyword;

[0051] The first determining unit is used to sort all the relevance scores from largest to smallest and determine the target texts corresponding to the top n relevance scores as the first candidate texts.

[0052] The calculation unit is used to calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database, respectively.

[0053] The second determining unit is used to determine the second candidate text based on all cosine similarities;

[0054] The output unit is used to perform deduplication on all first candidate texts and all second candidate texts, and to determine the final recommended text using a ranking model, and output the final recommended text.

[0055] Preferably, the acquisition unit includes:

[0056] The acquisition module is used to acquire the raw text;

[0057] The segmentation module is used to segment the original text according to each paragraph in the original text to obtain multiple paragraphs;

[0058] The first creation module is used to create a position index for each paragraph based on the chapter sequence position to which each paragraph belongs;

[0059] The first generation module is used to generate a JSON file based on each paragraph and its corresponding position index.

[0060] Preferably, the processing unit includes:

[0061] The first processing module is used to recursively segment the JSON data in the JSON file to obtain multiple preliminary text blocks, and to create a first sub-index for each preliminary text block in its respective paragraph.

[0062] The second creation module is used to create a copy of each initial text block and identify semantic units in the copy;

[0063] The chunking module is used to chunk the copy according to the semantic unit based on a preset chunking strategy to obtain semantic text blocks, and to create a second sub-index for each semantic text block in its respective paragraph;

[0064] The second generation module is used to generate a target JSON file based on each of the initial text blocks and its corresponding first sub-index, and to generate a target JSON file based on each of the semantic text blocks and its corresponding second sub-index.

[0065] Preferably, the storage unit includes:

[0066] The first calculation module is used to preprocess each target JSON file, calculate the word frequency of each word in the target JSON file, and the frequency of the word in all target JSON files to obtain the inverse document frequency.

[0067] The second calculation module is used to calculate the BM25 weight of the word based on the word frequency and the inverse document frequency;

[0068] The construction module is used to construct an inverted index list based on the BM25 weights of the words;

[0069] The first storage module is used to store the preprocessed target JSON file and the corresponding inverted index list into the BM25 database;

[0070] The second processing module is used to vectorize each JSON file after recursive character segmentation using a sentence embedding model, generating multiple high-dimensional vectors.

[0071] The recognition module is used to recognize each sentence corresponding to each of the high-dimensional vectors, generate fine-grained vectors, and store them in the vector database;

[0072] The second storage module is used to identify the semantics corresponding to each of the high-dimensional vectors, and to cluster each sentence corresponding to each of the high-dimensional vectors according to a preset clustering rule to obtain multiple coarse-grained category sets and store them in the vector database.

[0073] Based on the above embodiments of the present invention, a multi-knowledge granularity text retrieval method for RAG is provided. The original text is organized into a JSON file, and recursive character and semantic segmentation is performed to generate a target JSON file, which is then stored in a BM25 database and a vector database. When a user queries, a keyword list and query embedding vector are generated through preprocessing; target text in the BM25 database is searched, and a relevance score with the keywords is calculated to determine the first candidate text; a second candidate text is determined based on the query embedding vector; all first and second candidate texts are deduplicated, and a ranking model is used to determine and output the final recommended text. By considering the problem of different search granularities, the BM25 algorithm is used to solve the keyword granularity issue. Combining sentence-level and paragraph-level semantic retrieval strategies effectively addresses the challenges of varying text lengths and different retrieval purposes, thereby fully utilizing the advantages of traditional retrieval and semantic retrieval. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0075] Figure 1 A flowchart illustrating a multi-knowledge granularity text retrieval method for RAG provided in an embodiment of the present invention;

[0076] Figure 2 This is a flowchart for structuring and organizing original text, provided as an embodiment of the present invention.

[0077] Figure 3 This is a flowchart of recursive character segmentation and semantic segmentation of JSON data provided in an embodiment of the present invention;

[0078] Figure 4 A flowchart of a stored procedure provided in an embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of the retrieval and query process provided in an embodiment of the present invention;

[0080] Figure 6 This is a structural block diagram of a text retrieval device for RAG with multiple knowledge granularities, provided as an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] As the background technology indicates, keyword-based retrieval often relies on the accuracy of user input, which may lead to the omission of relevant information. Different retrieval methods also set different search granularities when processing information: Boolean searches may be too strict, while loose keyword matching may return a large number of irrelevant results, making it difficult for users to find the information they need. In addition, many retrieval systems use relevance-based ranking algorithms, which may not fully reflect the true value of the results in specific domains or under user preferences.

[0084] Therefore, this invention provides a multi-knowledge granularity text retrieval method for RAG. First, the original text is organized into a JSON file and recursively processed for character and semantic segmentation to generate a target JSON file, which is then stored in a BM25 database and a vector database. When a user-input query is received, a keyword list and query embedding vectors are preprocessed. Target text in the BM25 database is searched, and its relevance score to the keywords is calculated to determine the first candidate text. Second candidate text is determined based on the query embedding vector. All first and second candidate texts are deduplicated, and a ranking model is used to determine and output the final recommended text. By considering the issue of different search granularities, the BM25 algorithm is used to address keyword granularity, combined with sentence-level and paragraph-level semantic retrieval strategies, effectively addressing the challenges of varying text lengths and different retrieval purposes, thus fully utilizing the advantages of traditional retrieval and semantic retrieval. Finally, through the application of a hybrid retrieval method, this invention effectively combines keyword retrieval and semantic retrieval, improving overall system performance.

[0085] See Figure 1 The diagram shows a flowchart of a multi-knowledge granularity text retrieval method for RAG provided by an embodiment of the present invention.

[0086] It should be noted that this multi-knowledge granularity text retrieval method for RAG consists of three parts: text processing, text storage, and retrieval query. These three parts are explained below:

[0087] Step S101: Obtain the original text and perform structured processing to obtain a JSON file.

[0088] In the specific implementation step S101, the raw text is obtained. This raw text can be the raw text input by the user. This raw text is then structured and processed to obtain a JSON file. The specific processing steps are as follows: Figure 2 As shown, Figure 2 Includes:

[0089] Step S201: Obtain the original text.

[0090] The text can be a document.

[0091] In the specific implementation step S201, the original text to be processed is obtained.

[0092] Step S202: Divide the original text into multiple paragraphs based on each paragraph in the original text.

[0093] In the specific implementation step S202, the original text is divided according to the paragraph structure in the original text to obtain multiple paragraphs.

[0094] Step S203: Create a position index for each paragraph based on its position in the chapter sequence.

[0095] In the specific implementation step S203, a corresponding position index is created for each paragraph based on its position in the chapter sequence of the original text.

[0096] For example: the position index of the second paragraph of the first chapter in the original text is set to 1.2; the position index of the first paragraph of the third chapter in the original text is set to 3.1.

[0097] Understandably, in normal writing, there are often significant semantic differences between chapters and paragraphs, and each paragraph has its unique function. Therefore, by building an index for each paragraph, they can be distinguished from other paragraphs. During subsequent sentence context reconstruction, only relevant paragraphs are reconstructed, rather than adjacent paragraphs, which effectively reduces the length of the reconstruction.

[0098] In some embodiments, if it is detected that some paragraphs in the original text have corresponding subheadings, the subheadings are used as metadata of the original text and bound to the corresponding paragraphs so that a JSON file can be generated together with the paragraphs and their corresponding position indices.

[0099] Step S204: Generate a JSON file based on each paragraph and its corresponding position index.

[0100] In the specific implementation step S204, a JSON file is generated based on each paragraph and its corresponding position index.

[0101] Understandable, Figure 2 The content shown represents the first part of the text retrieval method provided in this embodiment of the invention: text preprocessing within text processing. Preprocessing the original text allows for better storage of the text in the database. The following explanation, in conjunction with step S102, details the preliminary segmentation and semantic segmentation within the text processing.

[0102] Step S102: Perform recursive character segmentation and semantic segmentation on the JSON data in the JSON file to obtain multiple text blocks containing sub-indexes, and generate the target JSON file based on the text blocks.

[0103] In the specific implementation step S102, the JSON data in the JSON file undergoes recursive character segmentation and semantic segmentation to obtain multiple text blocks containing sub-indexes. These text blocks include preliminary text blocks and semantic text blocks. The target JSON file is generated based on these text blocks. The specific segmentation process is as follows: Figure 3 As shown, Figure 3 Includes:

[0104] Step S301: Perform recursive character segmentation on the JSON data in the JSON file to obtain multiple preliminary text blocks, and create the first sub-index of each preliminary text block in its respective paragraph.

[0105] In the specific implementation step S301, according to the preset string length, the JSON data in the JSON file is recursively segmented into blocks to obtain multiple preliminary text blocks; and according to the paragraph to which each preliminary text block belongs, the first sub-index of the preliminary text block in its respective paragraph is created.

[0106] For example, the JSON data in a JSON file can be divided into multiple preliminary text blocks, each with a length of 500 characters, based on the length of each block.

[0107] Understandably, the purpose of recursive character segmentation is to divide the text into smaller units, making it easier for subsequent recursive refinement.

[0108] For example, if a certain initial text block belongs to the first position of the second paragraph of the first chapter in the text, then the first sub-index is set to 1.2.1.

[0109] It should be noted that creating a first sub-index for each initial text block helps ensure that the context of the sentence can be effectively restored during subsequent retrieval queries.

[0110] It is understood that the content shown in step S301 is the first part of the text retrieval method provided in this embodiment of the invention: the content of the preliminary segmentation in text processing.

[0111] Step S302: For each initial text block, create a copy of the initial text block and identify semantic units in the copy.

[0112] In practical applications, it's common to determine whether the initial text block is relatively long compared to the user's query. Specifically, if the effective information only accounts for one-tenth or even one-twentieth of the initial text block (500 characters long), the subsequent text vectorization process will further weaken the effective information, making it difficult to retrieve correctly. Therefore, further segmentation is necessary. In this case, semantic segmentation is used to cut each initial text block according to semantic similarity, ensuring the semantic integrity of the segments.

[0113] In the specific implementation step S302, for each initial text block, a copy of the initial text block is created, and the semantic units in the copy are identified, including identifying sentences, phrase structures, named entities (such as people's names, place names, organizations, etc.) and topics in the copy.

[0114] It should be noted that the semantic segmentation of the initial text block is performed on a copy of the initial text block.

[0115] Step S303: Based on the semantic unit, the copy is divided into blocks according to the preset slicing strategy to obtain semantic text blocks, and a second sub-index is created for each semantic text block in its respective paragraph.

[0116] In the specific implementation step S303, the copy is divided into semantic text blocks according to the semantic unit and based on a preset slicing strategy (such as a specific slicing rule, machine learning strategy or deep learning strategy). Then, according to the paragraph to which each semantic text block belongs, a second sub-index of the semantic text block in the paragraph to which it belongs is created.

[0117] Understandably, creating a second sub-index for each semantic text block also helps ensure that the context of the sentence can be effectively restored during subsequent retrieval queries.

[0118] Step S304: Generate a target JSON file based on each initial text block and its corresponding first sub-index, and generate a target JSON file based on each semantic text block and its corresponding second sub-index.

[0119] In the specific implementation step S304, a target JSON file is generated based on each preliminary text block and the first sub-index corresponding to each preliminary text block, and a target JSON file is generated based on each semantic text block and the second sub-index corresponding to each semantic text block.

[0120] It is understandable that, in combination with step S102 and Figure 3The content shown represents the first part of the text retrieval method provided in this embodiment of the invention: the complete explanation of text processing. In this embodiment, a combination of recursive character segmentation and semantic segmentation is used to improve the efficiency and accuracy of text processing while maintaining the semantic integrity of the text. Furthermore, during the segmentation process, metadata is used to preserve the position of the segment relative to the original text, thereby enabling rapid location of the original data during the retrieval process.

[0121] The second part of the text retrieval method provided in the embodiments of the present invention, namely, text entry into the database, will be explained below.

[0122] Understandably, BM25 is a retrieval algorithm based on term frequency and document frequency, offering advantages in efficiency and accuracy for keyword searching compared to semantic retrieval. BM25 effectively captures the importance of words and improves retrieval accuracy by scoring them based on their frequency of occurrence in documents and their weight within the entire document set. However, BM25 also faces the problem of data sparsity; many words may appear in only a few documents, resulting in low weight and potentially leading to the loss of important information. In contrast, semantic vectors, through machine learning and natural language processing techniques, can better capture the semantic relationships between words, understand user query intent, and handle polysemous and synonymous words, thereby improving the accuracy and comprehensiveness of retrieval.

[0123] Therefore, the embodiments of the present invention use both the BM25 database and the vector database for storage, which can combine the efficiency of traditional word frequency retrieval with the deep understanding of semantic retrieval, significantly improving the accuracy and relevance of retrieval.

[0124] Step S103: Store all target JSON files in the BM25 database, and store the JSON files after recursive character segmentation in the vector database.

[0125] In the specific implementation step S103, all target JSON files are stored in the BM25 database, and the JSON files after recursive character segmentation are stored in the vector database.

[0126] Specific stored procedures such as Figure 4 As shown, Figure 4 Includes:

[0127] Step S401: For each target JSON file, preprocess the target JSON file and calculate the word frequency of each word in the target JSON file and the frequency of the word in all target JSON files to obtain the inverse document frequency.

[0128] In the specific implementation step S401, for each target JSON file, the target JSON file is preprocessed (such as word segmentation, stop word removal, stemming, etc.), and the word frequency of each word in the preprocessed target JSON file and the frequency of each word appearing in all target JSON files are calculated to obtain the inverse document frequency (IDF).

[0129] It should be noted that the word frequency of each word in the preprocessed target JSON file is calculated according to formula (1).

[0130]

[0131] Where TF is the term frequency of each word in the preprocessed target JSON file; d is the preprocessed target JSON file (i.e., text); t is the term in the query; f(t,d) is the term frequency (TF) of term t in text d; |d| is the length (number of words) of text d; avgdl is the average text length of the entire text set; k1 and b are adjustable parameters. For example, k1 = 1.2, b = 0.75.

[0132] It should be noted that k1 is used to control the degree of word frequency saturation, that is, as f(t,d) increases, the rate at which the weights increase gradually decreases. b is used to control the normalization of document length; b=1 fully considers the document length, while b=0 ignores the document length.

[0133] It is understandable that the inverse document frequency is calculated based on formula (2).

[0134]

[0135] Where IDF(t) is the inverse document frequency; N is the total number of documents in the document set; and n(t) is the number of documents containing term t.

[0136] Step S402: Calculate the BM25 weight of the word based on word frequency and inverse document frequency.

[0137] In the specific implementation step S402, the BM25 weight of the word is calculated based on the word frequency and inverse document frequency, as well as formula (3).

[0138] ω(t,d)=IDF(t)*TF(t,d)(3)

[0139] Where ω(t,d) is the BM25 weight of the word; IDF(t) is the inverse document frequency; and TF(t,d) is the term frequency of term t in text d.

[0140] Step S403: Construct an inverted index list based on the BM25 weight of the words.

[0141] In the specific implementation step S403, for each word in the preprocessed target JSON file, find all paragraphs (paragraphs in the preprocessed target JSON file) containing that word, and construct an inverted index list for that word based on all paragraphs and the word's BM25 weight.

[0142] It's important to note that in the inverted index list for this term, each entry contains paragraph index information, i.e., identifiers pointing to specific paragraph locations. This helps to quickly locate relevant content and improves search speed.

[0143] Understandably, each word in the inverted index list also carries its BM25 weight, which reflects the word's importance within a specific paragraph. The BM25 weight considers factors such as term frequency and document frequency, helping the system assess the relevance of the paragraph to the query.

[0144] Step S404: Store the preprocessed target JSON file and the corresponding inverted index list into the BM25 database.

[0145] In the specific implementation step S404, the preprocessed target JSON file and the corresponding inverted index list of the preprocessed target JSON file are stored in the BM25 database.

[0146] Step S405: For each JSON file after recursive character segmentation, use a sentence embedding model to vectorize the JSON file after recursive character segmentation to generate multiple high-dimensional vectors.

[0147] In the specific implementation step S405, for each JSON file after recursive character segmentation, that is... Figure 3 The initial text blocks are processed using sentence embedding models (e.g., BGE). 1 The sentence embedding model vectorizes the initial text blocks to generate multiple high-dimensional vectors.

[0148] Step S406: Identify each sentence corresponding to each high-dimensional vector, generate fine-grained vectors, and store them in the vector database.

[0149] In the specific implementation step S406, each sentence corresponding to each high-dimensional vector is identified, and each sentence is individually generated into a fine-grained vector and stored in the vector database.

[0150] Understandably, fine-grained slicing of high-dimensional vectors and storing them in a vector database helps in subsequent queries that require precise matching of specific semantics. Fine-grained slicing and storage can provide more granular semantic information.

[0151] Step S407: Identify the semantics corresponding to each high-dimensional vector, and cluster each sentence corresponding to each high-dimensional vector according to the preset clustering rules to obtain multiple coarse-grained category sets and store them in the vector database.

[0152] In the specific implementation step S407, the semantics corresponding to each high-dimensional vector are identified, and several sentences corresponding to each high-dimensional vector are grouped into a larger coarse-grained category set according to the preset clustering rules. Then, all coarse-grained category sets are stored in the vector database.

[0153] In practical applications, the semantics corresponding to each high-dimensional vector are identified, and several sentences are merged into a larger text block (i.e., a coarse-grained category set) according to the topic or context (or according to criteria such as semantic similarity, topic consistency or time order) to obtain multiple text blocks.

[0154] For example, a series of related sentences can be combined to form a paragraph.

[0155] It is understandable that coarse-grained slicing of high-dimensional vectors and storing them in a vector database helps in subsequent queries that require understanding the contextual semantics. Coarse-grained slicing and storage can provide more complete semantic information.

[0156] Thus far, the vector database includes, but is not limited to, sentence-level embedding vectors and paragraph-level vectors.

[0157] It should be noted that, in combination Figure 4 The content shown here completes the description of the second part of the text retrieval method provided in this embodiment of the invention: text storage. In this embodiment, BM25 effectively captures the importance of words and improves retrieval accuracy by scoring them based on their frequency of occurrence in documents and their weight in the entire document set. By using two different granularities for document segmentation and storage in the vector database, the system can select the most suitable retrieval granularity according to query requirements, thereby improving retrieval accuracy and relevance.

[0158] The third part of the text retrieval method provided in the embodiments of the present invention, namely the retrieval query, will be explained below.

[0159] Step S104: When a query statement is received from the user, the query statement is preprocessed to obtain a list of keywords and a query embedding vector corresponding to the query statement.

[0160] In the specific implementation step S104, when the user input query statement is received, the query statement is segmented and denoised, and the processed query statement is organized into a list to obtain the keyword list corresponding to the query statement; the query statement is vectorized using a sentence embedding model (such as the bge-sentence-embedding model) to obtain the query embedding vector (text embedding) corresponding to the query statement.

[0161] Combination Figure 5 The diagram shown illustrates the retrieval process. After the user enters a query, the query is processed by word segmentation, stop word removal, and vectorization.

[0162] Step S105: For each keyword in the keyword list, find all target texts in the BM25 database that contain the keyword, and calculate the relevance score between each target text and the keyword.

[0163] In the specific implementation step S105, for each keyword in the keyword list, all target texts (e.g., corpus blocks) containing the keyword are searched from the BM25 database. The BM25 weight is calculated for each keyword, and the relevance score between each target text and the keyword is calculated.

[0164] It should be noted that in the process of calculating the BM25 weight for each keyword, BM25 will comprehensively consider the number of times the keyword appears in the target text (term frequency) and the rarity of the keyword in the entire target text set (IDF), and calculate the contribution of the keyword to each text. The specific calculation is shown in formula (4).

[0165]

[0166] Where q is the query; d is the target text; t is the keyword; and ω(t,d) is the BM25 weight.

[0167] Step S106: Sort all relevance scores from largest to smallest, and determine the target texts corresponding to the top n relevance scores as the first candidate texts.

[0168] In the specific implementation step S106, all relevance scores are sorted in descending order, the top n relevance scores are extracted from all sorted relevance scores, and the target text corresponding to the top n relevance scores is determined as the first candidate text.

[0169] Step S107: Calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and paragraph-level embedding vector in the vector database, respectively.

[0170] In the specific implementation step S107, the sentence-level cosine similarity between the query embedding vector and the sentence-level embedding vector in the vector database is calculated, and the paragraph-level cosine similarity between the query embedding vector and the paragraph-level embedding vector in the vector database is calculated.

[0171] Step S108: Determine the second candidate text based on all cosine similarities.

[0172] In the specific implementation step S108, all sentence-level cosine similarities are sorted in descending order; the top m sentence-level cosine similarities are extracted from the sorted sentence-level cosine similarities, and the texts corresponding to the top m sentence-level cosine similarities are determined as sentence-level candidate texts; all paragraph-level cosine similarities are sorted in descending order; the top m paragraph-level cosine similarities are extracted from the sorted paragraph-level cosine similarities, and the texts corresponding to the top m paragraph-level cosine similarities are determined as paragraph-level candidate texts; the sentence-level candidate texts and paragraph-level candidate texts are determined as the second candidate texts.

[0173] In practical applications, when selecting text at the sentence level, it's necessary to consider that a single sentence may not fully express its meaning. Therefore, it's essential to consider the content before and after each sentence, as well as the position index of each sentence within the original document. Using the position index, metadata (such as paragraph indexes) can be queried to extract relevant contextual content and reconstruct the complete paragraph. This not only helps users better understand the sentences but also provides richer informational background, thereby improving the readability and comprehension of the information.

[0174] It should be noted that the processes of determining the first candidate text and determining the second candidate text are parallel. The execution order in this embodiment of the invention is only for illustrative purposes and does not restrict the specific execution order.

[0175] Step S109: Perform deduplication on all first candidate texts and all second candidate texts, and use the ranking model to determine the final recommended text, and output the final recommended text.

[0176] In the specific implementation step S109, all first candidate texts and all second candidate texts are deduplicated. That is, there will be some duplicate search results in the target texts corresponding to the first n relevance scores, the texts corresponding to the first m paragraph level cosine similarity, and the texts corresponding to the first m sentence level cosine similarity. After deduplication, a candidate set of recommended texts is obtained. The candidate set of recommended texts is further filtered using a ranking model (such as the open-source model BGE-ReRanker-v2-MiniCPM) to determine the final recommended text and output the final recommended text.

[0177] It should be noted that the specific sorting process is as follows: First, each document in the query and candidate document set is treated as a text pair (query: candidate document); then, a prompt is designed to ask the large model whether the "candidate document" in the text pair (query: candidate document) can be an answer to the "query", and the large model only outputs "yes" or "no"; finally, the text pair and the prompt are combined and input into the MiniCPM re-ranking model, and the logit corresponding to the last token of the last layer is used as the ranking score to rank the candidate documents.

[0178] Understandably, the deduplicated results are fed into the MiniCPM re-ranking model for sorting. The sorted results are arranged in descending order of ranking score, ensuring that the most relevant content is displayed to the user first.

[0179] It's worth noting that by combining document-level and sentence-level similarity calculations, multi-level search results are achieved, ensuring both global relevance and local accuracy of information, thereby improving the comprehensiveness and accuracy of search results. Furthermore, the use of the MiniCPM re-ranking model for intelligent sorting makes the relevance of search results even more precise, further enhancing user satisfaction. Overall, the search query section, through multi-path retrieval at the word, sentence, and paragraph levels and large-scale model re-ranking technology, significantly improves the speed and accuracy of users obtaining the information they need, greatly enhancing the search experience and system performance.

[0180] This invention proposes an efficient, accurate, and flexible text file retrieval method that significantly improves retrieval performance. By considering different search granularities, this method employs the BM25 algorithm to address keyword granularity and combines sentence-level and paragraph-level semantic retrieval strategies to effectively address the challenges of varying text lengths and different retrieval purposes, thus fully leveraging the advantages of traditional retrieval and semantic retrieval. Furthermore, a re-ranking method based on a lightweight MiniCPM large model enables intelligent sorting of retrieval results according to the search input, achieving excellent ranking results. As an end-to-end text retrieval tool, this system meets user needs with high accuracy, particularly excelling in handling mismatched granularity of search queries. Simultaneously, the ability of the large model to predict the next token further enhances the ranking effect. Finally, through the application of a hybrid retrieval method, this invention effectively combines keyword retrieval and semantic retrieval, improving overall system performance.

[0181] Corresponding to the multi-knowledge granularity text retrieval method for RAG provided in the above embodiments of the present invention, see also... Figure 6The diagram shows a structural block diagram of a text retrieval device for RAG with multiple knowledge granularities provided in an embodiment of the present invention. The device includes: an acquisition unit 601, a processing unit 602, a storage unit 603, a preprocessing unit 604, a search unit 605, a first determination unit 606, a calculation unit 607, a second determination unit 608, and an output unit 609.

[0182] Unit 601 is used to retrieve the raw text and perform structured processing to obtain a JSON file.

[0183] Processing unit 602 is used to perform recursive character segmentation and semantic segmentation on the JSON data in the JSON file to obtain multiple text blocks containing sub-indexes, and generate the target JSON file based on the text blocks.

[0184] Storage unit 603 is used to store all target JSON files to the BM25 database and to store the JSON files after recursive character segmentation to the vector database.

[0185] The preprocessing unit 604 is used to preprocess the query statement when it receives the query statement input by the user, and obtain the keyword list and query embedding vector corresponding to the query statement.

[0186] The preprocessing unit 604 is specifically used to perform word segmentation and noise reduction on the query statement when it receives the query statement input by the user, so as to obtain the keyword list corresponding to the query statement; and to perform vectorization processing on the query statement using a sentence embedding model to obtain the query embedding vector corresponding to the query statement.

[0187] The lookup unit 605 is used to search for all target texts containing the keyword in the BM25 database for each keyword in the keyword list, and to calculate the relevance score between each target text and the keyword.

[0188] The first determining unit 606 is used to sort all relevance scores from largest to smallest and determine the target texts corresponding to the top n relevance scores as the first candidate texts.

[0189] The calculation unit 607 is used to calculate the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database, respectively.

[0190] The second determining unit 608 is used to determine the second candidate text based on all cosine similarities.

[0191] Output unit 609 is used to deduplicate all first candidate texts and all second candidate texts, and to determine the final recommended text using a ranking model, and output the final recommended text.

[0192] In this embodiment of the invention, as an end-to-end text retrieval tool, the system meets user needs with high accuracy, particularly excelling in handling mismatches in retrieval granularity. Furthermore, the ability to predict the next token using a large model further enhances the ranking results. Finally, through the application of a hybrid retrieval method, this invention effectively combines keyword retrieval and semantic retrieval, improving overall system performance.

[0193] Combination Figure 6 The content shown, the acquisition unit 601, includes: an acquisition module, a segmentation module, a first creation module and a first generation module.

[0194] The acquisition module is used to acquire the raw text.

[0195] The segmentation module is used to divide the original text into multiple paragraphs based on each paragraph in the original text.

[0196] The first creation module is used to create a position index for each paragraph based on its position in the chapter sequence.

[0197] The first generation module is used to generate JSON files based on each paragraph and its corresponding position index.

[0198] Combination Figure 6 The processing unit 602 shown includes: a first processing module, a second creation module, a slicing module, and a second generation module.

[0199] The first processing module is used to recursively segment JSON data in a JSON file to obtain multiple preliminary text blocks, and to create the first sub-index of each preliminary text block in its respective paragraph.

[0200] The second creation module is used to create a copy of the initial text block for each initial text block and to identify semantic units in the copy.

[0201] The chunking module is used to chunk the copy according to the semantic unit and based on a preset chunking strategy to obtain semantic text blocks, and to create a second sub-index for each semantic text block in its respective paragraph.

[0202] The second generation module is used to generate a target JSON file based on each initial text block and its corresponding first sub-index, and to generate a target JSON file based on each semantic text block and its corresponding second sub-index.

[0203] Combination Figure 6 The contents shown, storage unit 603, include: a first computing module, a second computing module, a construction module, a first storage module, a second processing module, an identification module, and a second storage module.

[0204] The first calculation module is used to preprocess the target JSON file for each target JSON file, calculate the word frequency of each word in the target JSON file, and the frequency of the word in all target JSON files, to obtain the inverse document frequency.

[0205] The second calculation module is used to calculate the BM25 weight of words based on word frequency and inverse document frequency.

[0206] The building module is used to construct an inverted index list based on the BM25 weights of words.

[0207] The first storage module is used to store the preprocessed target JSON file and the corresponding inverted index list into the BM25 database.

[0208] The second processing module is used to vectorize each JSON file after recursive character segmentation using a sentence embedding model, generating multiple high-dimensional vectors.

[0209] The recognition module is used to identify each sentence corresponding to each high-dimensional vector, generate fine-grained vectors, and store them in the vector database.

[0210] The second storage module is used to identify the semantics corresponding to each high-dimensional vector, and to cluster each sentence corresponding to each high-dimensional vector according to the preset clustering rules to obtain multiple coarse-grained category sets and store them in the vector database.

[0211] Combination Figure 6 The content shown includes cosine similarity at the sentence level and cosine similarity at the paragraph level; the second determining unit 608 includes:

[0212] The first sorting module is used to sort all sentence hierarchical cosine similarities in descending order.

[0213] The first extraction module is used to extract the first m sentence-level cosine similarities from all sorted sentence-level cosine similarities, and to determine the texts corresponding to the first m sentence-level cosine similarities as candidate texts at the sentence level.

[0214] The second sorting module is used to sort all paragraph levels by cosine similarity in descending order.

[0215] The second extraction module is used to extract the first m paragraph level cosine similarities from all sorted paragraph level cosine similarities, and to determine the text corresponding to the first m paragraph level cosine similarities as paragraph level candidate texts.

[0216] The determination module is used to identify sentence-level candidate texts and paragraph-level candidate texts as the second candidate texts.

[0217] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0218] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0219] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-knowledge granularity text retrieval method for RAG, characterized in that, The method comprises: obtaining original text and performing structured arrangement to obtain a JSON file; performing recursive character segmentation processing and semantic segmentation processing on the JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generating a target JSON file based on the text blocks; storing all target JSON files to a BM25 database, and storing the JSON file after recursive character segmentation processing to a vector database; when receiving a query sentence input by a user, preprocessing the query sentence to obtain a keyword list and a query embedding vector corresponding to the query sentence; for each keyword in the keyword list, finding all target texts containing the keyword in the BM25 database, and calculating a relevance score of each target text and the keyword; sorting all relevance scores from large to small, and determining the target texts corresponding to the first n relevance scores as first candidate texts; calculating the cosine similarity between the query embedding vector and the sentence-level embedding vector and the paragraph-level embedding vector in the vector database, respectively; determining second candidate texts according to all cosine similarities; performing deduplication processing on all first candidate texts and all second candidate texts, and determining a final recommended text by using a ranking model, and outputting the final recommended text.

2. The method of claim 1, wherein, The method comprises: obtaining original text and performing structured arrangement to obtain a JSON file; segmenting the original text according to each paragraph in the original text to obtain a plurality of paragraphs; creating a position index for each paragraph according to the chapter sequence position to which the paragraph belongs; generating a JSON file based on each paragraph and its corresponding position index.

3. The method of claim 1, wherein, The method comprises: performing recursive character segmentation processing on the JSON data in the JSON file to obtain a plurality of preliminary text blocks, and creating a first sub-index of each preliminary text block in the paragraph to which the preliminary text block belongs; for each preliminary text block, creating a copy of the preliminary text block and identifying semantic units in the copy; based on the semantic units, performing segmentation on the copy based on a preset segmentation strategy to obtain semantic text blocks, and creating a second sub-index of each semantic text block in the paragraph to which the semantic text block belongs; generating a target JSON file based on each preliminary text block and its corresponding first sub-index, and generating a target JSON file based on each semantic text block and its corresponding second sub-index.

4. The method of claim 1, wherein, The method comprises: For each target JSON file, the target JSON file is preprocessed, and the word frequency of each word in the target JSON file and the frequency of the word appearing in all target JSON files are calculated to obtain the inverse document frequency; Based on the word frequency and the inverse document frequency, the BM25 weight of the word is calculated; An inverted index list is constructed according to the BM25 weight of the word; The preprocessed target JSON file and the corresponding inverted index list are stored in a BM25 database; For each JSON file after recursive character segmentation processing, a sentence embedding model is used to perform vectorization processing on the JSON file after recursive character segmentation processing to generate a plurality of high-dimensional vectors; Each sentence corresponding to each high-dimensional vector is identified, a fine-grained vector is generated, and the fine-grained vector is stored in a vector database; The semantics corresponding to each high-dimensional vector are identified, and each sentence corresponding to each high-dimensional vector is clustered and grouped according to a preset clustering rule to obtain a plurality of coarse-grained category sets and store the coarse-grained category sets in the vector database.

5. The method of claim 1, wherein, When receiving a query sentence input by a user, the query sentence is preprocessed to obtain a keyword list and a query embedding vector corresponding to the query sentence, including: When receiving a query sentence input by a user, the query sentence is segmented and denoised to obtain a keyword list corresponding to the query sentence; The query sentence is vectorized using a sentence embedding model to obtain a query embedding vector corresponding to the query sentence.

6. The method of claim 1, wherein, The cosine similarity includes a sentence-level cosine similarity and a paragraph-level cosine similarity; The second candidate text is determined according to all cosine similarities, including: All sentence-level cosine similarities are sorted in descending order; The first m sentence-level cosine similarities are extracted from the sorted all sentence-level cosine similarities, and the texts corresponding to the first m sentence-level cosine similarities are determined as sentence-level candidate texts; All paragraph-level cosine similarities are sorted in descending order; The first m paragraph-level cosine similarities are extracted from the sorted all paragraph-level cosine similarities, and the texts corresponding to the first m paragraph-level cosine similarities are determined as paragraph-level candidate texts; The sentence-level candidate texts and the paragraph-level candidate texts are determined as the second candidate texts.

7. A multi-knowledge granularity text retrieval apparatus for RAG, characterized by, The apparatus includes: An acquisition unit configured to acquire an original text and perform structured arrangement to obtain a JSON file; A processing unit configured to perform recursive character segmentation processing and semantic segmentation processing on JSON data in the JSON file to obtain a plurality of text blocks containing sub-indexes, and generate a target JSON file based on the text blocks; A storage unit configured to store all target JSON files in a BM25 database, and store the JSON file after recursive character segmentation processing in a vector database; A preprocessing unit configured to, when receiving a query sentence input by a user, preprocess the query sentence to obtain a keyword list and a query embedding vector corresponding to the query sentence; The searching unit is configured to search, for each keyword in the keyword list, all target texts containing the keyword in the BM25 database, and calculate a relevance score of each target text and the keyword; The first determining unit is configured to sort all relevance scores from large to small, and determine target texts corresponding to the first n relevance scores as first candidate texts; The calculating unit is configured to calculate cosine similarity between the query embedding vector and sentence-level embedding vectors and paragraph-level embedding vectors in the vector database respectively; The second determining unit is configured to determine second candidate texts according to all cosine similarities; The output unit is configured to perform deduplication processing on all first candidate texts and all second candidate texts, determine a final recommended text by using a ranking model, and output the final recommended text.

8. The apparatus of claim 7, wherein, The obtaining unit comprises: An obtaining module is configured to obtain an original text; A segmentation module is configured to segment the original text according to each paragraph in the original text to obtain a plurality of paragraphs; A first creating module is configured to create a position index for each paragraph according to a chapter sequence position to which the paragraph belongs; A first generating module is configured to generate a JSON file based on each paragraph and the corresponding position index.

9. The apparatus of claim 7, wherein, The processing unit comprises: A first processing module is configured to perform recursive character segmentation processing on JSON data in the JSON file to obtain a plurality of preliminary text blocks, and create a first sub-index of each preliminary text block in the paragraph to which the preliminary text block belongs; A second creating module is configured to create a copy of each preliminary text block and identify semantic units in the copy; A segmentation module is configured to segment the copy based on a preset segmentation strategy according to the semantic units to obtain semantic text blocks, and create a second sub-index of each semantic text block in the paragraph to which the semantic text block belongs; A second generating module is configured to generate a target JSON file based on each preliminary text block and the corresponding first sub-index, and generate a target JSON file based on each semantic text block and the corresponding second sub-index.

10. The apparatus of claim 7, wherein, The storage unit comprises: A first calculating module is configured to, for each target JSON file, pre-process the target JSON file, calculate a term frequency of each word in the target JSON file, and calculate a frequency of the word appearing in all target JSON files to obtain an inverse document frequency; A second calculating module is configured to calculate a BM25 weight of the word based on the term frequency and the inverse document frequency; A constructing module is configured to construct an inverted index list according to the BM25 weight of the word; A first storage module is configured to store the pre-processed target JSON file and the corresponding inverted index list in a BM25 database; A second processing module is configured to, for each JSON file after recursive character segmentation processing, perform vectorization processing on the JSON file after recursive character segmentation processing by using a sentence embedding model to generate a plurality of high-dimensional vectors; An identifying module is configured to identify each sentence corresponding to each high-dimensional vector, generate a fine-grained vector, and store the fine-grained vector into a vector database; A second storing module is configured to identify a corresponding semantic of each high-dimensional vector, cluster and group each sentence corresponding to each high-dimensional vector according to a preset clustering rule, obtain a plurality of coarse-grained category sets, and store the coarse-grained category sets into the vector database.

Images