A method for removing duplicate knowledge from multiple sources in text recall in RAG

Abstract

The present invention belongs to the technical field of text recall deduplication, and specifically relates to a method for deduplication of multi-source knowledge in text recall in RAG, comprising the following steps: S1, document text knowledge storage; S11, text information identification and extraction for documents; S12, slicing required for document text when entering ES library; S13, based on ES slice data, secondary slicing before entering Faiss library and recording text index data; S14, secondary slice data entering Faiss library, recording library index; S15, text fusion, reorganization of ES text data and index data in vector library; S16, constructing a complete ES storage data structure, and entering data into ES library. This method significantly improves the problem of knowledge duplication in the RAG process and a series of related problems caused by solving the problems of computing resource consumption and negative question-answering effects, page source display of duplicate knowledge, and irrelevant knowledge from multiple sources, thereby improving the performance of the entire RAG process and the final user experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of text recall and deduplication, and in particular relates to a method for deduplicating multi-source knowledge in text recall in RAG. Background Art

[0002] The development of natural language processing has driven the continuous evolution of text generation models. From the earliest language models (LM) to later neural network-based generation models, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and transformer models, these models have made significant progress in generating text. However, text generation models face many challenges when dealing with knowledge-intensive tasks. These models usually lack the introduction and utilization of external knowledge, so they often fail to provide accurate and diverse information during the generation process. In order to solve the problems in knowledge-intensive tasks, researchers at Meta AI introduced a new method, Retrieval Augmented Generation (RAG). RAG combines information retrieval components and text generation models to improve the quality and diversity of generated text. Compared with traditional language model-based generation methods, RAG has many advantages. First, RAG does not require the entire model to be retrained, so it has higher efficiency. Second, RAG can obtain the latest or private knowledge at any time as needed, without being restricted by static parameterized knowledge, making the generated text more timely and accurate. Most importantly, when we use different databases to store data, we can also give full play to the storage and retrieval advantages of different databases, so that the recalled knowledge is rich and comprehensive, which significantly improves the effect of the answers generated by the final large model.

[0003] In the existing technology, RAG technology, as a natural language processing technology that combines retrieval and generation tasks, has demonstrated its unique advantages in many fields. However, with the continuous development of technology, RAG faces some technical problems in knowledge storage and recall. ES (Elasticsearch) is suitable for full-text search and storage and retrieval of structured data, while the vector library Faiss is suitable for vector retrieval scenarios and can also be combined with other NLP components to form more diverse solutions. For the purpose of question-answering effect, the final stored text is shorter than ES;

[0004] Although storing knowledge in ES and Faiss respectively seems to be able to fully utilize the advantages of the two storage solutions, it also brings new problems, such as the problem of duplicate recall. Duplicate knowledge recall will cause the system large model to spend extra time filtering duplicate knowledge and understanding semantics, resulting in response delays and reduced user experience. At the same time, duplicate knowledge may also interfere with the judgment of the model, making it difficult to filter out the most valuable content from a lot of information. In addition, duplicate recall results will result in a large amount of duplicate knowledge being displayed when we display the source of the answer. Removing duplicates by comparing rules alone will result in a mess of knowledge and lack of readability, resulting in a poor user experience. Summary of the invention

[0005] The purpose of the present invention is to provide a method for deduplicating multi-source knowledge in text recall in RAG. By solving the problems of computing resource consumption and negative question-answering effects, duplicate knowledge displayed in page sources, and unrelated knowledge from multiple sources, the problem of knowledge duplication in the RAG process and a series of related problems caused by it are significantly improved, thereby improving the performance of the entire RAG process and the end-user experience.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for removing duplicate knowledge from multiple sources in text recall in RAG, comprising the following steps:

[0008] S1, document text knowledge is stored in the database;

[0009] S11, identifying and extracting text information from the document;

[0010] S12. Slices required for document text to be entered into the ES library;

[0011] S13. Based on the ES slice data, perform secondary slicing before entering the Faiss library and record the text index data;

[0012] S14, import the secondary slice data into the Faiss library and record the index in the library;

[0013] S15, text fusion, reorganizing ES text data and index data in the vector library;

[0014] S16. Construct a complete ES storage data structure and store data in the ES database;

[0015] S2, recall knowledge to remove duplicates;

[0016] S21. Construct ES query conditions;

[0017] S22. Search the ES database and retain the Top-k results of the ES recall.

[0018] S23, calculate the Embedding vector of the input question;

[0019] S24, recall the most similar Faiss sample;

[0020] S25. Deduplication of indexes.

[0021] In a preferred solution, in the step of identifying and extracting text information from a document, the document text information is identified and extracted according to an API provided in Langchain or other toolkits, and the text information will serve as key data for subsequent algorithm processing.

[0022] In a preferred solution, in the step of slicing the document text when entering the ES library, the ES data is sliced ​​as required. Currently, a slicing rule with a larger granularity is used to make a longer text after slicing so as to retain paragraph information.

[0023] In a preferred solution, in the step of performing secondary slicing before entering the Faiss library based on the ES slice data and recording the text index data, secondary slice data is produced, and at the same time, it is necessary to record the mapping of each ES data (a piece of slice data in the text) to the ES block index list after Faiss slicing.

[0024] In a preferred solution, the secondary slice data is incorporated into the Faiss library, and in the step of recording the index in the library, the index position of the text in the Faiss library is generated according to the position information, combined with the ES block index list for fusion and construction of vector library index data.

[0025] In a preferred solution, in the text fusion, in the step of reorganizing the ES text data and the index data in the vector library, the secondary slice data is fused according to the ES block index list to form the data stored in the ES library, and the index in the vector library is found in docstore_ids through the location information recorded in the ES block index list, and then the index data is recorded in each ES data to be entered.

[0026] In a preferred solution, in the step of constructing a complete ES storage data structure and entering data into the ES library, a corresponding storage structure is generated based on the information sorted out from S11 to S15, and the text and related information are stored in the Faiss vector library and ES.

[0027] In a preferred solution, in the step of calculating the embedding vector of the input question, the question (Question) input by the user is converted into a vector form, so as to facilitate subsequent similarity retrieval in the vector database.

[0028] In a preferred solution, in the step of storing the document text knowledge, after the data is sliced ​​twice into the Faiss library and the index in the library is recorded, text fusion is not performed, and the ES text data and the index data in the vector library are reorganized.

[0029] The technical effects achieved by the present invention are:

[0030] The present invention can eliminate the problem of duplicate knowledge in multi-source recall results by fully implementing the two-step stages of document text knowledge storage and recall knowledge deduplication. When RAG retrieval is enhanced, its own data can be used to the maximum extent. This improvement not only reduces the system's repeated processing of duplicate knowledge, thereby reducing the consumption of computing resources, but also avoids the negative effects of question and answer caused by duplicate knowledge, improves the accuracy and efficiency of question and answer, and solves the consumption of computing resources and negative effects caused by duplicate knowledge after recall.

[0031] The present invention removes duplicates from the recalled knowledge, so that when the source is needed, a large amount of duplicate text does not need to be displayed, thereby improving the user experience and solving the problem of duplicate knowledge display in the page source.

[0032] The present invention integrates and associates knowledge data from different sources by describing and implementing the construction and storage of block indexes when document knowledge is entered into the database. This enables the system to understand and process knowledge from a global perspective, improving the interoperability of knowledge from multiple sources. At the same time, this also provides a more solid foundation for RAG's subsequent knowledge recall and final large model generation. Specifically, the system can associate knowledge data from different sources through block index storage algorithm technology to form a complete knowledge network. In this design, knowledge items can find related knowledge items from other sources, thereby forming a richer and more complete knowledge system, solving the problem of unrelated data structures when there are multiple sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a method for deduplication of multi-source knowledge in text recall of RAG according to the present invention;

[0034] Figure 2 It is a flow chart of document text knowledge storage of the present invention;

[0035] Figure 3 It is a flow chart of the present invention for recalling knowledge and removing duplicates. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0038] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0039] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0040] Please refer to the attached Figures 1 to 3 As shown, a method for removing duplicate knowledge from multiple sources of text recall in RAG includes the following steps:

[0041] S1, document text knowledge is stored in the database;

[0042] S11, identifying and extracting text information from the document;

[0043] S12. Slices required for document text to be entered into the ES library;

[0044] S13. Based on the ES slice data, perform secondary slicing before entering the Faiss library and record the text index data;

[0045] S14, import the secondary slice data into the Faiss library and record the index in the library;

[0046] S15, text fusion, reorganizing ES text data and index data in the vector library;

[0047] S16. Construct a complete ES storage data structure and store data in the ES database;

[0048] S2, recall knowledge to remove duplicates;

[0049] S21. Construct ES query conditions;

[0050] S22. Search the ES database and retain the Top-k results of the ES recall.

[0051] S23, calculate the Embedding vector of the input question;

[0052] S24, recall the most similar Faiss sample;

[0053] S25. Deduplication of indexes.

[0054] In the above, at the stage of document text knowledge storage, the document is firstly identified and extracted for text information, which includes identifying meaningful text units such as keywords, phrases, sentences, etc. in the document, and extracting them for subsequent processing. Specifically, text information is identified for each document, with the purpose of extracting key information in the document. Natural language processing techniques such as word segmentation, part-of-speech tagging, named entity recognition, etc. are used to identify and extract meaningful text units such as keywords, phrases, sentences, etc. in the document. The extracted text information is sliced ​​using appropriate algorithms and techniques according to the characteristics and needs of the document, and the text is sliced ​​into smaller fragments to generate a series of smaller text fragments for better management and retrieval. Then, secondary slicing is performed on the basis of the sliced ​​data to divide the text fragments into smaller fragments to meet the needs of the Faiss library. The sliced ​​data can be stored in the Faiss library, and the text index data of each slice is recorded to facilitate the subsequent precise positioning and access to these slices. Slice, Faiss library is an efficient vector search engine used to store and retrieve high-dimensional vector data. Each slice is converted into a high-dimensional vector and stored in the Faiss library. At the same time, the index of each vector in the Faiss library is recorded so that the subsequent steps can easily access and retrieve these vectors. After completing the slicing and storage of the document text, the data are recombined to achieve text fusion. Specifically, the text data in the ES library and the vector data in the Faiss library are integrated, and the corresponding text fragments and vectors are associated by matching the index data. In this way, we can use these fused data in subsequent retrieval and similarity calculations. Finally, a complete ES storage data structure is constructed and the data is stored in the ES library. The data structure includes various attributes and features of the document, as well as the text slices and vector data associated with it. Use appropriate data models and formats to store the data in the ES library in accordance with specifications for subsequent retrieval and management.

[0055] In the stage of recalling knowledge and deduplicating, after completing the storage of document text, we need to perform knowledge recall and deduplication operations to ensure the consistency and accuracy of the data. Specifically, in order to retrieve relevant knowledge from the ES library, it is necessary to construct appropriate query conditions. These query conditions can be defined based on document attributes, keywords, phrases, etc., so as to filter out qualified results. Use the constructed query conditions to search the ES library to obtain knowledge results that meet the requirements. In order to reduce redundancy and repetition, we only retain the most relevant Top-k results, where k is a pre-set threshold. Before calculating the similarity, we need to convert the input question into the corresponding Embedding vector. This process can be implemented using a pre-trained neural network model or a custom algorithm. The question is mapped into a high-dimensional space to obtain a vector representing its semantic features. Using the calculated Embedding vector of the question, we can perform a similarity search in the Faiss library to find the most similar sample. This process uses the efficient nearest neighbor search algorithm provided by the Faiss library to quickly find the vector closest to the query from a large number of vectors. Finally, the index data is used for judgment. If the same index is found in multiple results, it is considered a duplicate and only one of the results is retained. Through the above steps, the storage, recall and deduplication of document text knowledge are completed, ensuring the consistency and accuracy of the data, providing a reliable foundation for subsequent applications.

[0056] In a preferred embodiment, in the step of identifying and extracting text information from a document, the document text information is identified and extracted according to the API provided in Langchain or other toolkits, and the text information will serve as key data for subsequent algorithm processing.

[0057] In this implementation, by using API for text information recognition and extraction, key data can be quickly extracted from a large number of documents, saving time for manual processing. API usually has high accuracy and can reduce data processing problems caused by human errors. The extracted key data can be directly used as input for subsequent algorithm processing, simplifying the data processing process. By analyzing and mining key data, patterns and trends in the data can be discovered, providing strong support for decision-making.

[0058] Secondly, in the slicing step required for entering the document text into the ES library, the ES data is sliced ​​as required. Currently, a slicing rule with a larger granularity is used to obtain a longer text after slicing in order to retain paragraph information.

[0059] By using large-grained slices as mentioned above, it is possible to ensure that longer texts are retained so that the correlation information between paragraphs is not lost. This helps maintain the coherence of the context in subsequent text analysis, especially when dealing with natural language processing tasks that require understanding the content of the entire text. Large-grained slices can reduce the complexity of data processing because each slice contains more information, so the number of slices can be reduced during batch processing, which may reduce processing time and improve efficiency.

[0060] Secondly, in the step of performing secondary slicing before entering the Faiss library based on the ES slice data and recording the text index data, secondary slice data is produced. At the same time, it is necessary to record the mapping of each ES data (a piece of slice data in the text) to the ES block index list after Faiss slicing.

[0061] In the above, by further subdividing the text data into smaller segments (secondary slicing) and recording the mapping relationship between these segments and the original ES data blocks, a more sophisticated indexing system can be built. This fine-grained indexing helps improve the efficiency and accuracy of the retrieval algorithm, especially when processing large-scale data sets.

[0062] Furthermore, in the step of importing the secondary slice data into the Faiss library and recording the index in the library, the index position of the text in the Faiss library is generated according to the position information, combined with the ES block index list for fusion and construction of the vector library index data.

[0063] In the above, by recording the index position of the text in the Faiss library and combining it with the index list in ES, more flexible and powerful data management functions can be achieved, including quickly locating specific data, updating and maintaining data, and supporting more complex query and analysis operations, improving the efficiency and accuracy of the search, and providing support for long-term maintenance and complex queries of the data, while also leaving room for possible future technology upgrades.

[0064] Secondly, in the step of text fusion and reorganization of ES text data and index data in the vector library, the secondary slice data is fused according to the ES block index list to form the data stored in the ES library, and the index in the vector library is found in docstore_ids through the location information recorded in the ES block index list, and then the index data is recorded in each ES data to be entered.

[0065] In the above, by using the ES block index list as a reference, the original ES library text data and its index data in the vector library can be kept consistent. This consistency helps maintain the integrity and accuracy of the data. After fusing the text data with the corresponding vector index data, text-based and vector-based joint queries can be implemented, thereby providing a more flexible and accurate retrieval method. Recording the index data in the vector library in each ES data to be entered can make data storage more efficient, because this can process text and vector information simultaneously in a unified storage system, avoiding redundant storage between multiple systems. The fused data set supports complex data analysis tasks such as text mining and pattern recognition. These analyses can take advantage of the combined advantages of text information and vector index data to provide deeper insights and more accurate results.

[0066] Secondly, in the step of constructing a complete ES storage data structure and entering data into the ES library, a corresponding storage structure is generated based on the information sorted out from S11 to S15, and the text and related information are stored in the Faiss vector library and ES.

[0067] In the above, constructing a complete ES warehousing data structure based on the information from S11 to S15 and storing the data in the corresponding library can bring many benefits such as unified storage, efficient retrieval, data association, flexible expansion, resource optimization, processing performance, analysis convenience, large-scale operation and simplified maintenance.

[0068] Secondly, in the step of calculating the embedding vector of the input question, the question (Question) input by the user is converted into a vector form, so as to perform similarity retrieval in the vector database later.

[0069] In the above, converting the questions input by the user into vector form and performing similarity search in the vector database can significantly improve the accuracy and efficiency of the search, while providing a richer and more intelligent user experience.

[0070] Furthermore, in the step of storing the document text knowledge, after the data is sliced ​​twice and stored in the Faiss library, the index in the library is recorded without performing text fusion, and the ES text data and the index data in the vector library are reorganized.

[0071] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A method for deduplicating multi-source knowledge in text recall in RAG, characterized by: The following steps are involved: S1, document text knowledge is stored in the database; S11, identifying and extracting text information from the document; S12. Slices required for document text to be entered into the ES library; S13. Based on the ES slice data, perform secondary slicing before entering the Faiss library and record the text index data; S14, import the secondary slice data into the Faiss library and record the index in the library; S15, text fusion, reorganizing ES text data and index data in the vector library; S16. Construct a complete ES storage data structure and store data into the ES database; S2, recall knowledge to remove duplicates; S21. Construct ES query conditions; S22. Search the ES database and retain the Top-k results of the ES recall. S23, calculate the Embedding vector of the input question; S24, recall the most similar Faiss sample; S25, index deduplication; In the text fusion, in the step of reorganizing the ES text data and the index data in the vector library, the secondary slice data is fused according to the ES block index list to form the data stored in the ES library, and the index in the vector library is found in docstore_ids through the location information recorded in the ES block index list, and then the index data is recorded in each ES data to be entered.

2. According to claim 1, a method for deduplicating multi-source knowledge in RAG text recall, characterized in that: In the step of identifying and extracting text information from the document, the document text information is identified and extracted according to the API provided in Langchain or other toolkits, and the text information will be used as key data for subsequent algorithm processing.

3. According to claim 1, a method for deduplication of multi-source knowledge in RAG text recall, characterized in that: In the slicing step required for entering the document text into the ES library, the ES data is sliced ​​as required. Currently, a slicing rule with a larger granularity is used to obtain a longer text after slicing in order to retain paragraph information.

4. According to claim 1, a method for deduplication of multi-source knowledge in RAG text recall, characterized in that: In the step of performing secondary slicing before entering the Faiss library based on the ES slice data and recording the text index data, secondary slice data is produced, and at the same time, it is necessary to record the ES block index list after each ES data is mapped to the Faiss slice.

5. According to claim 4, a method for deduplication of multi-source knowledge in RAG text recall is characterized by: In the step of incorporating the secondary slice data into the Faiss library and recording the index in the library, the index position of the text in the Faiss library is generated, and the fusion and vector library index data are constructed based on the position information and combined with the ES block index list.

6. According to claim 1, a method for deduplication of multi-source knowledge in RAG text recall, characterized in that: In the step of constructing a complete ES storage data structure and entering data into the ES library, a corresponding storage structure is generated based on the information sorted out from S11 to S15, and the text and related information are stored in the Faiss vector library and the ES library.

7. According to claim 1, a method for removing duplicate knowledge from multiple sources in RAG text recall, characterized in that: In the step of calculating the embedding vector of the input question, the question input by the user is converted into a vector form so as to facilitate subsequent similarity retrieval in the vector database.

Images