RAG indexing method and device, computer equipment and storage medium

Abstract

The invention can be applied to intelligent customer services or intelligent assistants in the field of intelligent medical treatment and finance, and discloses an RAG indexing method and device, computer equipment and a storage medium. The RAG indexing method comprises the steps that a self-adaptive abstract strategy is adopted for an original document to construct a multi-level abstract; and forming a tree-shaped index to organize the document content of the original document according to the constructed multi-stage abstract and the document structure. By implementing the method, the retrieval speed and the retrieval precision can be improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and natural language processing technology, and can be applied to the fields of smart medical care and finance. In particular, it relates to a RAG indexing method, device, computer equipment and storage medium. Background Art

[0002] Intelligent customer service, or intelligent assistants, are one of the most widespread and important ways to implement natural language processing (NLP) technology in real-world scenarios. Search engines are the core modules of intelligent assistant systems and can be applied in smart healthcare or finance to query or manage reference materials such as medical records, medical papers, legal documents, or financial books. In the current field of natural language processing, retrieval-augmented generation (RAG) technology is a key strategy for improving the quality and accuracy of model generation, effectively leveraging large-scale text data sources to generate more accurate and detailed answers. Traditional RAG-based indexing methods primarily rely on information retrieval using vector databases combined with embedded vectors.

[0003] However, traditional RAG-based indexing methods usually divide text into small blocks for embedding and retrieval, which makes it difficult to ensure the coherence of text blocks and the integrity of information. This will cause some content to be broken up, making it difficult to accurately restore the context in the retrieved fragments. In addition, due to the segmentation of the text, the context is easily lost, resulting in semantic fragmentation, making the generated answers lack global understanding, low retrieval accuracy, and each retrieval requires matching multiple blocks, resulting in increased query costs and retrieval volume, and low retrieval speed. Summary of the Invention

[0004] The present invention provides a RAG indexing method, apparatus, computer equipment and storage medium to solve the technical problems existing in traditional RAG-based indexing methods, such as lack of global understanding of answers, low retrieval accuracy, increased query cost and retrieval volume, and low retrieval speed.

[0005] In a first aspect, a RAG indexing method is provided, including:

[0006] Adopting adaptive summarization strategy to construct multi-level summaries for original documents;

[0007] The document content of the original document is organized by forming a tree index based on the constructed multi-level summary combined with the document structure.

[0008] In a second aspect, a RAG indexing device is provided, including:

[0009] A multi-level summary building module is used to build a multi-level summary of the original document using an adaptive summary strategy;

[0010] The index generation module is used to form a tree-shaped index to organize the document content of the original document based on the constructed multi-level summary combined with the document structure.

[0011] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned RAG indexing method when executing the computer program.

[0012] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned RAG indexing method are implemented.

[0013] In the scheme implemented by the above-mentioned RAG indexing method, device, computer equipment and storage medium, a multi-level summary can be constructed by adopting an adaptive summary strategy for the original document; a tree index is formed based on the multi-level summary obtained by construction and the document structure to organize the document content of the original document. In the present invention, for the medical record management of the medical management system of smart medical care, or for the transaction file management of the transaction management system under financial business, a multi-level summary can be constructed by adopting an adaptive summary strategy for the original document; a tree index is formed based on the multi-level summary obtained by construction and the document structure to organize the document content of the original document. Using the tree index to organize data makes the query process more efficient, reduces the loading of irrelevant data, improves retrieval efficiency, and supports large-scale data sets. The tree index can form a directory structure to ensure that the upper and lower directory information can be referenced when answering, improves the accuracy and coherence of the answer, and has a stronger global understanding ability when generating the answer, improves the accuracy of the retrieval, and adds a summary to the index, which is conducive to quickly locating key information, reducing unnecessary full-text searches, reducing the length of the text to be processed, and reducing the computational burden of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 2. It is a schematic diagram of an application environment of a RAG indexing method according to an embodiment of the present invention;

[0016] Figure 2 is a flow chart of a RAG indexing method according to an embodiment of the present invention;

[0017] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S10;

[0018] Figure 4 is a structural diagram of a RAG indexing device according to an embodiment of the present invention;

[0019] Figure 5 is a structural diagram of a computer device in one embodiment of the present invention;

[0020] Figure 6 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] The RAG indexing method provided by the embodiment of the present invention can be applied in the following situations: Figure 1In the application environment, it is used in intelligent assistants or intelligent customer service in application scenarios such as medical care and finance, and is usually implemented through a server, wherein the client communicates with the server through a network. The server can receive user query and search requirements through the client, and construct a multi-level summary for the original document from the database based on the user query and search requirements using an adaptive summary strategy, and organize the document content of the original document by combining the constructed multi-level summary with the document structure to form a tree index, so as to perform retrieval based on the corresponding tree index combined with the key information in the user query and search requirements, so as to facilitate the feedback of the text information corresponding to the summary of the matching tree index as an answer to the client. In the present invention, for the medical record management of the medical management system of smart medical care, or for the transaction file management of the transaction management system under financial business, the original document can be automatically Adapt the summary strategy to construct a multi-level summary, and form a tree index to organize the document content of the original document based on the multi-level summary obtained by construction and in combination with the document structure. Use the tree index to organize data, so that the query process is more efficient, the loading of irrelevant data is reduced, the retrieval efficiency is improved, and large-scale data sets are supported. The tree index can form a directory structure to ensure that the upper and lower directory information can be referenced when answering, improve the accuracy and coherence of the answer, and have a stronger global understanding ability when generating answers, improve the accuracy of the retrieval, and add summaries to the index, which is conducive to quickly locating key information, reducing unnecessary full-text searches, reducing the length of text to be processed, and reducing the computational burden of the model. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0023] See also Figure 2 As shown, Figure 2 A flowchart of a RAG indexing method provided by an embodiment of the present invention includes the following steps:

[0024] S10: Adopt an adaptive summarization strategy to construct a multi-level summary of the original document.

[0025] The RAG indexing method provided by the present invention can be applied to intelligent customer service or intelligent assistants in various application scenarios, such as healthcare and finance. It is typically implemented through a server that can receive user query and search requests in real time. For example, in the medical application field, users often manage medical records via computers and often need to use intelligent assistants to organize or search these records. The original documents can be medical data documents, such as personal health records, prescriptions, and examination reports. Information query has become a channel for users to quickly obtain the information they need in many scenarios. Artificial intelligence models can be used to query the medical record information required by users from massive electronic medical records, helping to provide users with medical record references. For example, a user's query and search request may be the medication usage of a certain department on a certain date. Or, in the field of smart healthcare, users may use computers to search for medical information and often need to use intelligent assistants to search for medical papers. Or, in the financial application field, users may use computers to manage transaction files and use intelligent assistants to organize or search these files. For example, a user's query and search request may be the transaction files between A and B regarding a certain financial product on a certain date. After receiving the user's query and search request, the intelligent assistant needs to use an adaptive summarization strategy to construct a multi-level summary of the original documents in the database.

[0026] Constructing multi-level summaries can reduce redundant data, facilitate quick identification of key information, reduce unnecessary full-text searches, and improve search efficiency. In the adaptive summarization strategy, when the word count of a document falls below a preset word count or the number of sentences in a document falls below a preset sentence count, the original document can be retained as a summary.

[0027] Specifically, if Figure 3 As shown, in step S10, a multi-level summary is constructed for the original document using an adaptive summary strategy, which includes the following steps:

[0028] S11: Preprocess the original document to obtain a standard document. The standard document is then divided into text units based on the document structure or semantic information, with the text units having a granularity no larger than a paragraph. If the granularity of a text unit is no larger than a paragraph, the text unit may be a paragraph or a sentence with a granularity smaller than a paragraph. The document structure includes a table of contents, chapters, paragraphs, sentences, and user-defined tags.

[0029] Specifically, in step S11, the original document is pre-processed to obtain a standard document, specifically:

[0030] The original document is processed by removing HTML tags, invisible characters and redundant punctuation to obtain a cleaned document, and the text encoding and font format of the cleaned document are unified to obtain a standardized document.

[0031] S12: Use a clustering strategy to group text units to obtain text block groups.

[0032] Wherein, the clustering strategy includes a K-means clustering algorithm (K-means clustering algorithm) or a hierarchical clustering algorithm. Then, in step S12, the text units are grouped using the clustering strategy to obtain text block groups, specifically:

[0033] The content-related text units are merged based on semantic similarity according to the K-means clustering algorithm or the hierarchical clustering algorithm to obtain text blocks, and the text blocks are structuredly grouped in combination with the document structure of the obtained standard document to obtain text block class groups.

[0034] Among them, the text block group includes multiple text blocks of the same document structure. The text block refers to the text unit related to the content after merging. The document structure includes the table of contents, chapters, paragraphs, sentences and user-defined tags. The text blocks in the obtained text block group are relatively independent but consistent in theme, ensuring that the content of the same theme or chapter is classified into the same group. For example, when the text blocks are structurally grouped in combination with the chapters of the obtained standard document, the obtained text block group is a grouping of text blocks under different chapters, and a text block group includes multiple text blocks under the same chapter, each text block being a text unit related to different content. The text blocks with consistent themes are obtained by combining similarity merging and structural grouping, so that the text can be decomposed into multiple related subsets to generate summaries separately, effectively reducing the amount of context required to be processed for a single summary.

[0035] S13: A large language model (LLM) is used to generate a corresponding text summary for each text block in each text block group. A global summary is generated for each text summary using a recursive aggregation method. The text summary and the global summary form a multi-level summary.

[0036] Among them, by using a large language model to generate text summaries for each text block within each text block group, a lower-level summary is first generated for the low-level clustering. Multiple text summaries are generated for each text block group. The generated text summaries are then reconstructed using a recursive aggregation method to obtain a higher-level global summary, thereby enhancing the retention of global information through recursive aggregation. The text summary and the global summary form a multi-level summary. Through a multi-level summary generation method, key information is gradually refined to generate a structured multi-level summary while ensuring information integrity, avoiding the loss of key information. Hierarchical multi-level summaries are conducive to quickly locating key information, reducing unnecessary full-text searches, and improving the efficiency of query retrieval.

[0037] Specifically, the global summary may include a global summary for a similar group and / or a comprehensive global summary for a specific group. When the global summary is a global summary for a similar group, in step S13, the recursive aggregation method is used to generate a global summary for each text summary. Specifically, the recursive aggregation method is used to generate a global summary for the similar group for each text summary in the same text block group. When the global summary is a comprehensive global summary for a specific group, in step S13, the recursive aggregation method is used to generate a global summary for each text summary. Specifically, the recursive aggregation method is used to generate a comprehensive global summary for each text group for each text block group.

[0038] Specifically, in order to improve the information concentration of the summary, in step S13, a large language model is used to generate a corresponding text summary for each text block in each text block group, and a recursive aggregation method is used to generate a global summary for each text summary, including the following steps:

[0039] Generate a corresponding text summary for each text block in each text block group using a large language model according to a preset text summary length;

[0040] Using a redundancy elimination mechanism to compare and correct all text summaries; by comparing all text summaries obtained by using a redundancy elimination mechanism, identifying and removing repeated sentences or redundant phrases in all text summaries, and correcting all text summaries, it is possible to avoid information duplication and improve the compactness and information density of the content of the summaries;

[0041] According to the preset global summary length, the revised text summaries are recursively aggregated using a large language model to generate a global summary.

[0042] Among them, by generating corresponding text summaries and global summaries according to the preset text summary length and the preset global summary length respectively, it can be ensured that the length of the summaries at each level of the obtained multi-level summary meets the user's needs. The length of the text summary and the length of the global summary can be constrained by setting the specific corresponding number of text summary words and the global summary word count as the preset text summary length and the global summary length respectively. Alternatively, the length of the summary can be constrained by setting the corresponding number of text summary sentences and the global summary sentence count as the preset text summary length and the global summary length respectively. The length of the summary can also be constrained by setting the corresponding summary length control mark so that the length of the summary does not exceed the expected requirements. By eliminating redundancy and optimizing the summary, the phenomenon of information duplication across paragraphs or chapters can be avoided, the compactness and information density of the content of the summary can be improved, and the accuracy and information concentration of the summary can be improved.

[0043] S14: Record the mapping relationship between the multi-level summary and its corresponding text block. Wherein, the multi-level summary includes the text summary and the global summary, then step S14, that is, recording the mapping relationship between the multi-level summary and its corresponding text block, is specifically: recording the mapping relationship between the text summary and the text block corresponding to the text summary, and recording the mapping relationship between the global summary and the text block corresponding to the global summary. By recording the mapping relationship between the summary and its corresponding text block, the user can quickly locate the corresponding text block from the summary, thereby locating the corresponding text unit, and then quickly locate the content in the standard document. The mapping relationship between the summary and its corresponding text block can improve the interpretability of the summary and enhance the traceability of the retrieval process.

[0044] S20: A tree index is formed based on the constructed multi-level summary and the document structure to organize the document content of the original document.

[0045] Using tree indexes to organize data makes the query process more efficient, reduces the loading of irrelevant data, improves retrieval efficiency, supports large-scale data sets, and establishes a clear hierarchical index system, so that users can first obtain summary answers from high-level nodes when searching, and then go deeper into the lower levels to obtain detailed information as needed, achieving precise positioning from the overall overview to the local details. The tree index can form a directory structure to ensure that the upper and lower directory information can be referenced when answering, improve the accuracy and coherence of the answers, and have a stronger global understanding ability when generating answers, improve the accuracy of retrieval, and can adapt to documents of different sizes and complexities, with high retrieval efficiency and accuracy.

[0046] Among them, in S20, the document content of the original document is organized by forming a tree index based on the constructed multi-level summary and the document structure, specifically:

[0047] The constructed multi-level summaries are sorted and divided according to the document structure to form a tree index, and the document content of the original document is organized using the summaries corresponding to each layer in the tree index.

[0048] Specifically, the step of organizing and dividing the constructed multi-level summaries according to the document structure to form a tree index includes the following steps:

[0049] All summaries of the multi-level summaries grouped by chapters are summarized and summarized to form the root node of the tree index; the root node brings together the overall information of the document and is the highest level.

[0050] Obtaining a primary node of a tree index formed by summaries grouped by chapters of a multi-level summary; the primary node is a chapter summary node, which provides summary information of the document divided by chapters;

[0051] Obtaining a multi-level summary based on the summary grouped by paragraphs to form a secondary node of a tree index; the secondary node is a paragraph summary node, which captures important information within the chapter of the document;

[0052] The key sentences in each paragraph of the original document are obtained to form the leaf nodes of the tree index. The leaf nodes are directly text fragments in the original document, so that detailed content can be directly returned during retrieval.

[0053] Leaf nodes, secondary nodes, primary nodes, and the root node are connected by indexes. Leaf nodes store the original document content, enhancing information integrity and ensuring accuracy and contextual consistency. A tree-like index forms a directory structure, ensuring that answers can reference upper and lower directory information, providing a holistic understanding of the structure and making queries more intuitive and efficient.

[0054] It can be seen that in the above scheme, for the medical record management of the medical management system of smart medical care, or for the transaction file management of the transaction management system under financial business, a multi-level summary can be constructed by adopting an adaptive summary strategy for the original document. According to the constructed multi-level summary, a tree index is formed to organize the document content of the original document in combination with the document structure. The tree index is used to organize data, making the query process more efficient, reducing the loading of irrelevant data, improving retrieval efficiency, and supporting large-scale data sets. The tree index can form a directory structure to ensure that the upper and lower directory information can be referenced when answering, improving the accuracy and coherence of the answer, and having a stronger global understanding ability when generating answers, improving the accuracy of retrieval. Moreover, adding summaries to the index is conducive to quickly locating key information, reducing unnecessary full-text searches, reducing the length of text to be processed, and reducing the computational burden of the model.

[0055] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0056] In one embodiment, a RAG indexing device is provided, which corresponds one-to-one to the RAG indexing method in the above embodiment. Figure 4 As shown, the RAG indexing device includes a multi-level summary construction module 101 and an index generation module 102. The functional modules are described in detail as follows:

[0057] A multi-level summary construction module 101 is used to construct a multi-level summary of the original document using an adaptive summary strategy;

[0058] The index generation module 102 is configured to form a tree-shaped index based on the constructed multi-level summaries and the document structure to organize the document content of the original document.

[0059] In one embodiment, the multi-level summary construction module 101 is specifically configured to:

[0060] Preprocessing the original document to obtain a standard document, and dividing the obtained standard document into text units based on the document structure or semantic information, wherein the granularity of the text unit is no larger than a paragraph;

[0061] Clustering strategy is used to group text units and obtain text block groups;

[0062] A large language model is used to generate a corresponding text summary for each text block in each text block group. A global summary is generated for each text summary using a recursive aggregation method. The text summary and the global summary form a multi-level summary.

[0063] Records the mapping relationship between multi-level summaries and their corresponding text blocks.

[0064] In one embodiment, the multi-level summary construction module 101 is specifically configured to:

[0065] The original document is processed by removing HTML tags, invisible characters and redundant punctuation to obtain a cleaned document, and the text encoding and font format of the cleaned document are unified to obtain a standardized document.

[0066] In one embodiment, the multi-level summary construction module 101 is specifically configured to:

[0067] The content-related text units are merged based on semantic similarity according to the K-means clustering algorithm or the hierarchical clustering algorithm to obtain text blocks, and the text blocks are structuredly grouped in combination with the document structure of the obtained standard document to obtain text block class groups.

[0068] In one embodiment, the multi-level summary construction module 101 is specifically configured to:

[0069] Generate a corresponding text summary for each text block in each text block group using a large language model according to a preset text summary length;

[0070] All text summaries are compared and corrected using a redundancy elimination mechanism;

[0071] According to the preset global summary length, the revised text summaries are recursively aggregated using a large language model to generate a global summary.

[0072] In one embodiment, the index generation module 102 is specifically configured to:

[0073] The constructed multi-level summaries are sorted and divided according to the document structure to form a tree index, and the document content of the original document is organized using the summaries corresponding to each layer in the tree index.

[0074] In one embodiment, the index generation module 102 is specifically configured to:

[0075] Obtain all the summaries of the multi-level summaries grouped by chapters and summarize them to form the root node of the tree index;

[0076] Obtaining a primary node of a tree index of a multi-level summary grouped by chapters;

[0077] Obtaining the secondary nodes of the tree index formed by the summaries grouped according to the paragraphs of the multi-level summary;

[0078] The key sentences in each paragraph of the document content of the original document are obtained to form leaf nodes of the tree index.

[0079] The present invention provides a RAG indexing device, which constructs a multi-level summary of the original document by adopting an adaptive summary strategy, and forms a tree index to organize the document content of the original document based on the constructed multi-level summary combined with the document structure. The tree index is used to organize data, so that the query process is more efficient, the loading of irrelevant data is reduced, the retrieval efficiency is improved, and large-scale data sets are supported. The tree index can form a directory structure to ensure that the upper and lower directory information can be referenced when answering, thereby improving the accuracy and coherence of the answer, so that the global understanding ability is stronger when generating the answer, and the accuracy of the retrieval is improved. Moreover, adding the summary to the index is conducive to quickly locating key information, reducing unnecessary full-text searches, reducing the length of the text to be processed, and reducing the computational burden of the model.

[0080] For the specific definition of the RAG indexing device, please refer to the definition of the RAG indexing method above and will not be repeated here. Each module in the above-mentioned RAG indexing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0081] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a RAG indexing method.

[0082] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a RAG indexing method.

[0083] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0084] Adopting adaptive summarization strategy to construct multi-level summaries for original documents;

[0085] The document content of the original document is organized by forming a tree index based on the constructed multi-level summary combined with the document structure.

[0086] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0087] Adopting adaptive summarization strategy to construct multi-level summaries for original documents;

[0088] The document content of the original document is organized by forming a tree index based on the constructed multi-level summary combined with the document structure.

[0089] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0090] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0091] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0092] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A RAG indexing method, characterized in that: include: Adopting adaptive summarization strategy to construct multi-level summaries for original documents; The document content of the original document is organized by forming a tree index based on the constructed multi-level summary combined with the document structure.

2. The RAG indexing method according to claim 1, wherein: The method of constructing a multi-level summary of the original document using an adaptive summary strategy includes: Preprocessing the original document to obtain a standard document, and dividing the obtained standard document into text units based on the document structure or semantic information, wherein the granularity of the text unit is no larger than a paragraph; Clustering strategy is used to group text units and obtain text block groups; A large language model is used to generate a corresponding text summary for each text block in each text block group. A global summary is generated for each text summary using a recursive aggregation method. The text summary and the global summary form a multi-level summary. Records the mapping relationship between multi-level summaries and their corresponding text blocks.

3. The RAG indexing method according to claim 2, wherein: The original document is preprocessed to obtain a standard document, specifically: The original document is processed by removing HTML tags, invisible characters and redundant punctuation to obtain a cleaned document, and the text encoding and font format of the cleaned document are unified to obtain a standardized document.

4. The RAG indexing method according to claim 2, wherein: The clustering strategy is used to group text units to obtain text block groups, specifically: The content-related text units are merged based on semantic similarity according to the K-means clustering algorithm or the hierarchical clustering algorithm to obtain text blocks, and the text blocks are structuredly grouped in combination with the document structure of the obtained standard document to obtain text block class groups.

5. The RAG indexing method according to claim 2, wherein: The method of using a large language model to generate a corresponding text summary for each text block in each text block group and recursively aggregating each text summary to generate a global summary includes: Generate a corresponding text summary for each text block in each text block group using a large language model according to a preset text summary length; All text summaries are compared and corrected using a redundancy elimination mechanism; According to the preset global summary length, the revised text summaries are recursively aggregated using a large language model to generate a global summary.

6. The RAG indexing method according to claim 1, wherein: The document content of the original document is organized by combining the multi-level summary obtained by the construction with the document structure to form a tree index, specifically: The constructed multi-level summaries are sorted and divided according to the document structure to form a tree index, and the document content of the original document is organized using the summaries corresponding to each layer in the tree index.

7. The RAG indexing method according to claim 6, wherein: The step of arranging and dividing the constructed multi-level summaries according to the document structure to form a tree index specifically includes: Obtain all the summaries of the multi-level summaries grouped by chapters and summarize them to form the root node of the tree index; Obtaining a primary node of a tree index of a multi-level summary grouped by chapters; Obtaining the secondary nodes of the tree index formed by the summaries grouped according to the paragraphs of the multi-level summary; The key sentences in each paragraph of the document content of the original document are obtained to form leaf nodes of the tree index.

8. A RAG indexing device, characterized in that: include: A multi-level summary building module is used to build a multi-level summary of the original document using an adaptive summary strategy; The index generation module is used to form a tree-shaped index to organize the document content of the original document based on the constructed multi-level summary combined with the document structure.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the RAG indexing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the RAG indexing method according to any one of claims 1 to 7 are implemented.

Images