Application method of tree index structure driven by large model in RAG content analysis

Abstract

The invention provides an application method of a tree index structure driven by a large model in RAG content analysis. The method comprises the steps that an initial text is obtained according to the length of an original text and the length of a preset text; according to a middle text list corresponding to the initial text, tree-shaped storage data corresponding to the middle text list is obtained, the middle text comprises a middle paragraph text and a middle title path corresponding to the middle paragraph text, and the middle title path comprises a plurality of title texts with different title levels; acquiring attribute information corresponding to each node in the tree-shaped storage data, and storing the attribute information corresponding to the node into the node, so that the tree-shaped storage data is updated and a unique ID is allocated to the tree-shaped storage data; the tree-shaped storage data and the ID of the tree-shaped storage data are uploaded to the database corresponding to the RAG, it can be guaranteed that the semantic relation between the overall structure of the text and the initial text is not damaged, and the retrieval efficiency and the generation quality of the RAG can be improved.

Description

Technical Field

[0001] The present invention relates to the field of text processing technology, and in particular to an application method of a large model-driven tree index structure in RAG content parsing. Background Art

[0002] RAG (Retrieval Augmented Generation) is a technology that combines information retrieval and natural language generation. It aims to provide large language models with the ability to retrieve information from external data sources and generate more accurate and relevant answers based on this. In the RAG workflow, when a user asks a question or request, the system first retrieves relevant documents or fragments from its corresponding database. Then, the retrieved information will be integrated into the input prompt of the large language model as additional contextual information to help the model generate more accurate and relevant answers. Therefore, in order for RAG to effectively retrieve and utilize external information, it is necessary to upload the text to the database corresponding to RAG, referred to as text storage, so that RAG can retrieve quickly and effectively. Existing text storage methods include: segmenting the text according to sentences, paragraphs, fixed number of characters, semantics or summaries to obtain several text fragments, and uploading each text fragment as an independent storage unit to the database.

[0003] However, the above method also has the following technical problems:

[0004] Segmenting the text according to the above method may destroy the overall structure of the text and the semantic connection between different text fragments, resulting in context loss. This may also cause some text fragments to lose their original meaning due to lack of necessary context, thus affecting the retrieval efficiency and generation quality of RAG. Summary of the invention

[0005] In view of the above technical problems, the technical solution adopted by the present invention is:

[0006] The application method of the tree index structure driven by a large model in RAG content parsing includes the following steps:

[0007] S1. If the length of the original text is less than the preset text length, the original text is used as the initial text. If the length of the original text is not less than the preset text length, the original text is divided according to the preset text length to obtain a plurality of initial texts, and the length of the initial text is the same as the preset text length.

[0008] S2, obtain the intermediate text list A corresponding to the initial text = {A1, A2, ..., A i , ..., A m}, A iis the i-th intermediate text corresponding to the initial text, the value of i ranges from 1 to m, m is the number of intermediate texts corresponding to the initial text, the intermediate text includes intermediate paragraph text and intermediate title path corresponding to the intermediate paragraph text, wherein the intermediate title path includes several title texts with different title levels, the several title texts are connected in sequence from high to low in order of title level through preset connection characters, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected through preset connection characters.

[0009] S3, Generate rules based on tree structure, A i The middle title path and the middle paragraph text in the text are used to generate a tree structure to store data as A i The corresponding tree stores data.

[0010] S4, according to the node merging rule, A1, A2, ..., A i , ..., A m The corresponding tree-like storage data are merged to obtain the tree-like storage data corresponding to A.

[0011] S5. Obtain the attribute information corresponding to each node in the tree-like storage data corresponding to A, and store the attribute information corresponding to the node in the node, so that the tree-like storage data corresponding to A is updated and a unique ID is assigned to the tree-like storage data. The attribute information corresponding to the node includes a keyword list corresponding to the node text, the keyword list includes several keywords, a subject label corresponding to the node text, text position information corresponding to the node text and a comprehensive vector corresponding to the node, wherein, when the data stored in the node is a title text, the title text is used as the node text, and when the data stored in the node is a paragraph text, the paragraph text is used as the node text, and the text position information corresponding to the node text is information indicating the specific position of the node text in the original text.

[0012] S6. Upload the tree-like storage data corresponding to A and the ID of the tree-like storage data to the database corresponding to RAG.

[0013] The present invention has at least the following beneficial effects:

[0014] The present invention provides an application method of a large model-driven tree index structure in RAG content parsing, the method being capable of obtaining an initial text according to the length of an original text and a preset text length; obtaining an intermediate text list corresponding to the initial text, wherein the intermediate text includes an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, the intermediate title path including a plurality of title texts with different title levels, the plurality of title texts being sequentially connected by preset connection characters in order of the title levels from high to low, the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text being connected by preset connection characters; obtaining tree-like storage data corresponding to the intermediate text list according to the intermediate text; obtaining attribute information corresponding to each node in the tree-like storage data, and storing the attribute information corresponding to the node in the node, so that the tree-like storage data is updated and a unique ID is assigned to the tree-like storage data; uploading the tree-like storage data and the ID of the tree-like storage data to a database corresponding to the RAG. It can be seen that the present invention stores the titles and paragraphs in the text in a tree structure according to a hierarchical relationship to obtain tree-like storage data, which can ensure that the overall structure of the text and the semantic connection between the initial text are not destroyed, and uploads the tree-like storage data and the ID of the tree-like storage data to the database corresponding to RAG, which is beneficial to improving the retrieval efficiency and generation quality of RAG. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a method for applying a large model-driven tree index structure in RAG content parsing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar tasks, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0019] The embodiment of the present invention provides a method for applying a large model driven tree index structure in RAG content parsing, the method comprising the following steps: Figure 1 As shown:

[0020] S1. If the length of the original text is less than the preset text length, the original text is used as the initial text. If the length of the original text is not less than the preset text length, the original text is divided according to the preset text length to obtain a number of initial texts. The length of the initial text is the same as the preset text length. Those skilled in the art know that the preset text length is a text length pre-set by those skilled in the art according to actual needs, and will not be repeated here.

[0021] Specifically, before step S1, data cleaning is performed on the text to be processed through steps S01-S05 to obtain the original text. The text to be processed is the text that needs to be uploaded to the database corresponding to the RAG system. The database corresponding to the RAG system can be understood as a database that stores data that can be retrieved and used by the RAG system:

[0022] S01. De-noising the text to be processed to obtain a first key text. Those skilled in the art are aware that any de-noising method in the prior art belongs to the protection scope of the present invention and will not be described in detail herein.

[0023] S02. Obtain a text type corresponding to the first key text, where the text type includes Chinese text and English text. The Chinese text represents that the characters in the first key text are mainly Chinese characters, and the English text represents that the characters in the first key text are mainly English characters. A person skilled in the art knows that any method for obtaining the text type corresponding to a text in the prior art falls within the protection scope of the present invention, for example, obtaining the text type corresponding to the first key text based on the LangDetect language detection library, which will not be repeated here.

[0024] S03. When the text type corresponding to the first key text is Chinese text, all English punctuation characters in the first key text are replaced with Chinese punctuation characters corresponding to the English punctuation characters to obtain the second key text; when the text type corresponding to the first key text is English text, all Chinese punctuation characters in the first key text are replaced with English punctuation characters corresponding to the English punctuation characters to obtain the second key text.

[0025] S04. Input the second key text into a large language model trained by a preset prompt project to obtain meaningless text corresponding to the second key text, wherein the preset prompt project guides the large language model to process the input text through specific prompt words or instructions to ensure that the output text content is meaningless.

[0026] S05. Delete the meaningless text corresponding to the second key text from the second key text to obtain the original text.

[0027] Through the above steps, the text to be processed is denoised, invalid characters in the text to be processed are removed, such as extra spaces, line breaks, invisible characters, etc., to obtain the first key text, the format of punctuation characters in the first key text is unified to obtain the second key text, the second key text is input into a large language model trained by a preset prompt project to obtain meaningless text corresponding to the second key text, such as comments, advertisements, etc., the meaningless text corresponding to the second key text is deleted from the second key text to obtain the original text, which can ensure that the original text data is clean and has a clear structure.

[0028] S2, obtain the intermediate text list A corresponding to the initial text = {A1, A2, ..., A i , ..., A m}, A i is the i-th intermediate text corresponding to the initial text, the value of i ranges from 1 to m, m is the number of intermediate texts corresponding to the initial text, the intermediate text includes the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text, wherein the intermediate title path includes several title texts with different title levels, and the several title texts are connected in sequence from high to low in title level through preset connection characters, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected through preset connection characters, wherein those skilled in the art know that the preset connection characters are characters preset and set by those skilled in the art according to actual needs, for example: -, ^, &, which will not be repeated here.

[0029] Specifically, step S2 includes the following steps S21-S25:

[0030] S21. Obtain text structure features corresponding to the original text. The text structure features include explicit structure features and implicit structure features. The explicit structure features represent that titles and paragraphs are marked in the text, and the implicit structure features represent that titles and paragraphs are not marked in the text.

[0031] S22, when the text structure feature corresponding to the original text is a display structure feature, the original text is processed according to a preset regular expression to obtain a candidate text list B corresponding to the initial text = {B1, B2, ..., B j , ..., B n}, B j is the jth candidate text corresponding to the initial text, j ranges from 1 to n, n is the number of candidate texts corresponding to the initial text, the candidate texts include candidate paragraph texts and candidate title paths corresponding to the candidate paragraph texts, wherein the candidate title path includes several title texts with different title levels, and the several title texts are sequentially connected by preset connection characters in order of title levels from high to low, the candidate paragraph text and the candidate title path corresponding to the candidate paragraph text are connected by preset connection characters, and the candidate paragraph text is part of the text in the initial text; when the text structure feature corresponding to the original text is an implicit structure feature, the original text is input into a paragraph segmentation model to obtain B, wherein those skilled in the art know that any paragraph segmentation model in the prior art belongs to the protection scope of the present invention, for example: BERT-based paragraph segmentation model, Transformer-based paragraph segmentation model, which will not be repeated here.

[0032] S23. Get the value a, where the initial value of a is 1.

[0033] S24. When a=n, take B as A; when a≠n, go to step S25.

[0034] S25, when B a The candidate title path in B a+1 The candidate title paths in B are the same and a The candidate paragraph text in B a+1 When the semantic similarity between the candidate paragraph texts in B is greater than the preset similarity threshold, a+1 The candidate paragraph texts in B are concatenated a After the candidate paragraph text in B a+1 Delete from B so that B and B a Update and enter step S24; when B a The candidate title path in B a+1 The candidate header paths in B are not the same or a The candidate paragraph text in B a+1When the semantic similarity between the candidate paragraph texts in is not greater than the preset similarity threshold, let a=a+1 and go to step S24, B a is the ath candidate text corresponding to the initial text, B a+1 It is the a+1th candidate text corresponding to the initial text. Those skilled in the art know that any method for obtaining the semantic similarity between two texts in the prior art, such as Cosine similarity and KL divergence, belongs to the protection scope of the present invention, which will not be described in detail here.

[0035] In a specific embodiment, when the text structure feature corresponding to the original text is a display structure feature, B is obtained based on a natural language processing library. Those skilled in the art know that any natural language processing library in the prior art that can obtain structured text belongs to the protection scope of the present invention, such as NLTK, spaCy, which will not be repeated here.

[0036] Through the above steps, according to the text structure features corresponding to the original text, the candidate text list corresponding to the initial text is flexibly obtained, and the candidate text list is dynamically updated according to the semantic similarity between the candidate paragraph texts between two adjacent candidate texts and whether the candidate title paths are the same, and the candidate text list obtained after the last update is used as the intermediate text list, which can ensure that the intermediate paragraph texts in the intermediate texts in the intermediate text list remain semantically coherent and avoid the loss of meaning due to the lack of context. The tree-like storage data is obtained based on the intermediate text, which can ensure that the overall structure of the text and the semantic connection between the initial text are not destroyed, which is conducive to improving the retrieval efficiency and generation quality of RAG.

[0037] S3, Generate rules based on tree structure, A i The middle title path and the middle paragraph text in the text are used to generate a tree structure to store data as A i The corresponding tree-like storage data, wherein the tree structure generation rule is: several title texts in the middle title path that are connected in sequence from high to low title levels by preset connection characters are used as root nodes and child nodes stored in the tree structure storage data in sequence, and the middle paragraph texts are used as leaf nodes stored in the tree structure storage data in sequence.

[0038] S4, according to the node merging rule, A1, A2, ..., A i , ..., A m The corresponding tree-like storage data are merged to obtain the tree-like storage data corresponding to A, where the node merging rule is: when A1, A2, ..., A i , ..., A mIf there are several nodes storing the same data in the corresponding tree-shaped storage data, the several nodes storing the same data are merged into one node, and the child nodes of the several nodes storing the same data are also merged.

[0039] S5. Obtain the attribute information corresponding to each node in the tree-like storage data corresponding to A, and store the attribute information corresponding to the node in the node, so that the tree-like storage data corresponding to A is updated and a unique ID is assigned to the tree-like storage data. The attribute information corresponding to the node includes a keyword list corresponding to the node text, the keyword list includes several keywords, a subject label corresponding to the node text, text position information corresponding to the node text and a comprehensive vector corresponding to the node, wherein, when the data stored in the node is a title text, the title text is used as the node text, and when the data stored in the node is a paragraph text, the paragraph text is used as the node text, and the text position information corresponding to the node text is information indicating the specific position of the node text in the original text.

[0040] Specifically, step S5 includes the following steps:

[0041] S51. Perform word segmentation processing on the node text according to the target word segmentation library, and obtain an initial word list corresponding to the node text, wherein the initial word list includes several initial words, wherein when the text type corresponding to the first key text is Chinese text, the Chinese word segmentation library is used as the target word segmentation library, and when the text type corresponding to the first key text is English text, the English word segmentation library is used as the target word segmentation library. Those skilled in the art know that any Chinese word segmentation library and English word segmentation library in the prior art belong to the protection scope of the present invention, for example: jieba, spacy, which will not be repeated here.

[0042] S52. Traverse the initial word list. When the initial word is different from all the preset stop words in the preset stop word list, use the initial word as an intermediate word to obtain an intermediate word list corresponding to the node text. The intermediate word list includes several intermediate words. The preset stop word list includes several preset stop words. The preset stop words are stop words pre-set by technical personnel in this field according to actual needs.

[0043] S53. Process all the intermediate words in the intermediate word list corresponding to the node text according to a keyword acquisition algorithm to obtain a keyword list corresponding to the node text, wherein the keyword acquisition algorithm includes a TF-IDF algorithm and a TextRank algorithm.

[0044] In a specific embodiment, the intermediate word list corresponding to the node text is input into the deep learning model to obtain the keyword list corresponding to the node text.

[0045] S54. Input the node text into a text topic classification model to obtain a topic label corresponding to the node text. The text topic classification model includes an LDA classification model and a BERT classification model.

[0046] S55. Vectorize the node information set corresponding to the node to obtain a comprehensive vector corresponding to the node, wherein the node information set corresponding to the node includes: node text, a keyword list corresponding to the node text, a topic tag corresponding to the node text, and text position information corresponding to the node text. Those skilled in the art know that any vectorization processing method in the prior art belongs to the protection scope of the present invention, for example: obtaining a vector based on the bge-m3 model, which will not be repeated here.

[0047] Specifically, the comprehensive vector includes 4 vector values, which correspond one-to-one to the node text, the keyword list corresponding to the node text, the topic label corresponding to the node text, and the text position information corresponding to the node text.

[0048] Through the above steps, the attribute information corresponding to each node in the tree-like storage data is obtained, and the attribute information corresponding to the node is stored in the node, so that the node not only stores the node text, but also stores the keyword list corresponding to the node text. The keyword list includes several keywords, the subject label corresponding to the node text, the text position information corresponding to the node text and the comprehensive vector corresponding to the node. This can ensure that the overall structure of the text and the semantic connection between the initial text are not destroyed. Uploading the tree-like storage data and the ID of the tree-like storage data to the database corresponding to RAG is conducive to improving the retrieval efficiency and generation quality of RAG.

[0049] S6. Upload the tree-like storage data corresponding to A and the ID of the tree-like storage data to the database corresponding to RAG, wherein in the tree-like storage data corresponding to A, the child nodes of each node are stored in a list, which contains all the child nodes directly belonging to the node.

[0050] Specifically, after step S5 and before step S6, the following steps are also included:

[0051] S501. Establish several backward links for the root node in the tree storage data corresponding to A. The number of backward links of the root node is the same as the number of child nodes of the root node and the backward links of the root node correspond to the child nodes of the root node one by one. The backward links of the root node point to its corresponding child nodes.

[0052] S502. Establish a forward link and several backward links for the internal nodes in the tree storage data corresponding to A. The internal nodes are other nodes in the tree storage data corresponding to A except the root node and the leaf node. The forward link of the internal node points to the parent node of the internal node. The number of backward links of the internal node is the same as the number of child nodes of the internal node. The backward links of the internal nodes correspond one-to-one to the child nodes of the internal nodes. The backward links of the internal nodes point to their corresponding child nodes.

[0053] S503: Create a forward link for the leaf node in the tree-like storage data corresponding to A, and the forward link of the leaf node points to the parent node of the leaf node.

[0054] S504. Construct a unique node ID for each node in the tree storage data corresponding to A. The node ID includes the ID of the tree storage data where the node is located, the depth of the node in the tree storage data, and the sequence number of the node in its corresponding depth.

[0055] S505. Obtain the link combination list corresponding to A and upload the link combination list to the database corresponding to RAG. The link combination list corresponding to A includes the link combination corresponding to each node in the tree storage data corresponding to A. The link combination includes: node ID, forward link corresponding to the node ID, and backward link corresponding to the node ID.

[0056] Through the above steps, forward links and backward links are established for the nodes in the tree-like storage data, which can ensure that RAG can track the context through the forward links and backward links of the nodes when performing retrieval, which is beneficial to improving the retrieval efficiency and generation quality of RAG.

[0057] Specifically, after step S6, the following steps S10-S60 are also included:

[0058] S10. Obtain a result feedback code corresponding to the tree-shaped storage data corresponding to A, wherein the feedback code is a feedback code returned by the database corresponding to RAG after uploading the tree-shaped storage data corresponding to A to the database corresponding to RAG.

[0059] S20. When the result feedback code corresponding to the tree-like storage data corresponding to A is different from the preset feedback code, all data related to the tree-like storage data corresponding to A in the database corresponding to RAG are deleted, and the tree-like storage data corresponding to A and the ID of the tree-like storage data are re-uploaded to the database corresponding to RAG and enter step S10.

[0060] Specifically, when the result feedback code corresponding to the tree-shaped storage data corresponding to A is different from the preset feedback code, it indicates that the tree-shaped storage data corresponding to A fails to be uploaded.

[0061] S30. When the result feedback code corresponding to the tree-like storage data corresponding to A is the same as the preset feedback code, the initial text corresponding to A is input into the multimodal large model to generate the question text corresponding to A and the standard answer text of the question text corresponding to A.

[0062] Specifically, when the result feedback code corresponding to the tree-shaped storage data corresponding to A is different from the preset feedback code, it indicates that the tree-shaped storage data corresponding to A is uploaded successfully.

[0063] S40. Obtain the target answer text of the question text corresponding to A based on the RAG technology.

[0064] S50. Input the standard answer text of the question text corresponding to A and the target answer text of the question text corresponding to A into the referee model to obtain the score of the target answer text of the question text corresponding to A. The referee model refers to a machine learning or deep learning model used for judging or scoring in an automated evaluation system.

[0065] S60. When the score of the target answer text of the question text corresponding to A is less than the preset score, all data related to the tree-like storage data corresponding to A in the database corresponding to RAG are deleted, and the initial text corresponding to A is divided into two designated texts, and the designated texts are used as new initial texts to enter step S2.

[0066] Through the above steps, when the result feedback code corresponding to the tree-like storage data corresponding to the intermediate text list is different from the preset feedback code, it indicates that the tree-like storage data corresponding to the intermediate text list has failed to be uploaded, and all data related to the tree-like storage data corresponding to the intermediate text list in the database corresponding to RAG is deleted, and the tree-like storage data corresponding to the intermediate text list and the ID of the tree-like storage data are re-uploaded to the database corresponding to RAG. When the result feedback code corresponding to the tree-like storage data corresponding to the intermediate text list is different from the preset feedback code, it indicates that the tree-like storage data corresponding to the intermediate text list has been successfully uploaded. At this time, according to the initial text corresponding to the intermediate text list, the question text and the standard answer text of the question text are obtained. Target answer text, input the standard answer text and target answer text of the question text into the referee model to obtain the score of the target answer text. When the score of the target answer text of the question text is less than the preset score, delete all data related to the tree-like storage data corresponding to the intermediate text list in the database corresponding to RAG, and divide the initial text corresponding to the intermediate text list into two specified texts. Use the specified text as the new initial text, monitor the upload results and segmentation effects in real time according to the result feedback code corresponding to the tree-like storage data and the score of the target answer text, and dynamically adjust the initial text to ensure that the overall structure of the text and the semantic connection between the initial text are not destroyed, which is conducive to improving the retrieval efficiency and generation quality of RAG.

[0067] In a specific embodiment, after all the tree-like storage data corresponding to the initial text are uploaded to the database, an index data set is constructed and uploaded to the database, wherein the index data set includes a keyword mapping set and a target vector set, wherein the keyword mapping set includes the node ID of each node in each tree-like storage data and a keyword list corresponding to the node text corresponding to the node ID, the target vector set and the initial vector set that has been quantized and compressed, wherein the initial vector set includes a comprehensive vector corresponding to each node in the tree-like storage data, wherein those skilled in the art know that any quantization compression method in the prior art belongs to the protection scope of the present invention, such as: PQ quantization compression method, HNSW quantization compression method, which will not be repeated here.

[0068] In a specific embodiment, when using the RAG system to search based on search terms, the search terms and the keyword list in the keyword mapping set are processed based on an exact match retrieval method, the node ID corresponding to the search term is obtained and the node ID is used as the target node ID, all data stored in the node corresponding to the target node ID, all data stored in the node pointed to by the forward link of the node corresponding to the target node ID, and all data stored in the node pointed to by the backward link of the node corresponding to the target node ID are used as a recall result, and the comprehensive vector stored in the node corresponding to the target node ID is used as the key vector corresponding to the recall result.

[0069] In a specific embodiment, when using the RAG system to search based on search terms, the search terms and the keyword list in the keyword mapping set are processed based on a fuzzy search method, the node ID corresponding to the search term is obtained and the node ID is used as the target node ID, all data stored in the node corresponding to the target node ID, all data stored in the node pointed to by the forward link of the node corresponding to the target node ID, and all data stored in the node pointed to by the backward link of the node corresponding to the target node ID are used as a recall result, and the comprehensive vector stored in the node corresponding to the target node ID is used as the key vector corresponding to the recall result.

[0070] In a specific embodiment, when using the RAG system to search according to a search statement, the search statement is vectorized to obtain a statement vector corresponding to the search statement, and the vector similarity between the statement vector and the comprehensive vector in the target vector set is obtained. When the vector similarity is greater than the preset vector similarity, the node ID corresponding to the comprehensive vector corresponding to the vector similarity is used as the target node ID, all data stored in the node corresponding to the target node ID, all data stored in the node pointed to by the forward link of the node corresponding to the target node ID, and all data stored in the node pointed to by the backward link of the node corresponding to the target node ID are used as a recall result, and the comprehensive vector is used as the key vector corresponding to the recall result.

[0071] Through the above steps, an index dataset containing a keyword mapping set and a target vector set is constructed. Based on the index dataset, it supports exact match retrieval, fuzzy retrieval and sentence retrieval, can cope with diverse retrieval needs, and can quickly locate nodes related to the search terms or search data, reduce retrieval time and improve retrieval efficiency. The recall results include not only the data of the target node, but also the node data pointed to by the forward link and the backward link, ensuring the contextual integrity of the recall results, which is conducive to improving the retrieval efficiency and generation quality of RAG.

[0072] In a specific embodiment, when a RAG system is used to search and obtain several recall results, if the vector similarity between the key vectors corresponding to two recall results is greater than a preset vector similarity, one recall result is deleted from the two recall results. This can remove redundant information, streamline the recall results, and significantly improve the quality of the retrieval results.

[0073] In a specific embodiment, when the text to be processed is modified, the initial text corresponding to the modified part of the text to be processed is used as the text to be updated, and all data related to the text to be updated in the database corresponding to the RAG is deleted; based on the modified content in the text to be processed, the text to be updated is updated to obtain the updated text, and the updated text is used as the initial text and enters step S2, so that the tree-like storage data stored in the database corresponding to the RAG is updated, and according to the updated tree-like storage data in the database corresponding to the RAG, the link combination list and index data set in the database corresponding to the RAG are updated.

[0074] Through the above steps, incremental update or real-time modification of the text to be processed is supported. When the text to be processed is modified or updated, only the nodes in the tree storage data corresponding to the modified part or the updated part need to be updated, and there is no need to re-upload the entire modified or updated text to be processed, which reduces the amount of calculation and resource consumption.

[0075] Specifically, the application method of the large model-driven tree index structure in RAG content parsing can be used to parse various types of texts such as news information, academic papers, and legal documents.

[0076] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store a computer program related to a method in a method embodiment, and the computer program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0077] An embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the above embodiment when executing the computer program.

[0078] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0079] The present invention provides an application method of a large model-driven tree index structure in RAG content parsing, the method being capable of obtaining an initial text according to the length of an original text and a preset text length; obtaining an intermediate text list corresponding to the initial text, wherein the intermediate text includes an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, the intermediate title path including a plurality of title texts with different title levels, the plurality of title texts being sequentially connected by preset connection characters in order of the title levels from high to low, the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text being connected by preset connection characters; obtaining tree-like storage data corresponding to the intermediate text list according to the intermediate text; obtaining attribute information corresponding to each node in the tree-like storage data, and storing the attribute information corresponding to the node in the node, so that the tree-like storage data is updated and a unique ID is assigned to the tree-like storage data; uploading the tree-like storage data and the ID of the tree-like storage data to a database corresponding to the RAG. It can be seen that the present invention stores the titles and paragraphs in the text in a tree structure according to a hierarchical relationship to obtain tree-like storage data, which can ensure that the overall structure of the text and the semantic connection between the initial text are not destroyed, and uploads the tree-like storage data and the ID of the tree-like storage data to the database corresponding to RAG, which is beneficial to improving the retrieval efficiency and generation quality of RAG.

[0080] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. The application method of the tree index structure driven by a large model in RAG content parsing is characterized by: The method comprises the following steps: S1. If the length of the original text is less than the preset text length, the original text is used as the initial text. If the length of the original text is not less than the preset text length, the original text is divided according to the preset text length to obtain a plurality of initial texts, and the length of the initial text is the same as the preset text length. S2, obtain the intermediate text list A corresponding to the initial text = {A1, A2, ..., A i , ..., A m }, A i is the i-th intermediate text corresponding to the initial text, i ranges from 1 to m, m is the number of intermediate texts corresponding to the initial text, the intermediate text includes intermediate paragraph text and intermediate title path corresponding to the intermediate paragraph text, wherein the intermediate title path includes a plurality of title texts with different title levels, the plurality of title texts are sequentially connected by preset connection characters in the order of title levels from high to low, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected by preset connection characters; S3, Generate rules based on tree structure, A i The middle title path and the middle paragraph text in the text are used to generate a tree structure to store data as A i The corresponding tree stores data; S4, according to the node merging rule, A1, A2, ..., A i , ..., A m The corresponding tree-like storage data are merged to obtain the tree-like storage data corresponding to A; S5. Obtain the attribute information corresponding to each node in the tree-like storage data corresponding to A, and store the attribute information corresponding to the node in the node, so that the tree-like storage data corresponding to A is updated and a unique ID is assigned to the tree-like storage data, the attribute information corresponding to the node includes a keyword list corresponding to the node text, the keyword list includes a number of keywords, a subject label corresponding to the node text, text position information corresponding to the node text and a comprehensive vector corresponding to the node, wherein when the data stored in the node is a title text, the title text is used as the node text, when the data stored in the node is a paragraph text, the paragraph text is used as the node text, and the text position information corresponding to the node text is information indicating the specific position of the node text in the original text; S6. Upload the tree-like storage data corresponding to A and the ID of the tree-like storage data to the database corresponding to RAG.

2. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 1 is characterized in that: Before step S1, the text to be processed is cleaned to obtain the original text through the following steps. The text to be processed is the text that needs to be uploaded to the database corresponding to the RAG system: S01, performing denoising processing on the text to be processed to obtain the first key text; S02, obtaining a text type corresponding to the first key text, where the text type includes Chinese text and English text; S03. When the text type corresponding to the first key text is Chinese text, all English punctuation characters in the first key text are replaced with Chinese punctuation characters corresponding to the English punctuation characters to obtain a second key text; when the text type corresponding to the first key text is English text, all Chinese punctuation characters in the first key text are replaced with English punctuation characters corresponding to the English punctuation characters to obtain a second key text; S04, inputting the second key text into a large language model trained by a preset prompt project to obtain meaningless text corresponding to the second key text; S05. Delete the meaningless text corresponding to the second key text from the second key text to obtain the original text.

3. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 1 is characterized in that: Step S2 includes the following steps S21-S25: S21, obtaining text structure features corresponding to the original text, the text structure features include explicit structure features and implicit structure features, the explicit structure features represent that the text is marked with titles and paragraphs, and the implicit structure features represent that the text is not marked with titles and paragraphs; S22, when the text structure feature corresponding to the original text is a display structure feature, the original text is processed according to a preset regular expression to obtain a candidate text list B corresponding to the initial text = {B1, B2, ..., B j , ..., B n }, B j is the jth candidate text corresponding to the initial text, j ranges from 1 to n, n is the number of candidate texts corresponding to the initial text, the candidate texts include candidate paragraph texts and candidate title paths corresponding to the candidate paragraph texts, wherein the candidate title paths include a number of title texts with different title levels, the number of title texts are sequentially connected by preset connection characters in the order of title levels from high to low, the candidate paragraph texts and the candidate title paths corresponding to the candidate paragraph texts are connected by preset connection characters, and the candidate paragraph text is part of the text in the initial text; when the text structure feature corresponding to the original text is an implicit structure feature, the original text is input into the paragraph segmentation model to obtain B; S23, obtaining a value a, the initial value of a is 1; S24, when a=n, take B as A, when a≠n, go to step S25; S25, when B a The candidate title path in B a+1 The candidate title paths in B are the same and a The candidate paragraph text in B a+1 When the semantic similarity between the candidate paragraph texts in B is greater than the preset similarity threshold, a+1 The candidate paragraph texts in B are concatenated a After the candidate paragraph text in B a+1 Delete from B so that B and B a Update and enter step S24; when B a The candidate title path in B a+1 The candidate header paths in B are not the same or a The candidate paragraph text in B a+1 When the semantic similarity between the candidate paragraph texts in is not greater than the preset similarity threshold, let a=a+1 and go to step S24, B a is the ath candidate text corresponding to the initial text, B a+1 is the a+1th candidate text corresponding to the initial text.

4. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 1 is characterized in that: The tree structure generation rule is: several title texts in the middle title path that are connected in sequence from high to low title levels by preset connection characters are used as root nodes and child nodes in the storage data of the tree structure, and the middle paragraph text is used as leaf node data in the storage data of the tree structure.

5. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 1 is characterized in that: The node merging rule is: when A1, A2, ..., A i , ..., A m If there are several nodes storing the same data in the corresponding tree-shaped storage data, the several nodes storing the same data are merged into one node, and the child nodes of the several nodes storing the same data are also merged.

6. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 1 is characterized in that: Step S5 includes the following steps: S51, performing word segmentation processing on the node text according to the target word segmentation library, obtaining an initial word list corresponding to the node text, wherein the initial word list includes a plurality of initial words, wherein when the text type corresponding to the first key text is Chinese text, the Chinese word segmentation library is used as the target word segmentation library, and when the text type corresponding to the first key text is English text, the English word segmentation library is used as the target word segmentation library; S52, traversing the initial word list, when the initial word is different from all the preset stop words in the preset stop word list, taking the initial word as an intermediate word to obtain an intermediate word list corresponding to the node text, the intermediate word list includes a plurality of intermediate words, and the preset stop word list includes a plurality of preset stop words; S53, processing all the middle words in the middle word list corresponding to the node text according to the keyword acquisition algorithm to obtain the keyword list corresponding to the node text; S54, inputting the node text into a text topic classification model to obtain a topic label corresponding to the node text; S55. Vectorize the node information set corresponding to the node to obtain a comprehensive vector corresponding to the node, wherein the node information set corresponding to the node includes: node text, a keyword list corresponding to the node text, a topic tag corresponding to the node text, and text position information corresponding to the node text.

7. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 6 is characterized in that: The comprehensive vector includes 4 vector values, which correspond to the node text, the keyword list corresponding to the node text, the topic label corresponding to the node text, and the text position information corresponding to the node text.

8. The method for applying the large model-driven tree index structure in RAG content parsing according to claim 1 is characterized in that: After step S5 and before step S6, the following steps are also included: S501, establishing a number of backward links for the root node in the tree storage data corresponding to A, the number of the backward links of the root node is the same as the number of the child nodes of the root node and the backward links of the root node correspond to the child nodes of the root node one by one, and the backward links of the root node point to its corresponding child nodes; S502, establish a forward link and several backward links for the internal nodes in the tree-like storage data corresponding to A, the internal nodes are other nodes in the tree-like storage data corresponding to A except the root node and the leaf node, the forward link of the internal node points to the parent node of the internal node, the number of the backward links of the internal node is the same as the number of the child nodes of the internal node, and the backward links of the internal nodes correspond to the child nodes of the internal nodes one by one, and the backward links of the internal nodes point to their corresponding child nodes; S503, establishing a forward link for the leaf node in the tree storage data corresponding to A, where the forward link of the leaf node points to the parent node of the leaf node; S504, construct a unique node ID for each node in the tree storage data corresponding to A, where the node ID includes the ID of the tree storage data where the node is located, the depth of the node in the tree storage data, and the sequence number of the node in its corresponding depth; S505. Obtain the link combination list corresponding to A and upload the link combination list to the database corresponding to RAG. The link combination list corresponding to A includes the link combination corresponding to each node in the tree storage data corresponding to A. The link combination includes: node ID, forward link corresponding to the node ID, and backward link corresponding to the node ID.

Images