Application method of large model driven tree index structure in RAG content analysis

Abstract

The application provides an application method of a large model driven tree index structure in RAG content analysis, the method acquires an initial text according to the length of an original text and a preset text length; acquires tree-shaped storage data corresponding to an intermediate text list according to the initial text, wherein the intermediate text includes an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, and the intermediate title path includes a plurality of title texts with different title levels; acquires attribute information corresponding to each node in the tree-shaped storage data, and stores the attribute information corresponding to the node into the node, so that the tree-shaped storage data is updated and a unique ID is allocated to the tree-shaped storage data; and the tree-shaped storage data and the ID of the tree-shaped storage data are uploaded into a database corresponding to the RAG, so that the semantic connection between the overall structure of the text and the initial text is not damaged, and the retrieval efficiency and the generation quality of the RAG are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text processing, in particular to an application method of a tree index structure driven by a large model in RAG content analysis. BACKGROUND

[0002] RAG (Retrieval Augmented Generation) is a technology that combines information retrieval and natural language generation, aiming to provide large language models with the ability to retrieve information from external data sources and generate more accurate and relevant answers based on it. In the workflow of RAG, when a user presents a question or request, the system first retrieves relevant documents or fragments from its corresponding database. Then, the retrieved information is integrated into the input prompt of the large language model as additional context information to help the model generate more accurate and relevant answers. Therefore, in order to enable RAG to effectively retrieve and utilize external information, the text needs to be uploaded to the database corresponding to RAG, referred to as text loading, so that RAG can quickly and effectively retrieve. Existing text loading methods include: cutting the text into several text fragments according to sentences, paragraphs, fixed character quantities, semantics or summaries, and uploading each text fragment as an independent storage unit to the database.

[0003] However, the above method has the following technical problems:

[0004] According to the above method, the text may be cut, which may damage the overall structure of the text and the semantic relationship between different text fragments, resulting in a lack of context. This may also cause some text fragments to lose their original meaning due to the lack of necessary context, affecting the retrieval efficiency and generation quality of RAG. SUMMARY

[0005] To solve the above technical problems, the technical solution adopted by the present application is:

[0006] The application method of the tree index structure driven by the large model in the RAG content analysis, the method comprising the following steps:

[0007] S1, if the length of the original text is less than the preset text length, the original text is taken as the initial text, if the length of the original text is not less than the preset text length, the original text is divided according to the preset text length to obtain several initial texts, the length of the initial text is the same as the preset text length.

[0008] S2, obtaining the intermediate text list A = {A1, A2, …, A i , …, A m} corresponding to the initial text, A iFor the i-th intermediate text corresponding to the initial text, i is 1 to m, m is the number of intermediate texts corresponding to the initial text, the intermediate text includes an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, wherein the intermediate title path includes a plurality of title texts with different title levels, the plurality of title texts are sequentially connected by a preset connection character in the order from high to low of the title levels, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected by the preset connection character.

[0009] S3, generating a tree structure storage data as A i based on the tree structure generation rule, the intermediate title path and the intermediate paragraph text in A i corresponding to the tree structure storage data.

[0010] S4, according to the node merging rule, A1, A2, …, A i , …, A m corresponding to the tree structure storage data are merged to obtain the tree structure storage data corresponding to A.

[0011] S5, obtaining attribute information corresponding to each node in the tree structure storage data corresponding to A, and storing the attribute information corresponding to the node into the node, so as to update the tree structure storage data corresponding to A and assign a unique ID to the tree structure storage data, the attribute information corresponding to the node includes a keyword list corresponding to a node text, a plurality of keywords in the keyword list, a theme tag corresponding to the node text, text position information corresponding to the node text and a comprehensive vector corresponding to the node, wherein when the data stored in the node is a title text, the title text is taken as the node text, when the data stored in the node is a paragraph text, the paragraph text is taken as the node text, and the text position information corresponding to the node text is information representing the specific position of the node text in the original text.

[0012] S6, uploading the tree structure storage data corresponding to A and the ID of the tree structure storage data to the database corresponding to RAG.

[0013] The present application has at least the following beneficial effects:

[0014] The application provides an application method of a large model driven tree index structure in RAG content analysis, which can obtain an initial text according to the length of an original text and a preset text length; obtain an intermediate text list corresponding to the initial text, wherein the intermediate text includes an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, the intermediate title path includes a plurality of title texts with different title levels, the plurality of title texts are sequentially connected by a preset connection character in a descending order of the title levels, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected by the preset connection character; obtain tree storage data corresponding to the intermediate text list according to the intermediate text; obtain attribute information corresponding to each node in the tree storage data, and store the attribute information corresponding to the node into the node, so that the tree storage data is updated and a unique ID of the tree storage data is allocated; and upload the tree storage data and the ID of the tree storage data to a database corresponding to the RAG. It can be known that the title and the paragraph in the text are stored in a tree structure according to a hierarchical relationship to obtain the tree storage data, so that the semantic connection between the overall structure of the text and the initial text is not damaged, and the tree storage data and the ID of the tree storage data are uploaded to the database corresponding to the RAG, which is beneficial to improving the retrieval efficiency and generation quality of the RAG. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 The flowchart of the application method of the large model driven tree index structure in RAG content analysis is provided. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] It is to be understood that the terms "first", "second", and the like, used in the description and the claims of the application and the above-described drawings, are used to distinguish similar tasks, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0019] Embodiments of the application provide a method for applying a large model driven tree index structure in RAG content parsing, the method comprising the following steps, as shown in Figure 1

[0020] S1, if the length of the original text is less than the preset text length, the original text is taken as the initial text, if the length of the original text is not less than the preset text length, the original text is divided according to the preset text length to obtain a plurality of initial texts, the length of the initial text is the same as the preset text length, wherein the preset text length is a text length set by the person skilled in the art according to the actual demand, which will not be described here.

[0021] Specifically, before step S1, the data of the to-be-processed text is cleaned to obtain the original text through steps S01-S05, the to-be-processed text is the text that needs to be uploaded to the database corresponding to the RAG system, and the database corresponding to the RAG system can be understood as a database that stores data that can be retrieved and utilized by the RAG system:

[0022] S01, the to-be-processed text is denoised to obtain a first key text, wherein any denoising method in the prior art belongs to the protection scope of the present application, and will not be described here.

[0023] S02, obtaining the text type corresponding to the first key text, the text type includes Chinese text and English text, the Chinese text represents that the characters in the first key text are mainly Chinese characters, and the English text represents that the characters in the first key text are mainly English characters, wherein any method for obtaining the text type corresponding to the text in the prior art belongs to the protection scope of the present application, for example: obtaining the text type corresponding to the first key text based on LangDetect language detection library, which will not be described here.

[0024] ​S03, when the text type corresponding to the first key text is Chinese text, replacing all English punctuation characters in the first key text with Chinese punctuation characters corresponding to the English punctuation characters to obtain a second key text; when the text type corresponding to the first key text is English text, replacing all Chinese punctuation characters in the first key text with English punctuation characters corresponding to the Chinese punctuation characters to obtain a second key text.

[0025] S04, inputting the second key text into a large language model trained by a preset prompt engineering to obtain meaningless text corresponding to the second key text, wherein the preset prompt engineering guides the large language model to process the input text through specific prompt words or instructions to ensure that the output text content is meaningless.

[0026] S05, deleting the meaningless text corresponding to the second key text from the second key text to obtain the original text.

[0027] Through the above steps, the noise in the to-be-processed text is removed, such as extra spaces, line breaks, invisible characters, etc., to obtain the first key text, the format of the punctuation characters in the first key text is unified to obtain the second key text, the second key text is input into the large language model trained by the preset prompt engineering to obtain the meaningless text corresponding to the second key text, such as annotations, advertisements, etc., and the meaningless text corresponding to the second key text is deleted from the second key text to obtain the original text, which can ensure that the original text data is clean and the structure is clear.

[0028] S2, obtaining an intermediate text list A = {A1, A2, …, A i , …, A m} corresponding to the initial text, A i is the i-th intermediate text corresponding to the initial text, i is 1 to m, m is the number of intermediate texts corresponding to the initial text, and the intermediate text includes an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, wherein the intermediate title path includes a plurality of title texts with different title levels, and the plurality of title texts are sequentially connected by a preset connection character in order from high to low according to the title level, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected by a preset connection character, wherein the preset connection character is a character preset by a person skilled in the art according to actual needs, for example: -, ^, &. Details are not repeated here.

[0029] Specifically, in step S2, the following steps S21-S25 are included:

[0030] S21. Obtain the text structure features corresponding to the original text. The text structure features include explicit structure features and implicit structure features. Explicit structure features indicate that the text is labeled with titles and paragraphs, while implicit structure features indicate that the text is not labeled with titles and paragraphs.

[0031] S22. When the text structure feature corresponding to the original text is a display structure feature, process the original text according to a preset regular expression to obtain a candidate text list B = {B1, B2, ..., B...}. j , ..., B n}, B j Let j be the j-th candidate text corresponding to the initial text, where j ranges from 1 to n, and n is the number of candidate texts corresponding to the initial text. The candidate texts include candidate paragraph texts and candidate title paths corresponding to the candidate paragraph texts. The candidate title paths include several title texts with different title levels, which are sequentially connected by a preset connecting character in descending order of title level. The candidate paragraph texts and the candidate title paths corresponding to the candidate paragraph texts are connected by a preset connecting character. The candidate paragraph texts are a portion of the text in the initial text. When the text structure features corresponding to the original text are implicit structure features, the original text is input into the paragraph segmentation model to obtain B. It is known to those skilled in the art that any paragraph segmentation model in the prior art falls within the protection scope of this invention, such as the BERT-based paragraph segmentation model and the Transformer-based paragraph segmentation model, which will not be elaborated here.

[0032] S23. Get the value a, the initial value of a is 1.

[0033] S24. When a = n, treat B as A; when a ≠ n, proceed to step S25.

[0034] S25, When B a Candidate title paths and B a+1 The candidate title paths are the same and B a The candidate paragraph text in B a+1 When the semantic similarity between candidate paragraphs in the text exceeds a preset similarity threshold, B will be selected. a+1 Candidate paragraph text splicing in B a After the candidate paragraph text in B, a+1 Remove from B to make B and B a Perform the update and proceed to step S24; when B a Candidate title paths and B a+1 The candidate title paths are not the same or B a The candidate paragraph text in B a+1the semantic similarity between the candidate paragraph texts in the two adjacent candidate texts is not greater than a preset similarity threshold, let a=a+1 and enter step S24, B a is the a-th candidate text corresponding to the initial text, B a+1 is the a+1-th candidate text corresponding to the initial text, those skilled in the art know that any method for obtaining the semantic similarity between two texts in the prior art belongs to the protection scope of the present application, for example: Cosine similarity, KL divergence, which will not be repeated here.

[0035] In a specific embodiment, when the text structure feature corresponding to the original text is a display structure feature, B is obtained based on a natural language processing library, those skilled in the art know that any natural language processing library capable of obtaining structured text in the prior art belongs to the protection scope of the present application, for example: NLTK, spaCy, which will not be repeated here.

[0036] Through the above steps, the candidate text list corresponding to the initial text is flexibly obtained according to the text structure feature corresponding to the original text, the candidate text list is dynamically updated according to whether the semantic similarity between the candidate paragraph texts in the two adjacent candidate texts is the same, and the candidate text list obtained after the last update is taken as the intermediate text list, which can ensure that the intermediate paragraph texts in the intermediate texts in the intermediate text list are semantically coherent, avoid the loss of meaning due to the lack of context, obtain the tree-shaped storage data based on the intermediate text, which can ensure that the overall structure of the text and the semantic connection between the initial text are not destroyed, and is conducive to improving the retrieval efficiency and generation quality of RAG.

[0037] S3, based on a tree structure generation rule, A i , the intermediate title path and the intermediate paragraph text in the intermediate text list, generate a tree structure storage data as A i corresponding to the tree structure storage data, wherein the tree structure generation rule is that several title texts in the intermediate title path connected in turn by a preset connection character in the order of title level from high to low are stored as data of root nodes and child nodes in the tree structure storage data in turn, and the intermediate paragraph text is stored as data of leaf nodes in the tree structure storage data.

[0038] S4, according to a node merging rule, A1, A2, …, A i , …, A m corresponding to the tree structure storage data are merged to obtain the tree structure storage data corresponding to A, wherein the node merging rule is that when A1, A2, …, A i , …, A mIf there are several nodes with the same stored data in the corresponding tree-shaped stored data, the several nodes with the same stored data are merged into one node, and the child nodes of the several nodes with the same stored data are also merged.

[0039] S5, obtaining attribute information corresponding to each node in the tree-shaped stored data corresponding to A, and storing the attribute information corresponding to the node into the node, so as to update the tree-shaped stored data corresponding to A and assign a unique ID to the tree-shaped stored data, the attribute information corresponding to the node includes a keyword list corresponding to a node text, several keywords in the keyword list, a topic label corresponding to the node text, text position information corresponding to the node text, and a comprehensive vector corresponding to the node, wherein when the data stored in the node is a title text, the title text is taken as the node text, and when the data stored in the node is a paragraph text, the paragraph text is taken as the node text, and the text position information corresponding to the node text is information indicating the specific position of the node text in the original text.

[0040] Specifically, in step S5, the following steps are included:

[0041] S51, performing word segmentation processing on the node text according to a target word segmentation library to obtain an initial word list corresponding to the node text, the initial word list including several initial words, wherein when the text type corresponding to the first key text is Chinese text, a Chinese word segmentation library is taken as the target word segmentation library, and when the text type corresponding to the first key text is English text, an English word segmentation library is taken as the target word segmentation library, and those skilled in the art know that any Chinese word segmentation library and English word segmentation library in the prior art belongs to the protection scope of the present application, for example: jieba, spacy, which will not be described here.

[0042] S52, traversing the initial word list, and when the initial word is different from all preset stop words in a preset stop word list, taking the initial word as an intermediate word to obtain an intermediate word list corresponding to the node text, the intermediate word list including several intermediate words, the preset stop word list including several preset stop words, and the preset stop words being stop words preset by those skilled in the art according to actual needs.

[0043] S53, processing all intermediate words in the intermediate word list corresponding to the node text according to a keyword acquisition algorithm to obtain a keyword list corresponding to the node text, wherein the keyword acquisition algorithm includes a TF-IDF algorithm and a TextRank algorithm.

[0044] In one specific embodiment, the intermediate word list corresponding to the node text is input into a deep learning model to obtain the keyword list corresponding to the node text.

[0045] S54, input the node text into a text topic classification model to obtain a topic label corresponding to the node text, the text topic classification model including an LDA classification model and a BERT classification model.

[0046] S55, perform vectorization processing on a node information set corresponding to the node to obtain a comprehensive vector corresponding to the node, wherein the node information set corresponding to the node includes the node text, a keyword list corresponding to the node text, a topic label corresponding to the node text, and text location information corresponding to the node text, and any vectorization processing method in the prior art belongs to the protection scope of the present application, for example, a vector is obtained based on a bge-m3 model, which will not be described here.

[0047] Specifically, the comprehensive vector includes four vector values corresponding to the node text, the keyword list corresponding to the node text, the topic label corresponding to the node text, and the text location information corresponding to the node text.

[0048] Through the above steps, attribute information corresponding to each node in the tree-shaped storage data is obtained, and the attribute information corresponding to the node is stored in the node, so that the node not only stores the node text, but also stores the keyword list corresponding to the node text, the topic label corresponding to the node text, the text location information corresponding to the node text, and the comprehensive vector corresponding to the node, which can ensure that the overall structure of the text and the semantic connection between the initial text are not damaged, and uploading the tree-shaped storage data and the ID of the tree-shaped storage data to the database corresponding to the RAG is conducive to improving the retrieval efficiency and generation quality of the RAG.

[0049] S6, upload the tree-shaped storage data corresponding to A and the ID of the tree-shaped storage data to the database corresponding to the RAG, wherein in the tree-shaped storage data corresponding to A, the child nodes of each node are stored in a list, and the list includes all child nodes directly belonging to the node.

[0050] Specifically, after step S5 and before step S6, the following steps are further included:

[0051] S501, establish a plurality of backward links for a root node in the tree-shaped storage data corresponding to A, the number of the backward links of the root node being the same as the number of the child nodes of the root node, and the backward links of the root node corresponding to the child nodes of the root node one by one, the backward links of the root node pointing to the corresponding child nodes.

[0052] S502, establishing a forward link and a plurality of backward links for the internal nodes in the tree-structured storage data corresponding to A, the internal nodes being the nodes other than the root node and the leaf node in the tree-structured storage data corresponding to A, the forward link of the internal node pointing to the parent node of the internal node, the number of the backward links of the internal node being the same as the number of the child nodes of the internal node, the backward links of the internal node corresponding to the child nodes of the internal node one by one, and the backward links of the internal node pointing to the corresponding child nodes.

[0053] S503, establishing a forward link for the leaf node in the tree-structured storage data corresponding to A, the forward link of the leaf node pointing to the parent node of the leaf node.

[0054] S504, constructing a unique node ID for each node in the tree-structured storage data corresponding to A, the node ID including the ID of the tree-structured storage data where the node is located, the depth of the node in the tree-structured storage data, and the sequential number of the node in the corresponding depth.

[0055] S505, obtaining the link combination list corresponding to A and uploading the link combination list to the database corresponding to RAG, the link combination list corresponding to A including the link combination corresponding to each node in the tree-structured storage data corresponding to A, the link combination including the node ID, the forward link corresponding to the node ID, and the backward link corresponding to the node ID.

[0056] Through the above steps, the forward link and the backward link are established for the nodes in the tree-structured storage data, which can ensure that RAG can track the context through the forward link and the backward link of the nodes when performing retrieval, and is beneficial to improving the retrieval efficiency and the generation quality of RAG.

[0057] Specifically, after step S6, the following steps S10-S60 are further included:

[0058] S10, obtaining the result feedback code corresponding to the tree-structured storage data corresponding to A, the feedback code being the feedback code returned by the database corresponding to RAG after uploading the tree-structured storage data corresponding to A to the database corresponding to RAG.

[0059] S20, when the result feedback code corresponding to the tree-structured storage data corresponding to A is different from the preset feedback code, deleting all the data related to the tree-structured storage data corresponding to A in the database corresponding to RAG, and re-uploading the tree-structured storage data corresponding to A and the ID of the tree-structured storage data to the database corresponding to RAG and entering step S10.

[0060] Specifically, when the result feedback code corresponding to the tree-structured storage data corresponding to A is different from the preset feedback code, it indicates that the uploading of the tree-structured storage data corresponding to A fails.

[0061] S30, when the result feedback code corresponding to the tree-shaped storage data corresponding to A is the same as the preset feedback code, input the initial text corresponding to A into the multi-modal large model to generate the question text corresponding to A and the standard answer text of the question text corresponding to A.

[0062] Specifically, when the result feedback code corresponding to the tree-shaped storage data corresponding to A is not the same as the preset feedback code, it indicates that the tree-shaped storage data corresponding to A is successfully uploaded.

[0063] S40, obtaining the target answer text of the question text corresponding to A based on the RAG technology.

[0064] S50, inputting the standard answer text of the question text corresponding to A and the target answer text of the question text corresponding to A into the judging model to obtain the score of the target answer text of the question text corresponding to A, and the judging model refers to a machine learning or deep learning model used for judgment or scoring in the automatic evaluation system.

[0065] S60, when the score of the target answer text of the question text corresponding to A is less than the preset score, deleting all data related to the tree-shaped storage data corresponding to A in the database corresponding to RAG, and dividing the initial text corresponding to A into two specified texts, and inputting the specified texts as new initial texts into step S2.

[0066] Through the above steps, when the result feedback code corresponding to the tree-shaped storage data corresponding to the intermediate text list is not the same as the preset feedback code, it indicates that the tree-shaped storage data corresponding to the intermediate text list fails to upload, all data related to the tree-shaped storage data corresponding to the intermediate text list in the database corresponding to RAG is deleted, and the tree-shaped storage data corresponding to the intermediate text list and the ID of the tree-shaped storage data is re-uploaded to the database corresponding to RAG, when the result feedback code corresponding to the tree-shaped storage data corresponding to the intermediate text list is not the same as the preset feedback code, it indicates that the tree-shaped storage data corresponding to the intermediate text list is successfully uploaded, at this time, according to the initial text corresponding to the intermediate text list, the question text and the standard answer text and the target answer text of the question text are obtained, the standard answer text and the target answer text of the question text are input into the judging model to obtain the score of the target answer text, when the score of the target answer text of the question text is less than the preset score, all data related to the tree-shaped storage data corresponding to the intermediate text list in the database corresponding to RAG is deleted, and the initial text corresponding to the intermediate text list is divided into two specified texts, the specified texts are used as new initial texts, the result of uploading and the effect of cutting are monitored in real time according to the result feedback code corresponding to the tree-shaped storage data and the score of the target answer text, and the initial text is dynamically adjusted to ensure that the overall structure of the text and the semantic connection between the initial text are not damaged, which is conducive to improving the retrieval efficiency and generation quality of RAG.

[0067] In one specific embodiment, after all the initial text corresponding tree storage data is uploaded to the database, the index data set is constructed and uploaded to the database, the index data set includes a keyword mapping set and a target vector set, wherein the keyword mapping set includes the node ID of each node in each tree storage data and the keyword list corresponding to the node text corresponding to the node ID, and the target vector set is the initial vector set that has been quantized and compressed, and the initial vector set includes the comprehensive vector corresponding to each node in the tree storage data, wherein any quantization compression method in the prior art belongs to the protection scope of the present application, for example: PQ quantization compression method, HNSW quantization compression method, which will not be repeated here.

[0068] In one specific embodiment, when using the RAG system to search according to the search term, the search term and the keyword list in the keyword mapping set are processed based on the exact match retrieval method, the node ID corresponding to the search term is obtained and taken as the target node ID, all the data stored in the node corresponding to the target node ID, all the data stored in the node pointed to by the forward link of the node corresponding to the target node ID, and all the data stored in the node pointed to by the backward link of the node corresponding to the target node ID are taken as a recall result, and the comprehensive vector stored in the node corresponding to the target node ID is taken as the key vector corresponding to the recall result.

[0069] In one specific embodiment, when using the RAG system to search according to the search term, the search term and the keyword list in the keyword mapping set are processed based on the fuzzy retrieval method, the node ID corresponding to the search term is obtained and taken as the target node ID, all the data stored in the node corresponding to the target node ID, all the data stored in the node pointed to by the forward link of the node corresponding to the target node ID, and all the data stored in the node pointed to by the backward link of the node corresponding to the target node ID are taken as a recall result, and the comprehensive vector stored in the node corresponding to the target node ID is taken as the key vector corresponding to the recall result.

[0070] In a specific embodiment, when a search is performed according to a search statement using the RAG system, the search statement is vectorized to obtain a statement vector corresponding to the search statement, a vector similarity between the statement vector and a comprehensive vector in the target vector set is obtained, when the vector similarity is greater than a preset vector similarity, a node ID corresponding to the comprehensive vector corresponding to the vector similarity is taken as a target node ID, all data stored in the node corresponding to the target node ID, all data stored in the node pointed to by the forward link of the node corresponding to the target node ID, and all data stored in the node pointed to by the backward link of the node corresponding to the target node ID are taken as a recall result, and the comprehensive vector is taken as a key vector corresponding to the recall result.

[0071] Through the above steps, an index data set containing a keyword mapping set and a target vector set is constructed, accurate match search, fuzzy search and statement search are supported based on the index data set, diversified search requirements can be met, and a node related to a search word or search data can be quickly located, search time is reduced and search efficiency is improved, the recall result not only contains data of the target node, but also contains node data pointed to by the forward link and the backward link, context integrity of the recall result is ensured, and search efficiency and generation quality of the RAG are improved.

[0072] In a specific embodiment, when a search is performed using the RAG system, several recall results are obtained, and when a vector similarity between key vectors corresponding to two recall results is greater than a preset vector similarity, one of the two recall results is deleted, redundant information can be removed, the recall result can be simplified, and the quality of the search result can be significantly improved.

[0073] In a specific embodiment, when the to-be-processed text is modified, an initial text corresponding to a modified part in the to-be-processed text is taken as a to-be-updated text, and all data related to the to-be-updated text in the database corresponding to the RAG is deleted; the to-be-updated text is updated based on the modification content in the to-be-processed text to obtain an updated text, the updated text is taken as the initial text and enters step S2, so that the tree-shaped storage data stored in the database corresponding to the RAG is updated, and the link combination list and the index data set in the database corresponding to the RAG are updated according to the updated tree-shaped storage data in the database corresponding to the RAG.

[0074] Through the above steps, incremental updating or real-time modification of the to-be-processed text is supported, when the to-be-processed text is modified or updated, only the nodes in the tree-shaped storage data corresponding to the modified part or the updated part need to be updated, and the entire to-be-processed text that is modified or updated does not need to be uploaded again, thereby reducing the amount of calculation and resource consumption.

[0075] Specifically, the application method of the large model driven tree index structure in RAG content analysis can be used to analyze news information, academic papers, legal documents and various types of texts.

[0076] The embodiment of the application further provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to store a computer program related to a method in the method embodiment, and the computer program is loaded and executed by the processor to implement the method provided by the above embodiment.

[0077] The embodiment of the application further provides an electronic device, comprising a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided by the above embodiment when executing the computer program.

[0078] The embodiment of the application further provides a computer program product, which comprises program code, and when the program product is executed on an electronic device, the program code is used to make the electronic device execute the steps in the method according to various exemplary embodiments of the application described in the specification.

[0079] The application provides an application method of a large model driven tree index structure in RAG content analysis, which can obtain an initial text according to the length of an original text and a preset text length; obtain an intermediate text list corresponding to the initial text, wherein the intermediate text comprises an intermediate paragraph text and an intermediate title path corresponding to the intermediate paragraph text, the intermediate title path comprises a plurality of title texts with different title levels, the plurality of title texts are sequentially connected by a preset connection character in a descending order of the title levels, and the intermediate paragraph text and the intermediate title path corresponding to the intermediate paragraph text are connected by a preset connection character; obtain tree storage data corresponding to the intermediate text list according to the intermediate text; obtain attribute information corresponding to each node in the tree storage data, and store the attribute information corresponding to the node into the node, so as to update the tree storage data and assign a unique ID to the tree storage data; and upload the tree storage data and the ID of the tree storage data to a database corresponding to RAG. It can be known that the title and the paragraph in the text are stored in a tree structure according to a hierarchical relationship to obtain the tree storage data, so that the semantic connection between the overall structure of the text and the initial text is not damaged, and the tree storage data and the ID of the tree storage data are uploaded to the database corresponding to RAG, which is beneficial to improving the retrieval efficiency and generation quality of RAG.

[0080] While certain specific embodiments of the application have been described in detail herein for the purposes of exemplification, it will be understood by those skilled in the art that the examples are for illustration only and are not intended to limit the scope of the application. It will be further understood by those skilled in the art that various modifications can be made to the embodiments without departing from the scope and spirit of the application.

Claims

1. A method for applying a large model-driven tree index structure in RAG content resolution, characterized in that, The method comprises the following steps: S1, if the length of the original text is less than the preset text length, the original text is taken as the initial text, if the length of the original text is not less than the preset text length, the original text is divided according to the preset text length to obtain a plurality of initial texts, and the length of the initial text is the same as the preset text length; S2, obtaining a list of intermediate texts A = {A1, A2, …, Am} corresponding to the initial text, where Ai is the i-th intermediate text corresponding to the initial text, i is an integer from 1 to m, m is the number of intermediate texts corresponding to the initial text, and the intermediate texts include intermediate paragraph texts and intermediate title paths corresponding to the intermediate paragraph texts, wherein the intermediate title paths include a plurality of title texts with different title levels, the plurality of title texts are sequentially connected by a preset connection character in a descending order of the title levels, and the intermediate paragraph texts and the intermediate title paths corresponding to the intermediate paragraph texts are connected by the preset connection character. i m i ​​​ S3, generating rules based on tree structure, A i intermediate title path and intermediate paragraph text in A i corresponding tree-shaped storage data; S4, according to the node merging rule to A1, A2, …, A i , …, A m The corresponding tree-shaped storage data is merged to obtain the tree-shaped storage data corresponding to A. S5, obtaining attribute information corresponding to each node in the tree-shaped storage data corresponding to A, and storing the attribute information corresponding to the node into the node, so as to update the tree-shaped storage data corresponding to A and assign a unique ID to the tree-shaped storage data, the attribute information corresponding to the node includes a keyword list corresponding to the node text, a plurality of keywords in the keyword list, a topic label corresponding to the node text, text position information corresponding to the node text, and a comprehensive vector corresponding to the node, wherein when the data stored in the node is a title text, the title text is taken as the node text, when the data stored in the node is a paragraph text, the paragraph text is taken as the node text, and the text position information corresponding to the node text is information representing the specific position of the node text in the original text; S6, uploading the tree-shaped storage data corresponding to A and the ID of the tree-shaped storage data into the database corresponding to RAG.

2. The method of claim 1, wherein the method is applied to RAG content resolution. Before step S1, the to-be-processed text is also data cleaned to obtain the original text through the following steps: S01, denoising the to-be-processed text to obtain a first key text; S02, obtaining the text type corresponding to the first key text, the text type including Chinese text and English text; S03, when the text type corresponding to the first key text is Chinese text, replacing all English punctuation characters in the first key text with Chinese punctuation characters corresponding to the English punctuation characters to obtain a second key text; when the text type corresponding to the first key text is English text, replacing all Chinese punctuation characters in the first key text with English punctuation characters corresponding to the Chinese punctuation characters to obtain a second key text; S04, inputting the second key text into a large language model trained through a preset prompt engineering to obtain meaningless text corresponding to the second key text; S05, deleting the meaningless text corresponding to the second key text from the second key text to obtain the original text.

3. The method of claim 1, wherein the method is applied to RAG content resolution. In step S2, the following steps S21-S25 are included: S21, obtaining text structure features corresponding to the original text, the text structure features including display structure features and implicit structure features, the display structure features representing that the text is marked with a title and a paragraph, and the implicit structure features representing that the text is not marked with a title and a paragraph; S22. When the text structure feature corresponding to the original text is a display structure feature, process the original text according to a preset regular expression to obtain a candidate text list B = {B1, B2, ..., B...}. j , ..., B n }, B j Let J be the j-th candidate text corresponding to the initial text, where j ranges from 1 to n, and n is the number of candidate texts corresponding to the initial text. The candidate texts include candidate paragraph texts and candidate title paths corresponding to the candidate paragraph texts. The candidate title paths include several title texts with different title levels, which are connected sequentially by a preset connector character in descending order of title level. The candidate paragraph texts and their corresponding candidate title paths are connected by a preset connector character. The candidate paragraph texts are a portion of the text in the initial text. When the text structure features corresponding to the original text are implicit structure features, the original text is input into the paragraph segmentation model to obtain B. S23, obtaining a value a, the initial value of a being 1; S24, when a=n, taking B as A, when a≠n, entering step S25; S25, when the candidate title path in B a is same as the candidate title path in B a+1 and the semantic similarity between the candidate paragraph text in B a and the candidate paragraph text in B a+1 is greater than a preset similarity threshold, concatenating the candidate paragraph text in B a+1 after the candidate paragraph text in B a , deleting B a+1 from B to update B and B a and enter step S24; when the candidate title path in B a is not same as the candidate title path in B a+1 or the semantic similarity between the candidate paragraph text in B a and the candidate paragraph text in B a+1 is not greater than the preset similarity threshold, letting a=a+1 and entering step S24, B a being the a-th candidate text corresponding to the initial text, B a+1 being the a+1-th candidate text corresponding to the initial text.

4. The method of claim 1, wherein the method is applied to RAG content resolution. The tree structure generation rule is that a plurality of title texts in the intermediate title path connected in turn according to the title level from high to low through a preset connection character are stored as data of root nodes and child nodes in the storage data of the tree structure in turn, and the intermediate paragraph text is stored as data of a leaf node in the storage data of the tree structure.

5. The method of claim 1, wherein the method is applied to RAG content resolution. The node merging rule is: when A1, A2, …, A i , …, A m If there are several nodes with the same stored data in the corresponding tree-shaped stored data, the several nodes with the same stored data are merged into one node, and the child nodes of the several nodes with the same stored data are also merged.

6. The method of claim 1, wherein the method is applied to RAG content resolution. The step S5 comprises the following steps: S51, performing word segmentation processing on the node text according to a target word segmentation library to obtain an initial word list corresponding to the node text, the initial word list comprising a plurality of initial words, wherein when the text type corresponding to the first key text is Chinese text, a Chinese word segmentation library is taken as the target word segmentation library, and when the text type corresponding to the first key text is English text, an English word segmentation library is taken as the target word segmentation library; S52, traversing the initial word list, and when the initial word is different from all preset stop words in a preset stop word list, taking the initial word as an intermediate word to obtain an intermediate word list corresponding to the node text, the intermediate word list comprising a plurality of intermediate words, and the preset stop word list comprising a plurality of preset stop words; S53, processing all intermediate words in the intermediate word list corresponding to the node text according to a keyword acquisition algorithm to obtain a keyword list corresponding to the node text; S54, inputting the node text into a text topic classification model to obtain a topic label corresponding to the node text; S55, performing vectorization processing on the node information set corresponding to the node to obtain a comprehensive vector corresponding to the node, wherein the node information set corresponding to the node comprises the node text, the keyword list corresponding to the node text, the topic label corresponding to the node text, and text location information corresponding to the node text.

7. The method of claim 6, wherein the method is applied to RAG content resolution. The comprehensive vector comprises four vector values corresponding to the node text, the keyword list corresponding to the node text, the topic label corresponding to the node text, and the text location information corresponding to the node text respectively.

8. The method of claim 1, wherein the method is applied to RAG content resolution. After the step S5 and before the step S6, the following steps are further included: S501, establishing a plurality of backward links for the root node in the tree-shaped storage data corresponding to A, the number of the backward links of the root node being the same as the number of the child nodes of the root node, and the backward links of the root node corresponding to the child nodes of the root node one by one, the backward links of the root node pointing to the corresponding child nodes; S502, establishing one forward link and a plurality of backward links for the internal node in the tree-shaped storage data corresponding to A, the internal node being other nodes in the tree-shaped storage data corresponding to A except the root node and the leaf node, the forward link of the internal node pointing to the parent node of the internal node, the number of the backward links of the internal node being the same as the number of the child nodes of the internal node, and the backward links of the internal node corresponding to the child nodes of the internal node one by one, the backward links of the internal node pointing to the corresponding child nodes; S503, establishing one forward link for the leaf node in the tree-shaped storage data corresponding to A, the forward link of the leaf node pointing to the parent node of the leaf node; S504, constructing a unique node ID for each node in the tree-structured storage data corresponding to A, the node ID including the ID of the tree-structured storage data where the node is located, the depth of the node in the tree-structured storage data, and the sequential number of the node in the corresponding depth; S505, obtaining a link combination list corresponding to A and uploading the link combination list to a database corresponding to RAG, the link combination list corresponding to A including a link combination corresponding to each node in the tree-structured storage data corresponding to A, the link combination including: the node ID, a forward link corresponding to the node ID, and a backward link corresponding to the node ID.

Images