A method for enhancing RAG retrieval capability based on graph database technology

Abstract

The present invention discloses a method for enhancing RAG retrieval capability based on graph database technology, the method comprising the following steps: constructing a knowledge vector database; constructing a knowledge graph database; dividing the knowledge graph database into communities; obtaining a question input by a user and converting it into a user question vector, searching for a vector most relevant to the user question vector from the knowledge vector database; determining to which community of the knowledge graph database the searched-for vector most relevant to the user question vector belongs, and returning all domain knowledge involved in the corresponding community; constructing an intent expression based on the question input by the user and the returned domain knowledge of the corresponding community, and using the intent expression to ask questions to a large model. The present application integrates traditional vector databases and graph databases, greatly improving the depth and breadth of retrieval.

Description

Technical Field

[0001] The present invention relates to the field of natural language technology, and in particular to a method for enhancing RAG retrieval capability based on graph database technology. Background Art

[0002] With the continuous development of artificial intelligence technology, retrieval augmented generation (RAG) has become the most popular solution for providing domain knowledge to large models. Business intelligence (BI) can help enterprises make more reasonable decisions. Combining RAG with BI can not only reduce the risk of private data leakage, but also improve the accuracy of large models in answering questions, and make the interaction between people and data more convenient.

[0003] Currently, the retrieval of information related to user questions in RAG is done by searching the vector database through vector similarity to retrieve private data information with high similarity. However, since vector similarity does not satisfy the transitive characteristics, the information searched from the vector database based on the user question is only the content directly related to the question, and a lot of other related contextual information is lost, which ultimately leads to the inaccurate or insufficiently detailed answers to the questions by the large model. Therefore, it is very important to retrieve as much and accurate information related to the user question as possible to enrich the content of the large model prompt words and improve the accuracy of the large model's answers to questions. Summary of the invention

[0004] In view of the above-mentioned prior art, the present invention provides a method for enhancing RAG retrieval capability based on graph database technology, mainly to solve the technical problems existing in the above-mentioned background technology.

[0005] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0006] A method for enhancing RAG retrieval capability based on graph database technology, the method comprising the following steps:

[0007] Collecting domain knowledge, and importing the collected domain knowledge into a vector database template to form a knowledge vector database;

[0008] Based on the knowledge vector database, a format file is generated, each format file contains node ID and edge information, and the format file is imported into a graph database template to form a knowledge graph database;

[0009] Performing community division on the knowledge graph database;

[0010] Obtaining a question input by a user and converting it into a user question vector, and searching the knowledge vector database for a vector most relevant to the user question vector;

[0011] Determine which community of the knowledge graph database the searched vector most relevant to the user question vector belongs to, and return all domain knowledge involved in the corresponding community;

[0012] An intent expression is constructed based on the question input by the user and the domain knowledge of the corresponding community returned, and the intent expression is used to ask questions to the large model.

[0013] Optionally, the specific steps of forming a knowledge vector database include:

[0014] Collect domain knowledge documents that need to be stored, and pre-select a vector database as a vector database template, wherein the vector database includes at least a Faiss library, a Milvus library, a Chroma library, and an Elasticsearch library;

[0015] Preprocessing the collected domain knowledge documents includes: text cleaning, using regular expressions or other tools to remove HTML tags and non-alphanumeric characters in domain knowledge documents; word segmentation, using Chinese word segmentation tools to segment domain knowledge documents; stemming / lemmatization, extracting stems or lemmatization of domain knowledge documents; removing stop words, using a predefined stop word list to remove common words;

[0016] The preprocessed domain knowledge documents are vectorized using vector tools in the vector database, and the vectorized domain knowledge documents are imported into the vector database template to obtain a knowledge vector database.

[0017] Optionally, generating a format file based on the knowledge vector database specifically includes the following steps:

[0018] Calculate the similarity between each vector in the knowledge vector database and other vectors. If the similarity between two vectors is greater than a threshold, an edge is established between the two vectors, and the value of the similarity is the information of the edge.

[0019] Treat each vector in the knowledge vector database as a node and obtain a node ID;

[0020] The starting vector is used as the source node, the target vector is used as the target node, the source node ID and the target node ID are obtained respectively, and the source node ID, the target node ID and the similarity value are used as a line of format data, and multiple lines of format data constitute a format file.

[0021] Optionally, before importing the format file into the graph database template to form the knowledge graph database, it also includes: pre-selecting a graph database as the graph database template, and the graph database includes at least a Neo4j library, an OrientDB library, an Amazon-Neptune library, and a TigerGraph library.

[0022] Optionally, the Louvain algorithm is used to perform community partitioning on the graph database so as to divide the data into different categories, which specifically includes the following steps:

[0023] Obtain nodes and edges from a graph database, and construct a graph based on the nodes and edges;

[0024] The Louvain algorithm is used to divide the graph into communities, and each community contains a certain number of nodes and edges.

[0025] Optionally, obtaining a question input by a user and converting it into a user question vector, and searching the knowledge vector database for a vector most relevant to the user question vector, specifically includes:

[0026] Use the vector tool in the vector database to vectorize the questions input by the user, and convert the questions input by the user into user question vectors;

[0027] The similarity between the user question vector and each vector in the knowledge vector database is calculated, the vector set most relevant to the user question vector is found, and the corresponding node ID is obtained.

[0028] Optionally, determining to which community of the knowledge graph database the searched vector most relevant to the user question vector belongs specifically includes: searching in the constructed graph the community to which each vector in the vector set belongs based on a node ID corresponding to each vector in the vector set.

[0029] Optionally, all nodes in the community corresponding to each vector in the vector set are collected, the domain knowledge corresponding to the nodes are merged into a knowledge set, and the knowledge set is combined with the question input by the user to construct an intention expression to ask questions to the large model.

[0030] The beneficial effects of the present invention are as follows: the embodiments of the present application integrate the innovative design of the traditional vector database and the graph database to coordinate the best advantages of both in vector search and graph traversal, and in actual retrieval effect, greatly improve the depth and breadth of retrieval, so that the retrieval enhancement generation system can connect different knowledge points and provide a comprehensive and contextual knowledge vector graph;

[0031] It further introduces a graph database to search for domain knowledge related to the questions raised by users, thereby enriching the basic domain content provided to the big model, allowing the big model to understand the relevant questions more comprehensively and answer them;

[0032] When building a graph database, we use a threshold to filter out edges with weak similarity, which not only simplifies the graph model and speeds up graph traversal, but also reduces the interference of irrelevant or weakly relevant information on the large model.

[0033] Using the Louvain algorithm in the community discovery algorithm in the graph database can perform a global search in the entire graph instead of just optimizing locally, thereby obtaining higher quality relevant information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for enhancing RAG retrieval capability based on graph database technology in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further elaborated in detail below in conjunction with the drawings and specific embodiments of the specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the following description, the expression "some embodiments" is related to a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0036] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0037] It should be understood that the present invention can be implemented in different forms and should not be interpreted as being limited to the embodiments proposed herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and the scope of the present invention will be fully conveyed to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not intended to be a limitation of the present invention. When used herein, the singular forms of "one", "one" and "said / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "compose" and / or "include" when used in this specification determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0038] It should also be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "inside", "outside", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only implementation method.

[0039] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0040] Example 1

[0041] Please refer to the attached Figure 1 The present application provides a method for enhancing RAG retrieval capability based on graph database technology, the method comprising the following steps:

[0042] S1. Collect domain knowledge and import the collected domain knowledge into a vector database template to form a knowledge vector database;

[0043] S2. Generate a format file based on the knowledge vector database, each format file contains node ID and edge information, and import the format file into a graph database template to form a knowledge graph database;

[0044] S3, dividing the knowledge graph database into communities;

[0045] S4, obtaining the question input by the user and converting it into a user question vector, and searching the knowledge vector database for the vector most relevant to the user question vector;

[0046] S5. Determine which community of the knowledge graph database the searched vector most relevant to the user question vector belongs to, and return all domain knowledge involved in the corresponding community;

[0047] S6. Construct an intent expression based on the question input by the user and the domain knowledge of the corresponding community returned, and use the intent expression to ask questions to the large model.

[0048] For step S1, the specific steps of forming the knowledge vector database include:

[0049] Collect domain knowledge documents that need to be stored. In the embodiment of the present invention, domain knowledge refers to the professional knowledge and experience accumulated in a specific field or industry. It includes the understanding of the concepts, principles, technologies and practices in the field, and how to apply these understandings and skills to solve problems and complete tasks. Domain knowledge can be explicit, that is, knowledge that is clearly expressed, such as documents, reports and technical specifications; for example, in the field of enterprise planning and development, its domain knowledge specifically refers to various professional knowledge and information related to the enterprise business, which usually comes from: 1. Internal documents of the enterprise: including but not limited to employee manuals, operating guides, product manuals, technical documents, research reports, etc. 2. Business processes and policies: internal workflows, rules and regulations, compliance requirements, etc. 3. Employee exchanges and meeting records: meeting minutes, email exchanges, internal forum discussions, etc. 4. Case studies: analysis of successful or failed cases, project reviews and summaries, etc. 5. Industry standards and best practices: common standards and recommended practices in the industry. 6. Professional knowledge documents: white papers and technical articles written by experts. 7. Customer feedback and market research: feedback information from customers, market trend analysis, etc.

[0050] A vector database is pre-selected as a vector database template, and the vector database includes at least Faiss library, Milvus library, Chroma library, and Elasticsearch library. Faiss library is an efficient and flexible vector database developed by Facebook; Milvus library is a cloud-native vector database designed for processing super-large model vector data; Chroma library focuses on providing an efficient approximate nearest neighbor search vector library;

[0051] The collected domain knowledge documents are preprocessed, and the preprocessing process includes:

[0052] 1. Clean the text: Use regular expressions or other tools to remove HTML tags, non-alphanumeric characters, etc. from the text.

[0053] 2. Word segmentation: Use Chinese word segmentation tools (such as Jieba) to segment the text.

[0054] 3. Stemming / lemmatization: For some languages ​​(such as English), tools such as NLTK can be used for stemming or lemmatization.

[0055] 4. Remove stop words: Use a predefined stop word list to remove common words;

[0056] The preprocessed domain knowledge documents are vectorized using vector tools in the vector database, and the vectorized domain knowledge documents are imported into the vector database template to obtain a knowledge vector database.

[0057] For example, assume that the Faiss library is selected as the vector database, and the vector tool PQ (Product Quantizer) model in the database is used to perform text vectorization. The specific process is as follows:

[0058] 1. Load PQ model: Load the pre-trained PQ model.

[0059] 2. Document encoding: Input the preprocessed text into the PQ model to obtain the vector representation of each document.

[0060] 3. Format conversion: Convert each document and its corresponding vector representation into the format required by Faiss.

[0061] 4. Batch import: Use the API or tools provided by the Faiss library to batch import documents and vectors into the database.

[0062] This process ensures that domain knowledge is effectively converted into vector representations and stored in a vector database for subsequent similarity searches.

[0063] In an optional implementation, in step S2, a format file is generated based on the knowledge vector database, specifically comprising the following steps:

[0064] S201, calculating the similarity between each vector in the knowledge vector database and other vectors. If the similarity between two vectors is greater than a threshold, an edge is established between the two vectors. The value of the similarity is the information of the edge. The similarity calculation method includes cosine similarity, Euclidean distance, Jaccard similarity coefficient, etc. The present embodiment adopts the cosine similarity calculation method for calculation. The specific formula is common knowledge of those skilled in the art and is not specifically described in this embodiment.

[0065] Furthermore, the threshold in this embodiment is set to 0.8, and if the similarity between two vectors is higher than the threshold, an edge is established between the two nodes. However, it should be noted that those skilled in the art may also reset other thresholds according to actual conditions.

[0066] S202, treating each vector in the knowledge vector database as a node, and obtaining a node ID;

[0067] S203. Take the starting vector as the source node and the target vector as the target node, obtain the source node ID and the target node ID respectively, and use the source node ID, the target node ID and the similarity value as a line of format data. Multiple lines of format data constitute a format file. Exemplarily, the format file can be a CSV file, each line of which contains two vector IDs and similarity values.

[0068] Furthermore, before the format file is imported into the graph database template to form the knowledge graph database, it also includes: pre-selecting a graph database as the graph database template, wherein the graph database includes at least Neo4j library, OrientDB library, Amazon-Neptune library, and TigerGraph library. Among them, Neo4j library: an open source graph database; OrientDB library: a multi-model database with functions such as graph database, relational database, and document database; Amazon Neptune library: a fully managed graph data service; TigerGraph library: a high-performance graph database optimized for large-model graph data sets.

[0069] After the graph database is selected, the formatted file is imported into the graph database to eventually form a knowledge graph database.

[0070] In an optional implementation, in step S3, the Louvain algorithm is used to perform community partitioning on the graph database so as to divide the data into different categories, which specifically includes the following steps:

[0071] S301, obtaining nodes and edges from a graph database, and constructing a graph based on the nodes and edges;

[0072] S302, using the Louvain algorithm to divide the graph into communities, each community contains a certain number of nodes and edges.

[0073] For example, when the Louvain algorithm is used to divide a graph into communities, the steps include:

[0074] 1. Load graph data:

[0075] Load graph data from the graph database, including node and edge information.

[0076] 2. Build a graph network:

[0077] Build a graph network using a suitable graph library, adding nodes and edges to the graph.

[0078] 3. Use Louvain algorithm to divide the graph into communities.

[0079] The Louvain algorithm is a hierarchical clustering method that finds the best community structure by maximizing modularity. Modularity is an indicator to measure the quality of community structure, which is usually defined as the difference between the actual edge density and the expected edge density in the graph.

[0080] 3-1. Initialize community allocation:

[0081] Assign an initial community label to each node in the graph, usually each node forms a community of its own.

[0082] 3-2. Iterative optimization:

[0083] For each node, we iterate and try to move it to a neighboring community to maximize modularity, and repeat this process until no node movement can increase modularity.

[0084] This step may be repeated many times, merging communities each time, until a global optimum is reached.

[0085] 3-3. Get community division results:

[0086] The final community division result is a set of nodes, each set represents a community.

[0087] In an optional implementation, in step S4, the question input by the user is obtained and converted into a user question vector, and the vector most relevant to the user question vector is searched from the knowledge vector database, specifically including:

[0088] S401, using a vector tool in a vector database to vectorize the question input by the user, and converting the question input by the user into a user question vector, for example, using a PQ model to vectorize the question input by the user, and converting the question input by the user into a user question vector;

[0089] S402: Calculate the similarity between the user question vector and each vector in the knowledge vector database, find the vector set most relevant to the user question vector, and obtain the corresponding node ID.

[0090] Exemplarily, the calculation formula of cosine similarity is used to calculate the similarity between the user question vector and each vector in the knowledge vector database, the vectors in the database are sorted according to the cosine similarity, and the top 3 or top 5 vectors with the highest similarity are selected as the most relevant vector set. A minimum similarity threshold value, such as 0.7, can be set, and only vectors with a similarity greater than 0.7 will be considered, and multiple corresponding node IDs are obtained based on the multiple vectors.

[0091] In an optional implementation, determining to which community of the knowledge graph database the searched vector most relevant to the user question vector belongs specifically includes: based on the node ID corresponding to each vector in the vector set, searching in the constructed graph for the community to which each vector in the vector set belongs.

[0092] For example, the most relevant vector set is found from the vector database, and the IDs of these vectors are V1, V2, and V3 respectively. For each vector, a graph traversal algorithm is executed to find the community to which they belong. The graph traversal algorithm includes depth-first search (DFS), breadth-first search (BFS), etc., to explore all nodes and their connections in the community. When traversing the graph, certain restrictions can be set, such as the maximum depth of traversal or the maximum number of edges, to avoid searching for information that is too lengthy or irrelevant. For example, the result obtained is that V1 belongs to community A, V2 belongs to community B, and V3 also belongs to community C.

[0093] In an optional implementation, all nodes in the community corresponding to each vector in the vector set are collected, the domain knowledge corresponding to the nodes are merged into a knowledge set, and the knowledge set is combined with the question input by the user to construct an intention expression to ask questions to the large model.

[0094] For example, when it is known that V1 belongs to community A, V2 belongs to community B, and V3 also belongs to community C, traverse community A to obtain all nodes contained in community A, traverse community B to obtain all nodes contained in community B, traverse community C to obtain all nodes contained in community C, extract all domain knowledge associated with the above communities to form a knowledge set, and combine it with the user's question in a certain format to form a prompt word as the intent expression, and finally input the above intent expression into the big model to ask questions, and the big model will answer them.

[0095] For another example, when it is known that V1, V2, and V3 all belong to community A, only all nodes contained in community A are obtained, and all domain knowledge associated with the above nodes is extracted and combined with the user's question to form a prompt word as the expression of intent.

[0096] The large model refers to a large model, which usually refers to a large-scale pre-trained model. Such models are widely used in natural language processing (NLP), computer vision (CV) and other artificial intelligence fields. They are characterized by a very large parameter scale, usually containing billions or even more parameters, which enables them to learn extremely complex patterns and express highly abstract concepts. In this embodiment, the types of large models include at least the "ChatGPT" large model, the "Wenxin Yiyan" model, the "Tongyi Qianwen" large model, and the "chatglm" large model.

[0097] Example 2

[0098] The difference between this embodiment 2 and embodiment 1 is that, after determining to which community of the knowledge graph database the searched vector most relevant to the user question vector belongs, a graph traversal is performed in the corresponding community to find all nodes related to the most relevant vector.

[0099] For example, the IDs of the most relevant vectors are V1, V2, and V3, and V1 belongs to community A, V2 belongs to community B, and V3 belongs to community C. In community A, the nodes connected to V1 are traversed, in community B, the nodes connected to V2 are traversed, and in community C, the nodes connected to V3 are traversed. The domain knowledge associated with the above nodes is extracted and combined with the user's question in a certain format to form a prompt word as the intent expression. Finally, the above intent expression is input into the big model to ask questions, and the big model gives answers.

[0100] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for enhancing RAG retrieval capability based on graph database technology. It is characterized in that The method comprises the following steps: Collecting domain knowledge, and importing the collected domain knowledge into a vector database template to form a knowledge vector database; Based on the knowledge vector database, a format file is generated, each format file contains node ID and edge information, and the format file is imported into a graph database template to form a knowledge graph database; Performing community division on the knowledge graph database; Obtaining a question input by a user and converting it into a user question vector, and searching the knowledge vector database for a vector most relevant to the user question vector; Determine which community of the knowledge graph database the searched vector most relevant to the user question vector belongs to, and return all domain knowledge involved in the corresponding community; Constructing an intent expression based on the question input by the user and the domain knowledge of the corresponding community returned, and using the intent expression to ask questions to the large model; Based on the knowledge vector database, a format file is generated, which specifically includes the following steps: Calculate the similarity between each vector in the knowledge vector database and other vectors. If the similarity between two vectors is greater than a threshold, an edge is established between the two vectors, and the value of the similarity is the information of the edge. Treat each vector in the knowledge vector database as a node and obtain a node ID; Take the starting vector as the source node and the target vector as the target node, obtain the source node ID and the target node ID respectively, and take the source node ID, the target node ID and the similarity value as a line of format data, and multiple lines of format data constitute a format file; Use the Louvain algorithm to perform community partitioning on the graph database to divide the data into different categories. The specific steps include: Obtain nodes and edges from a graph database, and construct a graph based on the nodes and edges; The Louvain algorithm is used to divide the graph into communities, and each community contains a certain number of nodes and edges; Determining to which community of the knowledge graph database the searched vector most relevant to the user question vector belongs, specifically comprising: searching in the constructed graph the community to which each vector in the vector set belongs based on a node ID corresponding to each vector in the vector set; Collect all nodes in the community corresponding to each vector in the vector set, merge the domain knowledge corresponding to the nodes into a knowledge set, and construct an intention expression with the knowledge set and the question input by the user to ask questions to the large model; After determining the community to which each vector in the vector set belongs, a graph traversal is performed in the corresponding community to find out the nodes connected to each vector in the vector set.

2. According to claim 1, a method for enhancing RAG retrieval capability based on graph database technology, It is characterized in that The specific steps of forming a knowledge vector database include: Collect domain knowledge documents that need to be stored, and pre-select a vector database as a vector database template, wherein the vector database includes at least a Faiss library, a Milvus library, a Chroma library, and an Elasticsearch library; Preprocessing the collected domain knowledge documents includes: text cleaning, using regular expressions or other tools to remove HTML tags and non-alphanumeric characters in domain knowledge documents; word segmentation, using Chinese word segmentation tools to segment domain knowledge documents; stemming / lemmatization, extracting stems or lemmatization of domain knowledge documents; removing stop words, using a predefined stop word list to remove common words; The preprocessed domain knowledge documents are vectorized using vector tools in the vector database, and the vectorized domain knowledge documents are imported into the vector database template to obtain a knowledge vector database.

3. According to claim 2, a method for enhancing RAG retrieval capability based on graph database technology, It is characterized in that Before importing the format file into the graph database template to form a knowledge graph database, it also includes: pre-selecting a graph database as a graph database template, and the graph database includes at least a Neo4j library, an OrientDB library, an Amazon-Neptune library, and a TigerGraph library.

4. According to claim 3, a method for enhancing RAG retrieval capability based on graph database technology, It is characterized in that Obtaining the question input by the user and converting it into a user question vector, and searching the knowledge vector database for the vector most relevant to the user question vector, specifically including: Use the vector tool in the vector database to vectorize the questions input by the user, and convert the questions input by the user into user question vectors; The similarity between the user question vector and each vector in the knowledge vector database is calculated, the vector set most relevant to the user question vector is found, and the corresponding node ID is obtained.

Images