Implementation method of chat robot based on canal field

Abstract

The invention discloses a chat robot implementation method based on the canal field. The chat robot implementation method comprises the following steps: constructing a canal field knowledge base; partitioning the text, and extracting entities, relationships and attribute information from the partitioned text; constructing a knowledge graph, and performing community division on the knowledge graph by adopting a Leiden algorithm; generating a partition hierarchical abstract by using a large language model (LLM); after a query is received, generating an answer by using the generated partitioned hierarchical abstract; according to the method, more comprehensive and abundant answers can be generated, and a remarkable application value is shown in education and auxiliary decision-making scenes in the canal field.

Description

Technical Field

[0001] The present invention relates to the technical field of chat robots, and in particular to an implementation method of a chat robot based on the canal field. Background Art

[0002] The emergence of Generative Artificial Intelligence (GAI) tools such as Chat GPT, SoraBard, and Tongyi Qianwen is changing the way we understand and solve problems. OpenAI's GPT-3, released in 2020, has rapidly gained adoption in a wide range of scenarios thanks to its exceptional semantic understanding, matching, and generation capabilities, garnering significant attention. GAI has quickly been put to use in professional fields, generating diagnostic reports for pathology and other disciplines, automating contract generation, and facilitating news gathering and writing. It has become an indispensable tool in healthcare, law, journalism, and other fields.

[0003] The application of GAI in the canal sector also shows great potential. For example, companies such as ESRI and Google have launched GAI-based assistants to assist users in data querying, performing spatial analysis tasks, and automatically generating maps. However, the application of these technologies is still primarily targeted at professionals. For non-professionals, lack of relevant knowledge makes it difficult to manage and analyze canal information. Therefore, there is an urgent need for a question-answering robot that can answer questions, help them understand relevant knowledge, and then formulate countermeasures based on the results of spatial analysis.

[0004] Existing general-purpose GAI tools (such as ChatGPT and Wenyan Yixin) still have limitations in the canal field. Due to the complex, highly specialized, and interdisciplinary knowledge system in the canal field, they often cannot generate in-depth or specific answers to related questions. At the same time, for the same reason, the answers generated by existing canal-related question-answering robots may lack breadth and depth. Therefore, there is an urgent need to launch a question-answering robot that can understand the complex knowledge system in the canal field and generate more comprehensive and accurate answers to make up for the shortcomings of existing technologies.

[0005] Retrieval-Augmented Generation (RAG) technology can introduce external knowledge bases to provide support for text generation in large language models (LLM), thereby improving the accuracy and professionalism of answers. However, RAG technology still has shortcomings. It cannot effectively integrate information, especially in complex reasoning tasks. Its simple retrieval-generation process may find it difficult to capture the deep relationships between information. Recently, the emerging Graph RAG technology starts from building a knowledge graph, identifying entities in the corpus, establishing connections and forming a relationship network, so that text retrieval is no longer limited to a single sentence or paragraph, which makes up for the shortcomings of RAG. Microsoft open-sourced its Graph RAG project in July 2024. With its excellent performance and accessibility, the project is having a wide impact. There is broad potential to apply this technology to the canal field. Summary of the Invention

[0006] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide an implementation method of a chatbot based on the canal field. The present invention can generate more comprehensive and rich answers, and show significant application value in education and decision-making support scenarios in the canal field.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for implementing a chat robot based on the canal field, comprising the following steps:

[0008] Step 1: Build a canal domain knowledge base;

[0009] Step 2: Divide the text into blocks and extract entity, relationship and attribute information from the block text;

[0010] Step 3: Build a knowledge graph and use the Leiden algorithm to divide the knowledge graph into communities;

[0011] Step 4: Generate a partitioned hierarchical summary using the large language model (LLM); after receiving a query, generate an answer using the generated partitioned hierarchical summary.

[0012] As a further improvement of the present invention, the step 2 is specifically as follows:

[0013] Graph RAG is used to split the corpus into text blocks and process the overlapping parts to ensure semantic continuity; Graph RAG is used to determine the extraction targets from the text blocks, first the entities, including the names and types of the entities, and then the relationships and attributes between the entities, to generate the nodes and edge information of the knowledge graph, and iteratively extract the missing content.

[0014] As a further improvement of the present invention, the step 3 is specifically as follows:

[0015] Graph RAG uses the extracted entities and relationships to generate a preliminary knowledge graph, where entities are nodes and relationships are edges. It also generates embedding vectors for nodes and edges, calculates the similarity between nodes, and uses this to generate edge weights. It also uses a community monitoring algorithm to divide the knowledge graph into several communities, determine the strength of the relationships between nodes, and aggregate semantically similar nodes into the same community.

[0016] As a further improvement of the present invention, the step 4 is specifically as follows:

[0017] GraphRAG defines a hierarchy of summary generation, from high-level overviews to fine-grained community content. GraphRAG uses hierarchical community summaries to answer user questions, including: generating local answers from all relevant community summaries; integrating local answers into global answers; dynamically adjusting the level and scope of answers as user questions change; matching the most relevant communities or entities, and optimizing the relevance of retrieval results through vector similarity.

[0018] The beneficial effects of the present invention are:

[0019] Generative AI is becoming increasingly popular, but there are relatively few dedicated applications in the canal field, so answers to questions in this field often lack professionalism. Graph RAG builds a knowledge graph and integrates knowledge base context information to generate higher-quality answers, which has broad application prospects. The present invention uses Graph RAG to train LLM and launches a canal domain chatbot (domain chatbot) Canal Expert. In automated and manual evaluations, Canal Expert demonstrated its excellent ability to understand canal domain knowledge and generate answers, making it very suitable for application in canal domain education and decision-making support scenarios. This invention is the first to apply Graph RAG technology to the development of a canal domain chatbot. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 1 is a diagram of the training process of a chatbot in an embodiment of the present invention;

[0021] Figure 2 This is the knowledge graph generated in the embodiment of the present invention. DETAILED DESCRIPTION

[0022] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0023] Example

[0024] This example builds a canal knowledge base and uses Microsoft's publicly available GraphRAG technology to train an LLM, constructing a canal knowledge graph. This results in the development of a chatbot named Canal Expert. CanalExpert generates more comprehensive and informative responses, demonstrating significant application value in canal education and decision-making support scenarios.

[0025] 1. Canal Expert training process:

[0026] First, a canal field knowledge base is built, including introductory books, professional books, software operation manuals, etc.; then the text is divided into blocks to provide materials for the next step of processing; then key information such as entities, relationships and attributes are extracted from the block text; on this basis, a knowledge graph is built, and the Leiden algorithm is used to divide the graph into communities; LLM is used to generate partitioned hierarchical summaries; after receiving the query, the summary of the previous step is used to generate the answer. The above steps complete the training process of CanalExpert. The training process of Canal Expert is as follows. Figure 1 shown.

[0027] 1.1. Construction of canal knowledge base:

[0028] The quality of Canal Expert's answers, in terms of comprehensiveness, richness, and depth, largely depends on the professional knowledge system on which they are based. Therefore, this embodiment has built a canal field knowledge base that is both broad and deep. This knowledge base mainly consists of three parts:

[0029] (1) Software operation manual: The application in the canal field is highly dependent on software operation. From performing spatial analysis to mapping display, specific software is required. Therefore, knowledge related to software operation is very important.

[0030] (2) Introductory books: This collection includes classic introductory books covering the fundamental concepts of the canal field, including core topics such as physical models, spatial analysis, and coordinate systems. These books not only expand the breadth of the knowledge base, but also become the main components of the entities segmented by Graph RAG.

[0031] (3) Professional books: In order to further enhance the depth of the knowledge base and answer interdisciplinary questions, this embodiment collects professional books applied in the canal field.

[0032] After completing the document collection, a script was written using Python to complete a series of processes, including converting the documents into plain text, removing sensitive information such as names of people and places, integrating and storing them in .txt format, and finally forming a corpus.

[0033] 1.2, LLM training:

[0034] This example uses the state-of-the-art large language model Llama3.18B and the embedding model Nomic-Embed-Text, combined with Graph RAG for training. Llama3.18B supports multiple languages ​​and offers excellent performance, making it the preferred model for many pre-training tasks. Furthermore, its context length reaches 128KB, making it excellent for processing long texts and particularly well-suited for Canal Expert training. Nomic-Embed-Text is one of the most influential text embedding models, performing well for both short and long text tasks.

[0035] GraphRAG, l lama3.18B, and Nomic-Embed-Text are used together to complete the CanalExpert training according to the following process.

[0036] 1.2.1 Source document processing:

[0037] Graph RAG is used to split the corpus into text blocks, which will become the material for extracting the knowledge graph, and the overlapping parts are processed to ensure semantic continuity.

[0038] 1.2.2. Element instance extraction:

[0039] Graph RAG is used to identify extraction targets from text blocks, starting with entities, including their names and types, followed by relationships between entities and attributes. The extracted results are then managed as the foundation for building a knowledge graph. llama3.18B is responsible for identifying and extracting these targets from each text block, generating nodes and edges for the knowledge graph, and iteratively extracting missing content as necessary.

[0040] 1.2.3. Build a knowledge graph:

[0041] Graph RAG uses the extracted entities and relationships to generate a preliminary knowledge graph, with entities as nodes and relationships as edges. Nomi c-Embed-Text generates embedding vectors for nodes and edges, and calculates similarities between nodes (such as cosine similarity) to generate edge weights.

[0042] The generated knowledge graph is as follows Figure 2As shown, points are entities and lines are connections, and both can be clicked to view detailed information.

[0043] 1.2.4. Knowledge Graph Partitioning:

[0044] The knowledge graph is divided into several communities using a community monitoring algorithm. This algorithm determines the strength of relationships between nodes. This implementation utilizes the Leiden algorithm, which excels at identifying hierarchical community structures in large-scale graphs. Each community corresponds to a set of semantically related entities and relationships, which are mutually exclusive. This ensures a hierarchical community division, enabling Canal Expert to support fine-grained responses from global to local perspectives. In this step, Nomic-Embed-Text embedding vectors are used to optimize community divisions, making it easier for semantically similar nodes to be grouped into the same community.

[0045] 1.2.5. Community summary generation:

[0046] Graph RAG defines the hierarchy of summary generation, from high-level overviews to fine-grained community content. It manages the semantic information of each community. llama 3.18B generates natural language summaries using extracted entities, relationships, and community content, ensuring that summaries are both comprehensive and fluent.

[0047] 1.2.6. Answer generation:

[0048] Graph RAG uses hierarchical community summaries to answer user questions: Map phase: Generates local answers from all relevant community summaries. Reduce phase: Aggregates local answers into a global answer. The hierarchy and scope of answers are dynamically adjusted as the user's question changes. llama3.18B uses prompts to generate local answers (based on community summaries). All local answers are aggregated into a global answer, and the language style is optimized based on the user query. Nomic-Embed-Text matches the most relevant communities or entities, optimizing the relevance of search results through vector similarity.

[0049] The above-described embodiments merely represent specific implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

Claims

1. A method for implementing a chatbot based on the canal field, characterized in that: The following steps are involved: Step 1: Build a canal domain knowledge base; Step 2: Divide the text into blocks and extract entity, relationship and attribute information from the block text; Step 3: Build a knowledge graph and use the Leiden algorithm to divide the knowledge graph into communities; Step 4: Generate a partitioned hierarchical summary using the large language model (LLM); after receiving a query, generate an answer using the generated partitioned hierarchical summary.

2. The method for implementing a chatbot based on the canal field according to claim 1, characterized in that: The step 2 is specifically as follows: Graph RAG is used to split the corpus into text blocks and process the overlapping parts to ensure semantic continuity; Graph RAG is used to determine the extraction targets from the text blocks, first the entities, including the names and types of the entities, and then the relationships and attributes between the entities, to generate the nodes and edge information of the knowledge graph, and iteratively extract the missing content.

3. The method for implementing a chat robot based on the canal field according to claim 2, characterized in that: The step 3 is specifically as follows: Graph RAG uses the extracted entities and relationships to generate a preliminary knowledge graph, where entities are nodes and relationships are edges. It also generates embedding vectors for nodes and edges, calculates the similarity between nodes, and uses it to generate edge weights. Use the community monitoring algorithm to divide the knowledge graph into several communities, determine the strength of the relationship between nodes, and aggregate semantically similar nodes into the same community.

4. The method for implementing a chat robot based on the canal field according to claim 3, characterized in that: The step 4 is specifically as follows: Graph RAG defines a hierarchy of summary generation, from high-level overviews to fine-grained community content. Graph RAG uses hierarchical community summaries to answer user questions, including: generating local answers from all relevant community summaries; integrating local answers into global answers; dynamically adjusting the level and scope of answers as user questions change; matching the most relevant communities or entities, and optimizing the relevance of retrieval results through vector similarity.

Images