Knowledge Graph Enhanced Large Model Generation Method Driven by Logical Reasoning

Abstract

The present invention provides a method for generating a knowledge graph enhanced large model driven by logical reasoning, belonging to the technical field of generating knowledge graph enhanced large models. The method includes: Step 1, dataset preparation; Step 2, triple extraction and screening; Step 3, triple conversion and knowledge graph construction; Step 4, obtaining the original query problem; Step 5, extracting keywords of the user problem; Step 6, constructing a graph query engine and query execution; Step 7, knowledge integration and generation. Compared with the traditional manual construction method, the present invention not only significantly reduces the manual construction cost, but also improves the construction efficiency of the knowledge graph.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge graph enhanced large model generation, and more particularly to a method for generating a knowledge graph enhanced large model driven by logical reasoning. Background Art

[0002] With the development of information technology and artificial intelligence, intelligent question answering and recommendation systems based on large language models, as key tools for the intelligent digital transformation of various industries, are becoming increasingly important in tasks such as improving user experience and optimizing resource allocation.

[0003] From the perspective of the technical framework, in the process of intelligent question answering and recommendation, a large language model can parse a user's query, understand the intention behind the question, and give an answer based on an existing knowledge base or through reasoning; in addition, the large language model can also predict the content that the user may be interested in by combining user behavior, preferences, and the current context, so as to provide more personalized text for the user during the generation process. The architecture of such a large language model usually consists of a text generation module and a retrieval augmented generation module.

[0004] The text generation module (model) is the core functional module of a large language model, which is mainly responsible for generating natural language text that conforms to the context and logic. Traditional text generation models, such as sequence-to-sequence models (seq2seq), Transformer, etc., usually only rely on the content input by the user and the knowledge learned by the large language model during the pre-training stage for text generation. These models map the input content to the target output content through the encoding and decoding processes. However, due to the fact that the generation ability is limited by the coverage of the pre-training data, such models may not be able to accurately generate the required answers when facing dynamically changing information requirements.

[0005] To address this shortcoming, the Retrieval-Augmented Generation (RAG) module is introduced as an optional optimization module to further enhance the quality and relevance of text generation. The RAG module retrieves the most relevant fragments or documents from a database or knowledge base and combines this external information with the knowledge learned by the large language model during the pre-training phase to generate more factually and contextually relevant answers. Currently, the mainstream retrieval methods are based on vector retrieval techniques, such as Flat (Brute Force Search) retrieval and HNSW (Hierarchical Navigable Small World) retrieval. The Flat retrieval method finds the most matching result by calculating the similarity between the input query and all vectors in the database one by one. Although it has high precision, its computational efficiency is low. On the other hand, HNSW retrieval significantly improves the retrieval speed by constructing a hierarchical graph structure and is suitable for efficient retrieval of large-scale databases. These methods have significantly enhanced the large language model's ability to utilize external knowledge.

[0006] From the perspective of data organization, a knowledge graph is a technology that represents knowledge through a graph structure. It represents entities as nodes and the relationships between entities as edges, thus constructing a structured knowledge representation. The knowledge graph describes the semantic information in the real world in the form of triples (source entity, relationship, target entity), which is not only convenient for storage and management but also enables the discovery of hidden semantic relationships through graph reasoning. In a search engine, the knowledge graph supports semantic search and the dynamic display of knowledge panels by constructing an entity relationship network, improving the accuracy of user query results and the readability of information. In a recommendation system, the knowledge graph combines the ability to model the association between user interests and content, significantly improving the relevance and diversity of recommendations.

[0007] Disadvantages of the prior art:

[0008] 1. Insufficient ability for deep relationship modeling: The vector representation and similarity retrieval methods used in this technology mainly rely on the vector space model, which has a low dimension and can only handle simple query and matching tasks. However, when faced with multi-dimensional user questions containing complex logical relationships and multi-level dependencies, it is unable to effectively construct the associations between target data and entities such as users and relevant information data (such as scenic spots and locations / users / cultural backgrounds), and it is difficult to understand the complex interaction relationships, greatly limiting its application efficiency and accuracy in complex scenarios.

[0009] 2. Noise problems of the vectorized retrieval method: The traditional vectorized retrieval method used in this technology can only convert the user's question into a question vector, then index and retrieve the names of target data (such as scenic spots) by the question vector, and finally index the enhanced generated knowledge text by the names of target data (such as scenic spots). There is a certain amount of useless information noise in the question vector in this step, which will inevitably affect the accuracy of indexing and retrieval, and ultimately affect the system effect.

[0010] 3. One-sidedness problems of the vectorized retrieval method: The final retrieval result of the traditional vectorized retrieval method used in this technology is the name of the target data (such as scenic spots), and this name of the target data (such as scenic spots) is used to index relevant knowledge in the enhanced generation knowledge base. This way will lead to that knowledge retrieval can only index all relevant contents for a single direction (the name of the target data (such as scenic spots)). On the one hand, the knowledge involved in the retrieval is too narrow, and on the other hand, a large amount of unnecessary knowledge will be retrieved, which will affect the system retrieval efficiency to a certain extent.

[0011] 4. Adaptability in complex scenarios. For example, this technology cannot dynamically and effectively combine dimensional information such as user behavior / user preferences and cultural background to provide personalized recommendations.

[0012] 5. The transparency and interpretability of the recommended content are weak: The vectorized method used in this technology only performs matching through similarity, lacking an intuitive display and interpretation mechanism, resulting in an opaque answering or recommendation logic. It is difficult for users and developers to trace the specific logical path generated by it, resulting in a low system credibility.

[0013] 6. Lack of an automated process plan for the task division of the information extraction model: The information extraction model in this technology only realizes the extraction and collation of the required same type of knowledge (such as geographical information and historical information) from the source information by a single model, and does not consider how to divide the tasks into multiple subtasks and implement an automated process in complex scenarios, resulting in poor system performance in complex task scenarios or even unable to meet the application standards.

[0014] 7. The retrieval scheme is single and lacks flexibility: The retrieval scheme in this technology only adds a single technology based on the existing vectorized representation and similarity retrieval method, without considering the flexibility of retrieval, and cannot dynamically adjust the retrieval scheme according to different system running states and situations.

[0015] 8. Lack of application-level evaluation criteria and user feedback: The evaluation criteria for this technology are too metric-based and do not consider the subjective usage feelings of users. At the same time, due to not considering the feedback from users, the preferences of users for the answers returned by the system are not collected, resulting in difficulty in dynamically improving the system according to the feedback, and to a certain extent, the scalability of this technology is poor. Summary of the Invention

[0016] The present invention provides a method for generating a knowledge graph enhanced large model driven by logical reasoning, which includes two modules. One module is an intelligent knowledge extraction and graph generation module based on LLM and CoT, and the other module is a user question query index generation module.

[0017] To achieve the above object, the present invention adopts the following technical solutions:

[0018] A method for generating a knowledge graph enhanced large model driven by logical reasoning includes:

[0019] Step 1, dataset preparation: Collect datasets used in various industry fields;

[0020] Step 2, triple extraction and screening:

[0021] Perform entity / attribute extraction on the dataset collected in Step 1 to obtain entities / attributes;

[0022] Adopt a locally deployed large language model, use the Prompt for CoT provided by Microsoft on its Github as a preset prompt to obtain a logical chain mechanism, and based on the logical chain mechanism, adopt a step-by-step logical reasoning method to identify the relationships between entities from the information in each dimension of the dataset to obtain relationships;

[0023] Based on the entities / attributes and relationships, obtain triples;

[0024] Screen the triples so that the retained triple set has uniqueness, accuracy, and relevance;

[0025] Step 3, triple conversion and knowledge graph construction: Map the information of the triples screened in Step 2 to obtain a knowledge graph;

[0026] Step 4, obtaining the original query question: Input the query question;

[0027] Step 5, extracting keywords of the user question: Identify the key information of the question input in Step 4, or extract keywords and entity types from the question input in Step 4 according to a preset format;

[0028] Step 6, constructing a graph query engine and executing a query:

[0029] Based on the logical chain mechanism and the knowledge graph, clarify the key entities, the relationships between entities, and their attributes;

[0030] Establish a connection between the knowledge graph and the Neo4j database to generate and execute Cypher queries;

[0031] Dynamically generate corresponding Cypher query statements based on the key information identified in Step 5 or the extracted keywords and entity types;

[0032] Execute the Cypher query statement and return the query results through the Neo4j database connection. The query results are entities, relationships, and attributes;

[0033] Step 7, Knowledge Integration and Generation: Based on the large language model, analyze the query results in Step 6 to ensure the coherence and consistency of the information. Combine the knowledge base of the large language model and the query results, and through the logical reasoning ability within the large language model, generate a final answer that meets the user's needs.

[0034] In this specification, for the screening of triples in Step 2, a locally deployed large language model is used to accurately identify and extract entities and their corresponding attributes from structured and unstructured texts, and store them in a structured manner.

[0035] In this specification, the screening process of triples in Step 2 is as follows:

[0036] Carefully review each of the extracted triples one by one. By comparing the contents of different triples, judge whether there are duplicates. Based on reliable data sources, professional common sense, and text context information, evaluate the accuracy of the triples, clarify the core theme of the text, and judge whether the triples are closely related to the theme. Screen out the parts that deviate from the theme. For duplicate triples, keep one of them and remove the rest of the duplicates. For inaccurate or off-topic triples, directly eliminate them.

[0037] In this specification, in Step 3, the Python programming language and its related libraries are used to pre-design the template of the Cypher statement, and use the Python function body to batch process the triples extracted by the large language model, and map the entities, relationships, and attributes into the Cypher statement.

[0038] In this specification, Step 6 also includes a query based on multi-node traversal, which is specifically as follows:

[0039] Use the large language model to extract keywords, and determine which search method to use according to the number of keywords;

[0040] If the number of input keywords is 1, use the single-node search method; if the number of input keywords is multiple, use the multi-node search method for in-depth retrieval;

[0041] After the system executes the query according to the corresponding search method, parse the node, relationship, and attribute information returned by the query through Python and convert it into a structured data format to obtain the retrieval results;

[0042] The search results include all nodes on the path and their associated attribute and relationship information.

[0043] In this specification, in step 5, based on the deployed Mistral-7B-Instruct-v0.3 model and enabling its Function Calling function, when the user inputs a query question, the model automatically identifies the key information in the user input and returns the result in a structured format.

[0044] Or, based on LlamaFactory, fine-tune the LLMs of Mistral, set the preset output format to be consistent with the function calling method, that is, the output format includes the entity types and entity names required by the knowledge graph. When the user inputs a query question, the fine-tuned model extracts keywords and entity types according to the preset format.

[0045] In this specification, in step 5, based on LlamaFactory, fine-tune the LLMs of Mistral, set the preset output format to be consistent with the function calling method, that is, the output format includes the entity types and entity names required by the knowledge graph. When the user inputs a query question, the fine-tuned model extracts keywords and entity types according to the preset format.

[0046] In this specification, in step 6, use Python and its supported Neo4j database to establish a secure link with the Neo4j database Aura instance and dynamically generate and execute Cypher queries programmatically.

[0047] In this specification, in step 6, by defining a Python function that dynamically generates corresponding Cypher query statements based on the key information identified in step 5 or the extracted keywords and entity types.

[0048] In this specification, in step 6, by defining a Python function to execute the generated Cypher query statements, and return the query results through the Neo4j database link, store them in dictionary format, and convert them into a structured data format, presented in the form of a list. Each generated list element includes the entities, relationships, and attributes involved in the predefined logical chain.

[0049] In summary, the present invention has at least the following beneficial effects:

[0050] The present invention utilizes locally deployed large language models LLM A and LLM B, combined with the Chain of Thought (COT) logical chain mechanism, to accurately extract entities, attributes, and relationships from multi-dimensional information, and construct a complex and accurate knowledge graph. Compared with relying on traditional low-dimensional vector space models, when facing complex user questions (such as multi-entity association queries involving scenic spots, locations, users, cultural backgrounds, etc.), the present invention can effectively construct the associations between target data and various entities, understand complex interaction relationships, and automatically extract triples through large language models, greatly improving the efficiency and accuracy of information extraction. Compared with traditional manual construction methods, it not only significantly reduces the manual construction cost but also improves the construction efficiency of the knowledge graph. Based on the triples, a knowledge graph is constructed, and a complete and clear link is constructed in understanding user questions using keywords to generate more comprehensive and useful information. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0052] Figure 1 It is a schematic diagram of the method for generating a knowledge graph enhanced large model driven by logical reasoning involved in the present invention.

[0053] Figure 2 It is a schematic diagram of the working process of the dynamic strategy of different search methods involved in the present invention.

[0054] Figure 3 It is a schematic diagram of the comparison of evaluation indicators of the keyword extraction method involved in the present invention.

[0055] Figure 4 It is a schematic diagram of the comparison of user questionnaire result preferences involved in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] In the following text, only some exemplary embodiments are briefly described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and descriptions are considered to be exemplary in nature rather than restrictive.

[0057] The following disclosure provides many different embodiments or examples for implementing different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the embodiments of the present invention. In addition, the embodiments of the present invention may repeat reference numerals and / or reference letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0058] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0059] As Figure 1 shown, this embodiment provides a method for generating a knowledge graph enhanced large model driven by logical reasoning, including:

[0060] Module 1: Intelligent Knowledge Extraction and Graph Generation Module Based on LLM and CoT (Original Database + Prompt Engineering)

[0061] Step 1: Dataset Preparation

[0062] Condition: Collect datasets used in various industry fields, which are required to cover multiple dimensions, and multiple small dimensions are subdivided under the large dimension. These datasets can be sourced from public databases, internal data provided by enterprises, etc., and the data formats include but are not limited to text files, database files, etc.

[0063] Step 2: Triple Extraction and Screening

[0064] Entity / Attribute Extraction (LLM A)

[0065] Condition: Adopt the large language model LLM A deployed locally and use the preset prompt.

[0066] Operation: Accurately identify and extract entities and their corresponding attributes from structured and unstructured texts, and store them in a structured manner.

[0067] For example, the entity is "xx Palace", and its attributes "Type: Historical Site" and "Location: City, Province" are identified, and this information is saved in a structured way for subsequent use.

[0068] Relationship Extraction (LLM B)

[0069] Condition: Adopt the large language model LLM B with local deployment, utilize the Prompt forCoT provided by Microsoft on its Github as the preset prompt, and design a logical chain mechanism named Chain of Thought (COT).

[0070] Operation: Based on the COT logical chain mechanism, adopt a step-by-step logical reasoning method to identify the relationships between entities from various dimensions of information.

[0071] For example, for "xx Palace" and "xx Culture", identify that the relationship between them may be "represent" or "originate from", thus establishing a complete triple like "xx Palace - located in - x City" (entity - relationship - entity), realizing relationship extraction and establishing connections between entities.

[0072] Triple Screening

[0073] Condition: Manual operation by humans, with screening criteria for reference.

[0074] Operation: Team members carefully review each of the extracted triples one by one. By comparing the content of different triples, judge whether there are duplicates; based on reliable data sources, professional common sense, and text context information, evaluate the accuracy of the triples; clarify the core theme of the text, judge whether the triples are closely related to this theme, and screen out the parts that deviate from the theme. For duplicate triples, keep one of them and remove the rest of the duplicates; for inaccurate or off-topic triples, directly eliminate them, so as to ensure that the final set of retained triples is unique, accurate, and relevant.

[0075] Step 3: Triple Conversion and Knowledge Graph Construction

[0076] Required components: Use the Python programming language and its related libraries, and pre-design a template for Cypher statements (Neo4j query language).

[0077] Operation: Use the Python function body to batch process the triples extracted by LLM A and LLM B, and map information such as entities, relationships, and attributes into Cypher statements.

[0078] For example, converting a triple like "xx Palace - located in - x City" into the corresponding Cypher statement is "CREATE (:Place{name:\"xx Palace\"})-[:LOCATED_IN]->(:City{name:\"x City\"})", and creating nodes and edges in the Neo4j graph database through programming to form a complete knowledge graph.

[0079] Module 2: User Question Query Index Generation Module

[0080] Step 4: Obtain the original query question

[0081] Condition: The user inputs a query question, supporting the user to input questions in the form of natural language, such as "I want to eat xx hot pot. Which place in x city should I go to?" and questions in different fields.

[0082] Step 5: Extract keywords from the user's question

[0083] Method 1 (Function Calling function based on the Mistral-7B-Instruct-v0.3 model)

[0084] Condition: Deploy the Mistral-7B-Instruct-v0.3 model and enable its Function Calling function.

[0085] Operation: When the user inputs a query question, the model automatically identifies the key information in the user's input and returns the result in a structured format. For example, for the user's query "I want to eat xx hot pot. Which place in x city should I go to?", the system extracts "xx hot pot" and "x city" as keywords through the model and generates a structured output that meets the requirements of the knowledge graph, such as {"xx hot pot": "Food", "x city": "Location"}.

[0086] Method 2 (Fine-tuning the LLMs of Mistral based on LlamaFactory)

[0087] Condition: Use LlamaFactory to fine-tune the LLMs of Mistral, set the preset output format to be consistent with the function calling method, that is, the output format contains the entity types and entity names required by the knowledge graph.

[0088] Operation: When the user inputs a query question, the fine-tuned model extracts keywords and entity types according to the preset format to ensure accurate extraction of key information for subsequent queries.

[0089] Step 6: Construct and execute the graph query engine

[0090] Query based on the logical chain:

[0091] Such queries are carried out along specific relationship paths in the knowledge graph. For example, the user may ask "Which culture is related to the historical background of xx Palace?", and the engine will query along the logical chain of "xx Palace - represents - the culture of x city - originated from - xx place". The specific steps are as follows:

[0092] ① Sorting and defining logical chains: Sorting and defining relevant logical chains, clarifying key entities, relationships between entities, and their attributes. These predefined logical chains can effectively help the system perform structured retrieval in the knowledge graph, thereby quickly locating entities and relationships related to the query topic.

[0093] ② Establish a connection with the Neo4j database: Using Python and its supported neo4j database, a secure connection is established with the Neo4j database Halo instance, and Cypher queries are dynamically generated and executed programmatically.

[0094] ③ Dynamically generate Cypher query statements: By defining a Python function, the function dynamically generates the corresponding Cypher query statement based on the given query subject.

[0095] For example, for the query of the topic "xx hot pot", the generated Cypher query is as follows:

[0096] MATCH(c:美食{name:"xxhotpot"})

[0097] OPTIONAL MATCH(c)-[r:Provided by]-(l:Location)

[0098] RETURN c.name AS Cuisine,

[0099] collect(DISTINCT{

[0100] labels:labels(l),

[0101] properties: properties(l),

[0102] relationship:

[0103] type:type(r),

[0104] properties:properties(r)

[0105] }

[0106] })AS LocationAndRelationshipDetails.

[0107] The query searches for the relationship between "xx hotpot" and other places through the "food" node, and collects related entities and their attributes.

[0108] ④Execute the query and parse the results: Define a Python function to execute the generated Cypher query and return the query results through the Neo4j connection. Store them in dictionary format and convert them to a structured data format later, and finally present them in the form of a list. Output and application of structured data: Each generated list element includes the entities, relationships, and attributes involved in the predefined logical chain.

[0109] Query based on multi-node traversal:

[0110] Find relevant information by traversing multiple nodes in the graph. Such queries are usually used for more complex problems, such as "Which scenic spots are related to xx culture and located in city x?", and the engine will traverse multiple nodes related to "xx culture" and "city x" in the graph to find the scenic spots that meet the conditions. The specific steps are as follows:

[0111] ①Keyword extraction and search method judgment: Use the large language model to extract keywords and determine which search method to use according to the number of keywords. As Figure 2 shown, if the number of input keywords is 1, use the single-node search method; if the number of input keywords is multiple, use the multi-node search method for in-depth retrieval. This logical judgment ensures the flexibility of the retrieval strategy, enabling the system to dynamically adjust the search depth according to different query requirements.

[0112] ②-①Implementation of single-node in-depth retrieval: When it is recognized that the number of keywords is 1, perform a single-node search. This search strategy starts from the specified node and retrieves its directly related nodes and relationships layer by layer. The specific retrieval strategy algorithm is as follows. The Cypher query statement is as follows:

[0113] Retrieval strategy algorithm formula (case when the depth is 2):

[0114]

[0115] Formula interpretation:

[0116] 1. Initialization:

[0117] Select a starting node v.

[0118] Create an empty access set visited.

[0119] Create a stack stack and push the starting node (v,0), where 0 represents the current depth.

[0120] 2. Traversal process:

[0121] When the stack is not empty, perform the following operations:

[0122] Pop the top element (u, d) of the stack, where u is the node and d is the current depth.

[0123] If u is not in the visited set, add u to the visited set visited.

[0124] If the current depth d is less than 2, for each unvisited adjacent node w of u:

[0125] Add w to the visited set visited.

[0126] Push (w, d + 1) onto the stack.

[0127] 3. Termination condition:

[0128] When the stack is empty, the traversal ends.

[0129] Cypher query statement:

[0130] MATCH(n: Location {name: $keyword}) - [r1] - (m)

[0131] OPTIONAL MATCH(m) - [r2] - (o)

[0132] RETURN n AS StartNode, m AS MiddleNode, o AS EndNode, r1 AS StartRelationship, r2 AS EndRelationship;

[0133] This query parses the associated relationships and node attributes by retrieving two - layer - deep nodes related to the target node (such as "xx Street"). In this way, the system can quickly locate entities related to the query node and connect them with the associated nodes.

[0134] ② - ② Implementation of multi - node depth retrieval: When more than 1 keyword is recognized, the system performs multi - node depth retrieval. Based on the following shortest - path algorithm, the path from the start node to the end node is found through the following Cypher query:

[0135] Shortest - path algorithm (example):

[0136] 1. Determine the node set:

[0137] Determine the node set S to be retrieved.

[0138] 2. Run Dijkstra's algorithm for each node:

[0139] For each node s in the set S, use Dijkstra's algorithm to calculate the shortest path from s to all other nodes in the graph.

[0140] The mathematical formula of Dijkstra's algorithm is as follows:

[0141] ;

[0142] 3. Merge results:

[0143] For each node v in the graph, find the shortest paths from all nodes in the set S to v.

[0144] Select the minimum value among these paths as the shortest path from S to v.

[0145] 4. Return the set of shortest paths:

[0146] Return a set containing the shortest paths from S to all nodes in the graph.

[0147] Cypher query statement:

[0148] MATCH path=shortestPath((a{name:$start_node_name})-[*]-(b{name:$end_node_name}))

[0149] RETURN nodes(path) AS nodes, relationships(path) AS relationships;

[0150] This query will find the shortest path from the starting node (such as "xx Street") to the ending node (such as "x City's Folk Customs"), expand all the nodes and relationships in between layer by layer, and perform a detailed analysis through node attributes and relationship types. This method ensures that the deep connections between multiple nodes can be fully displayed, providing a complete knowledge chain for the RAG model.

[0151] ③ Parsing and display of query results:

[0152] After the system executes the query, it will parse the node, relationship, and attribute information returned by the query through Python and convert it into a structured data format. The name, label, attributes of each node, as well as the relationship type and direction between nodes, will be displayed in a clear hierarchical structure.

[0153] ④ Output and application of deeply structured data:

[0154] The results generated by each retrieval include all nodes on the path and their associated attribute and relationship information. This structured data provides deeper knowledge support for the reasoning process of the RAG model, enabling rapid location of complex relationship chains in the knowledge graph of multi-level nodes. Through this deep search method, the system effectively improves the knowledge utilization efficiency of RAG, laying a solid foundation for subsequent experiments and model optimization.

[0155] Step 7: Knowledge Integration and Generation

[0156] The third model, LLM C, combines the results of the graph query engine (i.e., Step 3) with its own internal knowledge (knowledge learned during the model pre-training phase) to generate content that better meets user preferences and needs, thereby enhancing the user experience and the accuracy of information.

[0157] Query Result Reception and Integration

[0158] Condition: Deploy the third model, LLM C, to enable it to receive and process the results returned by the graph query engine and integrate them with its own knowledge base.

[0159] Operation: The graph query engine returns the query results to LLM C. LLM C first analyzes the multi-entity and multi-relationship data returned by the graph query to ensure the coherence and consistency of the information, preparing for generating answers later.

[0160] Knowledge Integration and Generation

[0161] Condition: LLM C has an internal knowledge base and strong logical reasoning ability, enabling it to reason and generate by combining the externally input query results with its own knowledge.

[0162] Operation: LLM C combines its own knowledge base with the information returned by the graph query engine and, through its internal logical reasoning ability, generates a final answer that meets the user's needs. For example, if the user queries "What are the most culturally valuable scenic spots in City X?", LLM C will not only list the relevant scenic spots but also generate an explanation of the cultural value and historical background of these scenic spots, thereby enhancing the user experience and the accuracy of information.

[0163] Taking the experiment in the cultural and tourism field as an example, the BERTScore (semantic similarity score based on the BERT model) for 10 questions further quantifies the generation quality of the present invention. It can be analyzed from the following scoring table:

[0164] The average score is:

[0165] Precision(P): 0.6346; Recall(R): 0.6984; F1-Score(F1): 0.6645;

[0166] The BERTScore results show that the present invention can make good use of the structured information returned by the knowledge graph when processing complex queries, maintain a high semantic consistency, and perform better especially in the in-depth search of multiple keywords.

[0167] Specific case comparison

[0168] Taking the question "Which traditional festivals are celebrated on xx Street?" as an example, as shown in Table 1.

[0169] Table 1 Comparison of the effect demonstration of in-depth search of multiple keywords in the cultural and tourism field

[0170]

[0171] Baseline model (Answer 1): The answer is relatively general, mainly mentioning the geographical location of xx Street and the ethnic residence situation, without specifically listing the festivals.

[0172] The present invention (Answer 2): Details multiple traditional festivals related to xx Street and elaborates on the culture of xx Street, significantly improving the user experience.

[0173] Improving retrieval accuracy, breadth, and efficiency: The present invention comprehensively optimizes the retrieval task through the user question query index generation module. Taking the experiment in the cultural and tourism field as an example, in the step of extracting keywords from the user question, based on the Function Calling function of the Mistral-7B-Instruct-v0.3 model and the fine-tuning method of LlamaFactory based on the mistral model, it can accurately capture the keywords and entity types in the user input question. Through the comparative experiments of the Function Calling model, the fine-tuning (LlamaFactory) model, and the zero-shot model (a model without any processing), Function calling achieved a full score of 1.0 in all evaluation metrics (precision, recall, and F1-score), as Figure 3As shown, it performs best. LLMs can very accurately identify and classify entities and their related information in user queries. In addition, the present invention can intelligently select and dynamically adjust the retrieval strategy according to the question entity keywords extracted from the user's question. For example, in the cultural and tourism field, in the closest prior art to the present invention, for questions strongly related to "xx place", after the system performs vectorization matching, it can only output the name of the scenic spot "xx place", and then generate an enhanced module index for retrieval based on this name and return all the information about xx place. In the present invention (taking the query method based on multi-node traversal as an example), different node search methods will be performed according to the number of keywords. For questions strongly related to "xx place", in the present invention, it may obtain "xx place → special festival". According to this path, information about "special festivals related to xx place" will be retrieved, which can greatly improve the retrieval efficiency and effect compared to returning all the information about xx place.

[0174] For questions strongly related to "xx City xx Place", after the system performs vectorization matching, it can only output the name of the scenic spot "xx City xx Place", and then generate an enhanced module index for retrieval based on this name and return all the information about xx City xx Place. In the present invention, different node search methods are performed. For questions strongly related to "xx City xx Place", in the present invention, it may obtain combined retrieval chains such as "xx City xx Place → altitude", "xx City xx Place → scenery", "xx City xx Place → related traditional culture", etc. Information such as the altitude of xx City xx Place, recommended nearby scenery, and related traditional culture will be returned, which greatly improves the retrieval breadth compared to returning all the information about xx City xx Place.

[0175] Enhance the transparency and interpretability of recommended content: In the query result generation stage of the present invention, by combining the results of the graph query engine with its own knowledge through the third model LLM C, not only answers are given, but also explanatory content (such as the cultural value and historical background of scenic spots in the cultural and tourism field) can be generated.

[0176] Optimize the automated information extraction process: The intelligent knowledge extraction and graph generation module subdivides the information extraction task, uses different large language models to perform entity / attribute extraction and relationship extraction respectively, and realizes triple conversion and automatic construction of the knowledge graph with the help of Python programming and related libraries. Facing complex task scenarios, subdividing the extraction task into multiple extraction subtasks in different dimensions can greatly exert the capabilities of the LLM and improve the accuracy of the extracted knowledge; at the same time, the automated process processing enables the tasks of each module to proceed in an orderly manner, improving the working efficiency of the system in complex task scenarios.

[0177] Implement flexible adjustment of the retrieval scheme: The user question query index generation module automatically determines whether to adopt a single-node or multi-node deep retrieval strategy based on the number of keywords entered by the user. At the same time, it dynamically generates Cypher query statements adapted to different query topics based on Python functions, and can flexibly adjust the retrieval scheme according to the system running status and user questions. Taking the experiment in the culture and tourism field as an example, referring to the 6th point of the technical effect, in the diversified retrieval requirement test, the performance of the present invention compared with the benchmark model of single retrieval under 10 questionnaire questions is better in most questions (80% winning rate), indicating that the present invention can provide more detailed and accurate answers when dealing with complex problems, especially in complex fields such as culture and history, and the structured knowledge graph of the present invention provides strong support for the generation quality.

[0178] Introduce application-level evaluation and user feedback-driven optimization: The present invention pays attention to collecting user feedback on the query results, takes into account the subjective user experience, and establishes an application-level evaluation standard to ensure the scalability of the system. In subsequent research, based on this system and the collected feedback data, attempts will be made to dynamically improve text generation by continuously collecting feedback. Taking the experiment in the culture and tourism field as an example, specifically, the method of anonymous voting is adopted to record the preferences of the interviewees for the results generated by the original model and the present invention for the same problem respectively, and the results are sorted as Figure 4 shown.

[0179] In this experiment, we compared the performance of the benchmark model and the present invention under 10 questions and found that the present invention performed better in most questions (80% winning rate). This advantage can be more vividly demonstrated by comparing the answers to specific questions.

[0180] Overall experimental result analysis:

[0181] The invention performs better in most questions: From question 1 to question 10, in most cases, the present invention received higher user preference. For example, in question 5 and question 6, the present invention received support rates of 92.23% and 96.12% respectively, while the benchmark model only received 7.77% and 3.88%. This indicates that the present invention can provide more detailed and accurate answers when dealing with complex problems, especially in complex fields such as culture and history, and the structured knowledge graph of the present invention provides strong support for the generation quality.

[0182] The benchmark model performs better in some questions: In question 4 and question 10, the benchmark model performed relatively better, receiving user preferences of 74.76% and 53.4% respectively. This may be related to the lower complexity of these questions or the smaller dependence on structured knowledge, indicating that the benchmark model still has certain advantages when dealing with simple questions.

[0183] The embodiments described above are used to illustrate the present invention, not to limit the present invention. Therefore, changes in the exemplified numerical values or replacement of equivalent elements should still fall within the scope of the present invention.

[0184] From the above detailed description, those of ordinary skill in the art can clearly understand that the present invention can indeed achieve the foregoing objectives, which actually complies with the provisions of the Patent Law.

[0185] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the present invention. The above description is only the preferred embodiments of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0186] It should be noted that the above description of the process is only for illustration and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various corrections and changes can be made to the process under the guidance of this specification. However, these corrections and changes are still within the scope of this specification.

[0187] The basic concept has been described above. Obviously, for those of ordinary skill in the art after reading this application, the above invention disclosure is only an example and does not constitute a limitation to this application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are proposed in this application, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0188] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that the "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned two or more times in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0189] In addition, those of ordinary skill in the art can understand that various aspects of the present application can be illustrated and described by several patentable types or situations, including any new and useful process, machine, product, or composition of matter, or any new and useful improvement thereof. Therefore, various aspects of the present application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software. The above hardware or software can all be referred to as "units", "modules", or "systems". In addition, various aspects of the present application can take the form of a computer program product embodied in one or more computer-readable media, in which computer-readable program code is included therein.

[0190] The computer program code required for the operation of each part of the present application can be written in any one or more of the above programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C programming language, VisualBasic, Fortran2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages such as Python, Ruby, and Groovy, or other programming languages. The program code can run entirely on the user's computer, or run on the user's computer as an independent software package, or run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer in any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (for example, through the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0191] In addition, unless clearly stated in the claims, the order of the processing elements and sequences, the use of numbers and letters, or the use of other names in the present application are not used to limit the order of the processes and methods of the present application. Although some currently considered useful embodiments of the invention are discussed through various examples in the above disclosure, it should be understood that such details only serve the purpose of illustration, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of the present application. For example, although the implementation of the above various components can be embodied in a hardware device, it can also be implemented as a pure software solution. For example, it can be installed on an existing server or mobile device.

[0192] Similarly, it should be noted that, in order to simplify the description disclosed in the present application and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, sometimes multiple features are incorporated into one embodiment, drawing or description thereof. However, this method of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. On the contrary, the subject matter of the invention should have fewer features than the above-mentioned single embodiment.

Claims

1. A method for generating a knowledge graph enhanced large model driven by logical reasoning, characterized in that Including: Step 1, Dataset Preparation: Collect datasets used in various industry fields; Step 2, Triple Extraction and Screening: Extract entities and their corresponding attributes from the datasets collected in Step 1 to obtain entities and their corresponding attributes; Adopt a locally deployed large language model, use the Prompt for CoT provided by Microsoft on its Github as a preset prompt to obtain a logical chain mechanism, and based on the logical chain mechanism, use a step-by-step logical reasoning method to identify the relationships between entities from the information in each dimension of the dataset to obtain relationships; Based on the entities, their corresponding attributes, and relationships, obtain triples; Screen the triples to make the retained triple set unique, accurate, and relevant; Step 3, Triple Conversion and Knowledge Graph Construction: Map the information of the triples screened in Step 2 to obtain a knowledge graph; Step 4, Obtaining the Original Query Question: Input the query question; Step 5, Extracting Keywords from User Questions: Identify key information from the question input in Step 4, or extract keywords and entity types from the question input in Step 4 according to a preset format; Step 6, Constructing and Executing a Graph Query Engine: Based on the logical chain mechanism and the knowledge graph, clarify the key entities, the relationships between entities, and their attributes; Establish a connection between the knowledge graph and the Neo4j database to generate and execute Cypher queries; Based on the key information identified in Step 5 or the extracted keywords and entity types, dynamically generate corresponding Cypher query statements; Execute the Cypher query statements and return the query results through the Neo4j database link. The query results are entities, relationships, and attributes; Step 7, Knowledge Integration and Generation: Based on the large language model, analyze the query results in Step 6 to ensure the coherence and consistency of the information. Combine the knowledge base of the large language model and the query results, and through the logical reasoning ability within the large language model, generate a final answer that meets the user's needs; The query based on multi-node traversal is also included in Step 6, specifically as follows: Use the large language model to extract keywords and determine which search method to use according to the number of keywords; If the number of input keywords is 1, use the single-node search method; if the number of input keywords is multiple, use the multi-node search method for in-depth retrieval; After the system executes the query according to the corresponding search method, parse the node, relationship, and attribute information returned by the query through Python and convert it into a structured data format to obtain the retrieval result; The retrieval result includes all nodes on the path and their related attribute and relationship information; Single-node search method: Starting from a specified node, layer by layer retrieve its directly related nodes and relationships; Multi-node search method: Based on the shortest path algorithm, find the path from the starting node to the ending node:

1. Determine the node set: Determine the node set S to be retrieved; 2. Run the Dijkstra algorithm for each node: For each node s in the set S, use the Dijkstra algorithm to calculate the shortest path from s to all other nodes in the graph; 3. Merged Results: For each node v in the graph, find the shortest paths from all nodes in the set S to v; select the minimum value among these paths as the shortest path from S to v; 4. Return the Set of Shortest Paths: Return a set containing the shortest paths from S to all nodes in the graph.

2. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein In step 2, the triples are screened using a locally deployed large language model to accurately identify and extract entities and their corresponding attributes from structured and unstructured texts and store them in a structured manner.

3. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein The screening process of the triples in step 2 is as follows: Carefully review each of the extracted triples one by one. By comparing the contents of different triples, determine whether there are duplicates. Based on the source of the data, professional common sense, and text context information, evaluate the accuracy of the triples, clarify the core theme of the text, and determine whether the triples are closely related to this theme. Screen out the parts that deviate from the theme. For duplicate triples, keep one of them and remove the rest of the duplicates. For inaccurate or off-topic triples, directly eliminate them.

4. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein In step 3, the Python programming language and its related libraries are used. A template for Cypher statements is designed in advance, and Python function bodies are used to batch process the triples extracted by the large language model, mapping entities, relationships, and attributes into Cypher statements.

5. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein In step 5, based on the deployed Mistral-7B-Instruct-v0.3 model, its Function Calling function is enabled. When the user inputs a query question, the model automatically identifies the key information in the user input and returns the result in a structured format.

6. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, characterized in that, In step 5, based on LlamaFactory, the LLMs of Mistral are fine-tuned, and a preset output format is set to be consistent with the function calling method, that is, the output format includes the entity types and entity names required by the knowledge graph. When the user inputs a query question, the fine-tuned model extracts keywords and entity types according to the preset format.

7. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein, In step 6, Python and its supported Neo4j database are used to establish a secure connection to the Neo4j database Aura instance and dynamically generate and execute Cypher queries programmatically.

8. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein In step 6, a Python function is defined, which dynamically generates corresponding Cypher query statements according to the key information identified in step 5 or the extracted keywords and entity types.

9. The method for generating a knowledge graph enhanced large model driven by logical reasoning according to claim 1, wherein In step 6, a Python function is defined to execute the generated Cypher query statements, and the query results are returned through the Neo4j database connection, stored in dictionary format, and converted into a structured data format, presented in the form of a list. Each generated list element includes the entities, relationships, and attributes involved in the predefined logical chain.

Images