A LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement

Abstract

The present invention discloses an LLM reasoning method based on multi-step decomposition retrieval enhancement of knowledge graph. The present invention integrates the medical LLM capability into the entire retrieval enhancement reasoning process, and realizes accurate understanding and answering of the target text input by the user by combining with the medical knowledge graph, using multi-step decomposition, key entity extraction, candidate retrieval path scoring, key retrieval path discrimination verification, prompt word construction and reasoning; the present invention generates key retrieval paths by multi-step decomposition of the target text and heuristic search based on medical LLM scoring, which not only realizes the simplified processing of complex tasks and reduces the impact of sentence ambiguity, but also provides more accurate retrieval paths and specific answers for each subtask, thereby improving the accuracy of the overall answer.

Description

Technical Field

[0001] The present invention relates to a medical LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement, which belongs to the field of natural language processing. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have demonstrated powerful text generation and understanding capabilities and are widely used in multiple scenarios such as question-answering systems, intelligent customer service, and content creation. Through deep learning and massive data training, these models have the ability to understand and generate natural language, setting new standards for automatic language processing. However, although LLMs perform well in many tasks, they still face challenges in handling complex knowledge reasoning tasks because they rely on internal parameter storage and context understanding to process information when performing reasoning, such as untimely knowledge updates, insufficient coverage of knowledge in specific fields, lack of explainability, and the generation of false statements that do not match the facts, namely "factual hallucinations".

[0003] As a structured knowledge storage method, the Knowledge Graph (KG) organizes entities and their relationships in the real world through nodes (entities), edges (relationships) and attributes, providing strong support for accurate information retrieval and reasoning. Introducing the knowledge graph into the large language model can effectively supplement and update knowledge and enhance the model's ability to understand and answer complex questions. This combination can not only improve the accuracy of the large language model's answers to factual questions, but also demonstrate a stronger level of intelligence in logical reasoning, contextual understanding, and other aspects.

[0004] In recent years, researchers are actively exploring the technical path of integrating knowledge graphs with medical big language models, aiming to build a method that can dynamically extract rich medical information from knowledge graphs to make up for the limitations of the medical big language model's own knowledge coverage, thereby showing better performance in tasks such as medical question answering, disease diagnosis reasoning, and treatment plan recommendations. However, how to accurately extract key information from the target text input by the user for knowledge retrieval, how to design intelligent retrieval and reasoning mechanisms to achieve seamless integration of knowledge graphs and big language models, and ensure that the knowledge necessary to answer inquiries can be accurately retrieved, and the medical big language model can make full use of this knowledge to give accurate and reliable medical answers, are the key points and difficulties of current technological development.

[0005] In practice, it is found that there are currently several studies on enhancing the reasoning ability of the medical LLM model based on the knowledge graph, such as the domestic patent application number CN202410524696.1, published on June 14, 2024, which discloses a medical LLM reasoning method and device based on the knowledge graph. This method first recognizes the intent of the target text input by the user, then queries the subgraph related to the intent from the knowledge graph, and finally inputs the prompt words constructed with the subgraph and the target text into the medical LLM for logical reasoning, and outputs the reasoning result. The accuracy of this method will be affected by many factors such as the accuracy of intent recognition and the amount of effective information carried by the subgraph, because for complex, ambiguous or ambiguous sentences, it may not be possible to accurately understand their intent, which may cause the knowledge graph subgraph queried subsequently to not match the actual needs of the user. Secondly, this method does not crop the relevant subgraphs it retrieves, so the redundant information that the subgraph may contain will also affect the accuracy of the medical LLM model reasoning, which has certain limitations. For example, domestic patent application number CN202410521286.1, published on May 28, 2024, discloses a medical LLM reasoning method and related equipment based on knowledge graph. By obtaining the user's target text, if the text contains entities in the knowledge graph, the relevant subgraph is found, and the user's historical text information is combined to crop the unmatched edges in the subgraph to obtain the target subgraph. The target subgraph and the target text are constructed with prompt words and input into the medical LLM to perform logical reasoning and output the results. This method also does not take into account the possible irregularities, polysemy or ambiguity of sentences in the user's target text, which will directly affect the accuracy of the judgment of whether the text contains knowledge graph entities, and thus affect the effect of reasoning enhancement. In addition, this method uses historical text information for subgraph cropping, and does not consider the possibility that historical information may be irrelevant to the target problem. It is easy to mis-crop subgraph information and has certain limitations. Summary of the invention

[0006] The problem to be solved by the present invention is that the existing technical solutions for enhancing the reasoning ability of the medical LLM model based on knowledge graphs do not take into account the complex or ambiguous sentences that may exist in the user's target text and the sub-graph cropping is not fully considered.

[0007] In order to solve the above technical problems, the technical solution of the present invention is to disclose an LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement, which is characterized by comprising the following steps:

[0008] Step 1: After the user inputs the target text, the medical LLM is used to decompose the target text input by the user into multiple subtasks, and the subtasks are classified and merged;

[0009] Step 2: Use medical LLM to perform entity recognition and entity relationship extraction on each subtask, and output them in order of importance;

[0010] Step 3: Select multiple key entities and relationships from each subtask and apply them to the knowledge graph to find adjacent entities to form multiple candidate retrieval paths;

[0011] Step 4: Use the medical LLM to score the candidate retrieval paths obtained and select the triples most relevant to each subtask, and iteratively generate and update the key retrieval paths in a heuristic search manner;

[0012] Step 5: Verify the constructed key search paths through medical LLM to confirm whether they can be applied to answer the initial target text, thereby generating the final search path;

[0013] Step 6: Construct prompt words based on the retrieved knowledge and target text, input medical LLM for reasoning, and output the reasoning results corresponding to the target text.

[0014] Preferably, in step 1, the following steps are used to classify and merge the subtasks:

[0015] Step 101: extracting features for each subtask, where the features include the purpose of the subtask, the medical concepts involved, and the type of problem to be solved;

[0016] Step 102: label subtasks with similar feature information with the same purpose label;

[0017] Step 103: Select a representative subtask from the subtasks with the same purpose label and output it in a list format. Then, classify the subtasks by combining the decomposed subtasks with the constructed prompt words and inputting them into the medical LLM.

[0018] Preferably, step 2 comprises the following steps:

[0019] Step 201: Perform entity recognition and relationship extraction on the subtask description;

[0020] Step 202: Analyze the importance of the above entities and relationships to the description of the subtasks;

[0021] Step 203: sort the output entities and relationships in descending order according to importance, and sequentially combine the description of each subtask with the constructed prompt words and input them into the medical LLM to obtain the key entities of each subtask and the relationships between the entities;

[0022] Step 204: Select no more than 10 key entities and inter-entity relationships from each subtask and apply them to subsequent search path generation.

[0023] Preferably, step 4 comprises the following steps:

[0024] Step 401, analyzing the key elements and required information of the subtask;

[0025] Step 402: Analyze the relevance between entities and relationships in the triples of the knowledge graph and key elements in the subtasks;

[0026] Step 403: Evaluate the contribution score of the triple to the subtask based on the analysis result, with the score ranging from 0 to 1, and input the constructed prompt words into the medical LLM in combination with the triple and the subtask description to obtain the score value;

[0027] Step 404: Use the score obtained in step 403 as the estimated function value of the heuristic search, and perform heuristic search in the breadth and depth directions one by one starting from each candidate path head node, and iteratively select the triples with a score value greater than 0.5 as the most relevant triples to generate and update the key retrieval path for each subtask.

[0028] Preferably, step 5 comprises the following steps:

[0029] Step 501, analyzing the key elements and required information of the initial target text;

[0030] Step 502: Analyze the relevance between entities and relations in the triples and key elements in the target text;

[0031] Step 503: Evaluate whether the triple is related to the target text based on the analysis result: if so, output “yes”; otherwise, output “no”;

[0032] Step 504: Combine each triple and target text with the constructed prompt words and input them into the medical LLM in turn to obtain answers on whether each key search path is related to the initial target text, thereby screening out the final search path and applying it to the knowledge graph to retrieve the corresponding knowledge.

[0033] The present invention combines the powerful language processing capabilities of medical LLM and the rich structural information of knowledge graphs, and achieves accurate understanding and answers to the target text input by the user through step-by-step decomposition, extraction, retrieval, scoring, verification and reasoning. Each step is aimed at improving the accuracy, relevance and efficiency of the answer, ensuring the accuracy and relevance of the results. At the same time, the technical solution of the present invention has certain flexibility and scalability, and can adapt to the needs of different medical scenarios such as intelligent question and answer, auxiliary diagnosis and treatment plan recommendation, etc., providing strong support for the development of medical informationization and intelligence. In the technical solution disclosed in the present invention, the entire reasoning process is fully automatic, and each step is aimed at improving the accuracy, relevance and efficiency of the answer, which can provide users with better services and experience, and is suitable for wide application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1It is a flow chart of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0036] The embodiment of the present invention discloses an LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement. In the present invention, the basic component unit of the knowledge graph is a triple, and this triple structure clearly represents the relationship between entities in a symbolic form, which specifically includes the following steps:

[0037] After the user inputs the target text, the medical LLM is first used to understand the target text entered by the user and decompose it into multiple subtasks, and the decomposed subtasks are classified and merged;

[0038] Secondly, we use the medical LLM to perform entity recognition and entity relationship extraction for each subtask, and output them in order of importance. We select multiple key entities and relationships from each subtask and apply them to the knowledge graph to find adjacent entities and form multiple candidate retrieval paths.

[0039] Then, the score of the medical LLM on the correlation between the triple and the relevant subtask is used as the estimated function value to perform heuristic iterative search for key triples and generate and update the key retrieval path;

[0040] The constructed key retrieval paths are verified through medical LLM to confirm whether they can be applied to answer the initial target text, thereby generating the final retrieval path. The knowledge obtained from the final retrieval is used as background knowledge to construct prompt words together with the target text, which are input into the medical LLM for reasoning and the corresponding reasoning results are output.

[0041] The following is a detailed description of the technical means used in the above steps:

[0042] (1) Multi-step decomposition and classification of target text

[0043] Get the target text input by the user, and use the medical LLM's ability to understand the main intentions and key information such as disease description, symptom manifestations, treatment needs, etc. to decompose the target text input by the user and generate multiple subtasks. To ensure that each subtask focuses on a specific aspect or problem of the target text, such as disease diagnosis, treatment plan recommendation, etc., the decomposed subtasks are classified and merged through the medical LLM, and similar sub-problems are merged into one category. Specifically, the prompt words are constructed in the form of thought chains to inform the medical LLM to reason and output according to the following steps:

[0044] The first step is to extract features for each subtask. These features include the purpose of the subtask, the medical concepts involved, and the type of problem to be solved.

[0045] The second step is to label subtasks with similar feature information with the same purpose;

[0046] The third step is to select a representative subtask from the subtasks with the same purpose label and output it in a list format. The decomposed subtasks are combined with the constructed prompt words and input into the medical LLM for subtask classification. This step helps to reduce task redundancy and repetition and improve solution efficiency.

[0047] (2) Selection of key entities and relationships

[0048] For each subtask, the medical LLM is used to perform entity recognition and relationship extraction. At the same time, in order to further improve the retrieval efficiency, the medical LLM is required to sort and output entities and relationships in descending order of importance to the subtask. Specifically, the prompt words are constructed in the form of thought chains to inform the medical LLM to reason and output according to the following steps:

[0049] The first step is to perform entity recognition and relationship extraction on the subtask description;

[0050] The second step is to analyze the importance of the above entities and relationships to the description of the subtasks;

[0051] The third step is to output entities and relationships in descending order according to importance. Then, the description of each subtask is combined with the constructed prompt words and input into the medical LLM to obtain the key entities of each subtask and the relationships between entities;

[0052] The fourth step is to select no more than 10 key entities and inter-entity relationships from each subtask for use in subsequent retrieval path generation.

[0053] In the above steps, medical-related entities include disease names, symptom descriptions, drug names, etc., and the relationships between entities include the association between diseases and symptoms, the therapeutic effects between drugs and diseases, etc.

[0054] (3) Generation of candidate search paths

[0055] Based on the appropriate key entities and their relationships selected from each subtask, entities adjacent to these entities are found from the knowledge graph to form multiple retrieval paths. These candidate paths contain the key information and relationships required to answer user questions, and can provide rich background knowledge for subsequent reasoning.

[0056] (4) Heuristic Iteration to Construct Key Retrieval Paths

[0057] The medical LLM scores the relevance of the triples in the search path to its subtasks. Specifically, the prompt words are constructed in a thought chain to inform the medical LLM to reason and output according to the following steps:

[0058] The first step is to analyze the key elements and required information of the subtask;

[0059] The second step is to analyze the correlation between entities and relations in triples and key elements in subtasks;

[0060] The third step is to evaluate the contribution score of the triple to the subtask based on the analysis results, with the score ranging from 0 to 1. The constructed prompt words are combined with the triple and subtask description and input into the medical LLM to obtain the score value;

[0061] In the fourth step, the score obtained in the third step is used as the estimated function value of the heuristic search. Starting from each candidate path head node, heuristic search is performed one by one in the breadth and depth directions. Then, the triples with a score value greater than 0.5 are iteratively selected as the most relevant triples to generate and update the key retrieval path for each subtask.

[0062] (5) Final search path generation

[0063] The constructed key search paths are verified through the medical LLM to confirm whether they can be applied to answer the initial target text. The verification process mainly checks the direct relevance of each triple in the path to the user's question. Specifically, prompt words are constructed in the form of thought chains to inform the medical LLM to reason and output according to the following steps:

[0064] The first step is to analyze the key elements and required information of the initial target text;

[0065] The second step is to analyze the relevance of entities and relations in the triples with key elements in the target text;

[0066] The third step is to evaluate whether the triple is relevant to the target text based on the analysis results. If so, output “yes”, otherwise output “no”;

[0067] The fourth step is to combine each triple and target text with the constructed prompt words and input them into the medical LLM in turn to obtain the answer whether each key search path is related to the initial target text, so as to screen out the final search path and apply it to the knowledge graph to retrieve the corresponding knowledge.

[0068] (6) Prompt word construction and reasoning

[0069] The design of prompt words is to guide the medical LLM to make more accurate and targeted reasoning. According to the retrieved knowledge and target text, the prompt words are constructed. The specific template for constructing prompt words is as follows: Task: A\n Background knowledge: B\n Answer requirements: Please refer to the background knowledge to complete the task, and then combine the target text and the retrieved knowledge with the constructed prompt words to input the medical LLM for reasoning, and output the reasoning results.

[0070] The medical LLM used in the present invention is an LLM service model obtained after the pre-trained LLM basic model is pre-trained with medical field knowledge increments and fine-tuned, and the fine-tuning uses data pairs related to the target application scenario. Correspondingly, the knowledge graph used is the knowledge graph for the target application scenario; the target text input by the user received when the medical LLM interacts with the user can be text related to the target application scenario.

[0071] The present invention integrates the medical LLM capability into the entire retrieval enhancement reasoning process. By combining with the medical knowledge graph, the invention utilizes multi-step decomposition, key entity extraction, candidate retrieval path scoring, key retrieval path discrimination verification, prompt word construction and reasoning to achieve accurate understanding and answering of the user input target text.

[0072] The present invention generates key retrieval paths through multi-step decomposition of the target text and heuristic search based on medical LLM scoring, which not only simplifies the processing of complex tasks and reduces the impact of sentence ambiguity, but also provides a more precise retrieval path and specific answers for each subtask, thereby improving the accuracy of the overall answer.

Claims

1. An LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement, characterized in that: The following steps are involved: Step 1: After the user inputs the target text, the medical LLM is used to decompose the target text input by the user into multiple subtasks, and the subtasks are classified and merged; Step 2: Use medical LLM to perform entity recognition and entity relationship extraction on each subtask, and output them in order of importance; Step 3: Select multiple key entities and relationships from each subtask and apply them to the knowledge graph to find adjacent entities to form multiple candidate retrieval paths; Step 4: Use the medical LLM to score the candidate retrieval paths obtained and select the triples most relevant to each subtask, and iteratively generate and update the key retrieval paths in a heuristic search manner; Step 5: Verify the constructed key search paths through medical LLM to confirm whether they can be applied to answer the initial target text, thereby generating the final search path; Step 6: Construct prompt words based on the retrieved knowledge and target text, input medical LLM for reasoning, and output the reasoning results corresponding to the target text.

2. The LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement as claimed in claim 1, characterized in that: In step 1, the following steps are used to classify and merge subtasks: Step 101: extracting features for each subtask, where the features include the purpose of the subtask, the medical concepts involved, and the type of problem to be solved; Step 102: label subtasks with similar feature information with the same purpose label; Step 103: Select a representative subtask from the subtasks with the same purpose label and output it in a list format. The subtasks are classified by combining the decomposed subtasks with the constructed prompt words and inputting them into the medical LLM.

3. The LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement as claimed in claim 1 is characterized in that: Step 2 includes the following steps: Step 201: Perform entity recognition and relationship extraction on the subtask description; Step 202: Analyze the importance of the above entities and relationships to the description of the subtasks; Step 203: sort the output entities and relationships in descending order according to importance, and sequentially combine the description of each subtask with the constructed prompt words and input them into the medical LLM to obtain the key entities of each subtask and the relationships between the entities; Step 204: Select no more than 10 key entities and inter-entity relationships from each subtask and apply them to subsequent search path generation.

4. The LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement as claimed in claim 1, characterized in that: Step 4 includes the following steps: Step 401, analyzing the key elements and required information of the subtask; Step 402: Analyze the relevance between entities and relationships in the triples of the knowledge graph and key elements in the subtasks; Step 403: Evaluate the contribution score of the triple to the subtask based on the analysis result, with the score ranging from 0 to 1, and input the constructed prompt words into the medical LLM in combination with the triple and the subtask description to obtain the score value; Step 404: Use the score obtained in step 403 as the estimated function value of the heuristic search, and perform heuristic search in the breadth and depth directions one by one starting from each candidate path head node, and iteratively select the triples with a score value greater than 0.5 as the most relevant triples to generate and update the key retrieval path for each subtask.

5. The LLM reasoning method based on knowledge graph multi-step decomposition retrieval enhancement as claimed in claim 1, characterized in that: Step 5 includes the following steps: Step 501, analyzing the key elements and required information of the initial target text; Step 502: Analyze the relevance between entities and relations in the triples and key elements in the target text; Step 503: Based on the analysis result, evaluate whether the triple is related to the target text: if so, output "yes"; otherwise, output "no"; Step 504: Combine each triple and target text with the constructed prompt words and input them into the medical LLM in turn to obtain answers on whether each key search path is related to the initial target text, thereby screening out the final search path and applying it to the knowledge graph to retrieve the corresponding knowledge.

Images