A Disease Retrieval Method and Device Based on Knowledge Graph-Enhanced Large Language Model

Abstract

This invention discloses a disease retrieval method and apparatus based on a knowledge graph-enhanced large language model, relating to the fields of large language models and knowledge graph question answering technology. The method includes: constructing a question-answering dataset based on triples; storing and visualizing a knowledge graph of five specialized diseases using Neo4j; obtaining medical questions posed by users; progressively expanding the input questions to obtain sub-questions based on thought chain reasoning; performing relational path retrieval and knowledge subgraph retrieval based on a pre-trained model; fine-tuning the large language model to generate factual knowledge text based on the sampled answer subgraph; constructing a knowledge-enhanced prompt template by combining the input questions and factual knowledge, and feeding it into the large language model to obtain the final medical answer. This solution utilizes a knowledge graph-enhanced large language model, guiding it to reason and retrieve within efficient knowledge paths, enabling accurate and professional answers to users' specialized medical questions.

Description

Technical Field

[0001] This invention relates to the fields of large language models and knowledge graph question answering technology, and in particular to a disease retrieval method and apparatus based on a knowledge graph-enhanced large language model. Background Technology

[0002] Various specialized diseases present with diverse symptoms, requiring doctors' clinical experience and a series of examinations for diagnosis, making the diagnostic process complex and cumbersome. While many medical institutions have implemented digitalization of clinical diagnosis and treatment, data incompatibility and inapplicable methods frequently occur due to varying implementation times and the use of different information technologies. Therefore, simplifying, universalizing, digitizing, and intelligentizing the complex and cumbersome medical process is an effective way to address the challenges of diagnosing and treating specialized diseases in my country.

[0003] Applying large language models to healthcare, particularly in the field of disease prevention and control, represents a significant innovation in current research on disease retrieval question-answering systems. However, it is still limited by the following issues, preventing its direct application to clinical intelligent decision support: ① Large language models may produce illusions in specialized medical question-answering, indicating shortcomings in the accuracy and reliability of question-answering; ② Medical knowledge is constantly evolving, while the training cost of large language models is very high. When disease knowledge is updated, timely knowledge correction and retraining are required, which is excessively resource-intensive.

[0004] Therefore, for large language model retrieval and question answering of specialized diseases, the current common approach is to fine-tune the large language model or to construct a knowledge graph through traditional pipeline methods to assist in the retrieval and answering of specialized disease questions. There is currently no reliable, accurate, and human-like intelligent system that can provide professional retrieval and question answering for specialized diseases. Summary of the Invention

[0005] The purpose of this invention is to provide a disease retrieval method and apparatus based on a knowledge graph-enhanced large language model, in order to solve the problem that existing large language models cannot organize and utilize the latest professional disease knowledge and are prone to generating harmful hallucinatory facts.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide the following solutions:

[0007] A disease retrieval device based on a knowledge graph-enhanced large language model includes:

[0008] Data construction module: used to preprocess the collected multi-hop question-and-answer dataset, construct five sub-graphs of specialty disease knowledge graphs, and obtain text-based answers containing entity knowledge;

[0009] Subgraph retrieval module: Based on the medical question input by the user, it uses graph thinking chain to decompose the question, constructs a relational path reasoning model, samples triple knowledge, and retrieves the answer subgraph;

[0010] Knowledge Enhancement Module: Used to fine-tune the large language model, generating medical factual knowledge text based on the input and question-related knowledge subgraph to obtain a knowledge-enhanced retrieval model;

[0011] Answer generation module: This module combines medical factual knowledge with query questions to create knowledge-enhanced prompts, and then uses reasoning retrieval in a large language model to obtain the final medical answer.

[0012] Device and storage module: Used to process and store specialized disease knowledge graphs, providing basic hardware support for the execution of the functions of the aforementioned modules.

[0013] Furthermore, the data construction module is used to extract triple knowledge from the multi-hop question-answering dataset, construct an answer subgraph dataset based on a specific question based on the triple knowledge of topic entities, relational paths, answer entities, and their neighboring topic entities; and construct a subgraph of a specialty disease knowledge graph based on different topics, wherein the specialty disease knowledge graph topic categories include: disease_epidemiology, disease_diagnosis, disease_disease, disease_treatment, and disease_other.

[0014] Furthermore, the data construction module also includes:

[0015] The text generation and answer evaluation unit is used to generate text from the collected answer subgraph dataset using a large language model, obtaining a supervised fine-tuning dataset based on a specific question; among which...

[0016] The text generation and answer evaluation unit is specifically used to input the answer subgraph and the prompt template into the large language model to obtain the factual knowledge answer based on the specific question; encapsulate the answer as a knowledge enhancement prompt and input it into the large language model again; according to the response information, the factual knowledge answer containing at least one answer entity is taken as the correct answer; integrate the answer subgraph and the factual knowledge answer to divide the training, validation and test datasets as the supervised fine-tuning dataset of the knowledge enhancement module.

[0017] Furthermore, the subgraph retrieval module is used to expand the question step by step according to the user's input question and the graph thinking chain reasoning, decomposing it into several jump sub-questions. Each jump sub-question contains only one relation triplet knowledge based on the specialized disease knowledge graph, and obtains the step-by-step question planning steps.

[0018] Furthermore, the subgraph retrieval module also includes:

[0019] The relational path reasoning unit is used to predict the relationships of the problem step by step and generate a reasoning path based on the number of hops after the planning decomposition; among which,

[0020] The relation path reasoning unit is specifically used to construct a pre-trained language model and a linear classifier, embed the decomposed sub-problems and knowledge graph into a multi-dimensional vector, predict the relation category in the specialty disease knowledge graph at each hop, select several most relevant relations as the relations for the sub-problem based on the predicted relation probability distribution, iterate repeatedly during the next hop relation prediction reasoning, combine the new sub-problem vector and the currently predicted relation path reasoning to obtain the latest hop relation path, until the number of hops planned for decomposition is reached, calculate the relation path score for each prediction, measure the difference between the output relation path and the true triples and iterate repeatedly, use cross-entropy loss to optimize the parameters of the pre-trained language model, obtain the relation reasoning path set by sorting the relation path scores in descending order, and construct an answer subgraph based on a specific question.

[0021] Furthermore, the knowledge enhancement module is specifically used to convert the retrieved answer subgraph into factual knowledge text suitable for reasoning in the large language model, and to optimize and adjust the output of the large language model in a textual knowledge representation form that implies relational reasoning paths; during the fine-tuning of the large language model training, each time step predicts the probability distribution of all words in the current time step based on the correct embedding sequence of the previous time step, and optimizes the parameters of the large language model with cross-entropy loss; finally, the knowledge enhancement module will enable the large language model after instruction fine-tuning to have the ability to generate graph text.

[0022] Furthermore, the answer generation module is used to encapsulate the factual knowledge text and medical questions obtained from the large language model into knowledge-enhanced prompt templates, construct a knowledge-enhanced retrieval model, and obtain the final retrieval answer for the medical disease question.

[0023] Furthermore, the answer generation module also includes:

[0024] The model performance evaluation unit is used to measure the performance of the knowledge-enhanced retrieval model according to the evaluation indicators, and at the same time compare and analyze the performance of the large language model in the evaluation indicators of retrieval and question answering without knowledge enhancement, the performance of retrieval and question answering based on free-format text, and the performance of retrieval and question answering based on factual knowledge text in triplet format.

[0025] Furthermore, the device and storage module are used to provide a processor and memory device for the specialized disease retrieval system, which can respectively process readable instructions of the computer, execute the question-and-answer retrieval function in the specialized disease domain, and store triple datasets, supervised fine-tuning datasets, and specialized disease knowledge graph data.

[0026] Furthermore, this invention also provides a disease retrieval method based on a knowledge graph-enhanced large language model, wherein the method is applied to the apparatus described above, and the method includes:

[0027] Data construction method: Obtain a multi-hop question-answering dataset and perform data preprocessing; construct and store a knowledge graph of five themes of specialized diseases based on Neo4j; realize text generation and answer evaluation based on a large language model and answer subgraph dataset, and construct a supervised fine-tuning dataset;

[0028] Subgraph retrieval method: Obtain the medical questions raised by users, and expand the input questions step by step to obtain sub-questions based on graph thinking chain reasoning; perform relation path retrieval and knowledge subgraph retrieval based on pre-trained models;

[0029] Knowledge enhancement methods: fine-tuning a large language model to generate factual knowledge text based on sampled answer subgraphs; constructing a knowledge-enhanced retrieval model;

[0030] Answer generation method: Based on the input question and factual knowledge, knowledge-enhanced prompts are constructed, and the final medical search answer is obtained through a large language model; the performance of the knowledge-enhanced retrieval model is measured according to evaluation metrics.

[0031] Optionally, the data construction method is used to extract triple knowledge from the multi-hop question-answering dataset, construct an answer subgraph dataset based on a specific question based on the triple knowledge of topic entities, relation paths, answer entities and their neighboring topic entities; and construct a subgraph of specialized disease knowledge graph based on different topics.

[0032] The subject categories of the specialized disease knowledge graph include: Disease_Epidemiology, Disease_Diagnosis, Disease_Disease, Disease_Treatment, and Disease_Other.

[0033] The data construction method further includes: a text generation and answer evaluation method, which is used to generate text from the collected answer subgraph dataset using a large language model to obtain a supervised fine-tuning dataset based on a specific question;

[0034] Optionally, the text generation and answer evaluation method is further used for:

[0035] The answer subgraph and the prompt template are combined and input into a large language model to obtain factual knowledge answers based on specific questions;

[0036] The answer is encapsulated as a knowledge-enhanced hint and input into the large language model again. Based on the response information, the factual knowledge answer containing at least one answer entity is taken as the correct answer.

[0037] The training, validation, and test datasets are integrated with the answer subgraph and factual knowledge answer partitions to serve as the supervised fine-tuning dataset for the knowledge enhancement module.

[0038] The subgraph retrieval method is used to expand the question step by step according to the graph thinking chain reasoning based on the question input by the user, decompose it into several jump sub-questions, and each jump sub-question contains only a relation triple knowledge based on the specialized disease knowledge graph, thereby obtaining the step-by-step question planning steps.

[0039] The subgraph retrieval method further includes: a relational path reasoning method, which is used to predict the relationships of a problem step by step and generate a relational reasoning path based on a specific problem based on the number of hops after planning decomposition;

[0040] Optionally, the relational path reasoning method is further used for:

[0041] Construct a pre-trained language model and a linear classifier, embed the decomposed sub-problems and knowledge graph into a multi-dimensional vector, and predict the relationship category in the specialty disease knowledge graph in each hop;

[0042] Based on the predicted probability distribution of relationships, select several of the most relevant relationships as the relationships for this sub-problem;

[0043] The next-hop relation prediction reasoning is iterated repeatedly, combining the new sub-problem vector and the currently predicted relation path reasoning to obtain the latest next-hop relation path, until the number of hops of the planning decomposition is reached;

[0044] Calculate the relation path score for each prediction, measure the difference between the output relation path and the true triple, and iterate repeatedly to learn. Use cross-entropy loss to optimize the parameters of the pre-trained language model.

[0045] Obtain the set of relational reasoning paths by sorting them in descending order of relational path scores, and construct an answer subgraph based on a specific question.

[0046] The knowledge enhancement method is used to convert the retrieved answer subgraph into factual knowledge text suitable for reasoning in a large language model, and to optimize and adjust the output of the large language model in a textual knowledge representation form that implies relational reasoning paths.

[0047] During fine-tuning the training of the large language model, at each time step, the probability distribution of all words in the current time step is predicted based on the correct embedding sequence of the previous time step, and the parameters of the large language model are optimized by cross-entropy loss.

[0048] The final knowledge enhancement module will enable the finely tuned large language model to generate graph text.

[0049] The answer generation method is used to encapsulate factual knowledge text and medical questions obtained from large language model reasoning into knowledge-enhanced prompt templates, construct a knowledge-enhanced retrieval model, and obtain the final retrieval answer for medical disease questions.

[0050] The answer generation method further includes a model performance evaluation method, which measures the performance of the knowledge-enhanced retrieval model according to evaluation indicators, and simultaneously compares and analyzes the performance of the large language model in direct retrieval question answering without knowledge enhancement, in retrieval question answering based on free-format text, and in retrieval question answering based on factual knowledge text in triplet format.

[0051] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0052] This solution addresses the common problem of current large language models being unable to organize and utilize the latest specialized disease knowledge and prone to generating harmful illusory facts. It provides a disease retrieval device based on knowledge graph enhancement of large language models. By constructing a specialized disease domain knowledge graph, the large language model is optimized to efficiently grasp domain-specific knowledge and generate reliable retrieval question-and-answer factual text. This provides necessary system support for users with specialized disease-assisted retrieval question-and-answer needs or professional domain knowledge learning requirements.

[0053] This solution utilizes open-source multi-hop question-answering datasets to construct a knowledge graph for the specialized disease domain, based on five themes. A large language model generates factual knowledge text answers based on triples. Building upon this, a graph-based reasoning strategy guides the large language model to progressively expand and decompose the questions. A pre-trained language model predicts relational paths for sub-questions, and after sampling triples, an answer subgraph is obtained. By fine-tuning the large language model, it gains the ability to generate factual knowledge text answers based on the knowledge graph, thereby obtaining enhanced hints associated with the retrieved answer subgraph and providing professional knowledge base support for specialized disease domain question-answering retrieval. Finally, a knowledge-enhanced retrieval model based on the knowledge graph and the large language model is established, and answer generation is evaluated to achieve efficient and accurate responses to specialized disease domain knowledge.

[0054] This solution can effectively improve the accuracy and transparency of traditional large language models in question answering in the professional medical field. While providing a response, it can also provide a complete disease knowledge reasoning path, offering an innovative technical approach for specialized disease retrieval question answering and professional medical knowledge learning. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the structure of the disease retrieval device based on the knowledge graph-enhanced large language model provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the specialized disease knowledge graph model design provided in an embodiment of the present invention;

[0058] Figure 3 This is a partial schematic diagram of the visualization of a specialized disease knowledge graph provided in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram illustrating the principle of the subgraph retrieval model provided in this embodiment of the invention;

[0060] Figure 5 This is a graph showing the loss value and evaluation index of the relational path reasoning prediction model provided in this embodiment of the invention;

[0061] Figure 6 This is a schematic diagram illustrating the principle of the knowledge-enhanced retrieval model provided in this embodiment of the invention;

[0062] Figure 7 This is a schematic diagram of a specialized disease search and answering method provided in an embodiment of the present invention;

[0063] Figure 8 This is a block diagram of a disease retrieval device based on a knowledge graph-enhanced large language model provided in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0065] First Embodiment

[0066] To address the problems prevalent in current medical large language models, such as their inability to organize and utilize the latest professional disease knowledge and their tendency to generate harmful hallucinations, this embodiment provides a disease retrieval device based on a knowledge graph-enhanced large language model. This method can be implemented by an electronic device, which can be a terminal or a server.

[0067] This method is based on open-source multi-hop question-answering data. After data preprocessing, triple knowledge is obtained, and a knowledge graph of five themes in the specialty disease domain is constructed using Neo4j. Then, factual answer text is generated through a large language model, and the quality is evaluated based on whether the generated text contains at least one answer entity. Finally, factual knowledge answers based on triple knowledge are obtained, and training, validation, and test sets are divided as supervised fine-tuning datasets. For user-input questions, the questions are progressively expanded and planned using graph thinking chain reasoning. The large language model generates the reasoning solution steps and predicts the number of hops for the question. Based on the constructed pre-trained language model, potential relational path reasoning is performed on sub-questions to predict and mine background knowledge paths containing related to theme entities. Triples are then sampled to obtain answer subgraphs. The large language model is fine-tuned to enable it to generate factual knowledge text based on the answer subgraph. Through knowledge augmentation, the large language model is guided to answer user-input specialty disease domain questions with evidence, enabling intelligent and accurate retrieval and question answering in the specialty disease domain.

[0068] Based on the above, combined with Figure 1 As shown, the method in this embodiment includes the following steps:

[0069] S1 preprocesses the multi-hop question-answering dataset to construct knowledge graph subgraphs for five topics, obtaining text-based answers containing entity knowledge.

[0070] Specifically, in this embodiment, the implementation process of S1 is as follows:

[0071] S11: Obtain the multi-hop question-answering dataset and preprocess it to obtain the questions and their corresponding triples.

[0072] It should be noted that the multi-hop question-answering dataset used in this embodiment comes from the COKG multi-hop question-answering dataset in the OpenKG (http: / / openkg.cn / ) open knowledge graph community initiated and advocated by the Language and Knowledge Computing Professional Committee of the Chinese Information Processing Society of China. The preprocessing steps include reading the dataset and obtaining the knowledge triples for the corresponding questions using the split function, thereby extracting the knowledge subgraph based on the corresponding questions; obtaining a deduplicated relation list using the set collection and storing it as relation.txt.

[0073] S12, Construct a knowledge graph for specific disease areas.

[0074] The specific construction process of the knowledge graph for the specialty disease domain is as follows: ontology modeling, that is, using a top-down approach to model relation categories, including the name of each relation category, each relation subcategory under the relation category, the head entity type and tail entity type involved in the relation subcategory; based on the relation categories, a knowledge graph based on different relation topic categories can be constructed on the Neo4j graph database to realize the visualization of triple knowledge.

[0075] Specifically, in this embodiment, S12 includes the following steps:

[0076] S121. Based on the relationships extracted from relation.txt, text clustering is performed to determine the category of keywords. Then, through manual screening and summarization, the questions are divided into five categories: "Disease_Epidemiology", "Disease_Diagnosis", "Disease_Disease", "Disease_Treatment", and "Disease_Other".

[0077] Specifically, based on the COKG multi-hop question-answering dataset, the text is first clustered using the traditional style clustering algorithm K-Means to find the appropriate number of categories for each question. The specific question topics and examples are shown in Table 1.

[0078] Table 1. Topic Classification and Example Questions

[0079]

[0080]

[0081] S122 involves ontology modeling based on keyword categories and relationships, i.e., defining entity categories, relationship categories, and relationship subcategories. A schematic diagram of the knowledge graph pattern design is shown below. Figure 2 As shown.

[0082] S123, based on the Neo4j graph database, constructs a knowledge graph for a specific disease domain, enabling visualization of domain knowledge. A partial schematic diagram is shown below. Figure 3 As shown.

[0083] S13, construct a supervised fine-tuning dataset.

[0084] The specific construction process of the supervised fine-tuning dataset involves generating text from the collected answer subgraph dataset using a large language model (LLM), dividing the dataset into training, test, and validation sets after text generation and answer evaluation, and finally obtaining a supervised fine-tuning dataset based on a specific question.

[0085] Specifically, in this embodiment, S13 includes the following steps:

[0086] S131, Text generation: Combine the answer subgraph and the prompt template and input them into the large language model to obtain factual knowledge answers based on specific questions.

[0087] Specifically, the graph-to-text (KG-to-Text) prompts used to obtain factual knowledge answers through the large language model, KG2Text_Prompt, are shown in Table 2. KG2Text_Prompt must include an instruction and a knowledge subgraph.

[0088] Table 2. Description of the graph-to-text prompt: KG2Text_Prompt

[0089]

[0090] S132, Answer Evaluation: The answer from S131 is encapsulated as a knowledge-enhanced hint and re-inputted into the large language model. Based on the response information, the factual knowledge answer containing at least one answer entity is taken as the correct answer.

[0091] Specifically, the knowledge-enhanced prompt Augmented_Prompt used for answer evaluation through the large language model is shown in Table 3. Augmented_Prompt must contain an instruction, an input question, and factual information.

[0092] Table 3. Knowledge Enhancement Prompt Descriptions (Augmented_Prompt)

[0093]

[0094] S133 integrates the answer subgraph and factual knowledge answer partitioning into training, validation, and test datasets, serving as a supervised fine-tuning dataset for the knowledge enhancement module.

[0095] S2, Construct a subgraph retrieval model.

[0096] The specific process of constructing the subgraph retrieval model is as follows: The complex input problem is decomposed into several sub-problems using G-COT (Graph-Chain of Thoughts); relational paths are predicted using PLM (Pre-trained Language Model); finally, based on the relational paths obtained from the predicted probabilities, triple sampling is performed to retrieve knowledge subgraphs related to the specific problem. A schematic diagram of the subgraph retrieval model principle is shown below. Figure 4 As shown.

[0097] Specifically, in this embodiment, the implementation process of S2 is as follows:

[0098] S21, Problem planning and decomposition based on graph thinking chain.

[0099] It should be noted that, based on the user's input question, LLM uses a graph-based reasoning strategy to progressively expand the question, breaking it down into several sub-questions. Each sub-question contains only one relational triplet based on a specialized disease knowledge graph, thus obtaining progressive question planning steps.

[0100] The problem decomposition planning process uses the planning prompt example shown in Table 4, which must include an instruction, an example, and an input question. The Topic length obtained after problem decomposition planning is the number of hops for a given problem.

[0101] Table 4. Planning Prompt

[0102]

[0103] S22, Retrieval reasoning prediction based on relational paths.

[0104] It should be noted that the constructed relation path prediction model PLM is based on BERT (Bidirectional Encoder Representations from Transformers). Using BERT and a linear classifier, the decomposed sub-problems and the specialized disease knowledge graph are embedded into multi-dimensional vectors to predict the relation categories in the specialized disease knowledge graph at each hop. Based on the predicted relation probability distribution, several of the most relevant relations are selected as the relations for that sub-problem. During the next hop relation prediction inference, the process iterates repeatedly, combining the new sub-problem vector and the currently predicted relation path inference to obtain the latest next-hop relation path, until the planned number of hops for decomposition is reached. During training, the cross-entropy loss value is calculated on the training set to continuously optimize the model parameters. In each training cycle e... i The loss is then calculated on the validation set to monitor for overfitting. Finally, after model training is complete, the overall performance of the relation path retrieval inference prediction model is evaluated based on the test set.

[0105] Specifically, the training process for retrieval reasoning prediction based on relational paths is as follows:

[0106] The specific retrieval reasoning and prediction method based on relational paths is as follows:

[0107] enter:

[0108] The natural language question Q input by the user, the specialist disease knowledge graph KG, the size R of the relationship set in the graph, the problem hop count H, the preset number of candidate relationships N for each hop of the problem, and the number of iterations epoches = {e1, e2,..., e n}. Here, n represents the maximum number of iterations for model training.

[0109] Output:

[0110] The i-th relationship path p at the h-th hop h,i (1 ≤ h ≤ H ∧ i ≤ N < R), the score Score(p h,i ) of the i-th relationship path at the h-th hop, and the relationship path set p.

[0111] Step 1: Initialize the relationship path set p as an empty set

[0112] Step 2: Traverse the number of iterations e i , and perform the following process:

[0113] For each step h: that is, from 1 to H

[0114] If it is the first step: then

[0115] Construct a BERT encoder to embed the question Q into a vector V(Q), that is:

[0116] V(Q) = BERT(Q)

[0117] Based on a linear classifier, calculate the distribution D R of all relationships r1, r2,..., r r,1 in KG;

[0118]

[0119]

[0120] Obtain N relationships r with the highest probabilities i (i = 1, 2..., N ∧ i ≤ R), form the first-hop relationship path set p1, where the i-th relationship path p 1,i at the first hop is the relationship with the i-th highest relationship distribution probability;

[0121]

[0122] p1 = p 1,1 |p 1,2 |...|p 1,N

[0123] Add p1 to the relationship path set p;

[0124] p = {p1}

[0125] If it is step 2 to step H, then:

[0126] Connect problem Q with the path p1~p1 of the previous h-1 steps. h-1 Concatenate the data, perform classification based on BERT, and calculate the probability distribution D of the jump relationship in the KG. r,h ,Right now:

[0127] V(Q∨p1|p2|...|p h-1 =BERT(Q∨p1|p2|...|p h-1 )

[0128]

[0129]

[0130] Obtain the N relations with the highest probability r i (i = 1, 2, ..., N), forming the h-th hop relation path set p h ;

[0131]

[0132] p h =p h,1 |p h,2 |...|p h,N

[0133] Update the relation path set p to include all newly generated relation paths;

[0134] p = {p1, p2, ..., p} h}

[0135] Calculate until the iteration loop ends;

[0136] For each relation path p in P r :

[0137] Calculate p r The score of the probability of all relations in the middle, as p r The score, where the h-th hop and the i-th relation path is Score(p) h,i );

[0138]

[0139] The training loss value L is calculated using the cross-entropy loss function, and the parameters are optimized.

[0140]

[0141] Step 3: Return the set of relational paths p, and the scores for all relational paths;

[0142] Step 4: Evaluate the model performance on the test set.

[0143] Specifically, accuracy is used to evaluate the effectiveness of the aforementioned retrieval inference prediction method based on relation paths. Accuracy represents the proportion of correctly predicted relation paths (TP) to the total number of predicted relation paths (TP+FP), and it is calculated as follows:

[0144]

[0145] Specifically, the retrieval reasoning prediction model based on relational paths achieved a precision of 87% on the test set. During training, the model's training and validation loss values ​​and accuracy at each iteration were as follows: Figure 5 As shown.

[0146] S23, Knowledge Subgraph Retrieval Based on Triple Sampling.

[0147] It should be noted that in each knowledge subgraph retrieval process, the set of relational paths and the scores of all relational paths for the corresponding question are first obtained through a retrieval reasoning prediction algorithm based on relational paths. Then, the set of reasoning paths is obtained by sorting the scores of each hop relational path in descending order, and finally, a knowledge subgraph based on a specific question is constructed.

[0148] S3, Knowledge Enhancement and Answer Generation.

[0149] The specific process of knowledge enhancement and answer generation is as follows: The large language model is fine-tuned to generate medical factual knowledge text based on the input and question-related knowledge subgraph, resulting in a specialized disease knowledge graph text enhancement model. For a specific question input by the user, the answer subgraph retrieved in S2 is first converted into knowledge triples, and then the knowledge enhancement prompts are input into the LLM for answer generation. The LLM's answer is enhanced by the factual knowledge text after fine-tuning, reducing the generation of domain illusion in the LLM. The effect of graph text knowledge enhancement is evaluated using Hit@1. A schematic diagram of the knowledge enhancement retrieval model principle is shown below. Figure 6 As shown.

[0150] Specifically, in this embodiment, the implementation process of S3 is as follows:

[0151] S31, graph text generation based on large model fine-tuning.

[0152] It's worth noting that the `chinese-alpaca-2-7b` method was used for fine-tuning the large-scale model, employing Parameter-Efficient Fine-Tuning (PEFT). By constructing a knowledge graph and converting it to text prompts using `KG2Text_Prompt`, the knowledge subgraph is transformed into text in triplet form. Specifically, this converts the retrieved knowledge subgraph into factual knowledge text suitable for reasoning in the large language model, optimizing the model's output with a textual knowledge representation format that implies the reasoning path. During fine-tuning of the large language model training, at each time step, the probability distribution of all words in the current time step is predicted based on the correct embedding sequence from the previous time step, using cross-entropy loss to optimize the model's parameters.

[0153] Specifically, the training process of the graph text generation model based on large model fine-tuning is as follows:

[0154] The specific graph text generation algorithm based on large model fine-tuning is as follows:

[0155] enter:

[0156] Knowledge subgraph G: A collection containing triples TripleText, based on a KG-to-Text supervised fine-tuning dataset (including training set train_x, validation set valid_x, and test set test_x), with batch size batch_size and number of iterations epoches={e1,e2,...,e n} (where n represents the maximum number of training iterations), PEFT configuration PeftConfig: used to specify the target modules target_modules and hyperparameters for fine-tuning. Hyperparameters include: lora_rank: the rank of lora, i.e., the size of the low-rank matrix used in the model parameterization process; lora_dropout: the dropout rate, i.e., the proportion of randomly dropped neurons applied to the lora layer during model training; lora_alpha: the learning rate scaling factor for lora adaptation, where PeftConfig={target_modules,lora_rank,lora_dropout,lora_alpha},

[0157] chinese-alpaca-2-7b: A large language model of 7B size for Chinese understanding and generation tasks. Output:

[0158] Factual knowledge text Y: A sequence of natural language text describing the knowledge subgraph G.

[0159] Step 1: Based on the KG-to-Text supervised fine-tuning dataset, fine-tune the large language model using the PEFT method. The pre-trained large language model here is LLM. pre-trained The one used is chinese-alpaca-2-7b, which is about to be LLM. pre-trained The PEFT configuration, PeftConfig, is input into PeftModel for fine-tuning to obtain the fine-tuned Large Language Model (LLM). fine-tuned ;

[0160] LLM fine-tuned =PeftModel(LLM) pre-trained PeftConfig)

[0161] Number of iterations e i Perform the following process:

[0162] Import text data of size batch_size from the training set;

[0163] For each training text, the real output sequence and graph-to-text prompt KG2Text_Prompt are used to guide the LLM to generate the next correct output;

[0164] LLM fine-tuned .train(train_x)

[0165] The LLM parameters are updated by minimizing the cross-entropy loss function.

[0166] End the iteration loop;

[0167] Step 2: Construct the knowledge graph to a text prompt; that is, construct a KG2Text_Prompt prompt based on the input knowledge subgraph G.

[0168] KG2Text_Prompt=Template.format(TripleText)

[0169] Step 3: Input the graph-to-text prompt into the fine-tuned chinese-alpaca-2-7b to achieve KG-to-Text and return the factual knowledge text Y.

[0170] Y = LLM fine-tuned .generate(KG2Text_Prompt).

[0171] Specifically, the graph-to-text (KG) prompt example KG2Text_Prompt, as shown in Table 2, was used in the large model fine-tuning process for generating KG-to-text. The knowledge subgraph is the result obtained from the subgraph retrieval in S2.

[0172] S32, Knowledge-enhanced disease question-answering retrieval.

[0173] It should be noted that the factual knowledge text generated based on the large language model and the constructed knowledge graph is encapsulated along with the user-input medical question into a knowledge-enhanced prompt template. This factual knowledge text serves as background knowledge relevant to the user's medical question, thereby improving the performance of the LLM (Learning Resource Management). The knowledge-enhanced prompt is then input into the LLM for question answering to obtain the final retrieval answer to the medical question. This output answer is the knowledge-enhanced specialty disease retrieval result.

[0174] Specifically, the knowledge-enhanced disease question-answering retrieval process used the Augmented_Prompt example shown in Table 3. Here, the factual knowledge refers to the factual knowledge text output by KG-to-Text in S31.

[0175] S33, Evaluate the effectiveness of the knowledge-enhanced retrieval model.

[0176] It should be noted that the specific process for evaluating the effectiveness of the knowledge-enhanced retrieval model is as follows: A comparative evaluation experiment was conducted by measuring the performance of three retrieval question-answering methods: LLM performing retrieval question-answering directly without knowledge enhancement, LLM performing retrieval question-answering based on free-format text, and LLM performing retrieval question-answering based on factual knowledge text in triplet format. The evaluation process used Hit@1 to specifically evaluate the effectiveness of the knowledge-enhanced retrieval model. Hit@1 is used to assess whether the generated disease search answer contains at least one answer entity from a knowledge triplet related to the given question.

[0177] Specifically, the formula for calculating Hit@1 is:

[0178]

[0179] Specifically, |S| represents the total number of queries for knowledge triples related to a given question, and rank... i Let represent the rank of the i-th correct answer in the prediction results, and S denote the set of triples. Ⅱ is the indicator function; if rank... i If ≤1 is true (i.e., the correct answer entity is ranked first), then the function value is 1; otherwise, it is 0.

[0180] Specifically, the evaluation results of the knowledge-enhanced retrieval model are shown in Table 5:

[0181] Table 5 Evaluation Results of Knowledge-Enhanced Retrieval Model

[0182]

[0183] Second embodiment

[0184] Figure 8 This is a block diagram illustrating a disease retrieval device based on a knowledge graph-enhanced large language model, according to an exemplary embodiment. The device is used in a disease retrieval method based on a knowledge graph-enhanced large language model. (Refer to...) Figure 8 The device includes a data construction module 810, a text generation and answer evaluation unit 820, a subgraph retrieval module 830, a relational path reasoning unit 840, a knowledge enhancement module 850, an answer generation module 860, and a model performance evaluation unit 870. The specific methods implemented by this device include:

[0185] Data construction method: This method is used to implement the data construction module 810, which acquires a multi-hop question-and-answer dataset and performs data preprocessing. It also constructs and stores a knowledge graph of five themes of specialized diseases based on Neo4j, and implements text generation and answer evaluation based on a large language model and answer subgraph dataset, and constructs a supervised fine-tuning dataset.

[0186] Subgraph retrieval method: used to implement the subgraph retrieval module 830, obtain the medical questions proposed by the user, and gradually expand the input questions to obtain sub-questions based on graph thinking chain reasoning; perform relation path retrieval and knowledge subgraph retrieval based on a pre-trained model;

[0187] Knowledge augmentation methods: used to implement knowledge augmentation module 850, fine-tuning the large language model to generate factual knowledge text based on the sampled answer subgraph; constructing a knowledge augmentation retrieval model;

[0188] Answer generation method: This method is used to implement the answer generation module 860. It constructs knowledge-enhanced prompts based on the input question and factual knowledge, and obtains the final medical retrieval answer through a large language model. The performance of the knowledge-enhanced retrieval model is measured according to evaluation metrics.

[0189] The data construction method is used to implement the data construction module 810, extract triple knowledge from the multi-hop question-answering dataset, construct an answer subgraph dataset based on a specific question based on the triple knowledge of topic entities, relation paths, answer entities and their neighboring topic entities; and construct a subgraph of specialized disease knowledge graph based on different topics.

[0190] The subject categories of the specialized disease knowledge graph include: Disease_Epidemiology, Disease_Diagnosis, Disease_Disease, Disease_Treatment, Disease_Other, etc.

[0191] The data construction method further includes: a text generation and answer evaluation method, which is used to implement the text generation and answer evaluation unit 820, which generates text from the collected answer subgraph dataset through a large language model to obtain a supervised fine-tuning dataset based on a specific question;

[0192] Optionally, the text generation and answer evaluation method is further used for:

[0193] The answer subgraph and the prompt template are combined and input into a large language model to obtain factual knowledge answers based on specific questions;

[0194] The answer is encapsulated as a knowledge-enhanced hint and input into the large language model again. Based on the response information, the factual knowledge answer containing at least one answer entity is taken as the correct answer.

[0195] The training, validation, and test datasets are integrated with the answer subgraph and factual knowledge answer partitions to serve as the supervised fine-tuning dataset for the knowledge enhancement module.

[0196] The subgraph retrieval method is used to implement the subgraph retrieval module 830. Based on the question input by the user, the question is gradually expanded and decomposed into several sub-questions according to the graph thinking chain reasoning. Each sub-question contains only one relation triplet knowledge based on the specialized disease knowledge graph, and the step-by-step question planning steps are obtained.

[0197] The subgraph retrieval method further includes a relation path reasoning method, which is used to implement the relation path reasoning unit 840, and to perform step-by-step relation prediction on the problem and generate a relation reasoning path based on a specific problem according to the number of hops after planning decomposition.

[0198] Optionally, the relational path reasoning method is further used for:

[0199] Construct a pre-trained language model and a linear classifier, embed the decomposed sub-problems and knowledge graph into a multi-dimensional vector, and predict the relationship category in the specialty disease knowledge graph in each hop;

[0200] Based on the predicted probability distribution of relationships, select several of the most relevant relationships as the relationships for this sub-problem;

[0201] The next-hop relation prediction reasoning is iterated repeatedly, combining the new sub-problem vector and the currently predicted relation path reasoning to obtain the latest next-hop relation path, until the number of hops of the planning decomposition is reached;

[0202] Calculate the relation path score for each prediction, measure the difference between the output relation path and the true triple, and iterate repeatedly to learn. Use cross-entropy loss to optimize the parameters of the pre-trained language model.

[0203] Obtain the set of relational reasoning paths by sorting them in descending order of relational path scores, and construct an answer subgraph based on a specific question.

[0204] The knowledge enhancement method is used to implement the knowledge enhancement module 850, which converts the retrieved answer subgraph into factual knowledge text suitable for reasoning in a large language model, and optimizes and adjusts the output of the large language model in a textual knowledge representation form that contains relational reasoning paths.

[0205] During fine-tuning the training of the large language model, at each time step, the probability distribution of all words in the current time step is predicted based on the correct embedding sequence of the previous time step, and the parameters of the large language model are optimized by cross-entropy loss.

[0206] The final knowledge enhancement module will enable the finely tuned large language model to generate graph text.

[0207] The answer generation method is used to implement the answer generation module 860, which encapsulates the factual knowledge text and medical questions obtained by the large language model into a knowledge-enhanced prompt template, constructs a knowledge-enhanced retrieval model, and obtains the final retrieval answer for the medical disease question.

[0208] The answer generation method further includes a model performance evaluation method, which implements the model performance evaluation unit 870, measures the performance of the knowledge-enhanced retrieval model according to the evaluation index, and simultaneously compares and analyzes the performance of the large language model in direct retrieval question answering without knowledge enhancement, the performance of retrieval question answering based on free-format text, and the performance of retrieval question answering based on factual knowledge text in triplet format.

[0209] Third Embodiment

[0210] This embodiment provides an electronic device, which includes a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment, or an execution system including the method of the first embodiment.

[0211] The electronic device can vary considerably depending on its configuration or performance, and may include one or more processors (central processing units, CPUs) and one or more memories, wherein the memories store at least one instruction that is loaded by the processor and executed in accordance with the above method.

[0212] Fourth embodiment

[0213] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0214] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0215] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0216] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A disease retrieval device based on a knowledge graph-enhanced large language model, characterized in that, include: Data construction module: used to preprocess the collected multi-hop question-and-answer dataset, construct five sub-graphs of specialty disease knowledge graphs, and obtain text-based answers containing entity knowledge; Subgraph retrieval module: Based on the medical question input by the user, it uses graph thinking chain to decompose the question, constructs a relational path reasoning model, samples triple knowledge, and retrieves the answer subgraph; Knowledge Enhancement Module: Used to fine-tune the large language model, generating medical factual knowledge text based on the input and question-related knowledge subgraph to obtain a knowledge-enhanced retrieval model; Answer generation module: This module combines medical factual knowledge with query questions to create knowledge-enhanced prompts, and then uses reasoning retrieval in a large language model to obtain the final medical answer. Device and storage module: Used to process and store specialized disease knowledge graphs, providing basic hardware support for the execution of the functions of the aforementioned modules; The data construction module is used to extract triple knowledge from the multi-hop question-answering dataset, and construct an answer subgraph dataset based on specific questions according to the triple knowledge of topic entities, relation paths, answer entities, and their neighboring topic entities; and construct a subgraph of specialized disease knowledge graph based on different topics, wherein... The subject categories of the specialty disease knowledge graph include: Disease_Epidemiology, Disease_Diagnosis, Disease_Disease, Disease_Treatment, and Disease_Other; The subgraph retrieval module is used to expand the question step by step according to the user input question and the graph thinking chain reasoning, decomposing it into several jump sub-questions. Each jump sub-question contains only one relation triple knowledge based on the specialty disease knowledge graph, and obtains the step-by-step question planning steps.

2. The disease retrieval device based on a knowledge graph-enhanced large language model according to claim 1, characterized in that, The data construction module also includes: The text generation and answer evaluation unit is used to generate text from the collected answer subgraph dataset using a large language model, obtaining a supervised fine-tuning dataset based on a specific question; among which... The text generation and answer evaluation unit is specifically used to input the answer subgraph and the prompt template into the large language model to obtain the factual knowledge answer based on the specific question; encapsulate the answer as a knowledge enhancement prompt and input it into the large language model again; according to the response information, the factual knowledge answer containing at least one answer entity is taken as the correct answer; integrate the answer subgraph and the factual knowledge answer to divide the training, validation and test datasets as the supervised fine-tuning dataset of the knowledge enhancement module.

3. The disease retrieval device based on a knowledge graph-enhanced large language model according to claim 1, characterized in that, The subgraph retrieval module further includes: The relational path reasoning unit is used to predict relationships step-by-step based on the number of hops after planning decomposition and generate relational reasoning paths based on specific problems; among which, The relation path reasoning unit is specifically used to construct a pre-trained language model and a linear classifier, embed the decomposed sub-problems and knowledge graph into a multi-dimensional vector, predict the relation category in the specialty disease knowledge graph at each hop, select several most relevant relations as the relations for the sub-problem based on the predicted relation probability distribution, iterate repeatedly during the next hop relation prediction reasoning, combine the new sub-problem vector and the currently predicted relation path reasoning to obtain the latest hop relation path, until the number of hops planned for decomposition is reached, calculate the relation path score for each prediction, measure the difference between the output relation path and the true triples and iterate repeatedly, use cross-entropy loss to optimize the parameters of the pre-trained language model, obtain the relation reasoning path set by sorting the relation path scores in descending order, and construct an answer subgraph based on a specific question.

4. The disease retrieval device based on a knowledge graph-enhanced large language model according to claim 1, characterized in that, The knowledge enhancement module is specifically used to convert the retrieved answer subgraph into factual knowledge text suitable for reasoning in the large language model, and to optimize and adjust the output of the large language model in the form of textual knowledge representation containing relational reasoning paths. During the fine-tuning of the large language model training, at each time step, the probability distribution of all words in the current time step is predicted based on the correct embedding sequence of the previous time step, and the parameters of the large language model are optimized with cross-entropy loss. Finally, the knowledge enhancement module will enable the large language model after instruction fine-tuning to have the ability to generate graph text.

5. The disease retrieval device based on a knowledge graph-enhanced large language model according to claim 1, characterized in that, The answer generation module is used to encapsulate the factual knowledge text and medical questions obtained from the large language model into knowledge-enhanced prompt templates, construct a knowledge-enhanced retrieval model, and obtain the final retrieval answer for the medical disease question.

6. The disease retrieval device based on a knowledge graph-enhanced large language model according to claim 5, characterized in that, The answer generation module also includes: The model performance evaluation unit is used to measure the performance of the knowledge-enhanced retrieval model according to the evaluation indicators, and at the same time compare and analyze the performance of the large language model in the evaluation indicators of retrieval and question answering without knowledge enhancement, the performance of retrieval and question answering based on free-format text, and the performance of retrieval and question answering based on factual knowledge text in triplet format.

7. The disease retrieval device based on a knowledge graph-enhanced large language model according to claim 1, characterized in that, The device and storage module are used to provide the processor and memory devices for the specialized disease retrieval system. They can process readable instructions from the computer and execute question-and-answer retrieval functions in the specialized disease domain. They can also store triple datasets, supervised fine-tuning datasets, and specialized disease knowledge graph data.

8. A disease retrieval method based on a knowledge graph-enhanced large language model, characterized in that, The method is applied to the apparatus according to any one of claims 1-7, and the method includes: Data construction method: Obtain multi-hop question-answering dataset and perform data preprocessing; construct and store five-topic specialty disease knowledge graphs based on Neo4j; implement text generation and answer evaluation based on large language model and answer subgraph dataset, and construct supervised fine-tuning dataset; Subgraph retrieval method: Obtain the medical questions raised by users, and expand the input questions step by step to obtain sub-questions based on graph thinking chain reasoning; perform relation path retrieval and knowledge subgraph retrieval based on pre-trained models; Knowledge enhancement methods: fine-tuning a large language model to generate factual knowledge text based on sampled answer subgraphs; constructing a knowledge-enhanced retrieval model; Answer generation method: Based on the input question and factual knowledge, knowledge-enhanced prompts are constructed, and the final medical search answer is obtained through a large language model; the performance of the knowledge-enhanced retrieval model is measured according to evaluation metrics.

Images