Disease retrieval method and device based on knowledge graph enhanced large language model

Abstract

The invention discloses a disease retrieval method and device based on a knowledge graph enhanced large language model, and relates to the technical field of large language models and knowledge graph questions and answers. The method comprises the following steps: constructing a question and answer data set based on a triple form, and realizing storage and visualization of special disease knowledge maps of five themes based on Neo4j; acquiring a medical question asked by a user, and gradually expanding the input question based on thinking chain reasoning to obtain sub-questions; performing relation path retrieval and knowledge sub-graph retrieval based on the pre-training model; finely tuning the large language model to generate a fact knowledge text based on the sampled answer sub-graph; and constructing a knowledge enhancement prompt template in combination with the input question and the fact knowledge, and sending the knowledge enhancement prompt template into the large language model to obtain a final medical answer. According to the scheme, the knowledge graph is utilized to enhance the large language model, the large language model is guided to conduct reasoning and retrieval in the efficient knowledge path, and the special disease medical problem proposed by the user can be accurately and professionally answered.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models and knowledge graph question answering, and in particular to a disease retrieval method and device based on a knowledge graph-enhanced large language model. Background Art

[0002] Various specialized diseases present in a variety of forms and with diverse symptoms. Diagnosis relies on a physician's clinical experience and a range of examinations, resulting in a complex and cumbersome diagnostic process. Many medical institutions have already begun digitizing their clinical diagnosis and treatment processes, but due to varying development times and the use of different information technologies, data incompatibilities and inapplicable methods and processes are common. Therefore, simplifying, universalizing, informatizing, and intelligentizing the complex and cumbersome medical process is an effective approach to addressing the challenges of specialized disease diagnosis and treatment in my country.

[0003] Applying large language models to healthcare, particularly in the area of ​​specialized disease prevention and treatment, is a major innovation in current research on disease retrieval and question-answering systems. However, these systems are still limited by the following issues, hindering their direct application in intelligent clinical decision-making: 1. Large language models can produce illusory facts when answering specialized medical questions, and their accuracy and reliability are still insufficient; 2. Medical knowledge is rapidly evolving, and large language models are expensive to train. Updates to disease knowledge require timely knowledge revision and retraining, which is resource-intensive.

[0004] Therefore, for large language model retrieval and question answering of specialized diseases, the current common methods are to fine-tune the large language model or to build knowledge graphs through traditional pipeline methods to assist in specialized disease question answering. There is no reliable, accurate, and anthropomorphic intelligent system to provide professional specialized disease retrieval and question answering. Summary of the Invention

[0005] The purpose of the present invention is to provide a disease retrieval method and device based on a knowledge graph-enhanced large language model to solve the problem that existing large language models are unable to organize and utilize the latest professional disease knowledge and are prone to produce harmful hallucinations.

[0006] To solve the above technical problems, the embodiments of the present invention provide the following solutions:

[0007] A disease retrieval device based on a knowledge graph-enhanced large language model, comprising:

[0008] Data construction module: This module is used to preprocess the collected multi-hop question-answering dataset, construct subgraphs of the specialized disease knowledge graph for five topics, and obtain textual answers containing entity knowledge.

[0009] Subgraph retrieval module: This module is used to decompose the medical questions entered by users using graph thinking chains, construct a relational path reasoning model to sample triple knowledge, and retrieve the answer subgraph;

[0010] Knowledge Enhancement Module: This module is used to fine-tune the large language model and generate medical factual knowledge text based on the input knowledge subgraph related to the question to obtain a knowledge-enhanced retrieval model.

[0011] Answer generation module: This module combines medical facts and query questions into knowledge-enhanced prompts, and obtains the final medical answer through reasoning and retrieval in a large language model.

[0012] Equipment and storage module: used to process and store specialized disease knowledge graphs, and provide basic hardware support for the execution of the aforementioned module functions.

[0013] Furthermore, the data construction module is used to extract triple knowledge from the multi-hop question-answering dataset, and construct an answer subgraph dataset based on specific questions based on the triple knowledge of the subject entity, relationship path, answer entity and its subject entity's neighborhood; and construct a specialist disease knowledge graph subgraph based on different topics, wherein the specialist disease knowledge graph topic categories include: disease_epidemiology, disease_diagnosis, disease_disease, disease_treatment, disease_other.

[0014] Furthermore, the data construction module also includes:

[0015] The text generation and answer evaluation unit is used to generate text from the collected answer subgraph dataset through a large language model to obtain a supervised fine-tuning dataset based on a specific question;

[0016] The text generation and answer evaluation unit is specifically used to input the answer subgraph and prompt template combination into the large language model to obtain a factual knowledge answer based on a specific question; encapsulate the answer as a knowledge enhancement prompt and input it into the large language model again, and according to the reply information, the factual knowledge answer containing at least one answer entity is taken as the correct answer; integrate the answer subgraph and factual knowledge answer to divide the training, verification and test data sets as the supervised fine-tuning data sets of the knowledge enhancement module.

[0017] Furthermore, the subgraph retrieval module is used to gradually expand the problem according to the graph thinking chain reasoning based on the question input by the user, and decompose it into several jumping sub-problems. Each jumping sub-problem only contains a relationship triple knowledge based on the specialized disease knowledge graph to obtain step-by-step problem planning steps.

[0018] Furthermore, the subgraph retrieval module further includes:

[0019] The relational path reasoning unit is used to predict the relationship of the problem step by step and generate a reasoning path based on the specific problem according to the number of hops after the plan decomposition;

[0020] The relationship path reasoning unit is specifically used to construct a pre-trained language model and a linear classifier, embed the decomposed sub-problems and knowledge graph into a multi-dimensional vector, and predict the relationship categories in the specialized disease knowledge graph in each hop; select several most relevant relationships as the relationships of the sub-problem based on the predicted relationship probability distribution; repeatedly iterate during the next-hop relationship prediction reasoning, combine the new sub-problem vector and the currently predicted relationship path reasoning to obtain the latest hop relationship path until the number of hops of the planned decomposition is reached; calculate the relationship path score for each prediction, measure the gap between the output relationship path and the true triple, and repeatedly iterate and learn, using cross-entropy loss to optimize the parameters of the pre-trained language model; obtain a set of relationship reasoning paths in descending order of the relationship path scores, and construct an answer subgraph based on a specific question.

[0021] Furthermore, the knowledge enhancement module is specifically used to convert the retrieved answer subgraph into factual knowledge text suitable for large language model reasoning, and optimize and adjust the output of the large language model in a textual knowledge representation that contains relational reasoning paths; when fine-tuning the large language model training, each time step predicts the probability distribution of all words in the current time step based on the correct embedding sequence of the previous time step, and optimizes the parameters of the large language model with cross-entropy loss; finally, the knowledge enhancement module will enable the large language model after instruction fine-tuning to have the ability to generate graph text.

[0022] Furthermore, the answer generation module is used to encapsulate the factual knowledge text and medical questions obtained by reasoning with the large language model into a knowledge enhancement prompt template, construct a knowledge enhancement retrieval model, and obtain the final retrieval answer to the medical disease question.

[0023] Furthermore, the answer generation module further includes:

[0024] The model effect evaluation unit is used to measure the performance of the knowledge-enhanced retrieval model based on evaluation indicators, and at the same time compare and analyze the indicator evaluation effect of the large language model when directly performing retrieval and question answering without knowledge enhancement, the indicator evaluation effect when performing retrieval and question answering based on free-format text, and the indicator evaluation effect when performing retrieval and question answering based on factual knowledge text in triple format.

[0025] Furthermore, the device and storage module are used to provide a processor and memory device for a specialized disease retrieval system, which can respectively process computer-readable instructions and execute question-answering retrieval functions in the field of specialized diseases; store triple data sets, supervised fine-tuning data sets, and specialized disease knowledge graph data.

[0026] In addition, the present invention also provides a disease retrieval method based on a knowledge graph-enhanced large language model, which is applied to the above-mentioned device and includes:

[0027] Data construction method: We obtained a multi-hop question-answering dataset and performed data preprocessing. We then built a knowledge graph for specialized diseases on five topics based on Neo4j for storage and visualization. We also implemented text generation and answer evaluation based on a large language model and answer subgraph dataset, and constructed a supervised fine-tuning dataset.

[0028] Subgraph retrieval method: Obtain medical questions raised by users, and gradually expand the input questions to obtain sub-questions based on graph thinking chain reasoning; perform relationship path retrieval and knowledge subgraph retrieval based on pre-trained models;

[0029] Knowledge enhancement method: fine-tune the large language model to generate factual knowledge text based on the sampled answer subgraph; build a knowledge enhancement retrieval model;

[0030] Answer generation method: Construct knowledge-enhanced prompts based on input questions and factual knowledge, and obtain the final medical retrieval answer through a large language model; measure the performance of the knowledge-enhanced retrieval model based on evaluation indicators.

[0031] Optionally, the data construction method is used to extract triple knowledge from a multi-hop question-answering dataset, and construct an answer subgraph dataset based on specific questions based on the triple knowledge of the subject entity, relationship path, answer entity and its neighborhood triplet of the subject entity; and construct a specialized disease knowledge graph subgraph based on different topics.

[0032] Among them, the subject categories of the specialized disease knowledge graph include: disease_epidemiology, disease_diagnosis, disease_disease, disease_treatment, and disease_other.

[0033] The data construction method further includes: a text generation and answer evaluation method for generating text from the collected answer subgraph dataset using a large language model to obtain a supervised fine-tuning dataset based on a specific question;

[0034] Optionally, the text generation and answer evaluation method is further used to:

[0035] The answer subgraph and prompt template are combined and input into the large language model to obtain factual knowledge answers based on specific questions;

[0036] The answer is encapsulated as a knowledge-enhanced prompt and input into the large language model again. Based on the reply information, the factual knowledge answer containing at least one answer entity is regarded as the correct answer;

[0037] Integrate the answer subgraph and factual knowledge answers to divide the training, validation and test datasets as supervised fine-tuning datasets for the knowledge enhancement module.

[0038] Among them, the subgraph retrieval method is used to gradually expand the problem according to the graph thinking chain reasoning based on the question input by the user, and decompose it into several jumping sub-problems. Each jumping sub-problem only contains a relationship triple knowledge based on the specialized disease knowledge graph to obtain step-by-step problem planning steps.

[0039] The subgraph retrieval method further includes: a relational path reasoning method for performing step-by-step relational prediction on the problem and generating a relational reasoning path based on the specific problem according to the number of hops after the plan decomposition;

[0040] Optionally, the relationship path reasoning method is further used to:

[0041] Build a pre-trained language model and linear classifier, embed the decomposed sub-questions and knowledge graph into multi-dimensional vectors, and predict the relationship categories in the specialized disease knowledge graph at each hop;

[0042] According to the predicted probability distribution of relations, several most relevant relations are selected as the relations of this sub-problem;

[0043] During the next-hop relationship prediction reasoning, the algorithm repeatedly iterates and combines the new sub-problem vector and the currently predicted relationship path reasoning to obtain the latest one-hop relationship path until the number of hops required for the plan decomposition is reached.

[0044] Calculate the relationship path score for each prediction, measure the gap between the output relationship path and the true triple, and iterate learning repeatedly, using cross-entropy loss to optimize the parameters of the pre-trained language model;

[0045] Obtain a set of relational reasoning paths by sorting them in descending order of relational path scores, and construct an answer subgraph based on a specific question.

[0046] The knowledge enhancement method is used to convert the retrieved answer subgraph into factual knowledge text suitable for large language model reasoning, and optimize and adjust the output of the large language model in a textual knowledge representation form that contains relational reasoning paths;

[0047] When fine-tuning the large language model, at each time step, the probability distribution of all words in the current time step is predicted based on the correct embedding sequence of the previous time step, and the parameters of the large language model are optimized using cross-entropy loss.

[0048] Finally, the knowledge enhancement module will enable the large language model after fine-tuning the instructions to have the ability to generate graph text.

[0049] Among them, the answer generation method is used to encapsulate the factual knowledge text and medical questions obtained by large language model reasoning into a knowledge enhancement prompt template, construct a knowledge enhancement retrieval model, and obtain the final retrieval answer to medical disease questions.

[0050] Among them, the answer generation method also includes: a model effect evaluation method, which is used to measure the performance of the knowledge-enhanced retrieval model according to the evaluation indicators, and at the same time compare and analyze the indicator evaluation effect of the large language model directly performing retrieval and question answering under the condition of no knowledge enhancement, the indicator evaluation effect of performing retrieval and question answering based on free-format text, and the indicator evaluation effect of performing retrieval and question answering based on factual knowledge text in triple format.

[0051] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0052] This solution addresses the common issues with large language models, including their inability to organize and utilize the latest specialized disease knowledge and their tendency to generate harmful hallucinations. It provides a disease retrieval device that enhances large language models using knowledge graphs. By constructing a specialized disease domain knowledge graph, the large language model is optimized to efficiently capture domain expertise and generate reliable retrieval question and answer text. This provides essential system support for users seeking assisted retrieval and question answering for specialized diseases or learning specialized domain knowledge.

[0053] The question-and-answer data in the field of specialized diseases involved in this solution comes from an open-source multi-hop question-and-answer dataset. A knowledge graph in the field of specialized diseases based on five topics is constructed, and a large language model is used to generate text answers of factual knowledge based on triples. On this basis, a strategy based on graph thinking chain reasoning is used to guide the large language model to gradually expand and decompose the problem, and to perform relational path reasoning and prediction on the sub-problems based on the pre-trained language model. After sampling the triples, the answer subgraph can be obtained. By fine-tuning the large language model, it is enabled to have the ability to generate text answers of factual knowledge based on the knowledge graph, so as to obtain enhanced prompts associated with the retrieval answer subgraph, and provide auxiliary support for the professional knowledge base for question-and-answer retrieval in the field of specialized diseases. A knowledge-enhanced retrieval model of the large language model with knowledge graph enhancement is established, and answer generation evaluation is performed to achieve efficient and accurate response to knowledge in the field of specialized diseases.

[0054] This solution can effectively improve the accuracy and transparency of traditional large language models in question-and-answering in the professional medical field. While giving responses, it can also provide a complete disease knowledge reasoning path, offering an innovative technical route for specialized disease retrieval question-and-answering and professional medical knowledge learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 Schematic diagram of the structure of a disease retrieval device using a knowledge graph-enhanced large language model provided by an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the design of a knowledge graph model for specialized diseases provided by an embodiment of the present invention;

[0058] Figure 3 This is a partial schematic diagram of the visualization of the specialized disease knowledge graph provided by an embodiment of the present invention;

[0059] Figure 4 Schematic diagram of the principle of the subgraph retrieval model provided by an embodiment of the present invention;

[0060] Figure 5 : is a graph of loss values ​​and evaluation indicators of the relational path reasoning prediction model provided by an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the principle of the knowledge enhancement retrieval model provided by an embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram of a specialist disease search question and answer provided by an embodiment of the present invention;

[0063] Figure 8 This is a block diagram of a disease retrieval device using a knowledge graph-enhanced large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0065] First embodiment

[0066] In response to the widespread problems of current large medical language models, such as the inability to organize and utilize the latest professional disease knowledge and the susceptibility to harmful hallucinations, this embodiment provides a disease retrieval device based on a knowledge graph-enhanced large language model. The method can be implemented by an electronic device, which can be a terminal or a server.

[0067] This method is based on multi-hop question-answering data obtained from open source. After data preprocessing, triple knowledge is obtained, and a knowledge graph of specialized disease areas based on five topics is constructed based on Neo4j. Then, factual answer text is generated through a large language model, and quality assessment is performed based on whether the generated text contains at least one answer entity. Finally, factual knowledge answers based on triple knowledge are obtained, and training sets, validation sets, and test sets are divided as supervised fine-tuning datasets. For questions input by users, the questions are gradually expanded and planned in a graph thinking chain reasoning manner, and the large language model generates the reasoning steps for solving the questions and predicts the number of hops of the questions. Based on the construction of a pre-trained language model, potential relationship path reasoning and prediction are performed on sub-questions, and background knowledge paths associated with the subject entities are mined, so as to sample triples to obtain answer sub-graphs. The large language model is fine-tuned to enable it to have the ability to generate factual knowledge text based on the answer sub-graph. Through knowledge enhancement, the large language model is guided to answer the specialized disease questions input by users in a well-founded manner, which can realize intelligent and accurate retrieval and answering in the specialized disease area.

[0068] Based on the above, combined Figure 1 As shown, the method of this embodiment includes the following steps:

[0069] S1, perform data preprocessing on the multi-hop question answering dataset, construct knowledge graph subgraphs of five topics, and obtain textual answers containing entity knowledge.

[0070] Specifically, in this embodiment, the implementation process of the above S1 is as follows:

[0071] S11: Obtain a multi-hop question answering dataset and perform data preprocessing on it to obtain questions and corresponding triples.

[0072] It should be noted that the multi-hop question-answering dataset used in this embodiment is derived from the COKG dataset, part of the OpenKG (http: / / openkg.cn / ) open knowledge graph community, initiated and advocated by the Language and Knowledge Computing Committee of the Chinese Information Processing Society. The preprocessing steps include reading the dataset and using the split function to obtain the knowledge triples corresponding to the questions, thereby extracting the knowledge subgraph based on the corresponding questions. The set collection is then used to obtain a list of deduplicated relationships, which is stored in relation.txt.

[0073] S12, build a knowledge graph in the field of specialized diseases.

[0074] Among them, the specific construction process of the knowledge graph in the field of specialized diseases is as follows: ontology modeling, that is, using a top-down approach to model relationship categories, including the name of each relationship category, each relationship subcategory under the relationship category, the head entity type involved in the relationship subcategory, and the tail entity type; according to the relationship category, a knowledge graph based on different relationship subject categories can be constructed based on the Neo4j graph database to realize the visualization of triple knowledge.

[0075] Specifically, in this embodiment, the above S12 includes the following steps:

[0076] S121, based on the relations extracted from relation.txt, text clustering was performed to determine the subject word categories, and then the questions were divided into five categories of topics, namely "disease_epidemiology", "disease_diagnosis", "disease_disease", "disease_treatment" and "disease_others" through manual screening and induction.

[0077] Specifically, based on the COKG multi-hop question answering dataset, we first clustered the text using the traditional K-Means clustering algorithm to find the appropriate number of categories for question division. The final specific question topics and example descriptions are shown in Table 1:

[0078] Table 1. Topic classification and question examples

[0079]

[0080]

[0081] S122, based on the subject word categories and relationships, ontology modeling is performed, that is, entity categories, relationship categories, and relationship subcategories are defined. The schematic diagram of the knowledge graph model design is as follows: Figure 2 shown.

[0082] S123, based on the Neo4j graph database, builds a knowledge graph for specialized diseases to visualize domain knowledge. Figure 3 shown.

[0083] S13, construct a supervised fine-tuning dataset.

[0084] Among them, the specific construction process of the supervised fine-tuning dataset is to generate text from the collected answer subgraph dataset through a large language model (LLM), and then divide the dataset into training sets, test sets, and validation sets after text generation and answer evaluation, and finally obtain a supervised fine-tuning dataset based on specific problems.

[0085] Specifically, in this embodiment, the above S13 includes the following steps:

[0086] S131, text generation, combines the answer subgraph and prompt template into a large language model to obtain factual knowledge answers based on specific questions.

[0087] Specifically, the graph-to-text (KG-to-Text) prompt used to obtain factual knowledge answers through the large language model is shown in Table 2. KG2Text_Prompt must contain instructions and a knowledge subgraph.

[0088] Table 2 KG2Text_Prompt

[0089]

[0090] S132, answer evaluation, encapsulates the answer of S131 as a knowledge enhancement prompt and inputs it into the large language model again. According to the reply information, the factual knowledge answer containing at least one answer entity is regarded as the correct answer.

[0091] Specifically, the knowledge augmentation prompt Augmented_Prompt used for answer evaluation through the large language model is shown in Table 3. Among them, Augmented_Prompt must contain instructions (Instruction), input questions (Input), and factual knowledge (Fact).

[0092] Table 3 Augmented_Prompt

[0093]

[0094] S133, integrates the answer subgraph and factual knowledge answers to divide the training, validation and test datasets as the supervised fine-tuning dataset for the knowledge enhancement module.

[0095] S2, build subgraph retrieval model.

[0096] The specific process of building a subgraph retrieval model is as follows: decomposing and planning the complex input problem into several sub-problems through G-COT (Graph-Chain of Thoughts); predicting the relationship path through PLM (Pre-trained language model); and finally sampling triples based on the relationship path obtained by the predicted probability to retrieve the knowledge subgraph related to the specific problem. The schematic diagram of the subgraph retrieval model principle is shown in the figure. Figure 4 shown.

[0097] Specifically, in this embodiment, the implementation process of the above S2 is as follows:

[0098] S21, problem planning decomposition based on graph thinking chain.

[0099] It should be noted that, based on the question input by the user, LLM gradually expands the question with a graph thinking chain reasoning strategy and decomposes it into several jumping sub-problems. Each jumping sub-problem only contains a relational triplet knowledge based on the knowledge graph of specialized diseases to obtain step-by-step problem planning steps.

[0100] The problem decomposition and planning process uses the planning prompt example shown in Table 4. Planning_prompt must include instructions, examples, and input questions. The topic length obtained after problem decomposition and planning is the number of hops for a given problem.

[0101] Table 4 Problem Planning Prompt Description Planning_prompt

[0102]

[0103] S22, retrieval reasoning prediction based on relationship paths.

[0104] Among them, it should be noted that the constructed relational path prediction model PLM is an encoder based on BERT (Bidirectional Encoder Representations from Transformers). Through BERT and linear classifiers, the decomposed sub-problems and specialized disease knowledge graphs are embedded into multi-dimensional vectors to predict the relationship categories in the specialized disease knowledge graph in each hop. According to the predicted relationship probability distribution, several most relevant relationships are selected as the relationships of the sub-problem. During the next-hop relationship prediction reasoning, it is iterated repeatedly, and the new sub-problem vector and the currently predicted relationship path reasoning are combined to obtain the latest hop relationship path until the number of hops of the planned decomposition is reached. During the training process, the cross entropy loss value is calculated for the training set to continuously optimize the parameters of the model. In each training cycle e i The loss is then calculated on the validation set to monitor whether the model is overfitting. Finally, after the model training is completed, the overall effect of the relationship path retrieval and inference prediction model is evaluated based on the test set.

[0105] Specifically, the retrieval reasoning and prediction training process based on the relationship path is as follows:

[0106] The specific retrieval reasoning and prediction method based on relationship paths is:

[0107] enter:

[0108] The natural language question Q input by the user, the specialist disease knowledge graph KG, the size R of the relationship set in the graph, the problem hop count H, the number of candidate relationships N preset for each hop of the problem, and the number of iterations epoches = {e1, e2,..., e n}. Among them, n represents the maximum number of model training iterations.

[0109] Output:

[0110] The i-th relationship path p at the h-th hop h,i (1 ≤ h ≤ H ∧ i ≤ N < R), the score Score(p h,i ) of the i-th relationship path at the h-th hop, and the relationship path set p.

[0111] Step 1: Initialize the relationship path set p as an empty set

[0112] Step 2: Traverse the number of iterations e i , and perform the following process:

[0113] For each step h: that is, from 1 to H

[0114] If it is the first step: then

[0115] Construct a BERT encoder to embed the question Q into a vector V(Q), that is:

[0116] V(Q) = BERT(Q)

[0117] ?Based on a linear classifier, calculate the distribution D of all relationships r1, r2,..., r R in KG r,1 ;

[0118]

[0119]

[0120] Obtain N relationships r with the highest probabilities i (i = 1, 2..., N ∧ i ≤ R), and form the first-hop relationship path set p1, where the i-th relationship path p at the first hop 1,i is the relationship with the i-th highest relationship distribution probability;

[0121]

[0122] p1 = p 1,1 |p 1,2 |...|p 1,N

[0123] Add p1 to the relationship path set p;

[0124] p = {p1}

[0125] If it is steps 2 to H, then:

[0126] The relationship path between the problem Q and the previous h-1 steps p1~p h-1 Splicing, classification based on BERT, and calculation of the probability distribution D of the jump relationship in KG r,h ,Right now:

[0127] V(Q∨p1|p2|...|p h-1 )=BERT(Q∨p1|p2|...|p h-1 )

[0128]

[0129]

[0130] Get the N highest probability relationships r i (i=1,2...,N), forming the h-th hop relationship path set p h ;

[0131]

[0132] p h =p h,1 |p h,2 |...|p h,N

[0133] Update the relationship path set p to include all newly generated relationship paths;

[0134] p={p1,p2,...,p h}

[0135] Calculate until the iterative loop ends;

[0136] For each relation path p in P r :

[0137] Calculate p r The score of all relationship probabilities in , as p r The score of the h-th hop i-th relationship path is Score(p h,i );

[0138]

[0139] Calculate the training loss value L through the cross entropy Loss function and optimize the parameters;

[0140]

[0141] Step 3: Return the relationship path set p and all relationship path scores Score;

[0142] Step 4: Evaluate the model performance on the test set.

[0143] Specifically, the accuracy is used to evaluate the effectiveness of the above-mentioned relationship path-based retrieval reasoning prediction method. The accuracy (Precision) represents the ratio of the number of correctly predicted relationship paths (TP) to the total number of relationship paths in the prediction results (TP + FP) after the model prediction. It is calculated as follows:

[0144]

[0145] Specifically, the precision of the retrieval reasoning prediction model based on the relationship path on the test set is 87%. During the training process, the training and verification loss values ​​and accuracy of the model at each iteration are as follows: Figure 5 shown.

[0146] S23, knowledge subgraph retrieval based on triple sampling.

[0147] It should be noted that in each knowledge subgraph retrieval process, the relationship path set and all relationship path scores of the corresponding question will be obtained through the relationship path-based retrieval reasoning prediction algorithm, and then the reasoning path set will be obtained by sorting the relationship path scores of each hop in descending order, and finally a knowledge subgraph based on the specific question will be constructed.

[0148] S3, knowledge enhancement and answer generation.

[0149] The specific process of knowledge enhancement and answer generation is as follows: fine-tune the large language model so that it can generate medical factual knowledge text based on the knowledge subgraph related to the input question, so as to obtain a specialized disease knowledge graph text enhancement model; for the specific question input by the user, first convert the answer subgraph retrieved in S2 into knowledge triples, and then input it into the LLM through knowledge enhancement prompts for answer generation, and enhance the LLM's answer with the factual knowledge text fine-tuned by the LLM to reduce the generation of LLM domain hallucinations; evaluate the effect of graph text knowledge enhancement through Hit@1. The principle diagram of the knowledge enhancement retrieval model is shown in the figure below. Figure 6 shown.

[0150] Specifically, in this embodiment, the implementation process of the above S3 is as follows:

[0151] S31, graph-to-text generation based on fine-tuning of large models.

[0152] It should be noted that the large model fine-tuning process uses the Chinese-alpaca-2-7b framework, which is trained using Parameter-Efficient Fine-Tuning (PEFT). By constructing a graph-to-text prompt (KG2Text_Prompt), the knowledge subgraph is converted into text in the form of triples. Specifically, it is used to convert the retrieved knowledge subgraph into factual knowledge text suitable for large language model reasoning, and to optimize the output of the large language model using a text-based knowledge representation format that contains reasoning paths. When fine-tuning the large language model training, each time step predicts the probability distribution of all words in the current time step based on the correct embedding sequence of the previous time step, and the parameters of the large language model are optimized using cross-entropy loss.

[0153] Specifically, the training process of the graph-to-text generation model based on large model fine-tuning is as follows:

[0154] The specific graph-to-text generation algorithm based on large model fine-tuning is as follows:

[0155] enter:

[0156] Knowledge subgraph G: a set of triples TripleText, a supervised fine-tuning dataset based on KG-to-Text (including training sample set train_x, verification sample set valid_x, test sample set test_x), batch size batch_size, and number of iterations epoches = {e1, e2, ..., e n} (where n represents the maximum number of training iterations), PEFT configuration PeftConfig: used to specify the target module target_modules and hyperparameters for fine-tuning. The hyperparameters include: lora_rank: the rank of lora, that is, the size of the low-rank matrix used in the model parameterization process; lora_dropout: dropout rate, that is, the proportion of randomly discarded neurons applied to the lora layer during model training; lora_alpha: the learning rate scaling factor adapted by lora, where PeftConfig = {target_modules, lora_rank, lora_dropout, lora_alpha},

[0157] chinese-alpaca-2-7b: A 7-byte large language model for Chinese language understanding and generation tasks. Output:

[0158] Factual knowledge text Y: A natural language text sequence describing the knowledge subgraph G.

[0159] Step 1: Based on the KG-to-Text supervised fine-tuning dataset, fine-tune the large language model using the PEFT method. The pre-trained large language model LLM here pre-trained Using Chinese-alpaca-2-7b, LLM pre-trained And PEFT configuration PeftConfig are input together into PeftModel for fine-tuning to obtain the fine-tuned large language model LLM fine-tuned ;

[0160] LLM fine-tuned =PeftModel(LLM pre-trained ,PeftConfig)

[0161] The number of iterations e i , perform the following process:

[0162] Import text data of batch_size size in the training set;

[0163] For each training text, the real output sequence and the graph-to-text prompt KG2Text_Prompt are used to guide the LLM to generate the next correct output;

[0164] LLM fine-tuned .train(train_x)

[0165] The LLM parameters are updated by minimizing the cross entropy loss function.

[0166] End the iteration loop;

[0167] Step 2: Construct graph to text prompt; that is, construct KG2Text_Prompt prompt based on the input knowledge subgraph G

[0168] KG2Text_Prompt=Template.format(TripleText)

[0169] Step 3: Input the graph-to-text prompt into the fine-tuned chinese-alpaca-2-7b to implement KG-to-Text and return the factual knowledge text Y.

[0170] Y=LLM fine-tuned .generate(KG2Text_Prompt).

[0171] Specifically, the graph-to-text generation process based on large model fine-tuning uses the graph-to-text (KG-to-Text) prompt example KG2Text_Prompt shown in Table 2. The knowledge subgraph is the result of the subgraph retrieval in S2.

[0172] S32, disease question-answering retrieval based on knowledge enhancement.

[0173] It's important to note that the factual knowledge text generated using the large language model and the constructed knowledge graph is packaged with the user's medical question to form a knowledge-enhanced prompt template. This factual knowledge text serves as background knowledge related to the user's medical question, thereby improving the performance of the LLM. The knowledge-enhanced prompt is then fed into the LLM for question-and-answer processing, resulting in a final search answer to the medical question. This output answer is the knowledge-enhanced specialty disease search result.

[0174] Specifically, the knowledge augmentation-based disease question-answer retrieval process uses the knowledge augmentation prompt example Augmented_Prompt shown in Table 3. Here, the factual knowledge is the factual knowledge text output by KG-to-Text in S31.

[0175] S33, evaluate the effectiveness of knowledge-enhanced retrieval models.

[0176] It should be noted that the specific process for evaluating the effectiveness of the knowledge-augmented retrieval model involves conducting comparative evaluation experiments by measuring the performance of three retrieval question-answering methods: direct LLM retrieval without knowledge augmentation, LLM retrieval based on free-form text, and LLM retrieval based on triple-formatted factual knowledge text. The evaluation process also uses Hit@1 to specifically evaluate the effectiveness of the knowledge-augmented retrieval model. Hit@1 is used to assess whether the generated disease retrieval answer contains at least one answer entity from the knowledge triples relevant to the given question.

[0177] Specifically, the calculation formula of Hit@1 is:

[0178]

[0179] Specifically, |S| is the total number of queries of knowledge triples related to a given question, and rank i Represents the ranking of the correct answer to the i-th query in the prediction results, S represents the set of triples. II is the indicator function, if rank i If ≤1 is true (i.e. the correct answer entity ranks first), the function value is 1, otherwise it is 0.

[0180] Specifically, the specific evaluation results of the knowledge-enhanced retrieval model are shown in Table 5:

[0181] Table 5 Evaluation results of knowledge-enhanced retrieval model

[0182]

[0183] Second embodiment

[0184] Figure 8 This is a block diagram of a disease retrieval device for enhancing a large language model using a knowledge graph according to an exemplary embodiment. The device is used in a disease retrieval method for enhancing a large language model using a knowledge graph. Figure 8 The device includes a data construction module 810, a text generation and answer evaluation unit 820, a subgraph retrieval module 830, a relationship path reasoning unit 840, a knowledge enhancement module 850, an answer generation module 860, and a model effect evaluation unit 870. The specific methods implemented by the device include:

[0185] Data construction method: This is used to implement data construction module 810, obtain a multi-hop question-answering dataset, perform data preprocessing, and build a knowledge graph for specialized diseases on five topics based on Neo4j for storage and visualization. It also implements text generation and answer evaluation based on a large language model and answer subgraph dataset, and constructs a supervised fine-tuning dataset.

[0186] Subgraph retrieval method: used to implement the subgraph retrieval module 830, obtain the medical question raised by the user, and gradually expand the input question to obtain sub-questions based on graph thinking chain reasoning; perform relationship path retrieval and knowledge subgraph retrieval based on the pre-trained model;

[0187] Knowledge enhancement method: used to implement the knowledge enhancement module 850, fine-tune the large language model to generate factual knowledge text based on the sampled answer subgraph; and build a knowledge enhancement retrieval model;

[0188] Answer generation method: used to implement the answer generation module 860, construct knowledge enhancement prompts based on input questions and factual knowledge, and obtain the final medical retrieval answer through a large language model; measure the performance of the knowledge enhancement retrieval model according to evaluation indicators.

[0189] Among them, the data construction method is used to implement the data construction module 810, extract triple knowledge from the multi-hop question-answering dataset, and construct an answer subgraph dataset based on specific questions based on the neighborhood triple knowledge of the subject entity, relationship path, answer entity and its subject entity; and construct a specialized disease knowledge graph subgraph based on different topics.

[0190] Among them, the subject categories of the specialized disease knowledge graph include: disease_epidemiology, disease_diagnosis, disease_disease, disease_treatment, disease_others, etc.

[0191] The data construction method further includes: a text generation and answer evaluation method for implementing the text generation and answer evaluation unit 820, performing text generation on the collected answer subgraph dataset through a large language model to obtain a supervised fine-tuning dataset based on a specific question;

[0192] Optionally, the text generation and answer evaluation method is further used to:

[0193] The answer subgraph and prompt template are combined and input into the large language model to obtain factual knowledge answers based on specific questions;

[0194] The answer is encapsulated as a knowledge-enhanced prompt and input into the large language model again. Based on the reply information, the factual knowledge answer containing at least one answer entity is regarded as the correct answer;

[0195] Integrate the answer subgraph and factual knowledge answers to divide the training, validation and test datasets as supervised fine-tuning datasets for the knowledge enhancement module.

[0196] Among them, the subgraph retrieval method is used to implement the subgraph retrieval module 830. According to the question input by the user, the question is gradually expanded according to the graph thinking chain reasoning, and decomposed into several jumping sub-problems. Each jumping sub-problem only contains a relationship triple knowledge based on the specialized disease knowledge graph to obtain step-by-step problem planning steps.

[0197] The subgraph retrieval method further includes: a relational path reasoning method for implementing the relational path reasoning unit 840, which performs step-by-step relational prediction on the problem and generates a relational reasoning path based on the specific problem according to the number of hops after the plan decomposition;

[0198] Optionally, the relationship path reasoning method is further used to:

[0199] Build a pre-trained language model and linear classifier, embed the decomposed sub-questions and knowledge graph into multi-dimensional vectors, and predict the relationship categories in the specialized disease knowledge graph at each hop;

[0200] According to the predicted probability distribution of relations, several most relevant relations are selected as the relations of this sub-problem;

[0201] During the next-hop relationship prediction reasoning, the algorithm repeatedly iterates and combines the new sub-problem vector and the currently predicted relationship path reasoning to obtain the latest one-hop relationship path until the number of hops required for the plan decomposition is reached.

[0202] Calculate the relationship path score for each prediction, measure the gap between the output relationship path and the true triple, and iterate learning repeatedly, using cross-entropy loss to optimize the parameters of the pre-trained language model;

[0203] Obtain a set of relational reasoning paths by sorting them in descending order of relational path scores, and construct an answer subgraph based on a specific question.

[0204] The knowledge enhancement method is used to implement the knowledge enhancement module 850, which converts the retrieved answer subgraph into factual knowledge text suitable for large language model reasoning, and optimizes and adjusts the output of the large language model in a textual knowledge representation form that contains relational reasoning paths;

[0205] When fine-tuning the large language model, at each time step, the probability distribution of all words in the current time step is predicted based on the correct embedding sequence of the previous time step, and the parameters of the large language model are optimized using cross-entropy loss.

[0206] Finally, the knowledge enhancement module will enable the large language model after fine-tuning the instructions to have the ability to generate graph text.

[0207] Among them, the answer generation method is used to implement the answer generation module 860, encapsulate the factual knowledge text and medical questions obtained by large language model reasoning into a knowledge enhancement prompt template, construct a knowledge enhancement retrieval model, and obtain the final retrieval answer to the medical disease question.

[0208] Among them, the answer generation method also includes: a model effect evaluation method, which is used to implement the model effect evaluation unit 870, measure the performance of the knowledge enhancement retrieval model according to the evaluation index, and simultaneously compare and analyze the index evaluation effect of the large language model directly performing retrieval and answering questions without knowledge enhancement, the index evaluation effect of performing retrieval and answering questions based on free-format text, and the index evaluation effect of performing retrieval and answering questions based on factual knowledge text in triple format.

[0209] Third embodiment

[0210] This embodiment provides an electronic device, which includes a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment, or an execution system including the method described in the first embodiment.

[0211] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) and one or more memories, wherein the memory stores at least one instruction, which is loaded by the processor to execute the above method.

[0212] Fourth embodiment

[0213] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device. The instructions stored therein can be loaded by a processor in a terminal to execute the method.

[0214] Furthermore, it should be noted that the present invention may be provided as a method, apparatus, or computer program product. Thus, embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0215] It should also be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal device comprising the element.

[0216] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be noted that although the preferred embodiment of the present invention has been described, it is clear that those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered as within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A disease retrieval device based on knowledge graph enhanced large language model, characterized in that: include: Data construction module: used to preprocess the collected multi-hop question-answering dataset, construct subgraphs of the specialized disease knowledge graph for five topics, and obtain answers in text form containing entity knowledge; Subgraph retrieval module: It is used to decompose the medical questions input by users using graph thinking chains, construct a relational path reasoning model to sample triple knowledge, and retrieve the answer subgraph; Knowledge enhancement module: used to fine-tune the large language model and generate medical factual knowledge text based on the input knowledge subgraph related to the question to obtain a knowledge enhancement retrieval model; Answer generation module: used to combine medical factual knowledge and query questions into knowledge enhancement prompts, and obtain the final medical answer through reasoning and retrieval in a large language model; Equipment and storage module: used to process and store specialized disease knowledge graphs, and provide basic hardware support for the execution of the aforementioned module functions.

2. The disease retrieval device based on knowledge graph enhanced large language model according to claim 1 is characterized in that: The data construction module is used to extract triple knowledge from the multi-hop question-answering dataset, construct an answer subgraph dataset based on specific questions according to the triple knowledge of the subject entity, the relationship path, the answer entity and its subject entity's neighborhood; and construct a specialist disease knowledge graph subgraph based on different topics, wherein: The subject categories of the specialized disease knowledge graph include: disease_epidemiology, disease_diagnosis, disease_disease, disease_treatment, and disease_other.

3. The disease retrieval device based on knowledge graph enhanced large language model according to claim 2 is characterized in that: The data construction module also includes: The text generation and answer evaluation unit is used to generate text from the collected answer subgraph dataset through a large language model to obtain a supervised fine-tuning dataset based on a specific question; The text generation and answer evaluation unit is specifically used to input the answer subgraph and prompt template combination into the large language model to obtain a factual knowledge answer based on a specific question; encapsulate the answer as a knowledge enhancement prompt and input it into the large language model again, and according to the reply information, the factual knowledge answer containing at least one answer entity is taken as the correct answer; integrate the answer subgraph and the factual knowledge answer to divide the training, verification and test data sets as the supervised fine-tuning data sets of the knowledge enhancement module.

4. The disease retrieval device based on knowledge graph enhanced large language model according to claim 1, characterized in that: The subgraph retrieval module is used to gradually expand the problem according to the question input by the user based on the graph thinking chain reasoning, decompose it into several jumping sub-problems, each jumping sub-problem only contains a relationship triple knowledge based on the specialized disease knowledge graph, and obtain the step-by-step problem planning steps.

5. The disease retrieval device based on knowledge graph enhanced large language model according to claim 4 is characterized in that: The subgraph retrieval module further includes: The relational path reasoning unit is used to predict the relationship of the problem step by step according to the number of hops after the planning decomposition and generate a relational reasoning path based on a specific problem; wherein, The relationship path reasoning unit is specifically used to construct a pre-trained language model and a linear classifier, embed the decomposed sub-problems and knowledge graph into a multi-dimensional vector, and predict the relationship category in the specialist disease knowledge graph in each hop; select a number of most relevant relationships as the relationship of the sub-problem according to the predicted relationship probability distribution; repeatedly iterate during the next-hop relationship prediction reasoning, combine the new sub-problem vector and the currently predicted relationship path reasoning to obtain the latest hop relationship path until the number of hops of the planned decomposition is reached; calculate the relationship path score for each prediction, measure the gap between the output relationship path and the real triple, and repeatedly iterate and learn, and use the cross entropy loss to optimize the parameters of the pre-trained language model; obtain a relationship reasoning path set by sorting the relationship path score in descending order, and construct an answer subgraph based on a specific question.

6. The disease retrieval device based on knowledge graph enhanced large language model according to claim 1, characterized in that: The knowledge enhancement module is specifically used to convert the retrieved answer subgraph into factual knowledge text suitable for large language model reasoning, and optimize and adjust the output of the large language model in the form of textual knowledge representation that contains relational reasoning paths; when fine-tuning the large language model training, each time step predicts the probability distribution of all words in the current time step based on the correct embedding sequence of the previous time step, and optimizes the parameters of the large language model with cross entropy loss; finally, the knowledge enhancement module will enable the large language model after instruction fine-tuning to have the ability to generate graph text.

7. The disease retrieval device based on knowledge graph enhanced large language model according to claim 1 is characterized in that: The answer generation module is used to encapsulate the factual knowledge text and medical questions obtained by large language model reasoning into a knowledge enhancement prompt template, build a knowledge enhancement retrieval model, and obtain the final retrieval answer to the medical disease question.

8. The disease retrieval device based on knowledge graph enhanced large language model according to claim 7, characterized in that: The answer generation module also includes: The model effect evaluation unit is used to measure the performance of the knowledge-enhanced retrieval model according to the evaluation indicators, and to compare and analyze the indicator evaluation effect of the large language model in directly performing retrieval and question answering without knowledge enhancement, the indicator evaluation effect of performing retrieval and question answering based on free-format text, and the indicator evaluation effect of performing retrieval and question answering based on factual knowledge text in triple format.

9. The disease retrieval device based on knowledge graph enhanced large language model according to claim 1, characterized in that: The device and storage module are used to provide a processor and memory device for a specialized disease retrieval system, which can respectively process computer-readable instructions and execute question-answering retrieval functions in the field of specialized diseases; store triple data sets, supervised fine-tuning data sets, and specialized disease knowledge graph data.

10. A disease retrieval method based on knowledge graph enhanced large language model, characterized in that: The method is applied to the device according to any one of claims 1 to 9, and the method comprises: Data construction method: obtain multi-hop question-answering datasets and perform data preprocessing, build storage and visualization of specialized disease knowledge graphs for five topics based on Neo4j; implement text generation and answer evaluation based on large language models and answer subgraph datasets, and build supervised fine-tuning datasets; Subgraph retrieval method: obtain the medical questions raised by users, and gradually expand the input questions to obtain sub-questions based on graph thinking chain reasoning; perform relationship path retrieval and knowledge subgraph retrieval based on pre-trained models; Knowledge enhancement method: fine-tune the large language model to generate factual knowledge text based on the sampled answer subgraph; build a knowledge enhancement retrieval model; Answer generation method: construct knowledge-enhanced prompts based on input questions and factual knowledge, and obtain the final medical retrieval answers through a large language model; measure the performance of the knowledge-enhanced retrieval model based on evaluation indicators.

Images