A method, system, device and medium for enhancing medical large model answers

Abstract

The present invention relates to the field of large model technology, and discloses a method, system, device, and medium for enhancing answers to medical large models. The method uses a medical large model to perform medical named entity recognition on medical questions and obtained preliminary answer results; based on the obtained medical entity recognition results and preliminary answer results, a structured verification question set, an extended verification question set, and a counterfactual verification question set are generated; a first similarity result between the preliminary answer result and the verification answer results corresponding to the structured verification question set and the extended verification question set is identified, and a second similarity result between the preliminary answer result and the counterfactual verification answer results corresponding to the counterfactual verification question set is identified; a third similarity result between the preliminary answer result and an evidence file obtained from an evidence-based medicine database is identified; and the preliminary answer result is corrected according to the first similarity result, the second similarity result, and the third similarity result to obtain a target answer. The method of the present application enhances the credibility of the output of the medical large model.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a large medical model answer enhancement method, system, equipment and medium. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), intelligent online medical consultation technology based on medical big models has made significant progress and gained widespread application. Medical-specific big models such as GPT-4, Med-PaLM2, and Hua Tuo have emerged one after another, demonstrating significant potential to replace or assist doctors in making diagnosis and treatment decisions. These big models, deeply trained on massive amounts of medical text, including medical papers, clinical guidelines, and medical records, can rapidly generate medical recommendations during consultations, significantly improving medical efficiency and providing patients with more convenient access to medical consultations.

[0003] However, current medical big model technology still faces numerous challenges in practical application. For one thing, medical big models primarily operate based on probabilistic language modeling. Due to limitations in training data, their reasoning and factual consistency are weak. Consequently, in practical applications, simply using a big model to directly output answers can lead to responses that contradict actual medical facts. This can potentially mislead patients about their diagnosis and treatment, leading to potential medical risks.

[0004] It can be seen that how to improve the accuracy of the answers given by large medical models has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0005] The present invention provides a method and system for enhancing the answers of medical big models to solve the technical problem of providing a method that can improve the reasoning ability and answer credibility of medical big models, thereby improving the accuracy and credibility of the answers of medical big models and providing doctors and patients with accurate, reliable and quantifiable AI diagnosis and treatment support.

[0006] In a first aspect, the present invention provides a method for enhancing answers to large medical models, the method comprising:

[0007] Obtaining a preliminary answer result generated by the medical macromodel based on the medical question input by the user, and performing medical named entity recognition on the medical question and the preliminary answer result using the medical macromodel to obtain a medical entity recognition result;

[0008] Based on the medical entity recognition results, a structured verification question set is generated using pre-set verification question rules, and based on the medical entity recognition results and the preliminary answer results, a structured prompt method is used to guide the medical macro model to generate an extended verification question set and a counterfactual verification question set;

[0009] Inputting the structured verification question set and the extended verification question set into the medical macro model to generate verification answer results, and inputting the counterfactual verification question set into the medical macro model to generate counterfactual verification answer results;

[0010] identifying a first similarity result between the preliminary answer result and the verification answer result, and identifying a second similarity result between the preliminary answer result and the counterfactual verification answer result;

[0011] Obtaining an evidence document from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and identifying a third similarity result between the preliminary answer result and the evidence document;

[0012] The first similarity result, the second similarity result, and the third similarity result are analyzed, and the preliminary answer result is modified according to the analysis result to obtain a target answer.

[0013] Preferably, the medical named entity recognition is performed on the medical question and the preliminary answer using the medical big model to obtain a medical entity recognition result, including:

[0014] Using the medical big model to identify the diseases, symptoms, drugs, treatment plans and physiological indicators in the medical questions and the preliminary answers, to obtain a disease data set, a symptom data set, a drug data set, a treatment plan data set and a physiological indicator data set;

[0015] A medical entity recognition result is constructed based on the disease dataset, the symptom dataset, the drug dataset, the treatment plan dataset, and the physiological indicator dataset.

[0016] Preferably, the structured verification question set is generated based on the medical entity recognition result using pre-set verification question rules, including:

[0017] Based on the disease dataset, using pre-set disease verification question rules, generating a structured disease verification question;

[0018] Based on the symptom data set, a structured symptom verification question is generated using a pre-set symptom verification question rule;

[0019] Based on the drug dataset, a structured drug verification question is generated using pre-set drug verification problem rules;

[0020] Based on the treatment plan data set, a structured treatment plan verification question is generated using pre-set treatment plan verification problem rules;

[0021] Based on the physiological indicator data set, a structured physiological indicator verification question is generated using a pre-set physiological indicator verification question rule;

[0022] A structured verification question set is constructed based on the structured disease verification question, the structured symptom verification question, the structured drug verification question, the structured treatment plan verification question and the structured physiological indicator verification question.

[0023] Preferably, based on the medical entity recognition results and the preliminary answer results, a structured questioning method is used to guide the medical macro model to generate an extended verification question set and a counterfactual verification question set, including:

[0024] Based on the medical entity recognition result and the preliminary answer result, an extended verification question task is generated using a first structured question template, wherein the first structured question template is used to enable the medical macro model to generate extended verification questions to supplement medical knowledge blind spots not covered in the preliminary answer result;

[0025] Based on the medical entity recognition result and the preliminary answer result, a second structured question template is used to generate a counterfactual verification question task, wherein the second structured question template is used to enable the medical macro model to generate a counterfactual verification question to refute the preliminary answer result;

[0026] Inputting the extended verification question task into the medical macro model to generate the extended verification question set;

[0027] The counterfactual verification question task is input into the medical big model to generate the counterfactual verification question set.

[0028] Preferably, the identifying a first similarity result between the preliminary answer result and the verification answer result, and the identifying a second similarity result between the preliminary answer result and the counterfactual verification answer result, comprises:

[0029] Calculating a first semantic similarity between the preliminary answer result and the verification answer result, and obtaining a first similarity result between the preliminary answer result and the verification answer result based on the first semantic similarity;

[0030] A second semantic similarity between the preliminary answer result and the counterfactual verification answer result is calculated, and based on the second semantic similarity, a second similarity result between the preliminary answer result and the counterfactual verification answer result is obtained.

[0031] Preferably, obtaining evidence documents from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and calculating a third similarity result between the preliminary answer result and the evidence document, comprises:

[0032] generating a Boolean query structure based on the medical entity recognition result and a preset Boolean query template;

[0033] According to the Boolean query structure, dense paragraph retrieval technology is used to obtain retrieval results from the evidence-based medicine database;

[0034] sorting the search results using a text matching sorting method to obtain evidence documents;

[0035] A third semantic similarity between the preliminary answer result and the evidence document is calculated, and based on the third semantic similarity, a third similarity result between the preliminary answer result and the evidence document is obtained.

[0036] Preferably, analyzing the first similarity result, the second similarity result, and the third similarity result, and optimizing the preliminary answer result according to the analysis result to obtain a target answer includes:

[0037] Calculating and analyzing the first similarity result, the second similarity result, and the third similarity result to obtain a credibility score;

[0038] determining a relationship between the credibility score and a first threshold and a second threshold, and when the credibility score is greater than the first threshold, directly outputting the preliminary answer result as the target answer;

[0039] When the credibility score is less than or equal to the first threshold and greater than the second threshold, generating an uncertainty description, and outputting the preliminary answer result and the uncertainty description as the target answer;

[0040] When the credibility score is less than or equal to the second threshold, a supplementary explanation is generated, and the supplementary explanation and the medical question are re-input into the medical model until the credibility score of the regenerated preliminary answer result is greater than the second threshold, and the target answer is obtained.

[0041] In a second aspect, the present invention further provides a medical large model answer enhancement system, which implements the above-mentioned medical large model answer enhancement method, the system comprising: a preliminary answer result recognition unit, a verification question generation unit, a verification answer generation unit, a first recognition unit, a second recognition unit, and an answer correction unit;

[0042] The preliminary answer result recognition unit is used to obtain the preliminary answer result generated by the medical macro model based on the medical question input by the user, and use the medical macro model to perform medical named entity recognition on the medical question and the preliminary answer result to obtain a medical entity recognition result;

[0043] The verification question generating unit is configured to generate a structured verification question set based on the medical entity recognition result and a preset verification question rule, and to guide the medical macro model to generate an extended verification question set and a counterfactual verification question set based on the medical entity recognition result and the preliminary answer result using a structured prompting method;

[0044] The verification answer generation unit is configured to input the structured verification question set and the extended verification question set into the medical macro model to generate a verification answer result, and input the counterfactual verification question set into the medical macro model to generate a counterfactual verification answer result;

[0045] The first identification unit is configured to identify a first similarity result between the preliminary answer result and the verification answer result, and to identify a second similarity result between the preliminary answer result and the counterfactual verification answer result;

[0046] The second recognition unit is configured to obtain an evidence document from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and to identify a third similarity result between the preliminary answer result and the evidence document;

[0047] The answer correction unit is configured to analyze the first similarity result, the second similarity result, and the third similarity result, and correct the preliminary answer result according to the analysis result to obtain a target answer.

[0048] In a third aspect, the present invention also provides a computer device, which includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to execute the above-mentioned medical large model answer enhancement method.

[0049] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the above-mentioned medical large model answer enhancement method is implemented.

[0050] This application provides a method, system, device, and medium for enhancing answers to large medical models. Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0051] The medical big model answer enhancement method, system, device and medium disclosed in this application generate self-verification questions and counterfactual verification questions to perform medical fact consistency verification and counterfactual cross-validation, thereby ensuring that the logic of the answers generated by the medical big model is stable and self-consistent, avoiding low-level errors, and through evidence-based medicine verification and retrieval of the latest medical literature, ensuring that the answers generated by the medical big model are based on the latest authoritative data, thereby enhancing the credibility of the output of the medical big model and providing doctors and patients with accurate, reliable and quantifiable AI diagnosis and treatment support. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the steps of a method for enhancing a large medical model answer provided by a preferred embodiment of the present invention;

[0053] Figure 2 It is a structural diagram of a medical large model answer enhancement system provided by a preferred embodiment of the present invention;

[0054] Figure 3 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following is a detailed explanation of the embodiments of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limitations on the present invention. The accompanying drawings are for reference and illustration purposes only and do not constitute a limitation on the scope of patent protection of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0056] In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the two components. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for illustrative purposes only, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] In describing the present invention, it should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Those skilled in the art will understand the specific meanings of the above terms in the present invention in specific circumstances.

[0058] See also Figure 1 In an embodiment of the present invention, a method for enhancing a medical model answer is provided, the method comprising:

[0059] S1. Obtain the preliminary answer result generated by the medical model based on the medical question input by the user, and use the medical model to perform medical named entity recognition on the medical question and the preliminary answer result to obtain the medical entity recognition result; the user enters the medical question into the medical model. Input the medical model, and the medical model will respond to the input questions. Output the corresponding preliminary answer results , the preliminary answer results generated by the medical big model Relying on its own training data generation, due to limited training data and outdated knowledge, it cannot dynamically integrate and update the latest medical research and guidelines. There may be answers that are inconsistent with medical facts. The generated answers lack a credibility quantitative analysis mechanism. It is difficult for doctors and patients to judge whether the answers are scientific and credible. In view of this, it is necessary to conduct credibility quantitative analysis on the preliminary answer results. Before conducting credibility quantitative analysis, a medical big model is used to perform medical named entity recognition on the question and preliminary answer results to obtain medical entity recognition results. In this application, medical named entities include at least diseases ,symptom ,drug , treatment plan and physiological indicators The specific process of medical named entity recognition is: using a large medical model to identify diseases, symptoms, drugs, treatment plans and physiological indicators in the medical questions and preliminary answers. Specifically, a question task is generated based on the medical questions and preliminary answers. For example, if you are an experienced medical expert, please extract the medical named entities about diseases, symptoms, drugs, treatment plans and physiological indicators in the "medical questions" and "preliminary answers" to obtain disease datasets, symptom datasets, drug datasets, treatment plan datasets and physiological indicator datasets. The disease dataset is represented as:

[0060] D:

[0061] The symptom dataset is represented as:

[0062] S:

[0063] The drug dataset is represented as:

[0064] M:

[0065] The treatment plan dataset is represented as:

[0066] T:

[0067] The physiological indicator dataset is represented as:

[0068] B:

[0069] Based on the disease dataset, symptom dataset, drug dataset, treatment plan dataset and physiological indicator dataset, the medical entity recognition result is constructed. The medical entity recognition result is expressed as:

[0070]

[0071] S2. Based on the medical entity recognition result, a structured verification question is generated using a pre-set rule, and based on the medical entity recognition result and the preliminary answer result, a structured prompt method is used to guide the medical large model to generate an extended verification question and a counterfactual verification question; in an embodiment of the present application, different methods are used to generate structured verification questions, extended verification questions and counterfactual verification questions. For the generation of structured verification questions, a medical answer template is matched and a pre-set verification question rule is used to generate them. Specifically: based on a disease data set, a structured disease verification question is generated using a pre-set disease verification question rule; based on a symptom data set, a structured symptom verification question is generated using a pre-set symptom verification question rule; based on a drug data set, a structured drug verification question is generated using a pre-set drug verification question rule; based on a treatment plan data set, a structured treatment plan verification question is generated using a pre-set treatment plan verification question rule; based on a physiological indicator data set, a structured physiological indicator verification question is generated using a pre-set physiological indicator verification question rule; and a structured verification question is constructed based on the structured disease verification question, the structured symptom verification question, the structured drug verification question, the structured treatment plan verification question and the structured physiological indicator verification question.

[0072] Here is an example:

[0073] The user enters a medical question, Q: "Can a diabetic patient with stomach pain take ibuprofen?" The medical model generates a preliminary answer, R: "Ibuprofen may affect blood sugar levels and should be used with caution. A low-carb diet is recommended."

[0074] =

[0075] Based on the medical entity recognition results, the pre-set verification question rules are used to generate the results of structured verification questions as shown in Table 1.

[0076] Table 1

[0077]

[0078] Finally, a structured verification problem set is generated:

[0079]

[0080] in, is the number of structured verification questions.

[0081] An extended verification question set and a counterfactual verification question set are used to guide the generation of a medical big model based on the medical entity recognition results and preliminary answer results. Specifically, based on the medical entity recognition results and preliminary answer results, a first structured question template is used to generate an extended verification question task. The first structured question template is used to enable the medical big model to generate extended verification questions to supplement the medical knowledge blind spots not covered in the preliminary answer results; based on the medical entity recognition results and preliminary answer results, a second structured question template is used to generate a counterfactual verification question task. The second structured question template is used to enable the medical big model to generate counterfactual verification questions to refute the preliminary answer results; the extended verification question task is input into the medical big model to generate an extended verification question set; the counterfactual verification question task is input into the medical big model to generate a counterfactual verification question set.

[0082] Specifically, Structured Prompting is used to guide the large medical model to generate different types of extended verification questions. Few-shot Learning is used to provide structured question templates to improve the rationality and coverage of the large medical model in generating extended verification questions and counterfactual verification questions. The structured question template can also be dynamically adjusted. In this application, the structured question template corresponding to the extended verification question set is the first structured question template. An example of the first structured question template is as follows:

[0083] You are a senior medical expert responsible for optimizing an AI-generated medical consultation system. Your task is to generate 5-10 extended verification questions based on the AI-generated preliminary answer R and the medical entity recognition results E(Q,R). This will address potential medical knowledge blind spots in the AI's preliminary answer R, optimize the quality of the medical consultation, and ensure that the extended verification questions cover the following multiple levels:

[0084] 1. Knowledge supplement: Generate questions to fill blind spots in medical knowledge.

[0085] 2. Medical Mechanism Exploration: Generate questions to delve deeper into the physiological mechanisms of diseases or drugs.

[0086] 3. Disease Prevention and Management: Generate questions about disease management and preventive measures.

[0087] 4. Medical Guideline Comparison: Generate questions to compare the recommendations of different authoritative medical guidelines.

[0088] For example, for the user-entered medical question Q: "Can a diabetic patient with stomach pain take ibuprofen?", the medical model generates a preliminary answer R: "Ibuprofen may affect blood sugar levels and should be used with caution. A low-carb diet is recommended." The expanded validation question set generated by the medical model is:

[0089] 1. Knowledge Supplement: "What medications should diabetics avoid in their daily lives?"

[0090] 2. Medical Mechanism Exploration: "How does ibuprofen affect insulin sensitivity?"

[0091] 3. Disease Prevention and Management: "How should diabetic patients monitor their blood sugar levels when taking nonsteroidal anti-inflammatory drugs (NSAIDs)?"

[0092] 4. Comparison of medical guidelines: "Are there differences in the recommendations of different medical associations (ADA, WHO) regarding the use of NSAIDs in patients with diabetes?"

[0093] 5. Knowledge Supplement: "What alternative medicines can be used for pain management in patients with diabetes?"

[0094] The extended validation problem set generated by the medical large model is expressed as:

[0095]

[0096] in, The number of extended verification questions.

[0097] Merge the structured verification problem set and the extended verification problem set to obtain the verification problem set:

[0098]

[0099] By asking additional high-value verification questions, we ensure that the verification questions are comprehensive, scientifically reliable, and have medical logic, so that the answers generated by the medical big model have more medical depth and reliability.

[0100] The structured question template corresponding to the counterfactual verification question set is the second structured question template. An example of the second structured question template is as follows:

[0101] You are an experienced medical expert tasked with verifying the reliability of AI-generated answers to medical questions. Your task is to construct 3-5 high-quality counterfactual verification questions based on the AI-generated preliminary answer R and the medical entity recognition result E(Q,R). Methods for generating counterfactual verification questions include:

[0102] 1. Disease variants: changes in disease status, such as comorbidities or disease progression.

[0103] 2. Treatment variants: changing the treatment regimen or drug combination, taking into account interactions.

[0104] 3. Dosage variation: changing the dosage or duration of medication.

[0105] 4. Population variants: Modify patient demographic characteristics such as age, gender, and race.

[0106] 5. Temporal variation: Consider short-term vs. long-term effects, or effects after treatment has stopped.

[0107] The second structured question template enables the medical model to generate counterfactual verification questions to refute the initial answer. For example, for the user-input question Q: "Can a diabetic patient with stomach pain take ibuprofen?" the medical model generates the initial answer R: "Ibuprofen may affect blood sugar levels and should be used with caution. A low-carb diet is recommended." The expanded verification question set generated by the medical model is:

[0108] 1. Condition variant: "Is ibuprofen use riskier if the patient has chronic kidney disease?"

[0109] 2. Treatment variant: "If the patient is already taking metformin, will ibuprofen cause a drug interaction?"

[0110] 3. Dosage variation: “Will the blood sugar effect be more pronounced if patients take 800 mg of ibuprofen daily instead of 200 mg?”

[0111] 4. Population variant: "Is ibuprofen more dangerous for diabetic patients over 65 years of age than for younger patients?"

[0112] 5. Time variant: "Which situation has a greater impact on blood sugar, short-term use of ibuprofen vs. long-term use?"

[0113] The set of counterfactual verification problems is expressed as:

[0114]

[0115] in, The number of extended verification questions.

[0116] S3. Input the structured verification question set and the extended verification question set into the medical big model to generate a verification answer result, and input the counterfactual verification question set into the medical big model to generate a counterfactual verification answer result; in terms of the credibility assessment of the answer result, there is currently a lack of a scientific and effective quantitative mechanism to judge the credibility of the answer given by the medical big model. This makes it difficult for doctors to judge the scientific nature of the model recommendations when referring to them, and patients are also unable to determine the reliability of the answers provided by the model. This has become an important factor hindering the widespread and in-depth application of the medical big model in the medical field where decision-making accuracy and reliability are extremely high. Therefore, in this application, in order to ensure that the final answer generated by the medical big model is logical and avoid logical self-contradiction, the credibility of the preliminary answer results is verified from multiple angles, among which semantic consistency verification is performed using the verification answer results generated based on the structured verification question set and the extended verification question set, and cross-validation is performed using the counterfactual verification answer results generated based on the counterfactual verification question set. Before conducting credibility verification, it is necessary to input the structured verification question set and the extended verification question set into the medical big model to generate the verification answer results, and to input the counterfactual verification question set into the medical big model to generate the counterfactual verification answer results.

[0117] The verification answer result is expressed as:

[0118]

[0119] The counterfactual verification answer result is expressed as:

[0120]

[0121] S4. Identify a first similarity result between the preliminary answer result and the verification answer result, and identify a second similarity result between the preliminary answer result and the counterfactual verification answer result; first, perform consistency verification through the similarity between the preliminary answer question and the verification answer question.

[0122] Calculate the initial answer result and the verification answer result for each verification answer The first semantic similarity, in the preferred embodiment of the present application, the first semantic similarity is calculated using cosine similarity, and the calculation formula is:

[0123]

[0124] in, Indicates vectorization of text, and Cosine represents the cosine similarity calculation formula:

[0125]

[0126] Furthermore, the overall semantic consistency score of the preliminary answer result and the verification answer result is calculated using the following formula:

[0127]

[0128] in, Represents the overall semantic consistency score between the preliminary answer result and the verification answer result, .

[0129] The higher the overall semantic consistency score between the preliminary answer result and the verification answer result, the more consistent the preliminary answer result and the verification answer result are. The overall semantic consistency score between the preliminary answer result and the verification answer result is used as the first similarity score.

[0130] Calculate the initial answer result and the counterfactual verification answer result for each counterfactual verification answer The second semantic similarity, in the preferred embodiment of the present application, the second semantic similarity is also calculated using cosine similarity, and the calculation formula is:

[0131]

[0132] Furthermore, the overall semantic consistency score of the preliminary answer result and the counterfactual verification answer result is calculated using the following formula:

[0133]

[0134] in, Represents the overall semantic consistency score between the initial answer result and the counterfactual verification answer result, .

[0135] The higher the overall semantic consistency score between the preliminary answer result and the counterfactual verification answer result, the more self-consistent the preliminary answer result and the counterfactual verification answer result are, and medical hallucinations and low-level errors can be avoided. The overall semantic consistency score between the preliminary answer result and the counterfactual verification answer result is used as the second similarity score.

[0136] S5. According to the Boolean query structure generated based on the medical entity recognition result, evidence documents are obtained from the evidence-based medicine database, and the third similarity result between the preliminary answer result and the evidence document is identified; medical knowledge is updated and iterated rapidly, and new research results and clinical guidelines are constantly emerging, but the existing large medical models are difficult to dynamically and timely integrate this latest information, resulting in the knowledge they provide being outdated and unable to meet the needs of clinical practice for the latest medical knowledge.

[0137] In order to verify whether the preliminary answer results output by the medical big model can be supported by authoritative medical evidence, this application obtains the latest and authoritative evidence documents from the evidence-based medicine database to ensure that the preliminary answer results generated by the medical big model are supported by solid medical evidence.

[0138] Commonly used evidence-based medicine databases include PubMed, Cochrane, and ClinicalTrials.gov. Their contents mainly include: systematic reviews, such as Cochrane SR (Cochrane Database of Systematic Reviews), the UK National Institute for Health and Care Excellence (NICE Guidelines), and the World Health Organization Guidelines (WHO Guidelines); randomized controlled trials, such as PubMed Clinical Trials and ClinicalTrials.gov; observational studies, such as PubMed, Embase, and top medical journals; expert consensus, such as guidelines issued by organizations such as the American Diabetes Association (ADA), the European Society of Cardiology (ESC), the American Heart Association (AHA), and the Chinese Medical Association.

[0139] The retrieval of evidence documents from the evidence-based medicine database mainly includes the following contents: determining the most relevant medical query keywords, retrieving relevant evidence documents from the evidence-based medicine database, and in this application, generating an efficient Boolean query structure based on the medical entity recognition results and the preset Boolean query template to improve the retrieval accuracy.

[0140] Boolean queries use Boolean logical operators to connect key entities in the query, ensuring that the query covers all important concepts. This application flexibly adjusts Boolean query templates based on the needs of different questions. The following are some examples of Boolean query templates:

[0141] 1. Drugs affecting diseases or symptoms: ("{drug name (M)}" AND "{disease name (D)}") OR ("{drug name (M)}" AND "{symptom name (D)}").

[0142] 2. The relationship between disease and physiological indicators: "{disease name (D)}" AND "{physiological indicator name (B)}".

[0143] 3. Combination of disease and treatment plan: ("{disease name (M)}" AND "{treatment plan name (T)}") OR ("{disease name D)}" AND "{drug name (M)}").

[0144] The corresponding entities in the medical entity recognition results are substituted into the Boolean query template to generate a Boolean query structure. In order to improve the matching degree of the retrieval results, dense paragraph retrieval technology is used to obtain retrieval results from the evidence-based medicine database based on the Boolean query structure.

[0145] Dense paragraph retrieval technology utilizes deep learning models to learn dense vector representations of text, effectively capturing the text's semantic information and improving retrieval accuracy and efficiency. Deep learning models typically consist of an encoder and a retriever. First, the encoder encodes a text paragraph into a low-dimensional dense vector. During this process, the deep learning model learns the semantic features of the text and maps the semantic information in the text into a vector space. The retriever then uses these dense vectors to perform similarity calculations to find the paragraphs most relevant to the query. Common similarity calculation methods include cosine similarity and Euclidean distance. During training, using large-scale text data and corresponding query-paragraph pairs as training examples, the model learns how to better map text into the vector space, placing semantically similar texts closer together in the vector space, thereby achieving accurate retrieval. Compared to traditional retrieval methods, dense paragraph retrieval technology can better understand the semantic content of text and perform better when handling complex semantic queries, synonym substitution, and semantic entailment. It can return search results that better meet user needs, improving retrieval precision and recall.

[0146] Furthermore, a text matching sorting method is used to sort the documents in the search results to obtain evidence files. The text matching sorting methods used in this application include BM25 and TF-IDF. Taking BM25 as an example, the BM25 sorting formula is:

[0147]

[0148] in, Indicates that it contains keywords The number of documents, Represents keywords in the document The number of occurrences of Indicates the total number of documents, Indicates the length of the document. represents the average length of all documents, is a tuning parameter.

[0149] After sorting by BM25, the k most matching documents are selected to obtain the evidence file, which is expressed as:

[0150]

[0151] in, Indicates the Articles.

[0152] Furthermore, the third semantic similarity between the preliminary answer result and the evidence document is calculated, and then the overall semantic consistency score is calculated. The overall semantic consistency score between the preliminary answer result and the evidence document is used as the third similarity score. In this application, the calculation formula of the third similarity score is:

[0153]

[0154] in, , is the third similarity score. The higher the score, the more sufficient the medical evidence is. To obtain the third semantic similarity between the preliminary answer result and the evidence document, the semantic matching based on the pre-trained model BERT or the cosine similarity can be used.

[0155] Evidence-based medicine emphasizes making medical decisions based on the latest and authoritative clinical research evidence, rather than relying solely on expert opinions or personal experience. This ensures that the answers to medical questions generated by the medical big model are supported by evidence-based medicine, thereby improving the safety and credibility of the medical big model's question-and-answer results.

[0156] S6. Analyze the first similarity result, the second similarity result, and the third similarity result, and modify the preliminary answer result according to the analysis result to obtain a target answer; perform weighted summation on the first similarity result, the second similarity result, and the third similarity result to obtain a credibility score, calculated as follows:

[0157]

[0158] in, is the weight parameter, , The higher the value, the more sufficient the evidence supporting the preliminary answer is and the more credible it is.

[0159] In this application, a first threshold and a second threshold are set, and preliminary answer results with different credibility scores are processed differently. When the credibility score is greater than the first threshold, it indicates that the credibility of the preliminary answer result is high, and the preliminary answer result can be directly output as the target answer; when the credibility score is less than or equal to the first threshold and greater than the second threshold, it indicates that the preliminary answer result has a certain degree of uncertainty, and an uncertainty explanation is generated, and the preliminary answer result and the uncertainty explanation are output as the target answer; when the credibility score is less than or equal to the second threshold, it indicates that the credibility of the preliminary answer result is low, and a supplementary explanation is generated, and the supplementary explanation and the medical question are re-input into the medical model, and the answer is regenerated until the credibility score of the regenerated preliminary answer is greater than the second threshold, and the target answer is obtained.

[0160] This application uses the first threshold as 0.85 and the second threshold as 0.7 as an example, and the target output of the medical large model is shown in Table 2.

[0161] Table 2

[0162]

[0163] In a preferred embodiment of the present invention, a preliminary answer result generated by the medical big model based on the medical question input by the user is obtained, and the medical big model is used to perform medical named entity recognition on the medical question and the preliminary answer result to obtain a medical entity recognition result; based on the medical entity recognition result, a structured verification question set is generated using a pre-set verification question rule, and based on the medical entity recognition result and the preliminary answer result, a structured prompt method is used to guide the medical big model to generate an extended verification question set and a counterfactual verification question set; the structured verification question set and the extended verification question set are input into the medical big model to generate a verification answer result, and the counterfactual verification question set is input into the medical big model to generate a counterfactual verification answer result; a first similarity result between the preliminary answer result and the verification answer result is identified, and a second similarity result between the preliminary answer result and the counterfactual verification answer result is identified; according to the Boolean query structure generated based on the medical entity recognition result, an evidence file is obtained from the evidence-based medicine database, and a third similarity result between the preliminary answer result and the evidence file is identified; the first similarity result, the second similarity result and the third similarity result are analyzed, and the preliminary answer result is corrected according to the analysis result to obtain a target answer. The medical big model answer enhancement method disclosed in this application generates self-verification questions and counterfactual verification questions, performs medical fact consistency verification and counterfactual cross-validation, ensures that the logic of the answers generated by the medical big model is stable and self-consistent, avoids low-level errors, and through evidence-based medicine verification and retrieval of the latest medical literature, ensures that the answers generated by the medical big model are based on the latest authoritative data, enhances the credibility of the output of the medical big model, and provides doctors and patients with accurate, reliable, and quantifiable AI diagnosis and treatment support.

[0164] Accordingly, if Figure 2 As shown, based on a medical large model answer enhancement method, an embodiment of the present invention further provides a medical large model answer enhancement system, which implements the medical large model answer enhancement method disclosed in the embodiment of the present invention, including: a preliminary answer result recognition unit 1, a verification question generation unit 2, a verification answer generation unit 3, a first recognition unit 4, a second recognition unit 5 and an answer correction unit 6;

[0165] The preliminary answer result recognition unit 1 is used to obtain the preliminary answer result generated by the medical macro model based on the medical question input by the user, and use the medical macro model to perform medical named entity recognition on the medical question and the preliminary answer result to obtain a medical entity recognition result;

[0166] The verification question generating unit 2 is configured to generate a structured verification question set based on the medical entity recognition result and a preset verification question rule, and to guide the medical macro model to generate an extended verification question set and a counterfactual verification question set based on the medical entity recognition result and the preliminary answer result using a structured prompting method;

[0167] The verification answer generation unit 3 is used to input the structured verification question set and the extended verification question set into the medical macro model to generate verification answer results, and input the counterfactual verification question set into the medical macro model to generate counterfactual verification answer results;

[0168] The first identification unit 4 is configured to identify a first similarity result between the preliminary answer result and the verification answer result, and to identify a second similarity result between the preliminary answer result and the counterfactual verification answer result;

[0169] The second recognition unit 5 is configured to obtain an evidence document from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and to identify a third similarity result between the preliminary answer result and the evidence document;

[0170] The answer correction unit 6 is configured to analyze the first similarity result, the second similarity result, and the third similarity result, and correct the preliminary answer result according to the analysis result to obtain a target answer.

[0171] For the specific definition of a large medical model answer enhancement system, please refer to the above-mentioned definition of a large medical model answer enhancement method, which will not be repeated here. A person of ordinary skill in the art will appreciate that the various modules and steps described in conjunction with the embodiments disclosed in the present invention can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] like Figure 3 As shown, an embodiment of the present invention provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the steps in the above-mentioned medical large model answer enhancement embodiment are implemented, for example Figure 1 Steps S1 to S6 described in .

[0173] Those skilled in the art will understand that the schematic Figure 3These are merely examples of computer devices and do not constitute limitations on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0174] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0175] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and accessing the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0176] If the module integrated into the computer device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0177] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0178] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to perform the steps in the medical large model answer enhancement of the above embodiment, for example Figure 1 Steps S1 to S6 described in .

[0179] In summary, the embodiments of the present application provide a method, system, device and medium for enhancing the answer of a medical big model, which solves the technical problem of how to improve the accuracy of the answer of the medical big model. The method includes: obtaining the preliminary answer result generated by the medical big model based on the medical question input by the user, and using the medical big model to perform medical named entity recognition on the medical question and the preliminary answer result to obtain the medical entity recognition result; based on the medical entity recognition result, using pre-set verification question rules to generate a structured verification question set, and based on the medical entity recognition result and the preliminary answer result, using a structured prompt method to guide the medical big model to generate an extended verification question set and a counterfactual verification question set; The question set and the extended verification question set are input into the medical big model to generate the verification answer result, and the counterfactual verification question set is input into the medical big model to generate the counterfactual verification answer result; the first similarity result between the preliminary answer result and the verification answer result is identified, and the second similarity result between the preliminary answer result and the counterfactual verification answer result is identified; according to the Boolean query structure generated based on the medical entity recognition result, the evidence file is obtained from the evidence-based medicine database, and the third similarity result between the preliminary answer result and the evidence file is identified; the first similarity result, the second similarity result and the third similarity result are analyzed, and the preliminary answer result is corrected according to the analysis result to obtain the target answer. The medical big model answer enhancement method disclosed in this application generates self-verification questions and counterfactual verification questions, performs medical fact consistency verification and counterfactual cross-validation, ensures that the logic of the answer generated by the medical big model is stable and self-consistent, avoids low-level errors, and retrieves the latest medical literature through evidence-based medicine verification to ensure that the answer generated by the medical big model is based on the latest authoritative data, enhances the credibility of the output of the medical big model, and provides doctors and patients with accurate, reliable and quantifiable AI diagnosis and treatment support.

[0180] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the various technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The above-described embodiments merely represent several preferred implementations of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art could make several improvements and substitutions without departing from the technical principles of the present application, and such improvements and substitutions should also be considered within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be based on the scope of protection of the claims.

Claims

1. A method for enhancing the answer of a large medical model, characterized in that: The method comprises: Obtaining a preliminary answer result generated by the medical macromodel based on the medical question input by the user, and performing medical named entity recognition on the medical question and the preliminary answer result using the medical macromodel to obtain a medical entity recognition result; Based on the medical entity recognition results, a structured verification question set is generated using pre-set verification question rules, and based on the medical entity recognition results and the preliminary answer results, a structured prompt method is used to guide the medical macro model to generate an extended verification question set and a counterfactual verification question set; Inputting the structured verification question set and the extended verification question set into the medical macro model to generate verification answer results, and inputting the counterfactual verification question set into the medical macro model to generate counterfactual verification answer results; identifying a first similarity result between the preliminary answer result and the verification answer result, and identifying a second similarity result between the preliminary answer result and the counterfactual verification answer result; Obtaining an evidence document from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and identifying a third similarity result between the preliminary answer result and the evidence document; Analyzing the first similarity result, the second similarity result, and the third similarity result, and revising the preliminary answer result according to the analysis result to obtain a target answer; Based on the medical entity recognition result and the preliminary answer result, a structured prompt method is used to guide the medical large model to generate an extended verification question set and a counterfactual verification question set, including: Based on the medical entity recognition result and the preliminary answer result, an extended verification question task is generated using a first structured question template, wherein the first structured question template is used to enable the medical macro model to generate extended verification questions to supplement medical knowledge blind spots not covered in the preliminary answer result; Based on the medical entity recognition result and the preliminary answer result, a second structured question template is used to generate a counterfactual verification question task, wherein the second structured question template is used to enable the medical macro model to generate a counterfactual verification question to refute the preliminary answer result; Inputting the extended verification question task into the medical macro model to generate the extended verification question set; The counterfactual verification question task is input into the medical big model to generate the counterfactual verification question set.

2. The medical model answer enhancement method according to claim 1, characterized in that: The medical named entity recognition is performed on the medical question and the preliminary answer using the medical big model to obtain a medical entity recognition result, including: Using the medical big model to identify the diseases, symptoms, drugs, treatment plans and physiological indicators in the medical questions and the preliminary answers, to obtain a disease data set, a symptom data set, a drug data set, a treatment plan data set and a physiological indicator data set; A medical entity recognition result is constructed based on the disease dataset, the symptom dataset, the drug dataset, the treatment plan dataset, and the physiological indicator dataset.

3. The medical model answer enhancement method according to claim 2, characterized in that: Based on the medical entity recognition result, a structured verification question set is generated using pre-set verification question rules, including: Based on the disease dataset, using pre-set disease verification question rules, generating a structured disease verification question; Based on the symptom dataset, a structured symptom verification question is generated using a pre-set symptom verification question rule; Based on the drug dataset, a structured drug verification question is generated using pre-set drug verification problem rules; Based on the treatment plan data set, a structured treatment plan verification question is generated using pre-set treatment plan verification problem rules; Based on the physiological indicator data set, a structured physiological indicator verification question is generated using pre-set physiological indicator verification question rules; A structured verification question set is constructed based on the structured disease verification question, the structured symptom verification question, the structured drug verification question, the structured treatment plan verification question and the structured physiological indicator verification question.

4. The medical model answer enhancement method according to claim 1, wherein: The identifying a first similarity result between the preliminary answer result and the verification answer result, and the identifying a second similarity result between the preliminary answer result and the counterfactual verification answer result, includes: Calculating a first semantic similarity between the preliminary answer result and the verification answer result, and obtaining a first similarity result between the preliminary answer result and the verification answer result based on the first semantic similarity; A second semantic similarity between the preliminary answer result and the counterfactual verification answer result is calculated, and based on the second semantic similarity, a second similarity result between the preliminary answer result and the counterfactual verification answer result is obtained.

5. The medical model answer enhancement method according to claim 1, characterized in that: The step of obtaining an evidence document from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and identifying a third similarity result between the preliminary answer result and the evidence document, includes: generating a Boolean query structure based on the medical entity recognition result and a preset Boolean query template; According to the Boolean query structure, dense paragraph retrieval technology is used to obtain retrieval results from the evidence-based medicine database; sorting the search results using a text matching sorting method to obtain evidence documents; A third semantic similarity between the preliminary answer result and the evidence document is calculated, and based on the third semantic similarity, a third similarity result between the preliminary answer result and the evidence document is obtained.

6. The medical model answer enhancement method according to claim 1, characterized in that: The analyzing the first similarity result, the second similarity result, and the third similarity result, and correcting the preliminary answer result according to the analysis result to obtain a target answer, includes: Calculating and analyzing the first similarity result, the second similarity result, and the third similarity result to obtain a credibility score; determining a relationship between the credibility score and a first threshold and a second threshold, and when the credibility score is greater than the first threshold, directly outputting the preliminary answer result as the target answer; When the credibility score is less than or equal to the first threshold and greater than the second threshold, generating an uncertainty description, and outputting the preliminary answer result and the uncertainty description as the target answer; When the credibility score is less than or equal to the second threshold, a supplementary explanation is generated, and the supplementary explanation and the medical question are re-input into the medical model until the credibility score of the regenerated preliminary answer result is greater than the second threshold, and the target answer is obtained.

7. A medical large model answer enhancement system, which implements the medical large model answer enhancement method according to any one of claims 1 to 6, characterized in that: The system comprises: a preliminary answer result recognition unit, a verification question generation unit, a verification answer generation unit, a first recognition unit, a second recognition unit and an answer correction unit; The preliminary answer result recognition unit is used to obtain the preliminary answer result generated by the medical macro model based on the medical question input by the user, and use the medical macro model to perform medical named entity recognition on the medical question and the preliminary answer result to obtain a medical entity recognition result; The verification question generating unit is configured to generate a structured verification question set based on the medical entity recognition result and a preset verification question rule, and to guide the medical macro model to generate an extended verification question set and a counterfactual verification question set based on the medical entity recognition result and the preliminary answer result using a structured prompting method; The verification answer generation unit is configured to input the structured verification question set and the extended verification question set into the medical macro model to generate a verification answer result, and input the counterfactual verification question set into the medical macro model to generate a counterfactual verification answer result; The first identification unit is configured to identify a first similarity result between the preliminary answer result and the verification answer result, and to identify a second similarity result between the preliminary answer result and the counterfactual verification answer result; The second recognition unit is configured to obtain an evidence document from an evidence-based medicine database according to a Boolean query structure generated based on the medical entity recognition result, and to identify a third similarity result between the preliminary answer result and the evidence document; The answer correction unit is configured to analyze the first similarity result, the second similarity result, and the third similarity result, and correct the preliminary answer result according to the analysis result to obtain a target answer; Based on the medical entity recognition result and the preliminary answer result, a structured prompt method is used to guide the medical large model to generate an extended verification question set and a counterfactual verification question set, including: Based on the medical entity recognition result and the preliminary answer result, an extended verification question task is generated using a first structured question template, wherein the first structured question template is used to enable the medical macro model to generate extended verification questions to supplement medical knowledge blind spots not covered in the preliminary answer result; Based on the medical entity recognition result and the preliminary answer result, a second structured question template is used to generate a counterfactual verification question task, wherein the second structured question template is used to enable the medical macro model to generate a counterfactual verification question to refute the preliminary answer result; Inputting the extended verification question task into the medical macro model to generate the extended verification question set; The counterfactual verification question task is input into the medical big model to generate the counterfactual verification question set.

8. A computer device, characterized in that: The computer device includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to perform the medical large model answer enhancement method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the medical large model answer enhancement method according to any one of claims 1 to 6 is implemented.

Images