Medical evidence-based evidence sorting method and device, storage medium and computer equipment

Abstract

The embodiment of the invention provides a medical evidence-based evidence sorting method and device, a storage medium and computer equipment. The method is applied to a man-machine conversation application program based on a large language model. The method comprises the following steps: receiving a prompt which is input by a user in a man-machine conversation interface of the application program and triggers a medical evidence; generating at least two sub-queries based on the prompt; searching each sub-query, recalling an evidence subset corresponding to each sub-query, and forming an evidence set for the prompt based on the evidence subsets; and performing correlation operation on the evidences in the evidence set based on the prompt, and sorting the evidences in the evidence set based on a correlation operation result. In the technical scheme provided by the embodiment of the invention, the evidence in the evidence set is subjected to the correlation operation based on the prompt, and the evidence in the evidence set is sorted based on the correlation operation result, so that the searched evidence is matched with the user demand, and the authority and accuracy of medical evidence-based are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of evidence-based technology, and in particular to an evidence sorting method and device for medical evidence-based medicine, a storage medium and a computer device. BACKGROUND

[0002] Evidence-based medicine (EBM) in medicine refers to a medical practice mode that combines the current best research evidence, the professional skills of a clinician and the personal values and needs of a patient to jointly guide clinical decision-making. The core is to replace experience intuition with scientific evidence to ensure the objectivity and effectiveness of medical decision-making.

[0003] The evidence sorting result of medical evidence-based medicine is directly related to the effect of medical evidence-based medicine. If the evidence searched by medical evidence-based medicine does not match the user's needs, the authority and accuracy of medical evidence-based medicine will be greatly reduced. SUMMARY

[0004] Therefore, the embodiments of the present application provide an evidence sorting method and device for medical evidence-based medicine, a storage medium and a computer device, so as to match the searched evidence with the user's needs and improve the authority and accuracy of medical evidence-based medicine.

[0005] In one aspect, the embodiments of the present application provide an evidence sorting method for medical evidence-based medicine, applied to a man-machine dialogue application program based on a large language model, comprising: receiving a prompt for triggering medical evidence-based medicine input by a user in a man-machine dialogue interface of the application program; generating at least two sub-queries based on the prompt; searching each sub-query, recalling an evidence subset corresponding to each sub-query, and forming an evidence set for the prompt based on the evidence subset; performing a relevance operation on the evidence in the evidence set based on the prompt, and sorting the evidence in the evidence set based on the relevance operation result.

[0006] Optionally, the relevance operation on the evidence in the evidence set based on the prompt comprises: performing a relevance operation on the evidence in the evidence subset corresponding to each sub-query, and selecting a set number of most relevant evidences for each sub-query; performing a relevance operation on the evidence set composed of the set number of evidences for each sub-query and the prompt. Optionally, the prompt comprises a first number of elements, the sub-query comprises a second number of elements, and the relevance operation on the evidence in the evidence set based on the prompt comprises: performing a relevance operation on the evidences in the evidence subset corresponding to each sub-query, and selecting a set number of evidences most relevant to each sub-query; determining elements different between the prompt and each sub-query, to generate an element set; performing a relevance operation on the element set and a set number of evidences of each sub-query to form an evidence set. Optionally, after the evidences in the evidence set are sorted based on the relevance operation result, the method comprises: sorting the evidences according to the determined inquiry scene and the evidence grading rule corresponding to the inquiry scene.

[0007] Optionally, before the evidences are sorted according to the determined inquiry scene and the evidence grading rule corresponding to the inquiry scene, the method comprises: when the authority of the evidences in the relevance operation result is low, continuing to perform the step of sorting the evidences according to the determined inquiry scene and the evidence grading rule corresponding to the inquiry scene. Optionally, after the evidences are sorted according to the determined inquiry scene and the evidence grading rule corresponding to the inquiry scene, the method comprises: sorting the evidences of the same level in the sorted evidences according to their disclosure time.

[0008] Optionally, the inquiry scene and the evidence grading rule corresponding to the inquiry scene comprise: if the inquiry scene comprises a medical research scene, a medical teaching scene, or a medical basic knowledge query scene, the evidence grading rule corresponding to the inquiry scene comprises: setting a set of high-quality journals as high priority; setting evidences with research methods including meta-analysis, systematic review, and randomized controlled trial as high priority, setting a drug instruction or off-label use instruction as high priority, setting evidences with research methods including non-randomized controlled trial, observational study, and literature review as medium priority, setting contents of a medical website as medium priority, setting evidences with research methods including case report, animal experiment, and in vitro experiment as low priority; setting contents of a non-medical website as low priority; or, if the inquiry scene comprises a medical clinical scene, the evidence grading rule corresponding to the inquiry scene further comprises: setting a guideline as high priority.

[0009] Optionally, the generating of the at least two sub-queries based on the prompt comprises: generating a Cot based on the prompt according to an inference model; generating, based on the second large language model, Sub-questions from the Cot; generating, based on the second large language model, the at least 2 sub-queries from the Sub-questions.

[0010] Optionally, the searching each sub-query, recalling an evidence subset corresponding to each sub-query, and forming an evidence set for the prompt based on the evidence subset include: searching each sub-query in a self-built knowledge base and the Internet based on a third large language model, recalling an evidence subset corresponding to each sub-query by using a recall algorithm, and forming an evidence set for the prompt based on the evidence subset, wherein the knowledge base includes evidence grading rules corresponding to the inquiry scene.

[0011] Optionally, after the same level of evidence in the ranked evidence is sorted according to the disclosure time, the method further includes: generating, based on a fourth large language model and an inference model, a reply statement corresponding to the prompt from the sorted evidence, wherein the reply statement includes an evidence label generated from the sorted evidence, and the evidence label includes a research type, a source, an influence, and / or a grading of each evidence, wherein the evidence label is used to indicate the authority of the evidence. displaying the reply statement in a man-machine dialogue interface of the application program.

[0012] Optionally, the receiving a prompt input by a user in a man-machine dialogue interface of the application program includes: receiving an inquiry statement input by a user in a man-machine dialogue interface of the application program; generating, based on prompt engineering technology, the prompt from the inquiry statement; determining, based on a first large language model, an inquiry scene from the prompt, and judging whether the inquiry scene is a set inquiry scene capable of triggering medical evidence; if it is judged that the inquiry scene is an inquiry scene capable of triggering medical evidence, the step of generating at least 2 sub-queries based on the prompt is continued.

[0013] In another aspect, an embodiment of the present application provides an evidence sorting device for medical evidence, which is applied to a man-machine dialogue application program based on a large language model, and includes: a receiving module configured to receive a prompt input by a user in a man-machine dialogue interface of the application program, the prompt triggering medical evidence; a first generating module configured to generate at least 2 sub-queries based on the prompt; a second generation module configured to search each sub-query, recall an evidence subset corresponding to each sub-query, and form an evidence set for the prompt based on the evidence subset; an operation ordering module configured to perform a relevance operation on the evidence in the evidence set based on the prompt, and order the evidence in the evidence set based on the relevance operation result.

[0014] In another aspect, an embodiment of the present application provides a storage medium including a stored program, wherein the program, when executed, controls a device in which the storage medium is located to perform the evidence ordering method for medical evidence-based medicine.

[0015] In another aspect, an embodiment of the present application provides a computer device including a memory and a processor, the memory being configured to store information including program instructions, and the processor being configured to control execution of the program instructions, wherein the program instructions, when loaded and executed by the processor, implement the steps of the evidence ordering method for medical evidence-based medicine.

[0016] In the technical solution provided by the embodiment of the present application, a prompt triggering medical evidence-based medicine input by a user in a man-machine dialogue interface of an application program is received, at least two sub-queries are generated based on the prompt, each sub-query is searched, an evidence subset corresponding to each sub-query is recalled, and an evidence set for the prompt is formed based on the evidence subset. A relevance operation is performed on the evidence in the evidence set based on the prompt, and the evidence in the evidence set is ordered based on the relevance operation result. In the technical solution provided by the embodiment of the present application, a relevance operation is performed on the evidence in the evidence set based on the prompt, and the evidence in the evidence set is ordered based on the relevance operation result, so that the searched evidence matches the user's demand, and the authority and accuracy of medical evidence-based medicine are improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 A flowchart of an evidence ordering method for medical evidence-based medicine provided by an embodiment of the present application; Figure 2 A flowchart of another evidence ordering method for medical evidence-based medicine provided by an embodiment of the present application; Figure 3A flowchart of another medical evidence-based evidence ranking method provided by one embodiment of the present invention; Figure 4 It is a flowchart for performing correlation calculation on evidence in an evidence set based on prompt; Figure 5 This is another flowchart for performing correlation calculation on evidence in an evidence set based on prompt; Figure 6 A schematic diagram of a medical evidence-based evidence ranking device provided by one embodiment of the present invention; Figure 7 A schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0020] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0021] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0023] In a related technology, it is generally necessary to first perform Cot splitting on the prompt input by the user (the prompt includes multiple elements such as time, age, population, disease, etc.), and then convert it into a sub-query for searching (when the prompt is split into a sub-query, the elements in the prompt may be lost), and then perform relevance operation according to the sub-query (the lost elements are not operated by the relevance operation), resulting in a difference between the final searched evidence and the prompt. For example, the user wants to know the mortality rate of old people when doing A operation, if the element of the sub-query does not include the population element "old people", the relevant evidence of "children doing A operation" in the final searched evidence may be determined as strong correlation. Therefore, the technical solution in the related technology makes the searched evidence not match the user's demand, and the authority and accuracy of medical evidence are low.

[0024] To solve the technical problems in the related technology, an embodiment of the present application provides a medical evidence sorting method, Figure 1 The flow chart of the medical evidence sorting method provided by an embodiment of the present application is shown in Figure 1 The method comprises the following steps: Step 102, receiving the prompt input by the user in the man-machine dialogue interface of the application program to trigger the medical evidence.

[0025] In the embodiment of the present application, each step is executed by a computer device. For example, the computer device includes a computer, a mobile phone, a tablet computer, a wearable device, etc. Each step can be applied to a man-machine dialogue application program based on a large language model (Large Language Model, LLM for short).

[0026] In the embodiment of the present application, the prompt is an instruction or text input by the user to the model, which is used to guide the model to generate a specific output. The prompt can help the model understand the user's demand by explicitly requiring the task and providing context information, and directly affects the output quality of the model.

[0027] Step 104, generating at least two sub-queries based on the prompt.

[0028] In the embodiment of the present application, the Cot can be generated based on the reasoning model R1'671B according to the prompt, the sub-questions can be generated based on the second large language model 72B model 1 according to the Cot, and at least two sub-queries can be generated based on the second large language model according to the sub-questions.

[0029] Step 106, searching each sub-query, recalling the evidence subset corresponding to each sub-query, and forming an evidence set for the prompt based on the evidence subset.

[0030] In the embodiment of the present application, the third large language model 7B model 2 can be used to search the self-built knowledge base and the Internet based on each sub-query, recall the corresponding evidence subset of each sub-query by using a recall algorithm, form an evidence set for the prompt based on the evidence subset, and the knowledge base includes the evidence classification rules corresponding to the inquiry scene.

[0031] Step 108, correlation operation of the evidence in the evidence set based on the prompt, and sorting of the evidence in the evidence set based on the correlation operation result.

[0032] In the embodiment of the present application, the third large language model 7B model 2 can be used to search the self-built knowledge base and the Internet based on each sub-query, recall the corresponding evidence subset of each sub-query by using a recall algorithm, form an evidence set for the prompt based on the evidence subset, and the knowledge base includes the evidence classification rules corresponding to the inquiry scene.

[0033] The technical scheme provided by the embodiment of the present application receives the prompt for triggering medical evidence-based medicine input by the user in the man-machine dialogue interface of the application program, generates at least two sub-queries based on the prompt, searches each sub-query, recalls the corresponding evidence subset of each sub-query, forms an evidence set for the prompt based on the evidence subset, and performs correlation operation of the evidence in the evidence set based on the prompt, and sorts the evidence in the evidence set based on the correlation operation result. In the technical scheme provided by the embodiment of the present application, the evidence in the evidence set is subjected to correlation operation based on the prompt, and the evidence in the evidence set is sorted based on the correlation operation result, so that the searched evidence matches the user's demand, and the authority and accuracy of medical evidence-based medicine are improved.

[0034] Another evidence sorting method for medical evidence-based medicine provided by an embodiment of the present application, Figure 2 The flowchart of another evidence sorting method for medical evidence-based medicine provided by an embodiment of the present application, Figure 3 The flowchart of another evidence sorting method for medical evidence-based medicine provided by an embodiment of the present application, as shown in Figure 2 Or Figure 3 As shown in the drawings, the method comprises the following steps: Step 202, receiving the inquiry sentence input by the user in the man-machine dialogue interface of the application program.

[0035] In the embodiment of the present application, each step is executed by a computer device. For example, the computer device includes a computer, a mobile phone, a tablet computer, a wearable device, etc. Each step can be applied to a man-machine dialogue application program based on a large language model.

[0036] For example, the user can input the inquiry sentence "What is the dosage adjustment of drug a for people with disease A?" in the human-computer dialogue interface of the application.

[0037] Step 204, generating a prompt based on the inquiry sentence according to the prompt engineering technology.

[0038] In the embodiments of the present application, the history of the dialogue, user feedback, continuous dialogue content and the current inquiry sentence can be memorized and used to generate the prompt.

[0039] Specifically, the core intent of the inquiry sentence can be determined based on the prompt engineering technology according to the inquiry sentence, the appropriate template structure can be selected according to the inquiry scene, the content of the inquiry sentence can be mapped to the template structure, and the prompt can be generated by optimization and iteration.

[0040] For example, the template structure can be: markdownCopy Code = Role + Purpose + Expectation + Requirement - Role: the identity the model hopes to play (such as a pharmacist) - Purpose: the goal of playing this role (such as determining the dosage of drug a) - Expectation: the problem to be solved (such as the dosage adjustment of drug a for people with disease A) - Requirement: output format or style restrictions: ml-citation{ref="4" data="citationList"} Step 206, determining the inquiry scene based on the first large language model according to the prompt, and judging whether the inquiry scene is a set inquiry scene that can trigger medical evidence.

[0041] In the embodiments of the present application, the judgment of the inquiry scene can be realized by the word vector conversion and prediction mechanism of the first large language model. First, the prompt is converted into a digital vector representation (such as a 300-dimensional word vector), then the relevance between these word vectors is analyzed through a multi-layer Transformer architecture, and finally the most likely inquiry scene is predicted.

[0042] In the embodiments of the present application, the first large language model can be a small model suitable for processing simpler problems, such as a 7B model 1 (for example, LLaMA-7B or DeepSeek-7B), and the B in the 7B model 1 represents Billion, which is a unit of 10 billion parameters, and the 7B model represents a model with 700 million parameters.

[0043] In the embodiments of the present application, the inquiry scene capable of triggering medical evidence-based inquiry can be set according to actual conditions, for example, the inquiry scene capable of triggering medical evidence-based inquiry includes: medical research scene (academic search, evidence search, research dynamic tracking, research guidance), medical teaching scene (medical examination, clinical teaching, new technology teaching application, teaching evaluation method), medical basic knowledge query scene (medical concept query, medical knowledge query, popular science knowledge query), medical clinical scene (symptom analysis and diagnosis, treatment decision support, medication guidance, inspection, guideline query), and the like.

[0044] In the embodiments of the present application, if it is judged that the inquiry scene cannot trigger the inquiry scene capable of triggering medical evidence-based inquiry, an existing artificial intelligence (Artificial Intelligence, referred to as AI) model is called for processing. For example, if it is judged that the inquiry scene is a weather inquiry scene, the weather inquiry scene cannot trigger the inquiry scene capable of triggering medical evidence-based inquiry, and an existing AI model (such as DeepSeek-V3) can be called to process the weather inquiry scene.

[0045] Step 208, if it is judged that the inquiry scene is the inquiry scene capable of triggering medical evidence-based inquiry, at least two sub-queries are generated based on the prompt.

[0046] In the embodiments of the present application, step 208 includes: Step S1, generating Cot based on the reasoning model according to the prompt.

[0047] In the embodiments of the present application, the reasoning model can be a large model (GPT-4.5, Med-PaLM2, Qwen3 or BioMedLM), which can process large-scale literature and perform complex reasoning. For example, the large model includes Deepseek-R1’671B, and the 671B model represents a model with 671 billion parameters.

[0048] In the embodiments of the present application, Cot is a technical method for guiding AI models to generate intermediate reasoning steps, aiming to improve their logical reasoning ability in complex tasks. Its core is to require the model to explicitly output the complete reasoning path from input to output, rather than just providing the final answer.

[0049] For example, the Cot template can be: textCopy Code I need to solve [problem description]. Please help me think step by-step: 1. First, analyze the key points of the problem... 2. Then, consider possible solutions... 3. Evaluate the pros and cons of each solution... 4. Finally, present the most reasonable solution and explain the reasons.

[0050] Step S2, generating Sub-questions based on the second large language model according to Cot.

[0051] In the embodiment of the application, the second large language model includes a medium model, and the medium model can process relatively complex medical problems. For example, the medium model includes a 72B model 1 (LLaMA-72B), and the 72B model represents a model with a parameter amount of 720 billion.

[0052] In the embodiment of the application, Sub-questions refer to a strategy of analyzing and processing a complex problem by decomposing the complex problem into smaller and more specific components.

[0053] For example, if Cot contains mathematical reasoning (Xiao Ming has 5 apples, and the number of apples of Xiao Hong is 2 times more than that of Xiao Ming plus 3, how many apples does Xiao Hong have?), the Sub-questions include: 1, determine the number of apples of Xiao Ming (5); 2, calculate 2 times the number of apples of Xiao Ming (5*2=10); 3, add 3 to the 2 times (10+3=13); and 4, obtain the number of apples of Xiao Hong (13).

[0054] Step S3, generating at least two sub-queries based on the second large language model according to Sub-questions.

[0055] In the embodiment of the application, the complex Sub-questions are decomposed into at least two smaller and more specific sub-queries that are logically associated, for retrieving information with finer granularity.

[0056] Specifically, the Sub-questions can be decomposed in time sequence (splitting the steps of the sub-problems in chronological order), decomposed in logic (constructing a problem chain based on causal relationship), and decomposed in function (dividing the modules according to system functions) to generate at least two sub-queries.

[0057] Step 210, searching each sub-query in a self-built knowledge base and the Internet based on a third large language model, recalling an evidence subset corresponding to each sub-query by using a recall algorithm, and forming an evidence set for the prompt based on the evidence subset, wherein the knowledge base includes evidence hierarchical rules corresponding to the inquiry scene.

[0058] In the embodiment of the application, the third large language model can be a small model (such as Claude 3 / GLM-4), which is suitable for processing relatively simple problems, such as a 7B model 2 (for example, LLaMA-7B or DeepSeek-7B), and the 7B model represents a model with a parameter amount of 700 million.

[0059] Specifically, each sub-query can be vectorized and encoded using a third large language model, matched with the index of the self-built knowledge base through cosine similarity, weighted according to the evidence grading rules set by the knowledge base, and the knowledge base evidence is generated. The search engine API is called to obtain the TOP-K web page of each sub-query, and after denoising processing (removing ads / navigation bar), the Internet evidence is generated. The weighted hybrid ranking algorithm is used to integrate the knowledge base evidence and the Internet evidence to generate the integrated evidence. The MinHash-LSH algorithm is used for deduplication processing of the integrated evidence, and the recall algorithm is used to recall the evidence subset corresponding to each sub-query, and the evidence set corresponding to the prompt is formed based on the evidence subset.

[0060] In the embodiment of the application, the recall algorithm is used to recall the evidence subset corresponding to each sub-query from the massive data, and the recall algorithm can include: a collaborative filtering algorithm, a content-based recommendation algorithm, a model-based recommendation algorithm, a multi-path recall algorithm, and a deep learning driven vectorization recall algorithm, etc.

[0061] In the embodiment of the application, the knowledge base contains medical-related journals, papers, drug instructions, off-label use instructions, guidelines, case reports, medical conference documents, etc.

[0062] In the embodiment of the application, the evidence grading rules corresponding to the inquiry scene in the knowledge base include: if the inquiry scene includes medical research scene, medical teaching scene, and medical basic knowledge query scene, the evidence grading rules corresponding to the inquiry scene include: setting the high-quality journals as high priority; setting the evidence of research methods including meta analysis (Meta Analysis), systematic review (Systematic Review), randomized controlled trial (randomized controlled trial, RCT) as high priority, setting the drug instructions or off-label use instructions as high priority, setting the evidence of research methods including non-randomized controlled trial (Non-RCT Trial), observational study (Observational Study), literature review (Literature Review) as medium priority, and setting the evidence of research methods including case report (Case Report), animal trial (Animal Trial), in vitro trial (In Vitro Trial) as low priority; or, if the inquiry scene includes a medical clinical scene, the evidence grading rules corresponding to the inquiry scene further include: setting the guidelines as high priority. The guidelines can include clinical practice guidelines, expert consensus, expert opinion, scientific statement, technical specification and operation standard.

[0063] In the embodiment of the present application, the evidence grading rule corresponding to the inquiry scene in the Internet comprises: setting the content of the medical website as a medium priority, and setting the content of the non-medical website as a low priority.

[0064] In the embodiment of the present application, the evidence grading rule can also comprise the evidence pyramid proposed by the Medical Center of State University of New York in 2001, the evidence level grading method of the Joanna Briggs Institute (JBI), and the like.

[0065] In the embodiment of the present application, the priority of the evidence can be used to represent the authority of the evidence, and the evidence with high priority has high authority.

[0066] In the embodiment of the present application, when the evidence in the evidence set is graded and sorted according to the inquiry scene and the evidence grading rule corresponding to the inquiry scene, the evidence with low priority (low authority) can not be used.

[0067] In step 212, the evidence in the evidence set is subjected to relevance operation based on the prompt, and the evidence in the evidence set is sorted based on the result of the relevance operation.

[0068] In the embodiment of the present application, the prompt and the evidence in the evidence set can be mapped to the same vector space, and the cosine similarity is calculated to perform the relevance operation.

[0069] In the embodiment of the present application, the prompt and the evidence in the evidence set can be input into the BERT model to obtain the interactive relevance score to perform the relevance operation.

[0070] In the embodiment of the present application, the prompt and the evidence in the evidence set can be subjected to multi-dimensional weighted operation of comprehensive semantic similarity, keyword matching, and evidence authority to perform the relevance operation.

[0071] In the embodiment of the present application, the prompt comprises the main information of the inquiry sentence input by the user, and represents the main demand of the user.

[0072] In the embodiment of the present application, the core principles of refining the main requirements include: 1. The main requirements should contain all the essential key information in the user's query sentence; 2. Eliminate the modifying words or preferred expressions that have little effect on the understanding of the core requirements; 3. Based on the user's query sentence, no subjective judgment or additional information is added. The steps of refining the main requirements include: 1. Understand the user's input query sentence and determine the core requirements. 2. Keep the key information and eliminate the secondary information. 3. Verify the integrity of the main requirements. For example, the query sentence is "Guidelines on the treatment of children infected with Salmonella", and the main requirement is "the treatment of children infected with Salmonella", and the guide is a result preference, not a core requirement. For example, the query sentence is "Ozempic and Mounjaro GLP-1 receptor agonists health risks", and the main requirement is "GLP-1 receptor agonist health risks", and Ozempic and Mounjaro are GLP-1, which is only an example and needs to be deleted.

[0073] In the embodiment of the present application, the relevance operation includes direct satisfaction and indirect satisfaction. The direct satisfaction is that the range involved in the description completely overlaps the range in the problem, the indirect satisfaction is that the range involved in the description has an intersection with the range in the problem (including partial coverage) and the range is not directly mentioned in the description, but the conclusion that covers the range can be obtained through reasonable deduction.

[0074] For example, it is mentioned that 70-80 years old is a specific condition, then "75-90 years old" is a direct satisfaction, "postmenopausal" is an indirect satisfaction, "50 years old and continuously use for more than 20 years" is an indirect satisfaction, "average age 60 years old" is not satisfied, and "under 50 years old" is not satisfied.

[0075] Figure 4 It is a flowchart for performing relevance operation on evidence in an evidence set based on a prompt, as shown in Figure 4 Step 212 can include: Step A1, performing relevance operation on the evidence in the evidence subset corresponding to each subquery, and selecting the set number of most relevant evidences for each subquery.

[0076] Step A2, performing relevance operation on the evidence set composed of the set number of evidences of each subquery and the prompt.

[0077] In the embodiment of the present application, two relevance operations can be performed, first performing the relevance operation on the evidence in the evidence subset corresponding to each subquery in step A1, and then performing the relevance operation on the evidence set composed of the set number of evidences of each subquery in step A2.

[0078] Figure 5For another flowchart based on the prompt to perform the relevance operation on the evidence in the evidence set, as shown in Figure 5 Step 212 can further include: Step B1, performing the relevance operation on the evidence in the evidence subset corresponding to each sub-query, and selecting the set number of most relevant evidences for each sub-query.

[0079] Step B2, determining the elements that the prompt differs from each sub-query, and generating an element set.

[0080] Step B3, performing the relevance operation on the evidence set composed of the set number of evidences of each sub-query and the element set.

[0081] In the embodiment of the present application, the prompt includes a first number of elements, and the sub-query includes a second number of elements. For example, the prompt includes 6 elements “time”, “age”, “disease”, “population”, “gender” and “symptom”, a certain sub-query includes only 3 elements “time”, “disease” and “symptom”, and another sub-query includes 3 elements “time”, “disease” and “population”. The element set generated by the elements that the prompt differs from each sub-query is “age” and “gender”, and the relevance operation can be performed on the evidence set composed of the set number of evidences of each sub-query based on the element set (“age” and “gender”) that the prompt differs from the sub-query.

[0082] Step 214, performing the hierarchical sorting on the evidence in the evidence set according to the determined inquiry scene and the evidence hierarchical rule corresponding to the inquiry scene.

[0083] In the embodiment of the present application, when the authority of the evidence in the relevance operation result is low, step 214 is continuously executed. At this time, the relevance of the evidence in the relevance operation result is very high, but the authority of the evidence is low, and the evidence in the relevance operation result can be hierarchically sorted according to the evidence hierarchical rule corresponding to the inquiry scene.

[0084] In the embodiment of the present application, for the same query, a reply with high relevance is mentioned in a forum network, and a reply with relatively low relevance is mentioned in a medical guide, and the evidence related to the more authoritative medical guide is arranged in front in the embodiment of the present application.

[0085] For example, the evidence of the research method including meta-analysis, the evidence of the system review, the evidence of the research method including non-random control test, and the evidence in the forum network are obtained by the correlation operation, if the inquiry scene is the medical research scene, the evidence of the research method including meta-analysis, the evidence of the system review (high priority), the evidence of the research method including non-random control test (medium priority), and the evidence in the forum network (low priority) are ranked in order of priority, and the evidence with high priority is arranged in front of the evidence with medium priority, and the evidence with medium priority is arranged in front of the evidence with low priority.

[0086] Step 216, the evidence of the same level in the ranked evidence is sorted according to the publication time.

[0087] Specifically, the evidence of the same level with the publication time close to the current time can be arranged in front of the evidence of the same level with the publication time far from the current time.

[0088] For example, the evidence of the research method including meta-analysis and the evidence of the system review are both high-priority evidence, if the publication time of the evidence of the research method including meta-analysis is January 1, 2023, and the publication time of the evidence of the research method including system review is February 15, 2024, the publication time of the evidence of the research method including meta-analysis is earlier than the publication time of the evidence of the research method including system review, and the evidence of the research method including meta-analysis is arranged behind the evidence of the research method including system review.

[0089] Step 218, generating a reply sentence corresponding to the prompt based on the fourth large language model and the reasoning model according to the sorted evidence, the reply sentence including an evidence label generated according to the sorted evidence, the evidence label including the research type, the evidence source, the influence and / or the evidence classification of each evidence, wherein the evidence label is used to indicate the authority of the evidence.

[0090] In the embodiment of the application, the fourth large language model includes a medium model (such as Claude3 or Deepseek-V3), which can handle more complex medical problems. For example, the medium model can also include a 72B model 2 (LLaMA-72B), and the 72B model represents a model with 72 billion parameters.

[0091] Step 220, displaying the reply sentence in the man-machine dialogue interface of the application.

[0092] For example, the inquiry statement in step 202 "Does the dosage of drug a need to be adjusted for a person with disease A?" The corresponding reply statement can include "People with disease A usually do not need to adjust the dosage of drug a. According to the "drug a instruction" and "IDSA guidelines", drug a is mainly metabolized by the liver and excreted by the bile (about 40-65%), and only a small amount is excreted by the kidney (about 33-67%). Therefore, even if there is disease a, if the liver function is normal, generally no need to reduce the amount. Avoid the use of other nephrotoxic drugs (such as drug b) to prevent aggravation of kidney damage. Summary: People with disease A do not need to routinely reduce the dosage of drug a, but need to be evaluated comprehensively in combination with liver function status and clinical situation, and drug concentration needs to be monitored if necessary. In the technical solution provided by the embodiment of the present application, the prompt triggering medical evidence input by the user in the man-machine dialogue interface of the application program is received; at least two sub-queries are generated based on the prompt; each sub-query is searched to recall an evidence subset corresponding to each sub-query, and an evidence set for the prompt is formed based on the evidence subset; the evidence in the evidence set is subjected to a correlation operation based on the prompt, and the evidence in the evidence set is sorted based on the correlation operation result. In the technical solution provided by the embodiment of the present application, the evidence in the evidence set is subjected to a correlation operation based on the prompt, and the evidence in the evidence set is sorted based on the correlation operation result, so that the searched evidence matches the user's demand, and the authority and accuracy of the medical evidence are improved.

[0093] An embodiment of the present application provides an evidence sorting device for medical evidence, Figure 6 A schematic diagram of the evidence sorting device for medical evidence provided by an embodiment of the present application is shown in the figure, which comprises a receiving module 11, a first generating module 12, a second generating module 13 and an operation and sorting module 14. Figure 6

[0094] The receiving module 11 is used to receive the prompt triggering medical evidence input by the user in the man-machine dialogue interface of the application program.

[0095] The first generating module 12 is used to generate at least two sub-queries based on the prompt.

[0096] The second generating module 13 is used to search each sub-query to recall an evidence subset corresponding to each sub-query, and form an evidence set for the prompt based on the evidence subset.

[0097] The operation and sorting module 14 is used to subject the evidence in the evidence set to a correlation operation based on the prompt, and sort the evidence in the evidence set based on the correlation operation result.

[0098] ​In the embodiment of the present application, the operation and sorting module 14 is specifically configured to perform relevance operation on the evidence in the evidence subset corresponding to each sub-query, and select the set number of most relevant evidences for each sub-query; and perform relevance operation on the evidence set composed of the set number of evidences of each sub-query and the prompt.

[0099] In the embodiment of the present application, the prompt includes a first number of elements, and the sub-query includes a second number of elements. The operation and sorting module 14 is specifically configured to perform relevance operation on the evidence in the evidence subset corresponding to each sub-query, and select the set number of most relevant evidences for each sub-query; determine the elements different between the prompt and each sub-query, generate an element set; and perform relevance operation on the element set and the evidence set composed of the set number of evidences of each sub-query.

[0100] In the embodiment of the present application, the device further includes a first sorting module 15.

[0101] The first sorting module 15 is configured to sort the evidence according to the determined inquiry scene and the evidence grading rule corresponding to the inquiry scene.

[0102] In the embodiment of the present application, when the authority of the evidence in the relevance operation result is low, the first sorting module 15 is triggered to perform the step of sorting the evidence according to the determined inquiry scene and the evidence grading rule corresponding to the inquiry scene.

[0103] In the embodiment of the present application, the device further includes a second sorting module 16.

[0104] The second sorting module 16 is configured to sort the evidence of the same level in the sorted evidence according to the disclosure time thereof.

[0105] In the embodiment of the present application, the inquiry scene and the evidence grading rule corresponding to the inquiry scene include: if the inquiry scene includes a medical research scene, a medical teaching scene, and a medical basic knowledge query scene, the evidence grading rule corresponding to the inquiry scene includes: setting a high-quality journal as a high priority; setting the evidence of the research method including meta analysis, systematic review, and randomized controlled trial as a high priority, setting the drug instruction or off-label drug instruction as a high priority, setting the evidence of the research method including non-randomized controlled trial, observational study, and literature as a medium priority, setting the content of a medical website as a medium priority, setting the evidence of the research method including case report, animal experiment, and in vitro experiment as a low priority; and setting the content of a non-medical website as a low priority; or, if the inquiry scene includes a medical clinical scene, the evidence grading rule corresponding to the inquiry scene further includes: setting a guideline as a high priority.

[0106] In the embodiment of the present application, the first generation module 12 is specifically configured to generate Cot based on the inference model according to the prompt; generate Sub-questions based on the second large language model according to the Cot; and generate at least two sub-queries based on the second large language model according to the Sub-questions.

[0107] In the embodiment of the present application, the second generation module 13 is specifically configured to search each sub-query in a self-built knowledge base and the Internet based on the third large language model, recall the evidence subset corresponding to each sub-query by using a recall algorithm, and form an evidence set for the prompt based on the evidence subset, wherein the knowledge base includes evidence classification rules corresponding to the inquiry scene.

[0108] In the embodiment of the present application, the device further comprises a fourth generation module 17 and a display module 18.

[0109] The fourth generation module 17 is configured to generate a reply statement corresponding to the prompt based on the fourth large language model and the inference model according to the ranked evidence, wherein the reply statement includes evidence labels generated according to the ranked evidence, and the evidence labels include the research type, the evidence source, the influence and / or the evidence classification of each evidence, wherein the evidence labels are used to indicate the authority of the evidence.

[0110] The display module 18 is configured to display the reply statement in the man-machine dialogue interface of the application program.

[0111] In the embodiment of the present application, the receiving module 11 is specifically configured to receive an inquiry statement input by a user in the man-machine dialogue interface of the application program; generate a prompt based on the prompt engineering technology according to the inquiry statement; determine the inquiry scene based on the first large language model according to the prompt, and judge whether the inquiry scene is a set inquiry scene capable of triggering medical evidence; if it is judged that the inquiry scene is the inquiry scene capable of triggering medical evidence, the first generation module 12 continues to execute the step of generating at least two sub-queries based on the prompt.

[0112] The technical scheme provided by the embodiment of the present application comprises the following steps: receiving a prompt input by a user in a man-machine conversation interface of an application program; generating at least two sub-queries based on the prompt; searching each sub-query, recalling an evidence subset corresponding to each sub-query, and forming an evidence set for the prompt based on the evidence subset; performing a correlation operation on the evidence in the evidence set based on the prompt, and sorting the evidence in the evidence set based on the correlation operation result. In the technical scheme provided by the embodiment of the present application, the correlation operation is performed on the evidence in the evidence set based on the prompt, and the evidence in the evidence set is sorted based on the correlation operation result, so that the searched evidence matches the user's demand, and the authority and accuracy of the medical evidence are improved.

[0113] The evidence sorting device for medical evidence provided by the embodiment of the present application can be used to implement the medical evidence sorting method described above. Figure 1 The specific description can be referred to the embodiment of the medical evidence sorting method described above, and will not be described here.

[0114] The embodiment of the present application provides a storage medium, which comprises a stored program, wherein when the program is running, the device where the storage medium is located is controlled to perform each step of the embodiment of the medical evidence sorting method described above, and the specific description can be referred to the embodiment of the medical evidence sorting method described above.

[0115] The embodiment of the present application provides a computer device, which comprises a memory and a processor, the memory is used to store information comprising program instructions, and the processor is used to control the execution of the program instructions, the program instructions are loaded and executed by the processor, and each step of the embodiment of the medical evidence sorting method described above is realized, and the specific description can be referred to the embodiment of the medical evidence sorting method described above.

[0116] Figure 7 The embodiment of the present application provides a schematic diagram of a computer device. As shown in Figure 7 , the computer device 20 of the embodiment comprises a processor 21, a memory 22, and a computer program 23 stored in the memory 22 and executable on the processor 21, the computer program 23 is executed by the processor 21 to realize the evidence sorting method for medical evidence in the embodiment, to avoid repetition, which will not be described here. Alternatively, the computer program is executed by the processor 21 to realize the functions of each model / unit in the evidence sorting device for medical evidence in the embodiment, to avoid repetition, which will not be described here.

[0117] The computer device 20 comprises, but is not limited to, the processor 21 and the memory 22. Those skilled in the art can understand that Figure 7The computer device 20 is merely an example and does not constitute a limitation on the computer device 20, which can include more or fewer components than shown, or combine some components, or have different components, such as the computer device can also include an input / output device, a network access device, a bus, etc.

[0118] The processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0119] The memory 22 can be an internal storage unit of the computer device 20, such as a hard disk or a memory of the computer device 20. The memory 22 can also be an external storage device of the computer device 20, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 22 can include both an internal storage unit and an external storage device of the computer device 20. The memory 22 is used to store computer programs and other programs and data required by the computer device. The memory 22 can also be used to temporarily store data that has been output or will be output.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0121] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0122] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0123] In addition, each function unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function units.

[0124] The integrated unit implemented in the form of software function units can be stored in a computer readable storage medium. The software function unit stored in the storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a variety of media that can store program codes, such as a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

[0125] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A medical evidence-based evidence ranking method, applied to a human-computer dialogue application based on a large language model, characterized by: include: receiving a prompt for triggering medical evidence-based diagnosis input by a user in a human-computer dialogue interface of the application; Generate at least two sub-queries based on the prompt; Search each subquery, recall the evidence subset corresponding to each subquery, and form an evidence set for the prompt based on the evidence subset; A correlation operation is performed on the evidence in the evidence set based on the prompt, and the evidence in the evidence set is sorted based on the correlation operation result.

2. The method according to claim 1, characterized in that The performing a correlation operation on the evidence in the evidence set based on the prompt includes: Perform correlation calculation on the evidence in the evidence subset corresponding to each subquery, and select a set number of evidences that are most relevant to each subquery; A correlation operation is performed on the prompt and the evidence set consisting of a set number of evidences of each subquery.

3. The method according to claim 1, characterized in that The prompt includes a first number of elements, the subquery includes a second number of elements, and performing a correlation operation on the evidence in the evidence set based on the prompt includes: Perform correlation calculation on the evidence in the evidence subset corresponding to each subquery, and select a set number of evidences that are most relevant to each subquery; Determine the difference between the prompt and each of the subqueries, and generate an element set; A correlation operation is performed on the element set and an evidence set consisting of a set number of pieces of evidence for each subquery.

4. The method according to claim 1, wherein After sorting the evidence in the evidence set based on the correlation calculation result, the following steps are included: The evidence is ranked according to the determined inquiry scenario and the evidence grading rules corresponding to the inquiry scenario.

5. The method according to claim 4, characterized in that After sorting the evidence in the evidence set based on the correlation calculation result, and before sorting the evidence according to the determined inquiry scenario and the evidence grading rules corresponding to the inquiry scenario, the method includes: When the authority of the evidence in the correlation calculation result is low, the step of ranking the evidence according to the determined inquiry scenario and the evidence grading rule corresponding to the inquiry scenario is continued.

6. The method according to claim 4, characterized in that After ranking the evidence according to the determined inquiry scenario and the evidence grading rules corresponding to the inquiry scenario, the following steps are included: After the evidence is ranked, evidence of the same level will be ranked according to its disclosure time.

7. The method according to claim 4, characterized in that The inquiry scenarios and the corresponding evidence grading rules include: If the inquiry scenarios include medical research scenarios, medical teaching scenarios, and basic medical knowledge query scenarios, the evidence grading rules corresponding to the inquiry scenarios include: setting the set high-quality journals to high priority; setting the evidence of research methods including meta-analysis, systematic review, and randomized controlled trials to high priority, setting the drug instructions or off-label drug instructions to high priority, setting the evidence of research methods including non-randomized controlled trials, observational studies, and literature reviews to medium priority, setting the content of medical websites to medium priority, setting the evidence of research methods including case reports, animal experiments, and in vitro experiments to low priority, and setting the content of non-medical websites to low priority; or, If the inquiry scenario includes a medical clinical scenario, the evidence grading rule corresponding to the inquiry scenario further includes: setting the guideline to a high priority.

8. The method according to claim 1, characterized in that The generating of at least two sub-queries based on the prompt includes: Generate Cot according to the prompt based on the inference model; Generate sub-questions based on the Cot based on the second language model; The at least two sub-queries are generated according to the sub-questions based on the second language model.

9. The method according to claim 1, characterized in that The searching of each subquery, recalling the evidence subset corresponding to each subquery, and forming an evidence set for the prompt based on the evidence subset include: Based on the third language model, each subquery is searched in a self-built knowledge base and the Internet. A recall algorithm is used to recall the evidence subset corresponding to each subquery. Based on the evidence subset, an evidence set for the prompt is formed, where the knowledge base includes evidence grading rules corresponding to the inquiry scenario.

10. The method according to claim 6, characterized in that The evidence of the same level in the ranked evidence is ranked according to its disclosure time, including: Generate a reply statement corresponding to the prompt based on the sorted evidence based on the fourth language model and reasoning model. The reply statement includes evidence tags generated based on the sorted evidence. The evidence tags include the research type, evidence source, influence, and / or evidence rating of each piece of evidence. The evidence tags are used to indicate the authority of the evidence. The reply statement is displayed in the human-computer dialogue interface of the application.

11. The method according to claim 1, wherein The receiving of a prompt for triggering medical evidence-based diagnosis input by a user in a human-computer dialogue interface of the application comprises: Receiving a query statement input by a user in the human-computer dialogue interface of the application; Generate a prompt according to the query statement based on prompt engineering technology; Determining an inquiry scenario based on the prompt based on the first language model, and judging whether the inquiry scenario is a set inquiry scenario that can trigger medical evidence-based inquiry; If it is determined that the inquiry scenario is an inquiry scenario that can trigger medical evidence-based inquiry, the step of generating at least two sub-queries based on the prompt is continued.

12. A medical evidence-based evidence ranking device, used in a human-computer dialogue application based on a large language model, characterized in that: include: A receiving module, configured to receive a prompt inputted by a user in a human-computer dialogue interface of the application program to trigger medical evidence-based diagnosis; A first generation module, configured to generate at least two sub-queries based on the prompt; A second generation module is configured to search each subquery, recall an evidence subset corresponding to each subquery, and form an evidence set for the prompt based on the evidence subset; The operation and sorting module is used to perform a correlation operation on the evidence in the evidence set based on the prompt, and sort the evidence in the evidence set based on the correlation operation result.

13. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the medical evidence-based evidence ranking method according to any one of claims 1 to 11.

14. A computer device comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, wherein: When the program instructions are loaded and executed by the processor, the steps of the medical evidence-based evidence ranking method described in any one of claims 1 to 11 are implemented.

Images