Medical question-answering system based on multi-agent iterative retrieval enhanced generation

Abstract

The invention provides a medical question-answering system based on multi-agent iterative retrieval enhanced generation, which comprises a retrieval module used for extracting a plurality of keywords from a medical question of a user and screening out a plurality of segments of retrieval contents from a database according to the plurality of keywords; the multi-agent module is used for extracting effective retrieval information based on the multiple sections of retrieval contents, splitting the medical question into a plurality of sub-questions when the effective retrieval information is not extracted, calling the retrieval module to obtain retrieval contents related to the sub-questions, and synthesizing the effective retrieval information based on the retrieval contents of the sub-questions; and the generation module is used for generating answers based on a predefined prompt template according to the medical problems, the effective retrieval information and the internal knowledge of the large language model, and the answers comprise sources of answer contents. According to the method, complex questions can be decomposed, answers containing clear evidence sources are generated, and the accuracy and credibility of the answers are enhanced.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical information technology, specifically to the field of medical applications of large language models, and more specifically to a medical question-answering system based on multi-agent iterative retrieval enhancement generation. Background Technology

[0002] In the medical field, the application of large language models (LLMs) has brought new avenues for medical information retrieval and clinical decision support. For example, some systems utilize LLMs to quickly generate summaries of medical evidence and assist doctors in preliminary diagnostic analysis. However, existing LLM-based medical applications have many problems.

[0003] From a structural and technical perspective, LLM (Limited Learning Model) relies on statistical regularities in static training data to generate medical responses. This results in outdated information, a lack of deep understanding and accurate judgment of medical knowledge, and an inability to incorporate the latest medical guidelines and evidence, thus failing to reflect the latest advancements in the medical field. When dealing with complex medical problems, evidence content or sources are often fabricated, indicating unreliable evidence. Studies show that over 60% of the evidence summaries generated contain fictitious or erroneous associations, potentially leading to incorrect clinical decisions by physicians. Existing technologies, such as some medical question-answering systems, directly use unoptimized LLM without effective improvements tailored to the specific needs of the medical field. This often results in responses lacking sufficient medical expertise and accuracy, failing to meet the demands of complex clinical scenarios.

[0004] Therefore, existing medical question-answering systems directly use unoptimized LLM when applying it, without effectively improving it to address the specific needs of the medical field. This results in generated answers that often lack sufficient medical expertise and accuracy, as well as unreliable evidence.

[0005] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0006] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a medical question-answering system based on multi-agent iterative retrieval enhancement generation.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] According to a first aspect of the present invention, a medical question-answering system based on multi-agent iterative retrieval and enhanced generation is provided. The system includes a retrieval module, a multi-agent module, and a generation module. The retrieval module is used to extract multiple keywords from a user-submitted medical question based on a preset keyword extraction method, and to filter multiple segments of search content related to the medical question from a database based on these keywords. The multi-agent module is used to generate reference information to assist the generation module in generating answers, including valid search information. Valid search information is obtained by extracting multiple segments of search content, or, if no valid search information is extracted, the medical question is split into several sub-questions, and the retrieval module is invoked to obtain search content related to each sub-question. Valid search information is then synthesized based on the search content of each sub-question. The generation module is used to generate answers based on a predefined prompt template, the medical question, valid search information, and the internal knowledge of a large language model, including the traceable source of the answer content.

[0009] In some embodiments of the present invention, the multi-agent module includes: a first agent, used to extract effective retrieval information based on multiple segments of retrieval content using a large language model; a second agent, used to determine, when the first agent cannot extract effective retrieval information, whether its own internal knowledge is sufficient to accurately answer the medical question using a large language model; and a third agent, used to decompose the medical question into several sub-questions using a large language model when the second agent's judgment result is negative, call the retrieval module to obtain retrieval content related to each sub-question, and synthesize effective retrieval information based on the retrieval content of each sub-question.

[0010] In some embodiments of the present invention, the reference information further includes fact consistency indication information, wherein the multi-agent module further includes: a fourth agent, used to use a large language model to check whether the facts between the generated answer and the search content of the retrieval module are consistent, to obtain fact consistency indication information, and to call the generation module to regenerate the answer when the indication information indicates that the facts are inconsistent.

[0011] In some embodiments of the present invention, the multi-agent module adopts a combination of a first agent, a second agent, a third agent, and a fourth agent; wherein, the first agent is used to extract effective search information based on multiple search contents using a large language model; the second agent is used to determine whether its internal knowledge is sufficient to accurately answer the medical question when the first agent cannot extract effective search information; the third agent is used to decompose the medical question into several sub-questions using a large language model when the second agent's judgment result is negative, call the search module to obtain search contents related to each sub-question, and synthesize effective search information based on the search contents of each sub-question; the fourth agent is used to check whether the facts between the generated answer and the search contents of the search module are consistent using a large language model, obtain fact consistency indication information, and call the generation module to regenerate the answer when the indication information indicates fact inconsistency.

[0012] In some embodiments of the present invention, the retrieval module includes a keyword extraction agent, which is used to extract a preset number of keywords from medical questions using a preset keyword extraction method and to evaluate the importance of each keyword. The retrieval module is used to filter search content related to medical questions from the database based on all keywords extracted by the keyword extraction agent to obtain search results. When the search results are empty, the retrieval module iteratively searches in the following manner until the search results are not empty: removing the keyword with the lowest importance from all current keywords to obtain a set of remaining keywords, wherein the first removal is based on all keywords extracted by the keyword extraction agent, and subsequent removals are based on the previous set of remaining keywords; and filtering search content related to medical questions from the database based on the set of remaining keywords to obtain search results.

[0013] In some embodiments of the present invention, the system further includes a database module for connecting to a database; wherein the database includes online databases and custom local databases, the online databases include PubMed databases and Bing databases, and the local databases include OMIM databases, OncoKB databases, Textbooks databases, Guidelines databases and set medical literature or combinations thereof.

[0014] In some embodiments of the present invention, the database module is also used to periodically acquire the latest medical information and update the data in the local database according to the latest medical information.

[0015] According to a second aspect of the present invention, a medical question-answering method based on the system implementation of the first aspect of the present invention is provided, comprising: using a retrieval module to extract multiple keywords from a user-submitted medical question based on a preset keyword extraction method; filtering multiple segments of search content related to the medical question from a database based on the multiple keywords; using a multi-agent module to generate reference information to assist a generation module in generating an answer, including valid search information, wherein valid search information is obtained based on the extraction of multiple segments of search content, or, if no valid search information is extracted, the medical question is split into several sub-questions and the retrieval module is called to obtain search content related to each sub-question; and valid search information is synthesized based on the search content of each sub-question; and using a generation module to generate an answer based on a predefined prompt template, according to the medical question, valid search information, and the internal knowledge of a large language model, wherein the answer content includes a traceable source.

[0016] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the method steps of a second aspect of the present invention by executing the executable instructions.

[0017] Compared with the prior art, the advantages of the present invention are as follows:

[0018] First, the retrieval module of this invention initially retrieves multiple segments of relevant search content from the database based on keywords in the medical question. The multi-agent module then extracts effective search information from the search content, resulting in more accurate and useful medical information. Second, the multi-agent module can decompose complex medical questions for iterative retrieval, improving the quality of the retrieved information and thus enhancing the accuracy and professionalism of the subsequently generated answers. Finally, the generation module generates answers that include the source of the content, enhancing the credibility of the answers and facilitating user verification and traceability of the information. In particular, this invention combines medical information from the database with the internal knowledge of LLM to generate answers, ensuring that the answers reflect the latest advancements in the medical field and improving the real-time nature of the answers. Attached Figure Description

[0019] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0020] Figure 1 This is a schematic diagram illustrating the structural principle of a medical question-and-answer system according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of a medical question-and-answer method according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0023] As mentioned in the background section, existing medical question-answering systems suffer from several problems when applying LLM (Limited Learning Model). Because LLM learns from fixed, static training data, it suffers from untimely updates to medical information. Furthermore, directly using unoptimized LLM without effective improvements tailored to the specific needs of the medical field often results in generated answers that lack sufficient medical expertise and accuracy, as well as unreliable evidence.

[0024] While conducting research on medical artificial intelligence, the inventors discovered that existing LLM-based medical question-answering technologies suffer from unreliable evidence, insufficient professional knowledge, and untimely information due to limitations in their training data, lack of effective retrieval mechanisms, and inadequate understanding of medical knowledge. Therefore, the inventors found that adding database retrieval to medical question-answering systems and providing effective retrieval mechanisms and answer sources can overcome these problems.

[0025] Based on the above research and analysis, this paper proposes a medical question-answering system based on multi-agent iterative retrieval and enhanced generation, comprising a retrieval module, a multi-agent module, and a generation module. First, the retrieval module initially retrieves multiple segments of relevant search content from the database based on keywords in the medical question. The multi-agent module then extracts effective search information from the search content. This retrieval mechanism filters out more accurate and useful medical information. Second, when effective search information is not extracted, the multi-agent module also decomposes the medical question into multiple sub-questions. Based on the search content related to each sub-question obtained by the retrieval module, effective search information is synthesized. This allows the invention to decompose complex medical questions for iterative retrieval, improving the quality of search information and thus enhancing the accuracy and professionalism of the subsequently generated answers. Finally, the generation module generates answers including the source of the content based on the medical question, effective search information, and the internal knowledge of the LLM (Local Management Model). Clearly defined sources of evidence enhance the credibility of the answers, facilitate user verification and traceability of information, and improve the reliability of the evidence. In response to the problem that information in answers is not timely due to the internal knowledge of LLM learned solely from training data, this invention combines medical information in the database with the internal knowledge of LLM to generate answers, ensuring that the answers reflect the latest developments in the medical field and improving the real-time nature of the answers.

[0026] According to one embodiment of the present invention, see Figure 1This is a schematic diagram illustrating the structural principle of a medical question-and-answer system. The system includes a database module, a retrieval module, a multi-agent module, and a generation module. To better understand this invention, specific embodiments are described in detail below for each module.

[0027] I. Database Module

[0028] Existing LLM systems, which learn from fixed, static training data, cannot acquire the latest medical knowledge in real time and thus fail to reflect the latest medical information. Therefore, according to one embodiment of the present invention, a database module for connecting to databases is constructed in a medical question-and-answer system. This module connects to online databases and custom local databases. The online databases include PubMed and Bing databases, while the local databases include OMIM, OncoKB, Textbooks, Guidelines, and designated medical literature or combinations thereof. The online databases obtain the latest real-time data by calling API interfaces, while the local databases encode natural language text into vector database format for data storage. This embodiment achieves at least the following beneficial technical effects: users can customize the content and structure of the local database according to their needs, adding specific medical literature or data to meet personalized search and question-and-answer requirements. Simultaneously, the database provides rich and real-time information, generating answers based on search content selected from the database, ensuring the timeliness of medical information in the answers.

[0029] According to one embodiment of the present invention, the database module is further configured to periodically acquire the latest medical information and update the data in the local database based on the latest medical information. The updating method includes adding, deleting, and modifying relevant knowledge in the local database. The technical solution of this embodiment can achieve at least the following beneficial technical effects: the local database is set up in an easy-to-operate manner for users, who can easily add, delete, and modify relevant knowledge in the local database to build a personalized knowledge base. Furthermore, the latest knowledge can be retrieved as a supplement, thereby improving the timeliness of the large language model's answers. By combining the database and the internal knowledge of the LLM for question answering, compared to relying solely on the LLM for answers, the present invention enables the final answer to reflect the latest advancements in the medical field.

[0030] II. Search Module

[0031] According to one embodiment of the present invention, the retrieval module is connected to the database module. The retrieval module is used to extract multiple keywords from the medical questions raised by the user based on a preset keyword extraction method, and to filter out multiple segments of search content related to the medical questions from the database based on the multiple keywords.

[0032] According to one embodiment of the present invention, the retrieval module includes a keyword extraction agent, which is used to extract a preset number of keywords from medical questions using a preset keyword extraction method, and to evaluate the importance of each keyword. For example, an automatic keyword extraction agent (KEA) based on a large language model can extract a specific number of keywords from medical questions according to their importance from high to low using a large language model. The retrieval module employs a keyword extraction degradation algorithm (KED) to implement the above-mentioned keyword extraction method and to filter multiple segments of search content related to medical questions from the database based on multiple keywords. The retrieval module uses the KED algorithm to filter multiple segments of search content related to medical questions and uses the filtered multiple segments of search content as the search results.

[0033] According to one embodiment of the present invention, the retrieval module obtains retrieval results using the KED algorithm by: filtering search content related to medical issues from the database based on all keywords extracted by the keyword extraction agent, and obtaining retrieval results. When the retrieval results are empty, the retrieval module iteratively searches in the following manner until the retrieval results are not empty:

[0034] A degradation strategy is employed to remove the least important keywords from all current keywords, resulting in a set of remaining keywords. Initially, all keywords extracted by the keyword extraction agent are removed; subsequent removals are based on the previous set of remaining keywords. The remaining keyword set is then used to filter search results related to medical questions from the database, yielding the search results. This embodiment achieves at least the following beneficial effects: the more keywords there are, the stricter the search requirements, and the fewer results are retrieved. The search module of this invention is primarily optimized for searching long text questions. For example, a text question may contain dozens of keywords, making it likely that no relevant content will be found. The KED algorithm iteratively searches by removing one least important keyword at a time until the search results are not empty, thus maximizing the accuracy of the search while ensuring that relevant content is retrieved.

[0035] According to one embodiment of the present invention, when the retrieval module filters search results related to medical issues from the database based on all current keywords, it uses a semantic vector similarity encoding method to select the top k most relevant search results from the large amount of information in the database module. When k is 0, it indicates that the search results are empty, and the keywords need to be subtracted using the above method. Based on all remaining keywords, the top k most relevant search results are then selected again, and this process is repeated one or more times. When k is not 0, the database module outputs the top k most relevant search results to the retrieval module, and the retrieval module sends the top k most relevant search results to the multi-agent module.

[0036] Indicatively, such as Figure 1 As shown, KEA extracts five keywords, ordered in descending order of importance: K1, K2, K3, K4, and K5. If no results are found using these five keywords, a degradation strategy is employed to remove the least important keyword, K5, and the search continues using K1, K2, K3, and K4. If the search results are empty, the degradation strategy is again used to remove keyword K4, and the search continues using K1, K2, and K3. For example, consider the medical question: "What are the effects of combining antibiotics and immunotherapy?". First, the KEA search module extracts several keywords from the medical question: "antibiotics," "immunotherapy," and "combination therapy." Then, the search module connects these three keywords using AND and searches the database to obtain the search results. If no results are found, the degradation strategy in the KED algorithm is used, which involves removing the least important keyword and searching the database based on the remaining keywords. This process is repeated until results are found.

[0037] III. Multi-agent Module

[0038] According to one embodiment of the present invention, a multi-agent module is used to generate reference information to assist the generation module in generating answers, including effective retrieval information. Valid retrieval information is obtained by extracting multiple segments of retrieval content, or, if no effective retrieval information is extracted, the medical question is split into several sub-questions and the retrieval module is called to obtain retrieval content related to each sub-question. Valid retrieval information is then synthesized based on the retrieval content of each sub-question.

[0039] According to one embodiment of the present invention, the multi-agent module employs a multi-agent iterative retrieval enhancement algorithm to obtain effective retrieval information. The multi-agent iterative retrieval enhancement algorithm is mainly implemented through the collaborative efforts of four agents or a combination thereof:

[0040] The first agent (also known as the Passage Relevant Agent, or PRA for short) is used to extract effective search information based on multiple search segments using a large language model.

[0041] The second agent (also known as the self-knowledge agent, or SKA for short) is used to determine whether its own internal knowledge is sufficient to accurately answer the medical question when the first agent cannot extract effective retrieval information.

[0042] According to one embodiment of the present invention, the determination method for whether an answer is sufficient to provide an accurate answer includes: determining whether the coverage of the answer is greater than a coverage threshold, determining whether the accuracy of the answer is greater than an accuracy threshold, and / or determining whether the hallucination rate of the answer is less than a hallucination rate threshold. Specifically, SKA determines that the large language model itself is sufficient to independently and accurately answer the medical question only when all of the above conditions are met simultaneously or any two of them are met, and in this case, the answer is directly output through the large language model.

[0043] The third agent (also known as the Question Decomposition Agent, or QDA for short) is used to decompose the medical question into several sub-questions using a large language model when the second agent's judgment result is negative. It then calls the retrieval module to obtain the retrieval content related to each sub-question and synthesizes effective retrieval information based on the retrieval content of each sub-question.

[0044] The fourth agent (also known as the Passage Coherence Agent, or PCA for short) is used to use a large language model to check whether the generated answer is consistent with the facts retrieved by the retrieval module, obtain fact consistency indication information, and call the generation module to regenerate the answer when the indication information indicates that the facts are inconsistent.

[0045] According to one embodiment of the present invention, the multi-agent module can also be a combination of a first agent, a second agent, a third agent, and a fourth agent. That is, the multi-agent module includes four agents: PRA, SKA, QDA, and PCA. The multi-agent iterative retrieval enhancement algorithm is implemented through these four agents. In this case, the reference information generated by the multi-agent module includes valid retrieval information and factual consistency indication information. The multi-agent module can also include only three agents: PRA, SKA, and QDA. These three agents implement the multi-agent iterative retrieval enhancement algorithm. It should be understood that the present invention is not limited in this respect. Parameters can be set to select one or more agents. When dealing with complex problems, QDA can be selected to decompose the complex problem, and PRA, SKA, and / or PCA can be selected to collaborate with QDA to extract valid retrieval information. When dealing with simple problems, only PRA can be selected to directly extract valid retrieval information, without further decomposing the problem into sub-problems.

[0046] The technical solutions of the above embodiments can achieve at least the following beneficial technical effects: the activation of intelligent agents and the collaboration methods of multiple intelligent agents can be adjusted according to different application scenarios. When dealing with some simple problems, some intelligent agents can be reduced, thereby simplifying the collaboration process of intelligent agents and improving the efficiency of answering; when dealing with complex problems, QDA can be added to decompose complex problems for iterative processing, improving the quality of retrieved information, strengthening the collaboration and information sharing among intelligent agents, and improving the accuracy of answers. Furthermore, users can flexibly select the required intelligent agents according to their needs, thereby saving practical and computing resources.

[0047] It should be noted that all intelligent agents in this invention utilize a large language model to implement their functions for a specific task. For example, the first intelligent agent's specific task-specific function involves constructing task prompts using a specific prompt template combined with multiple search segments obtained from the retrieval module. These prompts are then input into the large language model, which extracts valid search information to achieve the first intelligent agent's function and complete the task. Similarly, the second intelligent agent's judgment process uses its corresponding prompt template to instruct the large language model to output specific content. For instance, it tells the model that if the answer is accurate, it outputs "yes," and if it is insufficient, it outputs "no."

[0048] IV. Generation Module

[0049] According to one embodiment of the present invention, the generation module is used to generate an answer based on a predefined prompt template, the medical question, valid retrieval information, and the internal knowledge of a large language model, including the traceable source of the answer content.

[0050] According to one embodiment of the present invention, the answer content generated by the generation module is presented in a format with traceable evidence sources. That is, the answer content includes one or more descriptions, each description corresponding to evidence citations and sources that generated that description. Figure 1 As shown, the answer generated by the module is in the following format:

[0051] {Description 1}[1][2];{Description 2}[1][3][4];{Description 3}[5];

[0052] Ref 1, Ref 2, Ref 3, Ref 4, Ref 5.

[0053] In this embodiment, descriptions 1, 2, and 3 all belong to the answer content. Each description corresponds to evidence citation and evidence source, such as reference [1] and reference [2] cited in description 1. The source of each of reference [1] and reference [2] is generated in the answer, and the source of reference [1] and reference [2] is represented as Ref 1 and Ref 2, respectively. The technical solution of this embodiment can achieve at least the following beneficial technical effects: by generating the corresponding evidence traceability format, the phenomenon of evidence illusion in the answer is effectively avoided, thereby improving the evidence validity and reliability of medical Q&A, and also facilitating users to verify and trace the information.

[0054] According to one embodiment of the present invention, taking the construction of a multi-agent module using four agents—PRA, SKA, QDA, and PCA—as an example, the execution process of the system of the present invention is described in full as follows:

[0055] First, when a medical question arises, the search module uses the KED algorithm to filter the database from the database module to obtain search results that include multiple search terms.

[0056] Subsequently, the PRA agent is invoked to extract valid search information based on multiple search segments using a large language model. If no valid search information is extracted, the SKA agent is invoked to determine whether the LLM can independently answer the question and output an answer that meets the overall preset quality standard. If the LLM can answer, the generation module directly uses the LLM to generate an answer based on the medical question. If it cannot answer, the QDA agent is invoked to break the question down into simpler sub-questions; such as... Figure 1 As shown, the problem is broken down into subproblem 1, subproblem 2, and subproblem 3;

[0057] Then, for each sub-question, the KED algorithm is used to extract and retrieve keywords by calling the retrieval module, and sub-retrieval paragraph 1, sub-retrieval paragraph 2 and sub-retrieval paragraph 3 corresponding to each sub-question are obtained, and effective retrieval information is synthesized based on the retrieval content of each sub-question;

[0058] Finally, the generation module regenerates an evidence-traceable format answer based on the knowledge retrieved from these sub-questions. At this point, the PCA agent checks whether the facts in the answer generated by the generation module are consistent with the facts in the retrieval module. If they are inconsistent, it means that the answer content conflicts with the facts, and the generation module will regenerate the answer. If they are consistent, the final answer is output, and the answer content is described as: Some studies have shown that vitamins may disrupt the balance of the gut microbiota, thereby negatively affecting the effect of immunotherapy. The evidence cited for this description is [1]...

[0059] According to one embodiment of the present invention, a medical question-answering method based on the system implementation of the above embodiment is provided. See also Figure 2 This is a flowchart illustrating a medical question-and-answer method. It includes: S1. Using a retrieval module, multiple keywords are extracted from the user's medical question based on a preset keyword extraction method. Based on these keywords, multiple segments of search content related to the medical question are then filtered from the database. S2. A multi-agent module generates reference information to assist the generation module in generating the answer, including valid search information. Valid search information is obtained based on multiple segments of search content, or, if no valid search information is extracted, the medical question is broken down into several sub-questions, and the retrieval module is called to obtain search content related to each sub-question. Valid search information is then synthesized based on the search content of each sub-question. S3. Using a generation module, an answer is generated based on a predefined prompt template, the medical question, valid search information, and the internal knowledge of a large language model. The answer includes traceable sources.

[0060] To verify the beneficial effects of the present invention, the inventors conducted the following experiments:

[0061] 1. The system of this invention (also known as Ling Shu, or LINS for short) is compared with the state-of-the-art (SOTA) results on six publicly available objective evaluation datasets: MedMCQA, PubMedQA*, MedQA-U, MedQA-M, MedQA-T, and Genetruing-disease. The SOTA results represent the optimal method for each dataset. Ling Shu achieved accuracies of 83.17%, 82.20%, 92.31%, 94.58%, 90.14%, and 86.66% on the six datasets, respectively. The accuracies of the optimal methods corresponding to the SOTA results on the previous six datasets were 79.10%, 69.00%, 78.63%, 91.10%, 87.47%, and 84.00%, respectively. These experimental results demonstrate that Ling Shu outperforms previous SOTA methods on all six datasets. This indicates that the system of this invention can effectively improve the accuracy and professionalism of the answers.

[0062] The references for the optimal methods corresponding to the state-of-the-art results on the six datasets are shown below:

[0063] The optimal approach corresponding to MedMCQA is: Can Generalist Foundation Models Outcompete Special-Purpose Turing? Case Study in Medicine.

[0064] The optimal method corresponding to PubMedQA* is: Benchmarking Retrieval-Augmented Generation for Medicin.

[0065] The optimal method corresponding to MedQA-U is: Capabilities of Gemini Models in Medicine.

[0066] The optimal method corresponding to MedQA-M is: Capabilities of GPT-4 on Medical Challenge Problems.

[0067] The optimal method corresponding to MedQA-T is: Capabilities of GPT-4 on Medical Challenge Problems.

[0068] The optimal approach for treating gene-turing disease is: Gene Turing tests, GPT models, and ingenomics.

[0069] 2. Retrieval experiments were conducted on the PubSearch dataset using both the retrieval module of this invention and direct retrieval methods. The KED algorithm used in the retrieval module of this invention achieved a retrieval accuracy of 89.3%, while the direct retrieval accuracy was 87.3%, meaning that the accuracy of question retrieval on the PubSearch dataset was improved by 2% compared to direct retrieval. For long questions, the retrieval accuracy of this invention was 92.8%, while the accuracy of direct retrieval was 52.8%, meaning that the accuracy of retrieval for long questions was improved by 40%. These experimental results demonstrate that the retrieval module of this invention can not only retrieve accurate, key, and highly relevant content information, but also ensures that important results are not missed when dealing with complex and lengthy medical questions, thus improving the comprehensiveness of the retrieval.

[0070] 3. The answers generated by Ling Shu and presented in a format with traceable evidence sources were evaluated by professional physicians. The evaluation results showed that Ling Shu can effectively assist physicians in evidence-based medicine practice on 87.00% of the questions, and can also help patients interpret medical orders on 90.09% of the questions.

[0071] Based on the above embodiments and experimental data, the beneficial effects achieved by the present invention include at least the following:

[0072] 1) Enhancing the evidentiary validity of medical question answering: Compared to pure large language models, which are prone to illusions and generate a large number of fabricated or forged evidence or connections, Ling Shu retrieves effective retrieval information through the KED algorithm in the retrieval module and the multi-agent iterative retrieval enhancement algorithm in the multi-agent module. Based on the effective retrieval information, it generates answers in a traceable evidence format, avoiding the phenomenon of evidence illusion in the answers, thereby enhancing the evidentiary validity of medical question answering.

[0073] 2) Enhance medical professionalism: By searching for professional medical papers or evidence-based guidelines, the large language model can be provided with professional and effective knowledge to help answer questions and inspire it to answer questions from a more professional perspective, thereby improving the medical professionalism of the answers.

[0074] 3) Improve information timeliness: The responses of large language models may be limited by the training data deadline, resulting in insufficient timeliness. Ling Shu can retrieve the latest knowledge from the database to supplement the internal knowledge of large language models, thereby improving the timeliness of their responses.

[0075] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0076] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0077] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0078] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A medical question-answering system based on multi-agent iterative retrieval and enhanced generation, characterized in that, The system includes a retrieval module, a multi-agent module, and a generation module; The retrieval module is used to extract multiple keywords from the medical questions raised by users based on a preset keyword extraction method, and then filter out multiple segments of search content related to the medical questions from the database based on the multiple keywords. The multi-agent module is used to generate reference information to assist the generation module in generating answers, including effective retrieval information. Specifically, effective retrieval information is extracted based on multiple segments of retrieval content, or when no effective retrieval information is extracted, the medical question is broken down into several sub-questions and the retrieval module is called to obtain retrieval content related to each sub-question. Effective retrieval information is then synthesized based on the retrieval content of each sub-question. The generation module is used to generate answers based on predefined prompt templates, medical questions, valid retrieval information, and the internal knowledge of a large language model, including the traceable source of the answer content.

2. The system according to claim 1, characterized in that, The multi-agent module includes: The first intelligent agent is used to extract effective search information based on multiple search segments using a large language model. The second intelligent agent is used to determine whether its internal knowledge is sufficient to accurately answer the medical question when the first intelligent agent is unable to extract valid retrieval information. The third agent is used to decompose the medical problem into several sub-problems using a large language model when the second agent's judgment result is negative. It then calls the retrieval module to obtain the retrieval content related to each sub-problem and synthesizes effective retrieval information based on the retrieval content of each sub-problem.

3. The system according to claim 2, characterized in that, The reference information also includes fact consistency indication information, wherein the multi-agent module further includes: The fourth agent is used to use a large language model to check whether the generated answer is consistent with the facts retrieved by the retrieval module, obtain fact consistency indication information, and call the generation module to regenerate the answer when the indication information indicates that the facts are inconsistent.

4. The system according to claim 1, characterized in that, The multi-agent module adopts a combination of a first agent, a second agent, a third agent, and a fourth agent; Among them, the first intelligent agent is used to extract effective search information based on multiple search segments using a large language model; The second intelligent agent is used to determine whether its internal knowledge is sufficient to accurately answer the medical question when the first intelligent agent is unable to extract valid retrieval information. The third intelligent agent is used to decompose the medical problem into several sub-problems using a large language model when the judgment result of the second intelligent agent is negative. It calls the retrieval module to obtain the retrieval content related to each sub-problem and synthesizes effective retrieval information based on the retrieval content of each sub-problem. The fourth agent is used to use a large language model to check whether the generated answer is consistent with the facts retrieved by the retrieval module, obtain fact consistency indication information, and call the generation module to regenerate the answer when the indication information indicates that the facts are inconsistent.

5. The system according to claim 1, characterized in that, The retrieval module includes a keyword extraction agent, which is used to extract a preset number of keywords from medical questions using a preset keyword extraction method, and to evaluate the importance of each keyword; The retrieval module is used to extract all keywords extracted by the intelligent agent and filter search results related to medical issues from the database to obtain search results. When the search results are empty, the retrieval module iterates the search in the following manner until the search results are not empty: Remove the least important keyword from all current keywords to obtain the remaining keyword set. The first removal is based on all keywords extracted by the keyword extraction agent, and subsequent removals are based on the previous remaining keyword set. Based on the remaining keyword set, search results are obtained by filtering search terms related to medical issues from the database.

6. The system according to any one of claims 1-5, characterized in that, The system also includes a database module for connecting to the database; The databases include online databases and custom local databases. The online databases include PubMed and Bing databases, while the local databases include OMIM, OncoKB, Textbooks, Guidelines, and a set of medical literature or a combination thereof.

7. The system according to claim 6, characterized in that, The database module is also used to periodically acquire the latest medical information and update the data in the local database based on the latest medical information.

8. A medical question-answering method based on the system implemented according to any one of claims 1-7, characterized in that, include: The retrieval module uses a preset keyword extraction method to extract multiple keywords from the medical questions raised by users, and then filters out multiple search results related to the medical questions from the database based on these keywords. The multi-agent module is used to generate reference information for the auxiliary generation module to generate answers, including effective retrieval information. Specifically, effective retrieval information is extracted based on multiple segments of retrieval content. Alternatively, if no effective retrieval information is extracted, the medical question is broken down into several sub-questions and the retrieval module is called to obtain retrieval content related to each sub-question. Effective retrieval information is then synthesized based on the retrieval content of each sub-question. The generation module uses a predefined prompt template to generate answers based on medical questions, valid retrieval information, and the internal knowledge of a large language model. The answers include traceable sources of content.

9. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method of claim 8.

10. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of claim 8 by executing the executable instructions.

Images