Medical knowledge question-answering method and system based on LLM and RAG
By segmenting medical textbooks and guidelines using chapter/key point awareness strategies and hybrid retrieval technology, and combining inverse ranking fusion and cross-ranking, the problems of knowledge illusion and untraceable evidence in medical scenarios of large language models are solved, improving the accuracy and interpretability of answers. This approach is suitable for medical knowledge question-and-answer systems deployed locally and offline.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HAINAN UNIV
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-21
AI Technical Summary
Existing large language models are susceptible to knowledge illusions, lack of traceability of evidence, and privacy compliance constraints in medical scenarios. Furthermore, the hierarchical numbering of long documents from multiple sources, such as Chinese medical textbooks and guidelines, is complex, with a high proportion of tables and flowcharts and significant differences in versions and years, resulting in insufficient accuracy and interpretability of the answers.
A chapter/key point awareness strategy is used to segment data in medical textbooks and guidelines. By combining sparse and dense retrieval, and through inverse ranking fusion and cross-ranking, a structured set of segments is generated, and answers are generated using lightly constrained prompt word templates.
It improves the accuracy and interpretability of answers to medical knowledge questions, adapts to local offline deployment needs, meets data security requirements, and provides a visualized evidence tracing and evaluation process.
Smart Images
Figure CN121901394A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and medical information retrieval technology, and in particular to a medical knowledge question-answering method and system based on LLM and RAG. Background Technology
[0002] In recent years, Large Language Models (LLMs) have made progress in open-domain question answering, but in medical scenarios they are easily limited by "knowledge illusion," lack of traceability of evidence, and privacy compliance constraints. Retrieval-Augmented Generation (RAG) can compensate for the lack of parameterized memory in models with external knowledge; however, existing solutions are mostly based on general English corpora, making it difficult to adapt to long documents from multiple sources such as Chinese medical textbooks and guidelines: their hierarchical numbering (Chapter 1 / Section 1 / I, / (I) / 1.) is complex, tables and flowcharts account for a high proportion, and there are significant differences in version and year. Common problems include word-by-word segmentation which interrupts key points and leads to fragmented evidence; pure sparse or pure dense retrieval is not robust to terminology aliases, units, and negative sentences. On the other hand, clinical and educational scenarios require local offline deployment to meet data security requirements; existing evaluations mostly rely on general semantic similarity and lack standardized processes and visual attribution for medical multiple-choice question (MCQ) evaluations. Therefore, there is an urgent need for a medical RAG method in a localized environment to improve the accuracy and interpretability of answers. Summary of the Invention
[0003] To address the aforementioned technical problems, this invention proposes a medical knowledge question-answering method and system based on LLM and RAG. In this method and system, a chapter / key point awareness strategy is employed for stable block segmentation, and sparse and dense retrievals are fused and cross-ranked to improve the accuracy and interpretability of the answers.
[0004] To achieve the above objectives, the technical solution of the present invention is as follows:
[0005] A medical knowledge question-answering method based on LLM and RAG includes the following steps:
[0006] A chapter / key point awareness strategy is used to segment data in medical textbooks and guidelines to obtain a structured segment set. Each segment in the structured segment set includes the segment plain text content and a metadata set. The segment ID of each segment is a combination of file path, segment number and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year and version.
[0007] Dense and sparse indexes are constructed based on chunked plain text content to obtain dense and sparse indexes.
[0008] Obtain the query input by the front-end user, calculate the semantic similarity between the query and the blocks in the dense index, sort them in descending order, and recall the previous query. Candidate blocks; calculate the literal matching score between the query and the words in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front The candidate blocks are deduplicated according to the block ID to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block plain text content, metadata set, semantic similarity and literal matching score.
[0009] The initial candidate block set is filtered and sorted in descending order through inverse ranking fusion and cross-ranking, and the output is the first... The candidate blocks serve as the final evidence block set, and each piece of evidence in the final evidence block set includes the block plain text content, metadata set, inverse ranking fusion score, and cross-ranking score;
[0010] Based on the final evidence block set, prompt words are generated using a lightly constrained prompt word template. The final evidence block set and prompt words are then input into a large language model to generate an answer. The answer, the final evidence block set, and the evidence table corresponding to the final evidence block set are sent to the front-end user and displayed.
[0011] Preferably, the chapter / key point perception strategy includes the following:
[0012] By combining regular expressions with state machines, standard hierarchical numbers in knowledge sources are identified, and data information such as document source, chapter title, publication year, and version is extracted.
[0013] Enforce chunking rules: For extracted key text within the same section, when the length of a single key text is ≥ When this point is in the text, it is treated as a separate block; for texts within the same section that are less than [a certain length], [the text is treated as a separate block]. The remaining key text is merged sequentially using a greedy algorithm, ensuring that the total length of each merged block does not exceed [a certain value]. For continuous natural paragraphs of ordinary text that are not extracted as key points, approximately [a certain amount] is retained between adjacent blocks during the segmentation process. Overlapping sections of length are used to ensure semantic continuity; list items and their corresponding explanatory text are merged into semantically consistent blocks, of which 250 characters or more... ≥350 words; 550 words or more ≥650 characters; 35 characters or more ≥45 characters.
[0014] Preferably, the dense index construction includes the following steps:
[0015] The BGE-M3 vector model is used to embed and encode the plain text content of each block to generate block vectors;
[0016] The block vector, block ID, and metadata set are associated and written into the Chroma vector library as a dense index and persisted. At the same time, L2 normalization is performed on the block vector.
[0017] Preferably, the sparse index construction includes the following steps:
[0018] The jieba word segmentation tool was used to segment the plain text content of each block to obtain the word segmentation results;
[0019] An inverted index for the BM25 sparse retrieval model is constructed based on the word segmentation results. The word segmentation inverted index, block ID, and metadata set are associated and used as a sparse index and persisted.
[0020] Preferably, the initial candidate block set is filtered and sorted in descending order through inverse ranking fusion and cross-ranking, and the output is the first... The candidate blocks, as the final set of evidence blocks, include the following steps:
[0021] Before Candidate blocks and front Candidate blocks are subjected to inverse ranking fusion processing to obtain inverse ranking fusion scores. These scores are then sorted in reverse order, and the first few blocks are output. Initial candidate blocks;
[0022] Before The candidate blocks are input into the bge-reranker-large cross-ranking algorithm, which calculates the cross-ranking score between each filtered candidate block and the query; based on the cross-ranking score, the top candidates are ranked... Candidate blocks are sorted in descending order to obtain the top The final sorting of candidate blocks.
[0023] Preferably, a weighted sum of the cross-ranking score and the inverse ranking fusion score of each candidate block is performed, and the sum is used to rank the top... Candidate blocks are sorted in descending order to obtain the top The final sorting of candidate blocks.
[0024] Preferably, a hypothetical explanation is generated based on the query using HyDE, and the final evidence block set and prompt words corresponding to the hypothetical explanation are input into the large language model to generate an answer.
[0025] Preferably, the evaluation process is also included, comprising the following steps:
[0026] Obtain a test dataset. Each question in the test dataset includes a question ID, a question, options, and an answer. The options include several basic symbols and the corresponding option content for each basic symbol.
[0027] The final evidence block set and prompt words corresponding to the questions in the test dataset are input into the large language model to generate an evaluation answer. The prompt words are used to constrain the large language model to output the predicted basic symbol at the end of the evaluation answer. The evaluation answer containing the predicted basic symbol is used as the output.
[0028] Accuracy is set as the evaluation metric. The predicted basic symbols of the last line of the evaluation answer are identified and extracted, and compared with the basic symbols corresponding to the answers in the test dataset to calculate the evaluation accuracy.
[0029] Based on the above, this invention also discloses a medical knowledge question-answering system based on LLM and RAG, comprising:
[0030] The data layer is used to store medical textbook and guideline data;
[0031] The structured processing layer is used to segment data in medical textbooks and guidelines using a chapter / key point awareness strategy to obtain a structured segment set. Each segment in the structured segment set includes the segment plain text content and a metadata set. The segment ID of each segment is a combination of file path, segment number and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year and version.
[0032] The index layer is used to build dense and sparse indexes;
[0033] The retrieval and reordering layer is used to obtain the query input by the front-end user, calculate the semantic similarity between the query and the blocks in the dense index, sort them in descending order, and recall the previous data. Candidate blocks; calculate the literal matching score between the query and the words in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front Candidate blocks are deduplicated based on their block IDs to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block's plain text content, metadata set, semantic similarity, and literal matching score. The initial candidate block set is then filtered and sorted in descending order using inverse ranking fusion and cross-ranking, and the first result is output. The candidate blocks serve as the final evidence block set, and each piece of evidence in the final evidence block set includes the block plain text content, metadata set, inverse ranking fusion score, and cross-ranking score;
[0034] The generation and visualization layer is used to generate prompt words based on the final evidence block set using a lightly constrained prompt word template, input the final evidence block set and prompt words into a large language model to generate an answer, and send the answer, the final evidence block set, and the evidence table corresponding to the final evidence block set to the front-end user for display.
[0035] Based on the above technical solution, the beneficial effects of this invention are as follows: This invention provides a medical knowledge question-answering method and system based on LLM and RAG. The method employs a chapter / key point awareness strategy to segment data in medical textbooks and guidelines, obtaining a structured segment set. Each segment in the structured segment set includes the segment's plain text content and a metadata set. The segment ID for each segment is a combination of file path, segment number, and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year, and version. Dense and sparse indexes are constructed based on the segment's plain text content to obtain dense and sparse indexes. The query input by the front-end user is obtained, and the semantic similarity between the query and the segments in the dense index is calculated and sorted in descending order. The system then retrieves the previous query. Candidate blocks; calculate the literal matching score between the query and the words in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front Candidate blocks are deduplicated based on their block IDs to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block's plain text content, metadata set, semantic similarity, and literal matching score. The initial candidate block set is then filtered and sorted in descending order using inverse ranking fusion and cross-ranking, and the first result is output. Candidate blocks serve as the final evidence block set. Each piece of evidence in the final evidence block set includes the block's plain text content, metadata set, bottom-ranked fusion score, and cross-ranking score. Based on the final evidence block set, prompt words are generated using a lightly constrained prompt word template. The final evidence block set and prompt words are input into a large language model to generate an answer. The answer, the final evidence block set, and the corresponding evidence table are sent to the front-end user for display. This invention employs a chapter / key point awareness strategy for stable block segmentation, integrates sparse and dense retrieval, and performs cross-ranking to improve answer accuracy and interpretability. Attached Figure Description
[0036] Figure 1 This is a schematic diagram of the RAG system for applying a medical knowledge question-and-answer method based on LLM and RAG in one embodiment;
[0037] Figure 2 This is a flowchart of a medical knowledge question-answering method based on LLM and RAG in one embodiment;
[0038] Figure 3 This is a diagram showing the front-end interactive interface of the RAG system in one embodiment, where (a) is the dialogue interface, (b) is the evidence tracing interface, and (c) is a diagram showing the evidence table interface with scoring.
[0039] Figure 4 This is a schematic diagram illustrating the model evaluation results based on an open-source medical multiple-choice dataset in one embodiment.
[0040] Figure 5 This is a bar chart showing the comparative experimental results of different systems (naked model (Qwen3-4B), naive RAG, and the RAG of this invention). Detailed Implementation
[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0042] like Figure 1 , 2 As shown, this embodiment provides a medical knowledge question-answering method based on LLM and RAG. This method can run offline in a standalone graphics processing unit (GPU) environment. After running, users enter their question in the front-end dialog box, click send, and wait for a response from the RAG system. Figure 3 (a) After the RAG system completes its response, you can click on the evidence tracing interface to view the evidence, such as... Figure 3 (b) You can also view the ranking of evidence in the evidence list, such as Figure 3 (c). The specific processing steps of this method include the following:
[0043] Step 1: Use a chapter / key point awareness strategy to segment the data in medical textbooks and guidelines into blocks to obtain a structured block set. Each block in the structured block set includes the block plain text content and a metadata set. The block ID of each block is a combination of file path, block number and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year and version.
[0044] In this embodiment, for medical knowledge base files such as medical textbooks and guidelines in Markdown format, standard hierarchical numbers in the knowledge source are identified by combining regular expressions and a state machine, and data information such as file source, chapter title, publication year, and version is extracted; block segmentation rules are executed to obtain block plain text content. Compared with fixed-length block segmentation, the chapter / key point awareness strategy retains sufficient context in long paragraphs such as "contraindications / processes / hierarchy / table descriptions," significantly reducing the occurrence of "hard-cut evidence." The specific processing is as follows:
[0045] Hierarchical parsing: By combining regular expressions with a state machine, standard hierarchical numbers such as "Chapter 1 / Section 1 / I、 / (I) / 1." are identified, and metadata such as file source, chapter title, publication year, and version are extracted and saved as metadata {source, title / section, year (optional), version (optional)}.
[0046] Semantic stable chunking: Chunking rules are executed according to uniform parameters—for each extracted key text within the same section, if the length of a single key text is ≥ T1 (300 characters), that key text is treated as a separate chunk; for the remaining key texts within the same section whose length is less than T1, a greedy approach is used to merge them sequentially, and the total length of each chunk after merging does not exceed T2 (600 characters); for ordinary body text that has not been extracted as key text, an overlap of about 40 characters is retained between adjacent chunks during chunking to ensure semantic continuity; list items and their corresponding explanatory body texts are merged into semantically consistent chunks;
[0047] Block ID deduplication and consistency verification: "File path + block sequence number + text hash value" is used as the unique block ID to eliminate duplicate blocks and avoid subsequent index conflicts.
[0048] Output: A standardized set of structured chunks, each chunk consisting of {text (chunk plain text content) and metadata (metadata collection)}.
[0049] Step 2: Construct dense and sparse indexes based on the chunked plain text content to obtain dense and sparse indexes.
[0050] In this embodiment, a hybrid (dense + sparse) retrieval / recall mechanism is constructed, including a dense index (BGE-M3 vector model + Chroma persistence) and a sparse index (jieba word segmentation tool + BM25 (BestMatch25) inverted index), ensuring a one-to-one correspondence between the index and the structured chunk set. However, this is not the only option; depending on business requirements, the vector embedder and vector storage in the dense index, and the word segmentation tool and neural sparse method in the sparse index, can be replaced with other tools with equivalent functionality.
[0051] The specific steps are as follows:
[0052] Dense index construction: The BGE-M3 vector model is used to embed and encode the text of each block to generate a high-dimensional semantic vector, i.e., the block vector. The "block vector + block ID + metadata" are associated and written into the Chroma vector library and persisted. At the same time, L2 normalization is performed on the vector to facilitate subsequent cosine similarity calculation.
[0053] Sparse index construction: The jieba word segmentation tool is used to segment the text of each block. Based on the word segmentation results, the inverted index of the BM25 sparse retrieval model is constructed to obtain the word segmentation inverted table. The "word segmentation inverted table + block ID + metadata" are associated and persisted.
[0054] Index consistency check: Verify that the block IDs in the dense index and sparse index correspond one-to-one, ensuring that the metadata of the same block is consistent in both indexes, and avoiding data conflicts during retrieval.
[0055] Output results: Chroma persistent dense index (including block vector, block ID, metadata) and BM25 persistent sparse index (including tokenized inverted index, block ID, metadata).
[0056] Step 3: Obtain the query input by the front-end user, calculate the semantic similarity between the query and the blocks in the dense index, sort them in descending order, and recall the previous... Candidate blocks; calculate the literal matching score between the query and the words in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front The candidate blocks are deduplicated according to the block ID to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block plain text content, metadata set, semantic similarity and literal matching score.
[0057] In this embodiment, a user query is received, triggering a Hybrid recall (including the HyDE process). The specific processing is as follows:
[0058] Search channel execution: Ordinary question-and-answer queries directly trigger Hybrid (dense + sparse) retrieval / recall;
[0059] Candidate Block Recall: Dense retrieval calculates the cosine similarity between the "query vector" and the block vectors in the dense index using the BGE-M3 vector model before recall. (50) Candidate blocks; Sparse retrieval calculates the literal matching score between the query and the word segments in the sparse index using the BM25 sparse retrieval model, before recall. (200) candidate blocks; the formula for calculating semantic similarity is as follows:
[0060]
[0061] In the formula, Indicates query With Documents The semantic similarity score under dense index ranges from [-1, 1], with a higher score indicating greater semantic similarity. Indicates query The embedded vectors (generated by vector models such as BGE-M3) are used to characterize the semantic features of the query. Document The embedding vectors (generated by the same vector model) are used to characterize the semantic features of the document. To query the dot product of the vector and the document vector, we measure the directional similarity of the vectors in space. The L2 norm (i.e., the magnitude of the vector) of the query vector and document vector is used for normalization to ensure that similarity is not affected by vector length.
[0062] The formula for calculating the literal matching score is as follows:
[0063]
[0064] In the formula, This represents the literal match score between the query and the document under the sparse index. The higher the score, the stronger the literal match. This represents a single term in the query (obtained through jieba word segmentation). Inverse document frequency (IDF) represents the frequency of a term and is used to measure the "rarity" of a term; rare terms have a higher IDF. It represents the frequency of a term in a document (i.e., the number of times it appears in the document), and measures the "local importance" of a term within the document. This represents the word frequency saturation adjustment factor (usually set to 1.2-2.0), which controls the degree of influence of word frequency on the score and prevents the score from increasing too quickly when the word frequency is too high. This represents the document length adjustment factor (value 0-1), which fully considers the document length when it is set to 0 and ignores the document length when it is set to 1, and is used to balance the matching deviation of documents of different lengths. Indicates the actual length of the document (number of words). The average length (number of words) of all documents is the proportion of documents relative to the average length.
[0065] Candidate block aggregation and deduplication: For regular question-and-answer queries, candidate blocks retrieved via dual-channel recall are merged; for MCQ queries, all subqueries are aggregated. Candidate blocks are deduplicated by block ID, and duplicate blocks are removed.
[0066] Output: The initial candidate block set after deduplication (including block text, metadata, semantic similarity and literal matching score).
[0067] Step 4: The initial candidate block set is filtered and sorted in descending order through inverse ranking fusion and cross-ranking, and the first block is output. The candidate blocks serve as the final evidence block set, and each piece of evidence in the final evidence block set includes the block plain text content, metadata set, inverse ranking fusion score, and cross-ranking score.
[0068] In this embodiment, highly relevant evidence is filtered using RRF (Reciprocal Rank Fusion) and cross-ranking, outputting the final sorted set of evidence blocks. The specific processing is as follows:
[0069] For each candidate block, the inverse ranking fusion score is calculated according to the RRF formula, as shown below:
[0070]
[0071] Where dense represents a dense index, bm25 represents a sparse index, and m ∈ {dense index, sparse index}. For block Rank in retrieval channel m .
[0072] Rearrange input filters: Press Candidate blocks are sorted in descending order before filtering. (20 items) are divided into blocks as cross-rearrangement inputs to balance retrieval coverage and computational efficiency;
[0073] Cross-reranker fine ranking: Employs the bge-reranker-large cross-reranker to calculate the relevance score between each filtered candidate block and the user's original query, i.e., the cross-reranking score. ;
[0074] Final sort: by As the primary ranking criterion (optional linear fusion) and ,default The highest weighted subset of evidence is used to obtain the final set of evidence blocks sorted in descending order of relevance.
[0075] Output: The final set of evidence blocks sorted by relevance priority (including block text, metadata, and inverse rank fusion score). Cross-sorting of scores ).
[0076] Step 5: Based on the final evidence block set, generate prompt words using a lightly constrained prompt word template; input the final evidence block set and prompt words into a large language model to generate an answer; send the answer, the final evidence block set, and the evidence table corresponding to the final evidence block set to the front-end user and display them.
[0077] In this embodiment, based on the final evidence blocks, attributable answers are generated by Qwen3-4B and displayed on the front end as "flowing single bubble + folded evidence card", supporting evidence tracing, function switch (HyDE / dialogue memory) and batch evaluation.
[0078] Prompt Constraint Configuration: Load the light constraint prompt template – “Answer only based on the provided evidence blocks; when evidence is insufficient, explicitly reply ‘uncertain / suggest supplementing relevant information’; indicate the reference evidence sorting number at the end of the answer (according to the sorting number in the final evidence block set); medical guideline evidence must indicate the year of publication”;
[0079] Streaming Iterative Generation: Activate the TextIteratorStreamer streaming output function of the Qwen3-4B generation model. First, send the final evidence block set back to the front end in the form of folded cards (including source file icons, chapter titles, and relevance weight bars). Then, update the response text of a single chat bubble incrementally with each token.
[0080] Attribution association processing: In the text of the answer, the corresponding block is referenced by the evidence sorting number (such as "[1]" "[3]"). The front end supports clicking the evidence sorting number to expand the full text of the corresponding block;
[0081] MCQ-specific processing: The last line of the MCQ query response will be forcibly output with uppercase option letters (A / B / C / D / E) for automatic extraction of prediction results during batch evaluation;
[0082] Interactive functionality support: The front-end provides a HyDE switch (linked with back-end parameters), a dialogue memory switch (the memory is only used for understanding queries; in case of conflict with evidence, the evidence prevails), and a clear function (clears historical conversation records, the evidence panel, and the back-end memory state); it also supports switching between evidence tables, displaying blocks such as source, section, etc. Text summaries, with support for sorting and exporting.
[0083] Output: Streaming updated attributable response text, front-end visual evidence panel (folded cards based on the final evidence chunk set + exportable evidence table).
[0084] Further applications: The last letter of the MCQ answer can be extracted by a batch evaluation script, compared with the standard answer to calculate accuracy, question-by-question details and other indicators, completing the "retrieval-generation-evaluation" closed loop.
[0085] One embodiment of a medical knowledge question-answering method based on LLM and RAG also includes an evaluation process, which includes the following steps:
[0086] Obtain a test dataset. Each question in the test dataset includes a question ID, a question, options, and an answer. The options include several basic symbols and the corresponding option content for each basic symbol.
[0087] The final evidence block set and prompt words corresponding to the questions in the test dataset are input into the large language model to generate an evaluation answer. The prompt words include constraints such as "output the selected basic symbol at the end of the answer" to make the large language model output the predicted basic symbol at the end of the evaluation answer. The evaluation answer containing the predicted basic symbol is used as the output.
[0088] Accuracy is set as the evaluation metric. The evaluation script identifies and extracts the predicted basic symbols from the last line of the evaluation answer, compares them with the basic symbols of the standard answer in the test dataset, counts the number of correctly predicted questions, and calculates the evaluation accuracy.
[0089] In this embodiment, the test dataset (open-source medical multiple-choice questions (single choice, A–E)) is read through the evaluation script. Each question structure includes a question ID, question, options, and answer (qid / question / options / answer). During reasoning, the last line of the generated evaluation answer is forcibly output as a capital letter (A / B / C / D / E). The system extracts the last capital letter from the last line of the answer as the predicted option, compares it with the standard answer, and obtains "question-by-question correctness, time taken, and evidence summary," exporting a comma-separated values (CSV) file and a JSON lines (JSONL) file. For the evaluation results, see [link to relevant documentation]. Figure 4 .
[0090] experiment
[0091] Three question-answering systems—No-RAG (bare model), Naive RAG, and the RAG of this invention—were compared. No-RAG (bare model): no retrieval; Qwen3-4B directly reads the question stem and options to answer. Naive RAG: fixed-length chunking + single-channel BM25 retrieval; no cross-reordering / HyDE; the first k segments are concatenated for LLM generation (this group serves as the baseline in the specification, falling between No-RAG and the improved RAG, used to characterize the level of "only adding retrieval but without structural improvement"). The RAG of this invention: chapter / key point-aware chunking + Hybrid (dense + sparse) retrieval / recall + RRF fusion + cross-reordering; HyDE is switched on and off as needed for experiments (Note: HyDE is only used for query vectors in dense retrieval and is not directly used as evidence).
[0092] Table 1 Evaluation results of three question-answering systems
[0093]
[0094] The evaluation results show that, under the same test dataset (N=300 evaluation questions), the RAG of this invention improves upon the naked model (Qwen3-4B) by 15 percentage points and upon the naive RAG by 7.5 percentage points. (See [link to relevant documentation]). Figure 5 The RAG of this invention achieves significant, stable and interpretable improvements over both the bare model and the naive RAG; the core gain comes from the collaborative design of "chapter / key point awareness block + Hybrid (BGE-M3 + BM25) + RRF + cross-rearrangement + HyDE", and is implemented with controllable latency in an offline single-card environment.
[0095] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0096] Based on the same inventive concept, this application also provides a system for a medical knowledge question-answering method based on LLM and RAG. The solution provided by this system is similar to the solution described in the above method, and therefore will not be repeated here.
[0097] In one embodiment, a medical knowledge question-answering system based on LLM and RAG is also provided, comprising:
[0098] The data layer is used to store medical textbook and guideline data;
[0099] The structured processing layer is used to segment data in medical textbooks and guidelines using a chapter / key point awareness strategy to obtain a structured segment set. Each segment in the structured segment set includes the segment plain text content and a metadata set. The segment ID of each segment is a combination of file path, segment number and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year and version.
[0100] The index layer is used to build dense and sparse indexes;
[0101] The retrieval and reordering layer is used to obtain the query input by the front-end user, calculate the semantic similarity between the query and the blocks in the dense index, sort them in descending order, and recall the previous data. Candidate blocks; calculate the literal matching score between the query and the word segmentation in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front Candidate blocks are deduplicated based on their block IDs to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block's plain text content, metadata set, semantic similarity, and literal matching score. The initial candidate block set is then filtered and sorted in descending order using inverse ranking fusion and cross-ranking, and the first result is output. The candidate blocks serve as the final evidence block set, and each piece of evidence in the final evidence block set includes the block plain text content, metadata set, inverse ranking fusion score, and cross-ranking score;
[0102] The generation and visualization layer is used to generate prompt words based on the final evidence block set using a lightly constrained prompt word template, input the final evidence block set and prompt words into a large language model to generate an answer, and send the answer, the final evidence block set, and the evidence table corresponding to the final evidence block set to the front-end user for display.
[0103] In the above embodiments, each level of a medical knowledge question-answering system based on LLM and RAG can be implemented entirely or partially through software, hardware, or a combination thereof. Each level can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0104] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.
Claims
1. A medical knowledge question-answering method based on LLM and RAG, characterized in that, Includes the following steps: A chapter / key point awareness strategy is used to segment data in medical textbooks and guidelines to obtain a structured segment set. Each segment in the structured segment set includes the segment plain text content and a metadata set. The segment ID of each segment is a combination of file path, segment number and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year and version. Dense and sparse indexes are constructed based on chunked plain text content to obtain dense and sparse indexes. Obtain the query input by the front-end user, calculate the semantic similarity between the query and the blocks in the dense index, sort them in descending order, and recall the previous query. Candidate blocks; calculate the literal matching score between the query and the words in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front The candidate blocks are deduplicated according to the block ID to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block plain text content, metadata set, semantic similarity and literal matching score. The initial candidate block set is filtered and sorted in descending order through inverse ranking fusion and cross-ranking, and the output is the first... The candidate blocks serve as the final evidence block set, and each piece of evidence in the final evidence block set includes the block plain text content, metadata set, inverse ranking fusion score, and cross-ranking score; Based on the final evidence block set, prompt words are generated using a lightly constrained prompt word template. The final evidence block set and prompt words are then input into a large language model to generate an answer. The answer, the final evidence block set, and the evidence table corresponding to the final evidence block set are sent to the front-end user and displayed.
2. The medical knowledge question-answering method based on LLM and RAG according to claim 1, characterized in that, The chapter / key point awareness strategy includes the following: By combining regular expressions with state machines, standard hierarchical numbers in knowledge sources are identified, and data information such as document source, chapter title, publication year, and version is extracted. Enforce chunking rules: For extracted key text within the same section, when the length of a single key text is ≥ When this point is in the text, it is treated as a separate block; for texts within the same section that are less than [a certain length], [the text is treated as a separate block]. The remaining key text is merged sequentially using a greedy algorithm, ensuring that the total length of each merged block does not exceed [a certain value]. For continuous natural paragraphs of ordinary text that are not extracted as key points, approximately [a certain amount] is retained between adjacent blocks during the segmentation process. Overlapping sections of length are used to ensure semantic continuity; list items and their corresponding explanatory text are merged into semantically consistent blocks, of which 250 characters or more... ≥350 words; 550 words or more ≥650 characters; 35 characters or more ≥45 characters.
3. The medical knowledge question-and-answer method based on LLM and RAG according to claim 1, characterized in that, The dense index construction includes the following steps: The BGE-M3 vector model is used to embed and encode the plain text content of each block to generate block vectors; The block vector, block ID, and metadata set are associated and written into the Chroma vector library as a dense index and persisted. At the same time, L2 normalization is performed on the block vector.
4. The medical knowledge question-and-answer method based on LLM and RAG according to claim 1, characterized in that, The sparse index construction includes the following steps: The jieba word segmentation tool was used to segment the plain text content of each block to obtain the word segmentation results; An inverted index for the BM25 sparse retrieval model is constructed based on the word segmentation results. The word segmentation inverted index, block ID, and metadata set are associated and used as a sparse index and persisted.
5. The medical knowledge question-answering method based on LLM and RAG according to claim 1, characterized in that, The initial candidate block set is filtered and sorted in descending order through inverse ranking fusion and cross-ranking, and the output is the first... The candidate blocks, as the final set of evidence blocks, include the following steps: Before Candidate blocks and front Candidate blocks are subjected to inverse ranking fusion processing to obtain inverse ranking fusion scores. These scores are then sorted in reverse order, and the first few blocks are output. Initial candidate blocks; Before The candidate blocks are input into the bge-reranker-large cross-ranking algorithm, which calculates the cross-ranking score between each filtered candidate block and the query; based on the cross-ranking score, the top candidates are ranked... Candidate blocks are sorted in descending order to obtain the top The final sorting of candidate blocks.
6. The medical knowledge question-and-answer method based on LLM and RAG according to claim 5, characterized in that, For each candidate block, a weighted sum of the cross-ranking score and the inverse ranking fusion score is taken, and the top... Candidate blocks are sorted in descending order to obtain the top The final sorting of candidate blocks.
7. The medical knowledge question-answering method based on LLM and RAG according to claim 1, characterized in that, Based on the query, HyDE is used to generate hypothetical explanations. The final evidence block set corresponding to the hypothetical explanations and the prompt words are input into the large language model to generate an answer.
8. The medical knowledge question-answering method based on LLM and RAG according to claim 1, characterized in that, It also includes the evaluation process, which includes the following steps: Obtain a test dataset. Each question in the test dataset includes a question ID, a question, options, and an answer. The options include several basic symbols and the corresponding option content for each basic symbol. The final evidence block set and prompt words corresponding to the questions in the test dataset are input into the large language model to generate an evaluation answer. The prompt words are used to constrain the large language model to output the predicted basic symbol at the end of the evaluation answer. The evaluation answer containing the predicted basic symbol is used as the output. Accuracy is set as the evaluation metric. The predicted basic symbols of the last line of the evaluation answer are identified and extracted, and compared with the basic symbols corresponding to the answers in the test dataset to calculate the evaluation accuracy.
9. A medical knowledge question-answering system based on LLM and RAG, characterized in that, include: The data layer is used to store medical textbook and guideline data; The structured processing layer is used to segment data in medical textbooks and guidelines using a chapter / key point awareness strategy to obtain a structured segment set. Each segment in the structured segment set includes the segment plain text content and a metadata set. The segment ID of each segment is a combination of file path, segment number and text hash value. The metadata set includes one or more of the following: file source, chapter title, publication year and version. The index layer is used to build dense and sparse indexes; The retrieval and reordering layer is used to obtain the query input by the front-end user, calculate the semantic similarity between the query and the blocks in the dense index, sort them in descending order, and recall the previous data. Candidate blocks; calculate the literal matching score between the query and the words in the sparse index and sort them in descending order, before recall. Candidate blocks; the preceding Candidate blocks and front The candidate blocks are deduplicated according to the block ID to obtain an initial candidate block set. Each candidate block in the initial candidate block set includes the block plain text content, metadata set, semantic similarity and literal matching score. This is used to filter and sort the initial candidate block set in descending order through inverse ranking fusion and cross-ranking, and output the first... The candidate blocks serve as the final evidence block set, and each piece of evidence in the final evidence block set includes the block plain text content, metadata set, inverse ranking fusion score, and cross-ranking score; The generation and visualization layer is used to generate prompts based on the final evidence block set using a lightly constrained prompt word template, and input the final evidence block set and prompt words into a large language model to generate an answer; The answer, the final evidence block set, and the evidence table corresponding to the final evidence block set are sent to the front-end user and displayed.