A Dynamic Retrieval Enhancement Method for Online Medical Question Answering

Abstract

This invention discloses an enhanced dynamic retrieval method for online medical question-and-answer systems, relating to the field of large language model technology. By introducing semantic entropy and a dynamic threshold setting mechanism based on semantic entropy, the system determines whether to trigger a retrieval based on the semantic entropy value of the generated content. This avoids excessive retrieval operations associated with traditional static thresholding methods, improving the targeting of retrieval, enhancing the professionalism and accuracy of answers, and reducing the possibility of the model generating "illusionary" or irrelevant responses. By combining a medical-specific model (Med-BERT) for medical entity extraction, this invention can accurately identify and classify medical elements in the generated text. Based on these medical elements, a more precise basis for triggering the system's retrieval mechanism can be provided.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, specifically to a dynamic retrieval and enhancement generation method for online medical question answering. Background Technology

[0002] With the rapid development of artificial intelligence technology, online medical question-answering systems have gradually become an important way for patients to obtain medical advice and health knowledge. However, when generating answers to online medical consultations, large language models may encounter the "illusion" problem, where the generated content does not match actual medical knowledge, and sometimes even provides information completely unrelated to the patient's question. Secondly, traditional question-answering systems are prone to producing answers that deviate from the context in long dialogues or multi-turn question-and-answer sessions, resulting in unrelevant or inaccurate generated content. Summary of the Invention

[0003] To overcome the shortcomings of the above-mentioned technologies, this invention provides a method for reducing the bias and "illusion" phenomenon in the content generated by large language models in medical question-answering systems, and for improving the accuracy and coherence of the generated content. The technical solution adopted by this invention to overcome its technical problems is as follows:

[0004] A method for enhancing the dynamic retrieval and generation of online medical question-and-answer systems includes:

[0005] a) The language model was trained using a Chinese medical dialogue dataset to obtain the trained language model;

[0006] b) Input the user's question q into the trained language model N times. Each time, the trained language model outputs a candidate answer, resulting in a candidate answer set A consisting of N candidate answers. num A num ={A1,A2,...,A i ,...,A N}, A i Let A be the i-th candidate answer, i∈{1,...,N}. i ={x1,x2,...,x j ,...,x u}, x j For the i-th candidate answer A i The j-th sub-word in the equation, j∈{1,...,u}, where u is the i-th candidate answer A. i The number of neutron words;

[0007] c) Construct a dictionary of sub-word generation probabilities (DIC) based on all sub-words of all N candidate answers;

[0008] d) Based on the i-th candidate answer A i The clustering result C is obtained. final Cfinal ={C1,C2,...,C k ,...,C M}, where C k Let be the kth cluster, k∈{1,...,M}, where M is the number of clusters after clustering;

[0009] e) Based on the k-th cluster C k Obtain a representative set of medical elements

[0010] f) Based on the representative medical element set Calculate the k-th cluster C k The generation probability P of the user input question q k ;

[0011] g) Based on the generation probability P k Calculate the semantic entropy SE(q) of the generated text;

[0012] h) Trigger a retrieval based on whether the semantic entropy SE(q) of the generated text exceeds a threshold, and generate the final answer based on the retrieved knowledge base content and question q.

[0013] Furthermore, in step a), the language model is the Qwen2.5-14B-Instruct model.

[0014] Furthermore, step a) includes the following steps:

[0015] a-1) Use the Chinese Medical Dialogue Dataset as the fine-tuning dataset; a-2) Use the fine-tuning dataset to fine-tune the Qwen2.5-14B-Instruct model using LoRA to obtain the trained Qwen2.5-14B-Instruct model.

[0016] Furthermore, the temperature parameter of the trained Qwen2.5-14B-Instruct model is set to T, where 0.7 ≤ T ≤ 1.2.

[0017] Furthermore, in step c), the i-th candidate answer A is extracted from the output layer of the Qwen2.5-14B-Instruct model. i The j-th subword x j The corresponding logits are then input into the softmax function, and the j-th word x is output. j The generation probability p j The i-th candidate answer A i The generation probabilities of all u subwords constitute the i-th candidate answer A. i The generation probability set Pi P i ={p1,p2,...,p j ,...,p u The set of generation probabilities for all N candidate answers constitutes the dictionary of word generation probabilities, DIC, where DIC = {P1, P2, ..., P...}. i ,...,P N}

[0018] Furthermore, step d) includes the following steps:

[0019] d-1) The i-th candidate answer A i The input is fed into the Sentence-BERT model, and the output is a high-dimensional semantic vector V. i V i ∈R d Where R is the real number space and d is the high-dimensional semantic vector V i The dimension d takes a value of 768 or 1024;

[0020] d-2) The j-th candidate answer A j The input is fed into the Sentence-BERT model, and the output is a high-dimensional semantic vector V. j V j ∈R d , j∈{1,...,N};

[0021] d-3) Calculate the high-dimensional semantic vector V i With high-dimensional semantic vector V j Cosine similarity S between i,j Construct an N×N similarity matrix S using the cosine similarity of all high-dimensional semantic vectors. In the similarity matrix S, all elements along the diagonal from the top-left element to the bottom-right element have a value of 1.

[0022] d-4) The i-th candidate answer A i The corresponding cluster is C i Let C be the set of independent clusters corresponding to all N candidate answers, where C = {C1, C2, ..., C}. i ,...,C N};

[0023] d-5) In the similarity matrix S, find the maximum value S of all elements except those in the diagonal formed by the elements from the top left corner to the bottom right corner. w,r Maximum value S w,r For the w-th candidate answer A w High-dimensional semantic vector V w With the r-th candidate answer A rHigh-dimensional semantic vector V r The cosine similarity between them, w∈{1,...,N}, r∈{1,...,N}, w≠r;

[0024] d-6) Through formula C wr =C w ∪C r ={A w A r Calculate the w-th candidate answer A w The corresponding cluster is C w With the r-th candidate answer A r The corresponding cluster is C r The new cluster C after merging wr Remove cluster C from the independent cluster set C. w and cluster C r And add a new cluster C wr This yields a new cluster set C′; d-7) using the formula Calculate the new cluster C wr With the k-th cluster C in the independent cluster set C k Similarity S wr,k In the formula S w,k For the w-th candidate answer A w High-dimensional semantic vector V w With the k-th candidate answer A k High-dimensional semantic vector V k Cosine similarity between them, k∈{1,...,N}, k≠w, S r,k For the r-th candidate answer A r High-dimensional semantic vector V r With the k-th candidate answer A k High-dimensional semantic vector V k The cosine similarity between them, where · represents the number of corresponding candidate answers within the cluster;

[0025] d-8) Construct a new similarity matrix S′ of (N-1)×(N-1). Where S wr,wr =1;

[0026] d-9) Replace the similarity matrix S in step c-5) with the new similarity matrix S′ and then repeat step c-5) to step c-8);

[0027] d-10) Repeat step c-9) 0 times until the minimum value of all elements in the new similarity matrix, excluding all elements in the diagonal formed by the elements from the top left corner to the bottom right corner, is greater than or equal to a threshold value, which is greater than or equal to 0.7 and less than or equal to 0.8. If so, this new similarity matrix is ​​taken as the final M×M similarity matrix S.final , The final similarity matrix S final The k-th row or k-th column in the diagram represents the k-th cluster C. k k∈{1,...,M} Where A kh For the k-th cluster C k The h-th candidate answer, h∈{1,...,C} k}, C k For the k-th cluster C k The number of candidate answers, and the total number of clusters (M) constitute the clustering result C. final C final ={C1,C2,...,C k ,...,C M}

[0028] Furthermore, step e) includes the following steps:

[0029] e-1) The MedBERT model was trained using the CMeEE Chinese Medical Entity Recognition dataset to obtain the trained MedBERT model;

[0030] e-2) Use the tokenizer of the trained MedBERT model to segment the k-th cluster C k The h-th candidate answer is A kh The subwords are split into a sequence of subwords, and each subword is converted into a unique identifier, resulting in an identifier sequence A. kh ′, A kh ′={x1′,x2′,...,x i ′,...,x t ′}, where x i Let ' be the identifier of the i-th subword, i∈{1,...,t}, and t be the identifier sequence A. kh The number of neutron words;

[0031] e-3) Establish identifier sequences A for the category labels dis, sym, ite, pro, and dru in the CMeEE Chinese Medical Entity Recognition dataset. kh The dis category set of ′ sym category collection ite category collection pro category collection dru category collection identifier sequence A kh The identifier x of the i-th subword in ' iThe input is fed into the trained MedBERT model, and the output is the probability of the i-th subword being the category label dis, category label sym, category label ite, category label pro, and category label dru in the CMeEE Chinese Medical Entity Recognition dataset. This subword is then placed into the category set corresponding to the category label with the highest probability value. After placing all t subwords into their respective category sets, the h-th candidate answer A is obtained. kh Medical element set E kh ,

[0032] e-4) The k-th cluster C k All C k The sets of medical elements in each candidate answer are merged to obtain the categorized set E. k ,

[0033] For set The i-th sub-word, i∈{1,...,z1}, where z1 is a set The number of neutron words;

[0034] For set The i-th sub-word in the set, i∈{1,...,z2}, where z2 is a set. The number of neutron words;

[0035] For set The i-th sub-word, i∈{1,...,z3}, where z3 is a set The number of neutron words;

[0036] For set The i-th sub-word, i∈{1,...,z4}, where z4 is a set The number of neutron words;

[0037] For set The i-th sub-word, i∈{1,...,z5}, where z5 is a set The number of neutron words;

[0038] e-5) will set For each of the z1 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z1 generation probabilities as the set. core elements e-6) will set For each of the z2 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z2 generation probabilities as the set. core elements e-7) will set For each of the z3 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z3 generation probabilities as the set. core elements e-8) will set For each of the z4 subwords, look up the generation probability of each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z4 generation probabilities as the set. core elements e-9) will set For each of the z5 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z5 generation probabilities as the set. core elements (e-10) Establish the k-th cluster C k A collection of representative medical elements

[0039] Furthermore, step f) includes the following steps:

[0040] f-1) Core elements Corresponding generation probability, core elements Corresponding generation probability, core elements Corresponding generation probability, core elements Corresponding generation probability, core elements The generation probability is obtained by summing the corresponding generation probabilities. f-2) via formula The k-th cluster C is calculated. k The generation probability P of the user input question q k .

[0041] Furthermore, in step g), the formula is used. The semantic entropy SE(q) of the generated text is calculated.

[0042] Furthermore, step h) includes the following steps:

[0043] h-1) Set the entropy threshold SE threshold 0.5≤SE threshold ≤0.8;

[0044] h-2) If the semantic entropy SE(q) of the generated text is less than the entropy threshold SE threshold Then from the candidate answer set A num Choose any one of the candidate answers as the final answer to the user's input question q;

[0045] h-3) If the semantic entropy SE(q) of the generated text is greater than or equal to the entropy threshold SE threshold Then, a knowledge base D consisting of n medical literature articles is obtained through the Elasticsearch search engine, where D = {d1, d2, ..., dn}. i ,...,d n}, d i For the i-th medical document, i∈{1,...,n}, based on the user input question q, the Elasticsearch search engine retrieves l medical documents from the medical question knowledge base D, forming a search result set D′. The search result set D′ is then concatenated with the user input question q and input into the trained Qwen2.5-14B-Instruct model, outputting the answer A. q As the final answer to the user's input question q.

[0046] The beneficial effects of this invention are: it is applicable to dynamic retrieval enhancement generation systems, especially in the field of medical consultation. This method has significant advantages in improving model accuracy, optimizing retrieval timing, and enhancing the quality of generated text. By introducing semantic entropy and a dynamic threshold setting mechanism based on semantic entropy, the system determines whether to trigger a retrieval based on the semantic entropy value of the generated content. This avoids excessive retrieval operations caused by traditional static threshold methods, improves the targeting of retrieval, enhances the professionalism and accuracy of responses, and reduces the possibility of the model generating "illusionary" or irrelevant answers. By combining a medical-specific model (Med-BERT) for medical entity extraction, this invention can accurately identify and classify medical elements in the generated text. Based on these medical elements, it can provide a more precise basis for triggering the system's retrieval mechanism. This can effectively improve the system's responsiveness to complex medical problems and enhance the model's adaptability. Detailed Implementation

[0047] The present invention will be further described below.

[0048] A method for enhancing the dynamic retrieval and generation of online medical question-and-answer systems includes:

[0049] a) The language model was trained using a Chinese medical dialogue dataset to obtain the trained language model.

[0050] b) Input the user's question q into the trained language model N times. Each time, the trained language model outputs a candidate answer, resulting in a candidate answer set A consisting of N candidate answers. num A num ={A1,A2,...,A i ,...,A N}, A i Let A be the i-th candidate answer, i∈{1,...,N}. i ={x1,x2,…,x j ,…,x u}, x j For the i-th candidate answer A i The j-th sub-word in the equation, j∈{1,…,u}, where u is the i-th candidate answer A. i The number of neutron words.

[0051] c) Construct a dictionary of sub-word generation probabilities (DIC) based on all sub-words of all N candidate answers.

[0052] d) Based on the i-th candidate answer A i The clustering result C is obtained. final C final ={C1,C2,…,C k ,…,C M}, where C k Let be the k-th cluster, k∈{1,...,M}, where M is the number of clusters after clustering.

[0053] e) Based on the k-th cluster C k Obtain a representative set of medical elements

[0054] f) Based on the representative medical element set Calculate the k-th cluster C k The generation probability P of the user input question q k .

[0055] g) Based on the generation probability P k Calculate the semantic entropy SE(q) of the generated text.

[0056] h) Trigger a retrieval based on whether the semantic entropy SE(q) of the generated text exceeds a threshold, and generate the final answer based on the retrieved knowledge base content and question q.

[0057] In one embodiment of the present invention, the language model in step a) is the Qwen2.5-14B-Instruct model. In another embodiment of the present invention, step a) includes the following steps:

[0058] a-1) Use the Chinese Medical Dialogue Dataset as the fine-tuning dataset.

[0059] a-2) Using the fine-tuning dataset, fine-tune the Qwen2.5-14B-Instruct model using LoRA (Low-Rank Adaptation) to obtain the trained Qwen2.5-14B-Instruct model. In one embodiment of the present invention, the temperature parameter of the trained Qwen2.5-14B-Instruct model is set to T, where 0.7 ≤ T ≤ 1.2.

[0060] In the candidate answer generation process, the candidate answers are generated step by step according to the sub-word sequence. Therefore, in one embodiment of the present invention, in step c), the i-th candidate answer A is extracted from the output layer of the Qwen2.5-14B-Instruct model. i The j-th subword x j The corresponding logits are then input into the softmax function, and the j-th word x is output. j The generation probability p j The i-th candidate answer A i The generation probabilities of all u subwords constitute the i-th candidate answer A. i The generation probability set P i P i ={p1,p2,...,p j ,...,p u The set of generation probabilities for all N candidate answers constitutes the dictionary of word generation probabilities, DIC, where DIC = {P1, P2, ..., P...}. i ,...,P N}

[0061] In one embodiment of the present invention, step d) includes the following steps:

[0062] d-1) The i-th candidate answer A i The input is fed into the Sentence-BERT model (SBERT), and the output is a high-dimensional semantic vector V. i V i ∈R d Where R is the real number space and d is the high-dimensional semantic vector V i The dimension d takes the value of 768 or 1024, depending on the version of the Sentence-BERT model selected. d-2) will select the j-th candidate answer A. jThe input is fed into the Sentence-BERT model, and the output is a high-dimensional semantic vector V. j V j ∈R d , j∈{1,...,N}.

[0063] d-3) Calculate the high-dimensional semantic vector V i With high-dimensional semantic vector V j Cosine similarity S between i,j Construct an N×N similarity matrix S using the cosine similarity of all high-dimensional semantic vectors. In the similarity matrix S, all elements along the diagonal from the top left to the bottom right corner have a value of 1 (meaning the similarity between a candidate answer and itself is 1).

[0064] d-4) To classify the generated candidate answers into clusters based on semantic similarity, a hierarchical clustering method is used. Specifically, the i-th candidate answer A i The corresponding cluster is C i Let C be the set of independent clusters corresponding to all N candidate answers, where C = {C1, C2, ..., C}. i ,...,C N}

[0065] d-5) In the similarity matrix S, find the maximum value S of all elements except those in the diagonal formed by the elements from the top left corner to the bottom right corner. w,r Maximum value S w,r For the w-th candidate answer A w High-dimensional semantic vector V w With the r-th candidate answer A r High-dimensional semantic vector V r The cosine similarity between them, w∈{1,...,N}, r∈{1,...,N}, w≠r.

[0066] d-6) Through formula C wr =C w ∪C r ={A w A r Calculate the w-th candidate answer A w The corresponding cluster is C w With the r-th candidate answer A r The corresponding cluster is C r The new cluster C after merging wr Remove cluster C from the independent cluster set C. w and cluster C r And add a new cluster C wr This yields a new cluster set C′.

[0067] d-7) By formula Calculate the new cluster C wr With the k-th cluster C in the independent cluster set C k Similarity S wr,k In the formula S w,k For the w-th candidate answer A w High-dimensional semantic vector V w With the k-th candidate answer A k High-dimensional semantic vector V k Cosine similarity between them, k∈{1,...,N}, k≠w, S r,k For the r-th candidate answer A r High-dimensional semantic vector V r With the k-th candidate answer A k High-dimensional semantic vector V k The cosine similarity between them, |·| represents the number of corresponding candidate answers within the cluster.

[0068] d-8) Construct a new similarity matrix S′ of (N-1)×(N-1). Where S wr,wr =1. Due to the cluster C in the similarity matrix S w and cluster C r Replaced with the new cluster C wr Therefore, the dimension of the new similarity matrix S′ is reduced by 1.

[0069] d-9) Replace the similarity matrix S in step c-5) with the new similarity matrix S′ and repeat step c-5) (value step c-8).

[0070] d-10) Repeat step c-9) 0 times until the minimum value of all elements in the new similarity matrix, excluding all elements in the diagonal formed by the elements from the top left corner to the bottom right corner, is greater than or equal to a threshold value, which is greater than or equal to 0.7 and less than or equal to 0.8. If so, this new similarity matrix is ​​taken as the final M×M similarity matrix S. final , The final similarity matrix S final The k-th row or k-th column in the diagram represents the k-th cluster C. k k∈{1,...,M} Where A kh For the k-th cluster C k The h-th candidate answer, h∈{1,...,|C k |},|C k | is the k-th cluster C k The number of candidate answers, and the total number of clusters (M) constitute the clustering result C. final Cfinal ={C1,C2,...,C k ,...,C M}

[0071] In one embodiment of the present invention, step e) includes the following steps:

[0072] e-1) The MedBERT model was trained using the CMeEE Chinese Medical Entity Recognition Dataset to obtain the trained MedBERT model.

[0073] e-2) Use the tokenizer of the trained MedBERT model to segment the k-th cluster C k The h-th candidate answer is A kh The subwords are split into a sequence of subwords, and each subword is converted into a unique identifier, resulting in an identifier sequence A. kh ′, A kh ′={x1′,x2′,...,x i ′,...,x t ′}, where x i Let ' be the identifier of the i-th subword, i∈{1,...,t}, and t be the identifier sequence A. kh The number of 'neutron words'.

[0074] e-3) Establish identifier sequences A for the category labels dis (disease name), sym (symptom), ite (examination items), pro (treatment plan), and dru (drug recommendation) in the CMeEE Chinese Medical Entity Recognition dataset. kh The dis category set of ′ sym category collection ite category collection pro category collection dru category collection identifier sequence A kh The identifier x of the i-th subword in ' i The input is fed into the trained MedBERT model, and the output is the probability of the i-th subword being the category label dis, category label sym, category label ite, category label pro, and category label dru in the CMeEE Chinese Medical Entity Recognition dataset. This subword is then placed into the category set corresponding to the category label with the highest probability value. After placing all t subwords into their respective category sets, the h-th candidate answer A is obtained. kh Medical element set E kh ,

[0075] e-4) The k-th cluster C k All Ck The sets of medical elements in each candidate answer are merged to obtain the categorized set E. k ,

[0076] For set The i-th sub-word, i∈{1,...,z1}, where z1 is a set The number of neutron words;

[0077] For set The i-th sub-word in the set, i∈{1,...,z2}, where z2 is a set. The number of neutron words;

[0078] For set The i-th sub-word, i∈{1,...,z3}, where z3 is a set The number of neutron words;

[0079] For set The i-th sub-word, i∈{1,...,z4}, where z4 is a set The number of neutron words;

[0080] For set The i-th sub-word, i∈{1,...,z5}, where z5 is a set The number of neutron words.

[0081] e-5) will set For each of the z1 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z1 generation probabilities as the set. core elements e-6) will set For each of the z2 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z2 generation probabilities as the set. core elements e-7) will set For each of the z3 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z3 generation probabilities as the set. core elements e-8) will set For each of the z4 subwords, look up the generation probability of each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z4 generation probabilities as the set. core elements e-9) will set For each of the z5 subwords, find the generation probability corresponding to each subword in the subword generation probability dictionary (DIC). Select the subword with the highest generation probability from all z5 generation probabilities as the set. core elements (e-10) Establish the k-th cluster C k A collection of representative medical elements

[0082] In one embodiment of the present invention, step f) includes the following steps:

[0083] f-1) Core elements Corresponding generation probability, core elements Corresponding generation probability, core elements Corresponding generation probability, core elements Corresponding generation probability, core elements The generation probability is obtained by summing the corresponding generation probabilities. f-2) via formula The k-th cluster C is calculated. k The generation probability P of the user input question q k .

[0084] In one embodiment of the present invention, step g) is performed using the formula The semantic entropy SE(q) of the generated text is calculated.

[0085] In one embodiment of the present invention, step h) includes the following steps:

[0086] h-1) Set the entropy threshold SE threshold 0.5≤SE threshold ≤0.8.

[0087] h-2) If the semantic entropy SE(q) of the generated text is less than the entropy threshold SE threshold This indicates that the model's generated results are relatively consistent, and the model has a high confidence level in its own answers. Therefore, from the candidate answer set A... num Choose any candidate answer from the options as the final answer to the user's input question q.

[0088] h-3) If the semantic entropy SE(q) of the generated text is greater than or equal to the entropy threshold SE threshold The model's generated results are considered scattered and have low confidence, suggesting the need to retrieve external documents for more accurate information. Upon triggering a search, medical information related to the user's question is retrieved from a medical knowledge base. This is achieved by using the Elasticsearch search engine to obtain a knowledge base D consisting of n medical documents, where D = {d1, d2, ..., dn}. i ,...,d n}, d i For the i-th medical document, i∈{1,...,n}, based on the user input question q, the Elasticsearch search engine retrieves l medical documents from the medical question knowledge base D, forming a search result set D′. The search result set D′ is then concatenated with the user input question q and input into the trained Qwen2.5-14B-Instruct model, outputting the answer A. q As the final answer to the user's input question q.

[0089] As shown in Table 1, to verify the effectiveness of the online medical question-answering dynamic retrieval enhancement generation method of the present invention, we conducted tests with different threshold SE values. threshold The experiments focused on balancing generation quality and retrieval efficiency. The Qwen2.5-14B-Instruct model was fine-tuned on a Chinese medical dialogue dataset, and different search engine optimization (SE) metrics were evaluated. threshold BLEU, ROUGE, and Search Ratio values.

[0090] In the experiment, we set up multiple SEs. threshold For each value, detailed tests were conducted. Experimental results show that when SE... threshold At lower scores, the generated answers perform better on BLEU and ROUGE scores, and the retrieval rate is also higher. However, with the increase in SE... threshold As the value increases, the quality of the generated answers and the retrieval efficiency tend to decrease.

[0091] Table 1 Different thresholds SE threshold experiment

[0092] <![CDATA[SE threshold ]]> BLEU ROUGE (Recall) Search Ratio 0.50 0.48 0.62 16.9 0.55 0.49 0.64 15.8 0.60 0.51 0.66 14.2 0.65 0.52 0.68 13.0 0.70 0.53 0.70 12.3 0.75 0.55 0.73 11.5 0.80 0.54 0.72 10.8 0.85 0.52 0.69 9.5

[0093] In the experiment, when SE threshold At a retrieval score of 0.75, the BLEU and ROUGE scores reached 0.55 and 0.73 respectively, with a retrieval rate of 11.5, compared to other search engines. threshold The value indicates high generation quality and low retrieval requirements. With SE thresholdAs the value increases, although the generated quality decreases slightly, the retrieval rate also decreases; while when the SE value increases, the retrieval rate also decreases. threshold When the value is low, although the quality of the generated data is improved, the retrieval rate increases significantly, and the computational burden is increased.

[0094] Therefore, by selecting SE threshold =0.75, which can effectively balance generation quality and retrieval efficiency, and can provide accurate and efficient medical question-and-answer services in practical applications.

[0095] As shown in Table 2, the traditional RAG method and the method of this invention are compared in terms of generation quality. Common automatic evaluation metrics (such as BLEU, ROUGE, METEOR, etc.) are used to measure their generation performance, and the results are analyzed.

[0096] Table 2 Comparison of Generation Quality between Traditional RAG Methods and the Method of This Invention

[0097] Performance indicators Method of the present invention Traditional RAG method BLEU 0.58 0.45 ROUGE (Recall) 0.74 0.60 METEOR 0.66 0.53

[0098] Compared to traditional RAG methods, the method of this invention demonstrates superior performance across multiple performance metrics, particularly in balancing generation quality and retrieval efficiency. Specifically, the method of this invention significantly outperforms traditional RAG methods in evaluation metrics such as BLEU, ROUGE, and METEOR, improving them by 0.13, 0.14, and 0.13 respectively, indicating that the system can generate higher quality and more accurate answers.

[0099] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for enhancing the dynamic retrieval and generation of online medical question-and-answer systems, characterized in that, include: a) The language model was trained using a Chinese medical dialogue dataset to obtain the trained language model; b) User input questions The inputs are fed into the trained language model respectively. Each time, the language model outputs a candidate answer after training, resulting in... A set of candidate answers consisting of 100 candidate answers , , For the first 10 candidate answers , , For the first One candidate answer The first in individual words, , For the first One candidate answer The number of neutron words; c) According to all Construct a dictionary of subword generation probabilities for all subwords of each candidate answer. ; d) Based on the first One candidate answer Obtain clustering results , ,in For the first A cluster, , The number of clusters after clustering; e) According to the Clusters Obtain a representative set of medical elements ; f) Based on the set of representative medical elements Calculate the first Clusters In user input question Generation probability on ; g) Based on generation probability Calculate the semantic entropy of the generated text ; h) Based on the semantic entropy of the generated text Whether a threshold is exceeded to trigger a search depends on the content and question retrieved from the knowledge base. Generate the final answer; Step e) includes the following steps: e-1) The MedBERT model was trained using the CMeEE Chinese Medical Entity Recognition dataset to obtain the trained MedBERT model; e-2) Use the tokenizer of the trained MedBERT model to segment the first... Clusters The Middle One candidate answer The process involves splitting the word into a sequence of subwords and converting each subword into a unique identifier, resulting in a sequence of identifiers. ; e-3) Establish identifier sequences for the category labels dis, sym, ite, pro, and dru in the CMeEE Chinese Medical Entity Recognition dataset. dis category set sym category collection ite category collection ,pro category collection dru category collection , identifier sequence The Middle Identifier of a subword The input is fed into the trained MedBERT model, and the output is the first... Each subword represents the probability of category labels dis, sym, ite, pro, and dru in the CMeEE Chinese Medical Entity Recognition dataset. The subword is then placed into the category set corresponding to the category label with the highest probability value. After placing each subword into its corresponding category set, we get the... One candidate answer Medical elements collection ; e-4) will be the Clusters All The sets of medical elements in each candidate answer are merged to obtain the categorized set. , ; e-5) respectively set ,gather ,gather ,gather and set All the corresponding subwords in the subword generation probability dictionary are respectively... Find the generation probability of each subword in the set, and select the subword with the highest generation probability from all the generation probabilities in each set to obtain the core elements of each set. , , , , ; e-6) Establish the first Clusters A collection of representative medical elements ; Step f) includes the following steps: f-1) respectively the core elements Core elements Core elements Core elements and core elements The generation probability is obtained by summing the corresponding generation probabilities. ; f-2) through formula Calculate the first Clusters In user input question Generation probability on .

2. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 1, characterized in that: In step a), the language model is the Qwen2.5-14B-Instruct model.

3. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 2, characterized in that, Step a) includes the following steps: a-1) Use the Chinese Medical Dialogue Dataset as the fine-tuning dataset; a-2) Use LoRA to fine-tune the Qwen2.5-14B-Instruct model using the fine-tuning dataset to obtain the trained Qwen2.5-14B-Instruct model.

4. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 3, characterized in that: The temperature parameters of the trained Qwen2.5-14B-Instruct model are set as follows: , .

5. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 2, characterized in that: In step c), the first... One candidate answer The first in Sub-words The corresponding logits are then input into the softmax function, and the output is the first logits. Sub-words generation probability , No. One candidate answer All The generation probability of each subword constitutes the first... One candidate answer Generation probability set , ,all The probability set of each candidate answer constitutes a dictionary of word generation probabilities. , .

6. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 5, characterized in that, Step d) includes the following steps: d-1) will the first One candidate answer The input is fed into the Sentence-BERT model, and the output is a high-dimensional semantic vector. , ,in, For the real number space, High-dimensional semantic vector Dimensions The value can be either 768 or 1024; d-2) will the first One candidate answer The input is fed into the Sentence-BERT model, and the output is a high-dimensional semantic vector. , , ; d-3) Calculate the high-dimensional semantic vector With high-dimensional semantic vectors Cosine similarity between Construct the cosine similarity between all high-dimensional semantic vectors Similarity matrix , Similarity matrix In the diagonal line formed by the element from the top left corner to the bottom right corner, all elements have a value of 1. d-4) One candidate answer The corresponding cluster is ,all The independent cluster set corresponding to each candidate answer is: , ; d-5) in the similarity matrix Find the maximum value of all elements in the array except for all elements along the diagonal formed by the top-left element to the bottom-right element. maximum value For the first One candidate answer High-dimensional semantic vectors With the One candidate answer High-dimensional semantic vectors Cosine similarity between them , , ; d-6) By formula Calculate the first One candidate answer The corresponding cluster is With the One candidate answer The corresponding cluster is The new cluster after merging , to set up independent clusters Remove clusters and cluster And add new clusters This yields a new cluster set. ; d-7) By formula Calculate the new cluster With independent cluster sets The Middle Cluster similarity In the formula For the first One candidate answer High-dimensional semantic vectors With the One candidate answer High-dimensional semantic vectors Cosine similarity between them , , For the first One candidate answer High-dimensional semantic vectors With the One candidate answer High-dimensional semantic vectors Cosine similarity between them This indicates the number of corresponding candidate answers within the cluster; d-8) Established The new similarity matrix , ,in ; d-9) The new similarity matrix Replace the similarity matrix in step c-5) Then repeat step c-5) to step c-8); d-10) Repeat step c-9) This process continues until the minimum value of all elements in the new similarity matrix, excluding all elements along the diagonal from the top-left to the bottom-right corner, is found to be greater than or equal to a threshold value between 0.7 and 0.

8. If the minimum value is greater than or equal to 0.7 and less than or equal to 0.8, then the new similarity matrix is ​​considered as... The final similarity matrix , The final similarity matrix The first in row or number Column represents the first Clusters , , ,in For the first Clusters The Middle 10 candidate answers , For the first Clusters The number of candidate answers, all Each cluster constitutes a clustering result. , .

7. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 6, characterized in that, Step e) includes the following steps: e-1) The MedBERT model was trained using the CMeEE Chinese Medical Entity Recognition dataset to obtain the trained MedBERT model; e-2) Use the tokenizer of the trained MedBERT model to segment the first... Clusters The Middle One candidate answer The process involves splitting the word into a sequence of subwords and converting each subword into a unique identifier, resulting in a sequence of identifiers. , ,in For the first Identifier of each subword, , For identifier sequence The number of neutron words; e-3) Establish identifier sequences for the category labels dis, sym, ite, pro, and dru in the CMeEE Chinese Medical Entity Recognition dataset. dis category set sym category collection ite category collection ,pro category collection dru category collection , identifier sequence The Middle Identifier of a subword The input is fed into the trained MedBERT model, and the output is the first... Each subword represents the probability of category labels dis, sym, ite, pro, and dru in the CMeEE Chinese Medical Entity Recognition dataset. The subword is then placed into the category set corresponding to the category label with the highest probability value. After placing each subword into its corresponding category set, we get the... One candidate answer Medical elements collection , ; e-4) will be the Clusters All The sets of medical elements in each candidate answer are merged to obtain the categorized set. , , , For set The Middle individual words, , For set The number of neutron words; , For set The Middle individual words, , For set The number of neutron words; , For set The Middle individual words, , For set The number of neutron words; , For set The Middle individual words, , For set The number of neutron words; , For set The Middle individual words, , For set The number of neutron words; e-5) will set All Each subword is in the subword generation probability dictionary Find the generation probability corresponding to each subword in the middle, and select all. Select the word with the highest generation probability from the given generation probabilities as a set. core elements ; e-6) will set All Each subword is in the subword generation probability dictionary Find the generation probability corresponding to each subword in the middle, and select all. Select the word with the highest generation probability from the given generation probabilities as a set. core elements ; e-7) will set All Each subword is in the subword generation probability dictionary Find the generation probability corresponding to each subword in the middle, and select all. Select the word with the highest generation probability from the given generation probabilities as a set. core elements ; e-8) will set All Each subword is in the subword generation probability dictionary Find the generation probability corresponding to each subword in the middle, and select all. Select the word with the highest generation probability from the given generation probabilities as a set. core elements ; e-9) will set All Each subword is in the subword generation probability dictionary Find the generation probability corresponding to each subword in the middle, and select all. Select the word with the highest generation probability from the given generation probabilities as a set. core elements ; e-10) Establish the first Clusters A collection of representative medical elements , .

8. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 7, characterized in that, Step f) includes the following steps: f-1) Core elements Corresponding generation probability, core elements Corresponding generation probability, core elements Corresponding generation probability, core elements Corresponding generation probability, core elements The generation probability is obtained by summing the corresponding generation probabilities. ; f-2) through formula Calculate the first Clusters In user input question Generation probability on .

9. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 1, characterized in that: In step g), the formula is used. The semantic entropy of the generated text is calculated. .

10. The online medical question-and-answer dynamic retrieval enhancement generation method according to claim 3, characterized in that, Step h) includes the following steps: h-1) Set the entropy threshold , ; h-2) If the semantic entropy of the generated text Less than the threshold of entropy Then from the set of candidate answers Choose any candidate answer from the options provided for the user input question. The final answer; h-3) If the semantic entropy of the generated text The threshold greater than or equal to entropy Then obtain it through the Elasticsearch search engine. A knowledge base of medical consultation questions composed of medical literature. , , For the first A medical literature, Based on user input questions Using the Elasticsearch search engine in a knowledge base for medical consultation questions Searched in The search results consist of medical literature articles. , retrieve the search results set Issues with user input After the concatenation operation, the data is input into the trained Qwen2.5-14B-Instruct model, and the answer is output. As a user input question The final answer.

Images