A two-stage knowledge reasoning distillation method, system, and medium for medical question answering tasks.

Abstract

This application discloses a two-stage knowledge reasoning distillation method, system, and medium for medical question answering tasks. The method includes: using a teacher model to parse and structure the original medical question answering dataset to obtain multiple medical knowledge triples; performing a first-stage knowledge distillation on a student model to obtain a trained student model, which is then validated to obtain misclassified samples; using the teacher model to parse and reason on the misclassified sample set to obtain multiple medical reasoning quadruples; performing a second-stage knowledge distillation on the student model to conduct reasoning transfer training, resulting in a retrained student model; and deploying the retrained student model to an edge computing platform to generate medical answers and corresponding medical reasoning paths based on received natural language consultation information. This application can improve model operating efficiency and enhance the model's understanding and reasoning transparency in the face of complex inputs.

Description

Technical Field

[0001] This application relates to the field of medical question answering, and in particular to a two-stage knowledge reasoning distillation method, system, and medium for medical question answering tasks. Background Technology

[0002] In the medical field, medical question-answering systems based on large language models (LLMs) are increasingly emerging and gradually penetrating key areas such as clinical decision support, online consultation, intelligent triage, and medical information retrieval. While the development of medical question-answering systems based on large language models currently holds great potential, it still suffers from key shortcomings in areas such as model compression, deployment feasibility, result interpretability, and robustness.

[0003] (1) The models are large in size and have high deployment costs. Current mainstream large models such as GPT-4, LLaMA2-70B and Qwen-72B often have a parameter scale of billions or even tens of billions, which puts extremely high demands on computing resources, memory and energy consumption. Most of these models rely on cloud deployment and need to run on high-end GPU server clusters, and cannot be directly applied to edge scenarios with limited computing power, such as community health service centers, mobile health terminals or smart wearable devices. Especially in environments with limited basic medical conditions, where there is a lack of high computing power equipment or stable network connection, these models cannot be deployed at all, which seriously restricts their application and technology promotion.

[0004] (2) There is a serious deficiency in interpretability. Most language models output only brief text responses, lacking clear reasoning paths and causal chains, and cannot answer questions such as "where does the answer come from" or "how to infer the conclusion based on symptoms". This "black box" decision-making process is difficult to meet the needs of doctors for abductive judgment and also hinders the acceptance and adoption of model results in real-world scenarios. Some current attempts have been made to provide explanations through attention visualization, cue word intervention, etc., but these methods are limited to formal salient cues and have not yet formed a structured and traceable medical reasoning mechanism. Therefore, due to the lack of the ability to map and reconstruct causal chains, when faced with complex decisions, fuzzy descriptions, or new sample distributions, the model is prone to "knowing the answer but not the reason", and thus cannot form a closed-loop system to enhance interpretability.

[0005] (3) Due to the significant decrease in model capacity after parameter compression, the robustness of small models to real user input is also significantly reduced. Real-world medical questions are typically characterized by colloquial language, incomplete information, and logical jumps. Patients often lack standard medical expression skills, which means that the input text may contain multiple parallel symptoms, vague time descriptions, and redundant information. Large models can achieve fault tolerance by relying on their massive context modeling capabilities, but distilled small models lack the perception and error correction mechanisms for complex structures, making them prone to problems such as symptom misunderstanding, disease confusion, or logical breaks. Multiple studies have pointed out that when faced with multi-disease fusion and cross-symptom input, the prediction accuracy of distilled models drops significantly, and even the risk of misdiagnosis and misjudgment arises.

[0006] Current distillation frameworks generally lack dedicated training mechanisms for misjudgment. Once a model misjudges during validation or real-world application, its erroneous paths and logic cannot be identified, recorded, or corrected. As distillation rounds progress, this logical blind spot accumulates and may even solidify erroneous habits during training, ultimately leading to model behavior where "the output seems reasonable, but the reasoning is completely wrong." Summary of the Invention

[0007] The purpose of this application is to provide a two-stage knowledge reasoning distillation method, system, and medium for medical question answering tasks, which can improve model running efficiency and enhance the model's ability to understand complex inputs and reasoning transparency.

[0008] To achieve the above objectives, this application provides the following solution.

[0009] In one aspect, this application provides a two-stage knowledge reasoning distillation method for medical question-answering tasks, comprising the following steps.

[0010] Obtain the pre-trained teacher model.

[0011] The teacher model described above is used to perform data parsing and structuring on the original medical question-and-answer dataset to obtain multiple medical knowledge triples and divide them into training and validation sets.

[0012] The training set is input into the student model for a first-stage knowledge distillation to guide the student model to learn the knowledge expression and answering ability of the teacher model in the medical question-answering task, resulting in a student model that has completed one training session. Based on the validation set, the student model that has completed one training session is validated to obtain misjudged samples. All the misjudged samples constitute the misjudged sample set.

[0013] The teacher model is used to perform data analysis and reasoning on the misjudged sample set to obtain multiple medical reasoning quadruples; the medical reasoning quadruples include medical reasoning paths from symptoms to diagnosis.

[0014] Multiple medical reasoning quadruples are used as training data and input into the student model for two-stage knowledge distillation to perform reasoning transfer training on the student model, resulting in a student model that has completed secondary training. After being deployed to an edge computing platform, the student model that has completed secondary training generates medical answers and corresponding medical reasoning paths based on the received natural language consultation information.

[0015] Secondly, this application provides a two-stage knowledge reasoning distillation system for medical question-answering tasks, including the following modules.

[0016] The model loading module is used to deploy the student model, which has been trained twice according to the two-stage knowledge reasoning distillation method for medical question answering tasks, to the edge computing platform.

[0017] The question-answering module is used to: receive natural language medical consultation information input by the user, call the student model that has been trained twice, and generate medical answers and corresponding medical reasoning paths.

[0018] The reasoning path display module is used to: display the medical reasoning path in a structured or visual form, and receive feedback error samples.

[0019] The error correction and update module is used to: periodically collect the error samples, upload the error samples to the server, and re-execute the steps of the two-stage knowledge reasoning distillation method for medical question answering tasks to achieve incremental model learning or remote update and redeployment.

[0020] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a two-stage knowledge reasoning distillation method for medical question-answering tasks.

[0021] According to the specific embodiments provided in this application, this application has the following technical effects: This application designs a two-stage distillation and transfer architecture. Based on the teacher model, semantic parsing is performed on a large-scale original medical question-and-answer dataset to construct medical knowledge triples. These triples are then input into the student model for a first-stage knowledge distillation, guiding the student model to learn the teacher model's knowledge representation and answering abilities in medical question-and-answer tasks, resulting in a fully trained student model. This mechanism substantially enhances the student model's ability to perceive "why the answer is valid," unlike traditional soft-label distillation which relies solely on "formal transfer" through distributional imitation, achieving distillation improvement at the "knowledge level." Then, the trained student model is evaluated, automatically identifying samples with significant deviations from the standard answer or the teacher's answer as misjudged samples. In the second stage, the student model is trained based on these misjudged samples, while simultaneously introducing medical reasoning paths to simulate the reasoning process in real doctor diagnoses, significantly strengthening the model's logical generation and self-correction capabilities, and enhancing its understanding and reasoning transparency in the face of complex inputs.

[0022] By introducing a structured triplet knowledge distillation and a reasoning chain transfer mechanism driven by misjudged samples, the logical capability transfer from large models to small models is realized, solving the key bottleneck problem of existing medical question-answering systems being "compressible but not interpretable, and generative but not reasonable." It has four major technical advantages: high performance, low resource consumption, high robustness, and strong deployment adaptability, improving model running efficiency. It can be widely applied to scenarios such as primary care, remote assistance, and mobile health, and has significant engineering value and promising prospects for promoting the popularization of intelligent medical systems to terminals. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic flowchart of a two-stage knowledge reasoning distillation method for medical question-answering tasks provided in an embodiment of this application.

[0025] Figure 2 This is a simplified flowchart illustrating a two-stage knowledge reasoning distillation method for medical question-answering tasks provided in an embodiment of this application.

[0026] Figure 3 This is a diagram illustrating the overall framework of the medical question-and-answer task in the Helicobacter pylori diagnostic task.

[0027] Figure 4This is a functional flowchart of the medical question-and-answer task in the Helicobacter pylori diagnosis task.

[0028] Figure 5 This is a schematic diagram of the test results for the Helicobacter pylori diagnostic task. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] This application aims to address key technical challenges in existing medical question-answering systems, such as excessively large models, lack of logical interpretability, high misjudgment rates, and the inability of distillation to transfer reasoning capabilities. By constructing a structured knowledge distillation mechanism and a reasoning chain transfer enhancement mechanism, it not only achieves efficient compression and transfer of medical knowledge from large models but also significantly improves the logical consistency, self-correction ability, and deployment adaptability of student models in medical question-answering scenarios, providing a practical and feasible technical path for the application of intelligent medical question-answering systems on edge terminals.

[0031] This application, with the goal of building and deploying student models, demonstrates how to start from medical question-and-answer data and gradually complete the training and compression of small model reasoning capabilities through multiple stages such as structured extraction, knowledge transfer, logical reasoning training and deployment optimization.

[0032] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] In one exemplary embodiment, such as Figure 1 As shown, a two-stage knowledge reasoning distillation method for medical question-answering tasks is provided. This method is executed by a computer device, specifically by a computer device such as a terminal or server alone, or by a terminal and server together. In the embodiments of this application, it includes the following steps 101 to 105.

[0034] Step 101: Obtain the pre-trained teacher model.

[0035] Step 102: The teacher model is used to parse and structure the original medical question-and-answer dataset to obtain multiple medical knowledge triples, which are then divided into training and validation sets. Each original medical question-and-answer dataset contains natural language consultation information for a patient. The original medical question-and-answer dataset is a large-scale dataset. The teacher model is then used to extract and construct semantically structured triples from the natural language contained within the dataset.

[0036] The medical knowledge tripartite is a tripartite data structure of question-medical answer-background knowledge. The question is determined based on the natural language consultation information and typically includes descriptions of symptoms, sensations, and onset time, such as "I've been experiencing frequent acid reflux and bloating lately," representing the chief clinical complaint. The medical answer is the medical suggestion generated by the teacher model after the natural language consultation information is input. It is a standardized medical response with clinical rationality and standardized terminology, such as "possibly related to Helicobacter pylori infection." The background knowledge is the medical evidence related to the natural language consultation information and the medical suggestion, such as "Gastroscopy results indicate Hp positive." The background knowledge includes, but is not limited to, disease definitions, pathogenesis, diagnostic guidelines, examination recommendations, and treatment methods, and can be mapped to predefined medical knowledge graph entities and their relationships.

[0037] Specifically, background knowledge includes not only supporting explanations for medical answers, but also medical terminology and sentence structures related to diseases, examination methods, clinical guidelines, drug treatments, and pathogenesis. This can be understood as fragments of professional knowledge supporting medical answers. This knowledge can be mapped to existing medical knowledge graphs, such as UMLS, SNOMED CT, or self-built disease-symptom-examination-treatment entity relationship graphs, forming a machine-resolvable structured medical semantic network.

[0038] Background knowledge can be obtained through medical knowledge graph queries, literature backtracking, or large language model completion. This structured medical knowledge triple not only enables semantic reconstruction of natural language question-answering samples but also provides a semantic foundation for subsequent training.

[0039] Step 103: Input the training set into the student model for a one-stage knowledge distillation to guide the student model to learn the knowledge expression method and answering ability of the teacher model in the medical question-answering task, and obtain a student model that has been trained once; Based on the validation set, validate the student model that has been trained once to obtain misjudged samples; All the misjudged samples constitute the misjudged sample set.

[0040] In the first-stage knowledge distillation process, the medical knowledge triples output by the teacher model are transferred to the student model to achieve knowledge compression and capability retention. Specifically, the medical knowledge triples are fed into the student model as input to guide it to learn to identify question elements, match answer patterns, and establish semantic associations with medical knowledge. The goal of the student model is to learn its language expression and knowledge mapping capabilities under limited parameter resources by simulating the teacher model's ability to understand and generate triples. Based on this, a dual loss mechanism is introduced for the training of the student model.

[0041] The dual loss mechanism comprises language modeling loss and distillation loss, thereby compressing the parameter scale while preserving key reasoning capabilities. The language modeling loss is used to encourage the student model to learn the ability to generate medical answers from the question and background knowledge, thus improving the quality of generation. The distillation loss is used to encourage the student model to approximate the teacher model's generation distribution in terms of output distribution. In specific applications, the distillation loss can employ Kullback-Leibler divergence.

[0042] Through multiple rounds of training, the student model not only learns how to generate reasonable answers based on medical knowledge, but also potentially grasps the semantic dependencies between questions, answers, and background knowledge, thus acquiring a certain semantic interpretation ability. The focus of this stage is to build the student model's basic expressive and cognitive abilities, making it a vehicle for subsequent logical training.

[0043] In specific applications, based on the validation set, the student model that has been trained once is validated to obtain misjudged samples. This step evaluates the prediction performance of the student model on the validation set, compares its output with the similarity of the teacher model or the standard answer, and quantifies the model output error. Specifically, it includes the following steps (1)-(3).

[0044] (1) Input the medical knowledge triples in the validation set into the student model after one training session to obtain the model's predicted medical answers.

[0045] (2) Based on the model, predict the medical answers and the corresponding medical answers in the medical knowledge triplet, and calculate the similarity.

[0046] (3) When the similarity meets the preset conditions, the medical knowledge triplet is marked as a misjudged sample. The preset conditions are: BLEU score is less than 0.6, or answer similarity is less than 75% (the specific values ​​can be adjusted by relevant technical personnel as needed). That is, samples that deviate significantly from the standard answer are defined as misjudged samples. These misjudged samples often reflect the model's inability to adapt to implicit logic, rare combinations, or spoken expressions. They are insufficient in understanding ambiguous questions, identifying implicit causes, and performing cross-sentence reasoning. They are representative and suitable as the core sample source for subsequent reasoning enhancement. The above process does not require manual annotation, has scalability and high efficiency, and can be applied to the iterative optimization of large-scale question-and-answer data.

[0047] Step 104: The teacher model is used to perform data analysis and reasoning on the misjudged sample set to obtain multiple medical reasoning quadruples; the medical reasoning quadruples include medical reasoning paths from symptoms to diagnosis.

[0048] Specifically, the medical reasoning quadruple is a quadruple data structure of question-student answer-medical answer-medical reasoning path; wherein, the student answer is a medical answer generated by a student model that has been trained once; and the medical reasoning path is a set of medical statements with sequential logical relationships, presenting a causal reasoning link from symptoms to diagnosis.

[0049] Medical reasoning chains typically consist of at least three logical chains. A standard template for a medical reasoning chain is "symptoms (representing the patient's chief complaint, such as 'upper abdominal distension' or 'acid reflux') → examination (such as 'recommended gastroscopy' or 'recommended Helicobacter pylori testing') → diagnosis (such as 'possible gastritis' or 'recommended omeprazole')". This can be expanded as needed into a composite structure such as "symptom manifestation → recommended examination → examination results → disease diagnosis". For example, in a Helicobacter pylori infection case, a medical reasoning chain might be generated: "Patient has long-term abdominal distension → recommended gastroscopy → examination results indicate HP positive → infection suspected". This not only replicates the reasoning process of the teacher's model but also provides a clear logical training objective for the student model. This causal path significantly improves the student model's understanding of medical reasoning mechanisms, unlike traditional black-box distillation methods.

[0050] Step 105: Multiple medical reasoning quadruples are used as training data and input into the student model for two-stage knowledge distillation to perform reasoning transfer training on the student model, resulting in a student model that has completed secondary training. This process enables the student model to have interpretable reasoning ability and improves its robustness and correction ability against misjudged samples.

[0051] In the two-stage knowledge distillation process, a reasoning supervision objective is constructed for the training of the student model, and a multi-task loss function is introduced. The reasoning supervision objective is used to guide the student model to generate the corresponding medical reasoning path simultaneously when generating medical answers, so that the final output form is "answer + reasoning". The multi-task loss function includes language modeling loss, reasoning path generation loss and error correction supervision loss, thereby synergistically optimizing the student model so that its generated results are significantly improved in three dimensions: language, logic and robustness, and a balanced optimization is established between logical consistency and answer quality.

[0052] The language modeling loss measures the coherence and accuracy of the student model's text generation; the inference path generation loss measures the consistency between the medical inference path generated by the student model and the output of the teacher model; the error correction supervision loss is used to penalize or reward the improvement of misjudged samples in the first-stage knowledge distillation process of the student model, guiding the correction of misjudged patterns. Correspondingly, the multi-task loss function is: total loss = α × language modeling loss + β × inference path generation loss + γ × error correction supervision loss; where α, β, and γ are hyperparameters that adjust the weights of different tasks, satisfying α + β + γ = 1.

[0053] In practical applications, the student model needs to generate medical answers consistent with the previous stage; secondly, it needs to simultaneously generate corresponding reasoning chains, requiring semantic fluency, structural completeness, and logical coherence; finally, after identifying previously misjudged questions, the student model needs to correctly correct its output and provide reasoning support. This multi-objective optimization not only improves the question-answering accuracy of the student model but also enhances its reasoning consistency and interpretability, thereby significantly improving the common problem of "distortion after compression" in small models.

[0054] In one specific application, both the teacher model and the student model are large language models; the ratio of the number of parameters in the student model to the number of parameters in the teacher model is less than a preset percentage; the teacher model is a large language model with billions of parameters, and the models used include, but are not limited to, Qwen-72B, LLaMA2-70B, and ChatGLM3-130B, which have rich medical text reasoning capabilities; the student model is a compressed lightweight language model with no more than 20% of the number of parameters in the teacher model, and the models used include, but are not limited to, LLaMA2-7B and Qwen-7B, which have edge deployment adaptability.

[0055] Due to the significant reduction in the number of model parameters, its inference efficiency is significantly improved, enabling it to run on mobile devices, consultation terminals, or clinic servers. Furthermore, the student model possesses structured dual-output capabilities, providing the inference chain along with the answer, allowing doctors or users to review the model's reasoning process and enhance their confidence in the results.

[0056] The retrained student model is deployed to an edge computing platform with limited computing power. This platform includes, but is not limited to, edge servers, embedded diagnostic tools, or mobile medical devices. The retrained student model offers the following advantages: efficient question-answering response (inference latency less than 250ms), visualized inference chain (supporting logical tracing of each diagnostic output), and fast loading and automatic updates (facilitating low-cost, long-term operation in primary healthcare scenarios). After deployment to the edge computing platform, the retrained student model generates medical answers and corresponding medical inference paths based on received natural language diagnostic information, ensuring reliable, real-time, and interpretable intelligent medical question-answering services in the edge environment.

[0057] like Figure 2 As shown, the specific strategies, methods, and logical relationships of each step in the two-stage knowledge reasoning distillation method are further explained in a straightforward manner, as follows: S1, generating structured medical knowledge triples; S2, performing structured knowledge distillation on the student model; S3, after completing the initial distillation training, identifying misjudged samples from the student model; S4, using the teacher model to perform re-reasoning on these misjudged samples, generating medical reasoning quadruples including the question, student answer, correct answer (i.e., medical answer), and medical reasoning path; S5, performing reasoning transfer training on the student model based on the above medical reasoning quadruples; S6, deploying the trained student model to an edge computing platform to complete model delivery. Experiments show that this model can complete question-answering tasks with a response time of less than 250ms in real-world application scenarios, while maintaining high stability and robustness.

[0058] In summary, this application, through a systematic two-stage design, introduces reasoning transfer and multi-task training mechanisms based on knowledge distillation, effectively realizing the compressibility, interpretability, and deployability of knowledge from large-scale medical language models, providing a practical technical solution for building medical question-answering systems for real-world scenarios.

[0059] To further illustrate the technical solution of this application, several typical implementation methods are described in detail below in conjunction with specific task scenarios.

[0060] In a specific application example, focusing on the question-and-answer diagnostic scenario for Helicobacter pylori infection, this study utilizes an open medical question-and-answer dataset to extract user questions related to stomach discomfort, acid reflux, and gastroscopy. The large language model Qwen-72B (Instruct) is then used as the teacher model for semantic extraction and reasoning generation. Figure 3The diagram illustrates the entire workflow from preprocessing raw medical question-and-answer data, extracting structured triples, first-stage knowledge distillation, second-stage reasoning transfer training, to deploying the model on an edge computing device; the specific steps are as follows:

[0061] (1) Construct a structured triple containing the question, the medical answer, and background knowledge, for example:

[0062] Question: "I have been experiencing upper abdominal pain recently. Do I need to get tested for Helicobacter pylori?"

[0063] Medical answer: "A C13 breath test is recommended."

[0064] Background information: "Helicobacter pylori infection can cause symptoms such as stomach pain and indigestion. The C13 breath test is a commonly used detection method."

[0065] These medical knowledge triples were input into the teacher model for text vectorization and structural semantic modeling, and then used as training data for the student model. The student model is a medium-sized model with approximately 7 bytes of parameters.

[0066] (2) In the first-stage knowledge distillation, a joint training strategy of language modeling and output distribution fitting is used to enable the student model to learn the knowledge expression methods and answering abilities of the teacher model in medical question-answering tasks. After training, as shown... Figure 5 As shown, the Top-1 accuracy of the model in the Helicobacter pylori diagnosis subtask increased from 22.4% of the original model to 26.2%, indicating that the student model trained using the method of this application has a continuously improving accuracy across multiple evaluation rounds, outperforming the traditional model.

[0067] (3) Complete reasoning transfer training in the two-stage knowledge distillation process. For example... Figure 4 As shown, the internal logical relationships of key modules such as triple construction, misjudged sample screening, inference path generation, and multi-task training are further refined. For misjudged samples, the teacher model is invoked to generate a complete inference path as a supervision signal for the next round of training. For example, for a mispredicted sample, the inference path given by the teacher model might be: "Upper abdominal pain → common symptoms of Helicobacter pylori infection → breath test → positive result → recommended use of amoxicillin and clarithromycin combination therapy". The student model needs to generate this logical chain simultaneously while answering the output. After training, the Top-1 accuracy further improved to 27.3%, and the output results showed a clear causal structure and logical consistency, significantly enhancing the explanatory power.

[0068] As can be seen from the above embodiments, this application has significant practical effects and technological advancements in the Helicobacter pylori diagnosis task. By constructing structured medical triplet inputs and introducing inference paths as supervision signals, the student model effectively transferred the teacher model's answering ability after the first-stage distillation, improving the Top-1 accuracy by 3.8 percentage points. After further enhancing the model's causal inference ability through inference transfer training, the accuracy further improved to 27.3%, demonstrating the enhancing effect of inference chain supervision on the model's logical consistency.

[0069] In one specific application embodiment, for the comprehensive diagnosis of gastrointestinal diseases, the approach is extended to a multi-disease comprehensive consultation task. A parallel question-and-answer dataset in Chinese and English covering twelve common digestive system diseases is constructed, including gastritis, gastric ulcer, indigestion, and irritable bowel syndrome. The two-stage knowledge reasoning distillation method of this application is applied for model training, where the teacher model is an LLaMA2-70B and the student model is an INT8-quantized 6B model. After the first stage of knowledge distillation, the model already possesses strong cross-disease answering capabilities and can associate key symptoms with diagnostic suggestions. After introducing reasoning chain training in the second stage of knowledge distillation, the model can output a clearly structured medical logical expression chain, for example:

[0070] Question: "I feel bloated, have a decreased appetite, and frequently experience acid reflux."

[0071] The student model answered: "It might be gastritis; an endoscopy is recommended."

[0072] The medical reasoning path output by the student model is: "bloating + acid reflux + poor appetite → gastric mucosal irritation → common manifestations of gastritis".

[0073] The student model was evaluated and found to have improved the average accuracy of 4.6% on twelve disease seed tasks. It also demonstrated strong generalization and interpretability by outputting "symptom combination - examination suggestion - diagnosis conclusion".

[0074] As can be seen from the above embodiments, this application demonstrates good task generalization ability in multi-disease gastrointestinal consultation scenarios. On a comprehensive question-answering dataset covering twelve digestive system diseases, the student model, without adding additional parameters, achieves an average accuracy improvement of 4.6% compared to the traditional distillation method, and can output a logical path of "symptom combination - examination suggestion - diagnosis conclusion," significantly enhancing the clinical interpretability of the output. This indicates that inference chain transfer is not only applicable to single-disease tasks, but can also improve the stability and comprehension ability of the student model in multi-disease and multilingual mixed input scenarios.

[0075] In a specific application embodiment, this application selects an edge computing platform as the deployment target to test the trained student model, focusing on its edge deployment and operational performance. A lightweight inference framework and local caching mechanism, combined with a Web API interface, are used to achieve offline medical question-answering capabilities. Deployment results show that the student model's initialization loading time is no more than 6 seconds, and the average inference time per round of question answering is less than 250 milliseconds, supporting real-time interactive scenarios. Through an embedded front-end interface, the model can not only provide standard answers but also synchronously display the inference path, such as: "symptom input → examination suggestion → disease judgment → treatment suggestion." Doctors can manually verify and edit the model's inference process, further enhancing security and reliability. Furthermore, this application supports operation without internet access, making it suitable for low-resource environments such as remote areas, community clinics, and mobile medical terminals, offering significant advantages such as flexible deployment, efficient inference, rapid response, and transparent structure.

[0076] As can be seen from the above embodiments, in terms of edge deployment, the number of parameters in the student model trained by this application is approximately one-fifth that of the teacher model, exhibiting high compression and execution efficiency. After deployment on a terminal with 8GB of video memory, the model loading time is less than 6 seconds, and the inference response time is less than 250 milliseconds, meeting the dual requirements of low latency and offline operation for mobile healthcare and remote consultation. Simultaneously, by displaying the model's inference path in real time through the front-end system, doctors can review and intervene in the model's diagnostic logic, further improving the credibility and practical usability of the medical question-and-answer system.

[0077] In summary, this application is applicable to medical question-answering model compression and reasoning enhancement in edge deployment scenarios. The proposed two-stage reasoning distillation method achieves the organic unity of knowledge compression, logic transfer and interpretable output while maintaining high generation quality. It is significantly better than existing simple output distillation methods or lightweight question-answering models, and has important engineering practical value and promotion prospects.

[0078] This application's two-stage structure not only forms a progressive enhancement mechanism in the technical path but also constructs a three-dimensional output capability of "answer + reasoning + error correction" at the model behavior level, significantly outperforming traditional question-answering systems that only output answers. This application significantly improves the accuracy, explanatory scores, and misjudgment correction capabilities of the small model in multiple medical question-answering tasks (such as disease diagnosis, symptom assessment, and drug recommendation). Especially when deployed on edge devices with 8GB of video memory, the model loading speed is more than 30% faster than the traditional BERT baseline, and the average response time per round of inference is less than 250 milliseconds, fully meeting the timeliness requirements of primary care and mobile healthcare scenarios.

[0079] Especially given the current trend towards lightweight design and edge deployment, there is a pressing need for a technical approach that balances performance compression with logical reasoning transfer. This application addresses this issue by resolving key technical challenges in existing medical question-answering systems, such as model redundancy, lack of logical interpretability, high misjudgment rates, and the inability of distillation to transfer reasoning capabilities. By constructing a structured knowledge distillation mechanism and a reasoning chain transfer enhancement mechanism, it not only achieves efficient compression and transfer of medical knowledge from large models but also significantly improves the logical consistency, self-correction capabilities, and deployment adaptability of student models in medical question-answering scenarios. This provides a practical technical path for the application of intelligent medical question-answering systems on edge terminals.

[0080] Based on the same inventive concept, this application also provides a two-stage knowledge reasoning distillation system for medical question-answering tasks to implement the above-described method. The solution provided by this system is similar to the solution described in the above-described method; therefore, the specific limitations in one or more system embodiments provided below can be found in the limitations of the method described above, and will not be repeated here. This application's system includes:

[0081] The model loading module is used to deploy the secondary-trained student model, obtained using the two-stage knowledge reasoning distillation method described above for medical question answering tasks, to an edge computing platform. The secondary-trained student model for medical question answering can simultaneously output standardized medical answers and corresponding medical reasoning paths when receiving natural language question input, and possesses the following capabilities: automatically determining whether the input requires additional reasoning chains; generating clearly structured and logically complete reasoning chains for ambiguous, redundant, or semantically missing inputs; and outputting a combination of "answer + reasoning process" for easy review and verification by doctors. Specifically, the trained student model can be initialized and loaded on GPUs, NPUs, or embedded devices.

[0082] The question-answering module is used to: receive natural language medical consultation information input by the user, call the student model that has been trained twice, and generate medical answers and corresponding medical reasoning paths.

[0083] The reasoning path display module is used to: display the medical reasoning path in a structured or visual form, support doctors to make quick judgments, and receive feedback on error samples.

[0084] The error correction and update module is used to: periodically collect the error samples and misjudged samples, upload the error samples and misjudged samples to the server, and re-execute the steps of the two-stage knowledge reasoning distillation method for medical question answering tasks to achieve incremental model learning or remote update and redeployment.

[0085] Specifically, the misjudged samples are obtained by validating the student model after one training session based on the validation set after the first stage of knowledge distillation. All misjudged samples constitute the misjudged sample set, which reflects the low-confidence samples that occur during the operation.

[0086] In one specific application embodiment, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described two-stage knowledge reasoning distillation method for medical question-answering tasks.

[0087] In summary, this application provides a method for medical question-answering model compression and inference enhancement for resource-constrained environments. This method comprises two stages: structured knowledge distillation and inference chain transfer. Through this strategy, while maintaining a diagnostic accuracy of ≥90% for multiple diseases, the number of model parameters is compressed by more than 80%, and a traceable medical inference chain is explicitly output, significantly improving interpretability and robustness. This application can run in real-time on edge devices with 8GB of video memory (single inference latency <250ms), meeting the application needs of primary healthcare and mobile terminals.

[0088] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0089] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0090] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A two-stage knowledge reasoning distillation method for medical question-answering tasks, characterized in that, The method includes: Obtain the pre-trained teacher model; The teacher model described above is used to perform data parsing and structured processing on the original medical question-and-answer dataset to obtain multiple medical knowledge triples and divide them into training and validation sets. The training set is input into the student model for a first-stage knowledge distillation to guide the student model to learn the knowledge representation and answering ability of the teacher model in the medical question-answering task, resulting in a student model that has completed one training cycle. Based on the validation set, the student model that has completed one training cycle is validated to obtain misjudged samples. All the misjudged samples constitute the misjudged sample set. The teacher model is used to perform data analysis and reasoning on the misjudged sample set to obtain multiple medical reasoning quadruples; the medical reasoning quadruples include medical reasoning paths from symptoms to diagnosis; Multiple medical reasoning quadruples are used as training data and input into the student model for two-stage knowledge distillation to perform reasoning transfer training on the student model, resulting in a student model that has completed secondary training. After being deployed to an edge computing platform, the student model that has completed secondary training generates medical answers and corresponding medical reasoning paths based on the received natural language consultation information. One original medical question-and-answer data point in the original medical question-and-answer dataset includes a patient's natural language consultation information; The medical knowledge triplet is a triplet data structure of question-medical answer-background knowledge; wherein, the question is determined based on the natural language consultation information; the medical answer is the medical advice generated by the teacher model after the natural language consultation information is input into the teacher model; and the background knowledge is the medical evidence information related to the natural language consultation information and the medical advice. The medical reasoning quadruple is a quadruple data structure of question-student answer-medical answer-medical reasoning path; wherein, the student answer is a medical answer generated by a student model that has been trained once; the medical reasoning path is a set of medical statements with sequential logical relationships, presenting a causal reasoning link from symptoms to diagnosis.

2. The two-stage knowledge reasoning distillation method for medical question-answering tasks according to claim 1, characterized in that, The method further includes: In the first-stage knowledge distillation process, a dual loss mechanism is introduced for training the student model; wherein, the dual loss mechanism includes language modeling loss and distillation loss; the language modeling loss is used to enable the student model to learn the ability to generate medical answers from questions and background knowledge, so as to improve the generation quality; the distillation loss is used to enable the student model to approximate the generation distribution of the teacher model in terms of output distribution.

3. The two-stage knowledge reasoning distillation method for medical question-answering tasks according to claim 1, characterized in that, The method further includes: In the two-stage knowledge distillation process, an inference supervision objective is constructed for the training of the student model, and a multi-task loss function is introduced. The reasoning supervision objective is used to guide the student model to generate corresponding medical reasoning paths simultaneously when generating medical answers; the multi-task loss function includes language modeling loss, reasoning path generation loss, and error correction supervision loss. The language modeling loss is used to measure the coherence and accuracy of the student model in text generation; the reasoning path generation loss is used to measure the consistency between the medical reasoning path generated by the student model and the output of the teacher model; the error correction supervision loss is used to punish or reward the improvement of the student model in the process of first-stage knowledge distillation, and guide the correction of misjudgment patterns.

4. The two-stage knowledge reasoning distillation method for medical question-answering tasks according to claim 3, characterized in that, The multi-task loss function is: Total loss = α × Language modeling loss + β × Inference path generation loss + γ × Error correction and supervision loss; Where α, β, and γ are hyperparameters that adjust the weights of different tasks, satisfying α+β+γ=1.

5. The two-stage knowledge reasoning distillation method for medical question-answering tasks according to claim 1, characterized in that, Based on the validation set, the student model trained once is validated to obtain misclassified samples, including: The medical knowledge triples from the validation set are input into the student model after one training cycle to obtain the model's predicted medical answers; Based on the model, predict the medical answers and the corresponding medical answers in the medical knowledge triplet, and calculate the similarity; When the similarity meets the preset conditions, the medical knowledge triplet is marked as a misjudged sample.

6. The two-stage knowledge reasoning distillation method for medical question-answering tasks according to claim 1, characterized in that, The background knowledge includes disease definition, pathogenesis, diagnostic guidelines, examination recommendations, and treatment methods.

7. The two-stage knowledge reasoning distillation method for medical question-answering tasks according to claim 1, characterized in that, Both the teacher model and the student model are large language models; the ratio of the number of parameters in the student model to the number of parameters in the teacher model is lower than a preset percentage; The teacher model uses one of the following models: Qwen-72B, LLaMA2-70B, and ChatGLM3-130B; the student model uses one of the following models: LLaMA2-7B and Qwen-7B. The edge computing platform is an edge server, an embedded diagnostic tool, or a mobile medical device.

8. A two-stage knowledge reasoning distillation system for medical question-answering tasks, characterized in that, The system includes: The model loading module is used to: deploy the student model, which has been trained twice, to the edge computing platform by the two-stage knowledge reasoning distillation method for medical question answering tasks according to any one of claims 1-7; The question-answering module is used to: receive natural language medical consultation information input by the user, call the student model that has been trained twice, and generate medical answers and corresponding medical reasoning paths; The reasoning path display module is used to: display the medical reasoning path in a structured or visual form, and receive feedback error samples; The error correction and update module is used to: periodically collect the error samples and misjudged samples, upload the error samples and misjudged samples to the server, and re-execute the steps of the two-stage knowledge reasoning distillation method for medical question answering tasks according to any one of claims 1-7, so as to realize incremental learning of the model or remote update and redeployment.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the two-stage knowledge reasoning distillation method for medical question-answering tasks as described in any one of claims 1-7.

Images