Clinical diagnosis and treatment text-oriented large model increment training and domain knowledge distillation method

Abstract

The invention discloses a clinical diagnosis and treatment text-oriented large model incremental training and domain knowledge distillation method. The method comprises the steps of constructing a clinical diagnosis and treatment text data set, processing data by adopting a desensitization mechanism combining differential privacy and federal learning, completing structured labeling based on a standard term system, and grading data through model fitting difficulty scores; based on a parameter hierarchical isolation and dynamic replay buffer strategy, clinical new knowledge is injected according to a concept layer, a data layer and a task layer, and incremental training is executed; configuring a teacher-student model architecture containing a clinical term enhancement layer, designing a multi-objective joint loss function, and optimizing a lightweight student model through task-aware multi-course distillation; based on a clinical exclusive quantitative evaluation system and an expert feedback mechanism, a dynamic optimization closed loop is constructed, and model parameters and a training strategy are iteratively adjusted. The privacy security and data quality can be guaranteed, the incremental updating efficiency is remarkably improved, the lightweight model performance is broken through, and the clinical suitability is continuously optimized.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of natural language processing and medical artificial intelligence. Specifically, it is a large-scale model optimization technique for the characteristics of clinical diagnosis and treatment text data. This technique enables large models to efficiently absorb dynamically updated clinical knowledge, while constructing lightweight models through domain-adaptive knowledge distillation. It is suitable for medical information scenarios such as structured analysis of electronic medical records, clinical decision support, intelligent question answering of diagnosis and treatment texts, and medical knowledge retrieval. It is especially suitable for application environments with large data volumes and rapid knowledge updates, such as tertiary hospitals and regional medical centers. Background Technology

[0002] With the deepening application of artificial intelligence technology in the medical field, Large Language Models (LLMs) have demonstrated significant value in clinical diagnostic text processing due to their powerful text understanding and generation capabilities. For example, LLM-based electronic medical record analysis systems can automatically extract key information such as disease diagnoses and treatment plans, assisting medical staff in improving work efficiency; clinical decision support models can provide doctors with medication suggestions and differential diagnosis references based on diagnostic texts. However, the practical application of LLMs in clinical diagnostic text processing scenarios still faces two major technical bottlenecks, which severely restrict the realization of their actual value.

[0003] The first core bottleneck is the contradiction between high model update costs and severe knowledge forgetting. Clinical diagnosis and treatment texts have typical dynamic characteristics: on the one hand, a single tertiary hospital can generate tens of millions of electronic medical records, diagnosis and treatment records, and other textual data annually. Moreover, the discovery of new disease subtypes, revisions to diagnostic criteria (such as ICD coding updates), and iterations of treatment plans (such as the application of new targeted drugs) lead to continuous updates of clinical knowledge, requiring the model to absorb new knowledge in a timely manner. On the other hand, clinical data is strictly constrained by regulations such as the "Medical Data Security Management Standard" and the "Personal Information Protection Law," with extremely high data privacy requirements, making it impossible to conduct large-scale centralized training like general text data. Traditional batch learning models require retraining the model based on all historical data and new data, which is not only computationally expensive—training an LLM with 1 billion parameters requires 100+ GPU hours of computing power—but also has a long training cycle (usually 3-7 days), making it difficult to meet the needs of rapid updates in clinical knowledge. More seriously, batch learning is prone to "catastrophic forgetting" of the model, that is, after learning new knowledge, the model's mastery of old clinical knowledge (such as classic disease diagnosis logic and traditional treatment plans) decreases significantly.

[0004] The second core bottleneck is the contradiction between the high computational demands of large models and the need for lightweight deployment in clinical settings. General-purpose LLMs typically have billions to hundreds of billions of parameters, and their inference process consumes substantial computational resources, with inference response times often exceeding 500ms. This makes it difficult to meet the real-time response requirements in clinical scenarios. For example, when doctors review electronic medical records, the model needs to return key information extraction results within 100-200ms. To address this issue, existing technologies often employ knowledge distillation methods, transferring knowledge from large models (teacher models) to lightweight models with fewer parameters (student models). However, existing knowledge distillation techniques have significant limitations in clinical diagnostic text scenarios: First, they neglect the accuracy requirements of clinical terminology. General distillation methods only focus on semantic similarity of text and do not specifically optimize for professional terms such as disease names, drug dosages, and surgical names, resulting in a 20%-30% decrease in the accuracy of student models in clinical terminology recognition compared to teacher models. Second, they lack robustness guarantees under adversarial perturbations. Perturbations such as typos (e.g., misspellings of "lung Ca" and "lung Ca") and differences in abbreviations (e.g., "heart attack" and "myocardial infarction") in clinical texts can easily lead to serious problems such as diagnostic logic breaks and missed contraindications in lightweight models. Third, they do not incorporate clinical diagnostic logic. The distillation process only focuses on matching the probability distribution of the model output and does not consider the completeness of the clinical reasoning chain of "symptoms → examination → diagnosis → treatment." As a result, although student models have fewer parameters, they cannot reproduce the clinical reasoning ability of teacher models, greatly reducing their practical application value.

[0005] In summary, existing technologies cannot simultaneously meet the clinical needs of efficient model updates, accurate knowledge retention, and lightweight and reliable deployment. There is an urgent need for an integrated technical solution that combines incremental training and domain-adaptive knowledge distillation to solve the challenges of implementing LLM in clinical diagnostic text processing scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a method for incremental training of large models and domain knowledge distillation for clinical diagnosis and treatment texts, so as to solve the problems raised in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for incremental training of large models and domain knowledge distillation for clinical diagnosis and treatment texts, comprising the following steps:

[0008] S01. Construct a clinical diagnosis and treatment text dataset, process the data using a desensitization mechanism combining differential privacy and federated learning, complete structured annotation based on a standard terminology system, and classify the data levels by model fitting difficulty score;

[0009] S02. Based on parameter hierarchical isolation and dynamic replay buffering strategies, new clinical knowledge is injected into three levels: concept layer, data layer, and task layer, and incremental training is performed.

[0010] S03. Configure a teacher-student model architecture with a clinical term enhancement layer, design a multi-objective joint loss function, and optimize the lightweight student model through task-aware multi-round curriculum distillation;

[0011] S04. Based on a clinical-specific quantitative evaluation system and an expert feedback mechanism, construct a dynamic optimization closed-loop to iteratively adjust model parameters and training strategies.

[0012] Preferably, the desensitization mechanism in step 1 is specifically as follows: perform k-anonymization (k≥10) on patient identification information, and generate 768-dimensional feature vectors for the core content of diagnosis and treatment through local preprocessing by federal nodes; the structured annotation is based on the ICD-11 coding and SNOMED CT terminology system, and uses the BIOES specification to annotate 5 types of entities such as diseases and symptoms and extract 8 types of entity-relationship-entity triples; the formula for the model fitting difficulty score is:

[0013] MFD = 0.3×term matching degree + 0.4×logical chain integrity + 0.3×entity recall rate, and accordingly divide the data into three levels: MFD≤0.4 (easy), 0.4<MFD<0.7 (medium), and MFD≥0.7 (difficult).

[0014] Preferably, the parameter hierarchical isolation in step 2 divides the model into a general basic layer (the first 12 layers of Transformer and the basic word embedding matrix) and a domain adaptation layer (the last 6 layers of Transformer and the medical-specific word embedding matrix). The incremental training only updates the parameters of the adaptation layer, and the learning rate is set to 2e-5; the initial capacity of the dynamic replay buffer pool is 50,000, and historical data is sampled in a 3:4:3 ratio. It expands by 20% per round and preferentially supplements samples with MFD≥0.6. During training, buffer pool samples and new samples are mixed in a 1:1 ratio.

[0015] Preferably, the three-level incremental training in step 2 specifically includes:

[0016] Concept layer: Convert the newly added guidelines into a knowledge graph, fuse the knowledge through an entity alignment module with a cosine similarity ≥0.92, and update the domain word embedding matrix;

[0017] Data layer: Use a dynamic temperature adjustment strategy for pre-training, with a temperature of 0.8 for the symptom description section, 0.5 for the treatment plan section, and 0.6 for the examination result section;

[0018] Task layer: For downstream tasks such as diagnostic classification, use a 5-shot few-shot learning strategy to fine-tune the model.

[0019] Preferably, the teacher-student model architecture in step 3 is as follows: the teacher model is selected as LLaMA2-70B, and the student model is modified based on Qwen-7B (the number of parameters is 1 / 10 of the teacher model); the clinical terminology enhancement layer integrates a medical knowledge graph embedding interface; the multi-objective joint loss function formula is:

[0020] L total =0.4L KD +0.2L grad +0.1L perturb +0.2L CE +0.1L align

[0021] The weights are dynamically adjusted in 0.05 steps per round using the validation set.

[0022] Preferably, in the multi-objective joint loss function: L KD The cross-entropy of the soft tag with a temperature coefficient T of 3 was used for calculation; L perturb Adversarial perturbations with L2 norm constraints added to the input embedding (ε = 1e-5, ||δ||2 ≤ ε); L align The formula for calculating the edit distance of the logical chain of symptoms, examinations, diagnosis, and treatment is as follows:

[0023]

[0024] Preferably, the multi-round course distillation in step 3 includes 3 rounds of iteration: the first round samples easy, medium and difficult data in a 5:4:1 ratio to train terminology matching ability; the second round strengthens logical reasoning in a 3:4:3 ratio; and the third round improves complex scenario processing ability in a 1:4:5 ratio. Each round samples diagnostic, treatment and question-and-answer task samples in a 4:3:3 ratio, and reasoning tasks are trained using a dual-mode of "thinking chain + structured conclusion".

[0025] Preferably, the formula for the clinically specific quantitative assessment system mentioned in step 4 is:

[0026] Medical Score = 0.6 × Knowledge Graph Matching Degree + 0.3 × Diagnostic Logic Chain Completeness + 0.1 × Readability

[0027] The knowledge graph matching score is the average of the accuracy rates of entity link and relationship matching. The logical chain completeness requires that the missing rate of necessary nodes be ≤5%. When the score is lower than 0.85, parameter adjustment is triggered. Error cases reported by experts are automatically assigned MFD=0.9 and included in the incremental dataset, triggering the next round of fine-tuning.

[0028] This invention employs a four-level integrated architecture encompassing data preprocessing, incremental training, knowledge distillation, and dynamic optimization, offering the following significant advantages over existing technologies:

[0029] (1) Dual protection of privacy and data quality: A two-layer desensitization mechanism (k-anonymity and federated feature transmission) is adopted to strictly comply with medical data privacy compliance requirements and effectively avoid the risk of privacy leakage caused by cross-domain transmission of original diagnosis and treatment data; the data is structured and labeled based on the ICD-11 and SNOMED CT standard terminology system, which greatly improves the accuracy of clinical entity recognition and lays a high-quality data foundation for model training; the training data is allocated on demand by combining the MFD difficulty grading mechanism to avoid invalid data occupying training resources and significantly improve the efficiency of subsequent training stages.

[0030] (2) Significantly improved incremental update efficiency: Through the parameter hierarchical isolation design, the parameter update range of incremental training is focused on the domain adaptation layer, without the need to adjust all parameters of the model, which greatly reduces the consumption of computing resources; coupled with the dynamic replay buffer strategy, while learning new knowledge, the memory of old knowledge is continuously strengthened, which effectively alleviates the catastrophic forgetting problem common in traditional batch training, significantly shortens the single-round model update cycle, and meets the needs of rapid iteration of clinical knowledge.

[0031] (3) Lightweight model performance breakthrough: A clinical terminology enhancement layer is added to the student model to enhance the understanding and matching ability of medical professional terms; combined with a multi-objective joint loss function, the distillation process is optimized from multiple dimensions such as knowledge transfer, gradient alignment, robustness improvement, label supervision, and logical matching, so that the lightweight student model can effectively reduce the number of parameters while retaining the core clinical capabilities of the teacher model; it not only ensures the accuracy of clinical terminology recognition and the integrity of diagnosis and treatment logic, but also improves the stability of the model in adversarial perturbation scenarios and avoids the risk of clinical decision-making caused by text interference.

[0032] (4) Continuous optimization of clinical adaptability: Construct a clinical-specific assessment system and expert feedback mechanism to form a dynamic optimization closed loop of training, assessment, feedback and fine-tuning; through expert labeling of model output errors and multi-dimensional scoring, accurately locate the weak links of the model, and achieve targeted optimization by combining targeted fine-tuning strategies; regularly evaluate the optimization effect and dynamically adjust the feedback sampling frequency to ensure that the model performance continues to iterate with changes in clinical needs, and continuously improve the practical application value in scenarios such as electronic medical record analysis and clinical decision support, effectively reducing the workload of medical staff. Attached Figure Description

[0033] Figure 1 This is a diagram of the overall technical architecture of the present invention. Detailed Implementation

[0034] This invention employs a four-level integrated architecture encompassing data preprocessing, incremental training, knowledge distillation, and dynamic optimization. (See reference...) Figure 1 This approach achieves efficient infusion and lightweight deployment of clinical knowledge through modular design. The specific technical solution includes the following four core steps:

[0035] 1. Construction and Preprocessing of Clinical Diagnosis and Treatment Text Dataset

[0036] This step aims to build high-quality, privacy-preserving two-level datasets—basic and incremental—to provide data support for subsequent model training. It specifically includes three sub-steps:

[0037] Multi-source data fusion and two-layer anonymization: Collecting multi-source clinical text data including electronic medical records (outpatient and inpatient records), clinical guidelines (such as the "Chinese Guidelines for the Prevention and Treatment of Hypertension"), treatment records (such as surgical records and ward round records), and medical literature abstracts, constructing a basic dataset (a complete collection of historically accumulated treatment texts, with a scale of no less than 1 million records) and an incremental dataset (newly added treatment texts, updated weekly / monthly, with a single update scale of no less than 10,000 records). A two-layer anonymization mechanism combining differential privacy and federated learning is employed to ensure data privacy: k-anonymization (k≥10, ensuring each anonymization group contains information from at least 10 patients, avoiding individual identification) is implemented for identifying information such as patient names, ID numbers, and medical record numbers; core treatment content (such as symptom descriptions, examination results, and treatment plans) is preprocessed locally through federated learning nodes (such as hospital local servers), converting the text into 768-dimensional feature vectors (generated based on a clinically pre-trained word embedding model). Only the feature vectors are transmitted to the aggregation node, avoiding cross-domain transmission of the original treatment text and fundamentally reducing the risk of privacy leakage.

[0038] Structured annotation under the standard terminology system: Based on the ICD-11 disease coding system and the SNOMED CT clinical terminology system, the text data is structured and annotated to ensure that the annotation results conform to clinical norms. The annotation content includes: ① Entity annotation: Using the BIOES (Begin-Inside-Outside-End-Single) annotation standard, five core entities are annotated: disease (e.g., "type 2 diabetes"), symptoms (e.g., "polydipsia and polyuria"), drugs (e.g., "metformin"), surgery (e.g., "laparoscopic cholecystectomy"), and examination indicators (e.g., "fasting blood glucose"); ② Relationship extraction: Structured diagnosis and treatment triplets of entity-relationship-entity are extracted, and eight key clinical relationships are defined, such as causal association (e.g., "polydipsia and polyuria - causal association - type 2 diabetes"), treatment correspondence (e.g., "metformin - treatment correspondence - type 2 diabetes"), and contraindication conflict (e.g., "insulin - contraindication conflict - hypoglycemia"), forming structured knowledge units to support the model's learning of clinical logic.

[0039] Model Fitting Difficulty Grading: A Model Fitting Difficulty (MFD) scoring mechanism is used to grade the difficulty of the dataset, providing a basis for subsequent incremental training and knowledge distillation tailored instruction. The MFD scoring formula is:

[0040] MFD = 0.3 × Term matching degree + 0.4 × Logical chain integrity + 0.3 × Entity recall rate

[0041] Among them: The term matching degree refers to the matching ratio of clinical terms in the text with the SNOMED CT term library; the logical chain integrity refers to the completeness of the clinical reasoning chain of symptoms → examinations → diagnoses → treatments in the text (a complete chain is recorded as 1, missing 1 link is recorded as 0.75, and so on); the entity recall rate refers to the recognition ratio of the core entities in the text by the model. The weight coefficients (0.3, 0.4, 0.3) are calibrated through 500 clinical expert-annotated samples to ensure that the scoring results conform to clinical cognition. The data is divided into three levels according to the MFD value: MFD ≤ 0.4 is the "easy" difficulty level (such as a conventional medical record with clear terms and a complete logical chain), 0.4 < MFD < 0.7 is the "medium" difficulty level (such as a medical record with 1 - 2 rare terms and a slightly missing logical chain), and MFD ≥ 0.7 is the "difficult" difficulty level (such as a difficult medical record containing rare disease terms and a complex logical chain).

[0042] 2. Three-level incremental training: Achieving precise injection of new knowledge and retention of old knowledge

[0043] This step injects clinical new knowledge in layers through the parameter hierarchical isolation and dynamic replay buffer strategy, while avoiding the model from forgetting old knowledge, specifically including three sub-links:

[0044] Parameter hierarchical isolation design: The large model parameters are divided into a general basic layer and a domain adaptation layer to achieve isolated learning of new and old knowledge. Among them, the general basic layer includes the first 12 layers of the Transformer encoder and the basic word embedding matrix, which is responsible for extracting the basic semantic features of the text. The parameters of this layer remain fixed during the incremental training process to avoid forgetting old knowledge due to parameter updates; the domain adaptation layer includes the last 6 layers of the Transformer encoder and the medical-specific word embedding matrix (pre-trained based on the clinical term library), which is responsible for the learning and update of clinical domain knowledge. Only the parameters of this layer are updated during the incremental training, and the parameter update learning rate is set to 2e - 5 (verified through multiple groups of experiments, this learning rate can avoid parameter oscillation while ensuring the learning effect of new knowledge).

[0045] Dynamic replay buffer construction: To further mitigate catastrophic forgetting, a dynamic replay buffer is constructed to store key samples from historical data, which are then mixed with incremental data for training. The initial buffer capacity is set at 50,000 samples, and samples are sampled from the basic dataset in a fixed ratio of easy:medium:difficult = 3:4:3. After each round of incremental training, the buffer capacity is expanded by 20% of the previous round's data volume. The expanded samples are preferentially selected from the "difficult" samples with MFD ≥ 0.6 in the previous round of training (these samples contain complex clinical knowledge and have a greater impact on model performance). During incremental training, the buffer samples:new samples = 1:1 are input into the model to ensure that the model continuously encounters old knowledge while learning new knowledge, thus strengthening memory.

[0046] Three-tiered incremental knowledge injection: New clinical knowledge is injected in a hierarchical manner, following the order of concept layer, data layer, and task layer, ensuring the accuracy and effectiveness of knowledge injection: ① Concept layer injection: New clinical guidelines (such as the new version of the "Guidelines for the Diagnosis and Treatment of Chronic Obstructive Pulmonary Disease") are transformed into a knowledge graph (nodes are entities such as diseases and symptoms, and edges are relationships). This is then fused with the model's existing knowledge graph through an entity alignment module (cosine similarity threshold ≥0.92 to ensure alignment accuracy), updating the medical terminology embedding matrix in the domain adaptation layer, enabling the model to quickly grasp new clinical concepts; ② Data layer injection: Incremental preprocessing is performed on the incremental diagnostic and treatment text. For training, a dynamic temperature adjustment strategy was used to optimize the pre-training effect. The temperature coefficient of the symptom description text was set to 0.8 (to retain a certain semantic diversity), the temperature coefficient of the treatment plan text was set to 0.5 (to ensure the accuracy of treatment terminology), and the temperature coefficient of the examination result text was set to 0.6 (to balance accuracy and diversity). ③ Task layer injection: For downstream clinical tasks such as diagnosis classification, treatment suggestion generation, and diagnosis and treatment question answering, a 5-shot few-shot learning strategy was adopted. Each task was configured with 5 high-quality samples labeled by clinical experts, and the parameters of the domain adaptation layer were fine-tuned to enable the model to quickly adapt to specific task requirements.

[0047] 3. Domain-Adaptive Knowledge Distillation: Constructing a Lightweight and Highly Reliable Student Model

[0048] This step efficiently transfers clinical knowledge from the teacher model to the lightweight student model through a customized teacher-student model architecture, multi-objective loss functions, and curriculum distillation strategies. It specifically includes three sub-steps:

[0049] Teacher-student model architecture configuration: A high-performance pre-trained large model was selected as the teacher model, and a lightweight student model adapted to clinical scenarios was constructed. The teacher model is LLaMA2-70B (70 billion parameters), which excels in general text understanding tasks and can meet clinical knowledge requirements after clinical pre-training. The student model is based on Qwen-7B (7 billion parameters, 1 / 10 the size of the teacher model), with a "clinical terminology enhancement layer" added to the Transformer architecture. This layer integrates a medical knowledge graph embedding interface, allowing real-time querying of the relationships between clinical terms in the input text and the knowledge graph (such as contraindications and indications for "aspirin"), and converting the relationship information into feature vectors to integrate into the model, thus enhancing the student model's ability to understand clinical terms.

[0050] Multi-objective joint loss function design: To ensure that the student model aligns with the teacher model in terms of "terminology accuracy, logical completeness, and robustness," a multi-objective joint loss function is designed, with the following formula:

[0051] L total =0.4L KD +0.2L grad +0.1L perturb +0.2L CE +0.1L align

[0052] The definitions and functions of each loss item are as follows: ① Standard distillation loss (L KD ): Using soft-label cross-entropy with a temperature coefficient T of 3, the difference in the probability distribution of the teacher-student model output is calculated. The formula is:

[0053]

[0054] N is the number of samples, C is the number of classes, and p is the model output probability, ensuring that the student model learns the generalization knowledge of the teacher model; ② Gradient calibration loss (L grad ): Calculate the gradient direction difference of intermediate layer features in the teacher and student models, alleviate gradient conflicts during the distillation process through gradient alignment, and improve the training stability of the student model; ③ Perturbation feature loss (L perturb ): Adding L2 norm-constrained adversarial perturbations (ε=1e-5,||δ||2≤ε) to the embedding vectors of the input text, enabling the student model to learn the robust feature representations of the teacher model under perturbations, thus enhancing the model's resistance to interference; ④ Cross-entropy loss (L CE ): Loss is calculated based on hard labels annotated by experts (such as disease diagnosis results, entity categories) to ensure the accuracy of student models on key clinical tasks; ⑤ Logical alignment loss (L align): This is achieved by calculating the edit distance (a measure of the difference between two logical chains) of symptoms, examinations, diagnoses, and treatments generated by the teacher-student model. The formula is as follows:

[0055]

[0056] Ensure that the student model replicates the clinical reasoning logic of the teacher model. The initial values ​​of the weight coefficients of the loss function (0.4, 0.2, 0.1, 0.2, 0.1) are determined through validation on the validation set and dynamically adjusted after each training round based on the performance on the validation set, with an adjustment step size of 0.05, to ensure that the loss function always adapts to the needs of the current training stage.

[0057] Task-Aware Multi-Round Course Distillation: Based on the principles of progressing from easy to difficult and maintaining task balance, a three-round course distillation strategy is designed to gradually improve student model performance: ① Round 1 (Basic Ability Development): Using a data ratio of easy:medium:difficult = 5:4:1, focusing on training clinical terminology matching ability, and enabling student models to master basic clinical terminology through a large number of simple samples; ② Round 2 (Logic Ability Enhancement): Using a data ratio of easy:medium:difficult = 3:4:3, strengthening training in diagnostic and treatment logical reasoning ability, with a focus on optimizing L... align Loss, ensuring the student model can generate a complete clinical reasoning chain; ③ Round 3 (Complexity Enhancement): Using an easy:medium:difficult data ratio of 1:4:5, improve the ability to handle complex scenarios, focusing on optimizing L perturb To mitigate losses and enhance the model's robustness to complex medical records and perturbed text, task samples are evenly sampled in a 4:3:3 ratio for diagnosis, treatment, and question-and-answer tasks during each distillation round. This ensures balanced performance of student models across various clinical tasks. For reasoning tasks (such as diagnostic logic generation), the teacher model is guided to generate dual-mode responses of thought chains and structured conclusions (e.g., "Patient presents with polydipsia and polyuria → fasting blood glucose 10.2 mmol / L → meets the diagnostic criteria for type 2 diabetes → metformin treatment recommended") for student models to learn from. This enables student models not only to output correct results but also to reproduce the clinical reasoning process.

[0058] 4. Dynamic optimization closed loop: ensuring the model continuously adapts to clinical needs.

[0059] Expert feedback integration mechanism: A clinical expert feedback interface is established, comprising two sub-modules: text annotation and scoring. Clinical experts can use the annotation module to categorize and label error types in the model output, specifically classifying them into three core error categories: ① Terminology errors (e.g., confusion of disease names, incorrect drug dosage labeling, e.g., misclassifying "type 2 diabetes" as "type 1 diabetes"); ② Logical breaks (e.g., missing links in the reasoning chain, reversed causal relationships, e.g., skipping "examination results" and directly inferring "diagnosis" from "symptoms"); ③ Omission of contraindications (e.g., failure to identify drug contraindications or surgical contraindications, e.g., recommending aspirin to patients with a history of peptic ulcer disease). After annotation, the system automatically assigns a "difficult" difficulty label (MFD=0.9) to the error case (this type of sample contains model weaknesses and is crucial for optimizing model performance) and includes it in the next round of incremental datasets to ensure targeted model learning.

[0060] The scoring module uses a 1-5 point scale (1 point is very poor, 5 points is excellent). Experts score the model output from three dimensions: terminology accuracy, logical completeness, and clinical applicability. Each dimension has equal weight (each accounts for 1 / 3), and the final score is the average of the three dimensions. When a sample's score is ≤3 points, the system triggers a targeted fine-tuning mechanism: extracting the corresponding clinical scenario (e.g., "medication recommendations for diabetes mellitus complicated with nephropathy"), matching 5-10 high-quality historical samples from the same scenario, constructing a specialized fine-tuning dataset, and using a learning rate of 1e-5 to locally fine-tune the model's domain adaptation layer. The number of fine-tuning iterations is set to 10 rounds (experiments have verified that this parameter can correct model errors in a short time and avoid overfitting).

[0061] To achieve efficient feedback flow, the system sets a feedback processing cycle: weekly, it summarizes expert-annotated error cases and low-scoring samples, automatically generating a "Model Optimization Requirement Report" which includes error type distribution, high-frequency error scenarios, and suggested fine-tuning parameters; monthly, it evaluates the model optimization effect based on feedback data. If the incidence of a certain type of error decreases by more than 40%, the sampling frequency of feedback samples for that scenario is reduced, and vice versa, the sampling frequency is increased, forming a dynamic closed loop of feedback, fine-tuning, evaluation, and iteration.

[0062] The following embodiments are provided based on the above technical solution:

[0063] Implement environmental preparation

[0064] (I) Hardware Environment

[0065] This embodiment uses a GPU cluster as the computing hardware support, specifically configured as follows: eight NVIDIA A100 GPU servers (80GB of VRAM per card), achieving high-speed data transmission within the cluster via a PCIe 4.0 interface, with a transmission bandwidth of 64GB / s; the accompanying CPUs are Intel Xeon Platinum 8375C (32 cores, 64 threads), with 256GB of DDR4 memory and a 10TB SSD array for storage, ensuring rapid read / write and secure storage of massive amounts of clinical data. Simultaneously, a hardware co-tuning module is deployed in the cluster to dynamically allocate GPU computing power according to the training task load, avoiding resource waste.

[0066] (II) Software Environment

[0067] On the software side, the training environment is built based on the PyTorch 2.0 deep learning framework, integrating a HIPAA (Health Insurance Portability and Accountability Act) protocol-compatible module to ensure that the entire data processing and model training process complies with medical data privacy protection requirements. The Transformers 4.30.2 library is used to build the large model architecture, and the Datasets 2.14.5 library is used for dataset management. An ethics review feedback module has been developed, which can record key operations such as data usage and parameter adjustment during model training in real time, facilitating subsequent ethical compliance traceability. The operating system used is Ubuntu 22.04LTS, combined with Docker containerization technology, to achieve rapid deployment and version control of the training environment.

[0068] (III) Dataset Preparation

[0069] The base dataset is the publicly available electronic medical record dataset MIMIC-III (Medical Information Mart for Intensive Care III) (already anonymized), containing approximately 400,000 inpatient medical records. It covers diagnostic and treatment texts from 16 clinical departments, including internal medicine, surgery, and emergency medicine. The data includes basic patient information (after anonymization), symptom descriptions, examination results, diagnostic conclusions, and treatment plans. This dataset was preprocessed using the structured annotation method under the standard terminology system described in this invention. The entity annotation accuracy was verified by clinical experts to be 98.2%, generating approximately 1.2 million structured diagnostic and treatment triples.

[0070] Incremental Dataset: This dataset contains 100,000 newly added medical records from a tertiary-level hospital between January and March 2024, including 60,000 outpatient medical records, 30,000 inpatient medical records, and 10,000 surgical records. It includes newly added clinical knowledge such as records of novel antibiotic use and cases of rare diseases (e.g., hereditary hemochromatosis). The dataset requires processing using a multi-source data fusion and two-layer anonymization process: first, k-anonymization (k=15) is applied to patient identification information; then, the core medical content is transformed into a 768-dimensional feature vector via the hospital's local federated nodes and transmitted to the GPU cluster aggregation node.

[0071] Validation and test sets: 5,000 samples of varying difficulty were labeled by 5 clinical experts with the title of associate chief physician or above as the validation set, including 1,750 samples of "easy" difficulty, 2,000 samples of "medium" difficulty, and 1,250 samples of "difficult" difficulty; another 10,000 independent samples were labeled as the test set, covering common diseases and difficult cases in key departments such as respiratory medicine, cardiology, and endocrinology, to evaluate the final performance of the model.

[0072] II. Specific Implementation Steps

[0073] (I) Construction and Preprocessing of Clinical Diagnosis and Treatment Text Dataset

[0074] Two-layer desensitization operation: A k-anonymization processing script was written in Python to replace identifying information such as patient names, ID numbers, and medical record numbers in the incremental dataset (e.g., replacing "Zhang San" with "Patient A", and desensitizing ID numbers in the format "110101********1234"); Based on the FedAvg federated learning algorithm, a feature extraction model (using a BERT-base-clinical pre-trained model) was deployed on the hospital's local server to convert the medical text into a 768-dimensional feature vector, which was sent to the cluster aggregation node through an encrypted transmission protocol (SSL / TLS1.3). No original text data was transferred during the transmission. According to the privacy leakage risk assessment, the privacy protection level of this desensitization mechanism reached the sensitive data protection level in the national "Information Security Technology Health and Medical Data Security Guidelines".

[0075] Structured annotation implementation: An annotation tool based on ICD-11 and SNOMED CT terminology database was developed, supporting BIOES annotation specifications and triple extraction functionality. The annotation team consisted of 3 clinicians and 5 natural language processing engineers. They were first trained to standardize annotation criteria through annotation of 200 samples, and then annotated both the base dataset and the incremental dataset. After annotation, a two-person cross-validation and expert review mechanism was used to ensure annotation quality: samples with a consistency of 95% or higher in two-person cross-validation passed directly; samples with a consistency below 95% were arbitrated by clinical experts. The final structured annotation completion rate was 100%, with an annotation error rate of less than 1.8%.

[0076] MFD Difficulty Grading Implementation: An MFD scoring algorithm is implemented using Python. Inputting labeled text data, the algorithm automatically calculates three metrics: terminology matching degree, logical chain completeness, and entity recall. The MFD value is generated using the formula: "MFD = 0.3 × Terminology Matching Degree + 0.4 × Logical Chain Completeness + 0.3 × Entity Recall". For example, a "Routine Treatment Record for Type 2 Diabetes" (with clear terminology and a complete logical chain) has an MFD value of 0.32, classifying it as "Easy" difficulty; a "Treatment Record for Rare Disease Hereditary Hemochromatosis Complicated with Liver Cirrhosis" (containing rare terminology and a complex logical chain) has an MFD value of 0.85, classifying it as "Difficult". After grading, the datasets are stored in an "Easy:Medium:Difficult" ratio for easy access during subsequent training.

[0077] (II) Implementation of Three-Level Incremental Training

[0078] Parameter layering and isolation configuration: A large model architecture was built based on the PyTorch framework, dividing the Transformer encoder into 18 layers. The first 12 layers were designated as "general base layers," loading pre-trained LLaMA2-7B base model parameters (frozen and not updated); the last 6 layers were designated as "domain adaptation layers," loading medical-specific parameters pre-trained based on the MIMIC-III dataset, with these parameters set to be updatable. The parameter update learning rate was determined through comparative experiments: 1e-5, 2e-5, and 3e-5 learning rates were tested respectively. The results showed that the model converged fastest (loss stabilized after 15 iterations) and achieved the best new knowledge learning effect (97% accuracy in recognizing new terms). Therefore, 2e-5 was ultimately selected as the learning rate for the domain adaptation layer.

[0079] Dynamic replay buffer pool operation: The initial buffer pool capacity is 50,000 records. Samples are taken from the basic dataset in a ratio of "easy:medium:difficult = 3:4:3" (15,000 records for "easy" difficulty, 20,000 records for "medium" difficulty, and 15,000 records for "difficult" difficulty). After each round of incremental training, the buffer pool is expanded by 20% of the previous round's data volume. The expanded samples are preferentially selected from "difficult" difficulty samples with MFD ≥ 0.6 (such as difficult cases and rare disease diagnosis texts). Taking the first round of incremental training as an example, the amount of new data is 30,000 records, and the buffer pool is expanded by 6,000 samples (all of which are "difficult" difficulty samples with MFD ≥ 0.6), and the buffer pool capacity reaches 56,000 records after expansion. During training, the buffer pool samples are mixed into the model in a ratio of "buffer pool samples: new samples = 1:1", that is, 256 samples are input in each batch (128 buffer pool samples + 128 new samples).

[0080] Implementation of Level 3 Incremental Knowledge Injection:

[0081] Conceptual layer injection: The "Guidelines for the Prevention and Treatment of Type 2 Diabetes in China (2024 Edition)" published in 2024 was transformed into a knowledge graph, containing over 5,000 nodes and over 8,000 edges, including relationship types such as "disease-symptom," "disease-drug," and "drug-contraindications." Through an entity alignment module (with a cosine similarity threshold set to 0.92), it was fused with the model's existing knowledge graph. For example, "novel SGLT2 inhibitor" was aligned with the existing "hypoglycemic drugs" node. The medical terminology embedding matrix of the domain adaptation layer was updated, improving the model's accuracy in understanding newly added drug terms to 96%.

[0082] Data layer injection: Incremental pre-training was performed on the incremental dataset, employing a dynamic temperature adjustment strategy: the temperature of symptom description text (e.g., "The patient experiences thirst and polyuria, with a daily water intake of approximately 3000ml") was set to 0.8 to preserve the semantic diversity of symptom descriptions; the temperature of treatment plan text (e.g., "Metformin extended-release tablets 0.5g, orally twice daily") was set to 0.5 to ensure the accuracy of key information such as drug dosage and usage; and the temperature of examination result text (e.g., "Fasting blood glucose 8.5mmol / L, glycated hemoglobin 7.2%) was set to 0.6 to balance the accuracy and flexibility of the examination indicator descriptions. Pre-training iterated for 20 rounds, with a batch size of 128 per round, using the AdamW optimizer (weight decay of 0.01).

[0083] Task-layer injection: For the three downstream clinical tasks of "disease diagnosis classification," "treatment suggestion generation," and "diagnosis and treatment Q&A," each task is configured with 5 expert-annotated Few-Shot samples (e.g., the sample for the "diabetes diagnosis classification" task includes "symptoms + examination results + diagnosis conclusion"). Taking the "treatment suggestion generation" task as an example, the input is "Patient, male, 55 years old, diagnosed with type 2 diabetes for 3 years, poor blood glucose control (fasting blood glucose 9.0 mmol / L), no history of drug allergies." The expert-annotated reference output is "Recommended treatment plan adjustment: Metformin extended-release tablets 1.0g, twice daily orally, combined with empagliflozin 10mg, once daily orally; monitor fasting blood glucose and 2-hour postprandial blood glucose at least 3 times a week; control diet, with daily carbohydrate intake accounting for 50%-60% of total calories, and appropriate exercise (e.g., brisk walking for 30 minutes / day)." Based on this sample, the model domain adaptation layer is fine-tuned, with 10 iterations and a learning rate set to 1e-5 to ensure that the model quickly adapts to task requirements.

[0084] (III) Implementation of Domain-Adaptive Knowledge Distillation

[0085] Teacher-student model architecture: The teacher model uses the LLaMA2-70B pre-trained model, which is pre-trained (30 iterations) using the MIMIC-III dataset and clinical guideline texts to enable it to understand basic clinical knowledge. The student model is based on the Qwen-7B model with 7 billion parameters (1 / 10 of the teacher model). A "clinical terminology enhancement layer" is added between the encoder and decoder of the Transformer architecture. This layer integrates a medical knowledge graph embedding interface (using the Neo4j graph database to store the knowledge graph), which can query the association information of clinical terms in the input text in real time (e.g., inputting "aspirin" will automatically query its indication "secondary prevention of coronary heart disease", contraindication "gastric ulcer", adverse reaction "gastrointestinal bleeding", etc.), and convert the association information into a 512-dimensional feature vector, which is fused with the original feature vector of the model to enhance the student model's understanding of clinical terms.

[0086] Application of Multi-Objective Joint Loss Function: A multi-objective joint loss function is implemented in the PyTorch framework, with initial weights set to ω1 = 0.4, ω2 = 0.2, ω3 = 0.1, ω4 = 0.2, and ω5 = 0.1. Taking a "pneumonia diagnosis sample" as an example, the following loss terms are calculated: ① L_KD: The teacher model's output probability for "pneumonia" diagnosis is 0.95, and the student model's initial output probability is 0.78. Calculated using the soft-label cross-entropy with a temperature coefficient T = 3, L_KD = 0.12; ② L_grad: The gradient direction difference between the 8th layer Transformer encoder of the teacher and student models is calculated, L_grad = 0.08; ③ L_perturb: An L2 norm constraint of ε = 1e-5 is added to the input text embedding to counteract perturbations. The student model... The output probability change after the change is 0.05, L_perturb=0.05; ④L_CE: Based on the expert-annotated hard label "pneumonia", the classification loss of the student model is L_CE=0.10; ⑤L_align: The edit distance of the diagnosis and treatment logic chain generated by the teacher and student models is 1, and the total length of the logic chain is 4, L_align=1-(1 / 4)=0.75; Finally, L_total=0.4×0.12+0.2×0.08+0.1×0.05+0.2×0.10+0.1×0.75=0.162. After each round of training, the weights are adjusted according to the performance of the validation set. For example, if the validation finds that the integrity of the student model's logic chain is low, ω5 is increased from 0.1 to 0.15. In the next round of training, L_align's contribution to the total loss increases, driving the model to optimize the inference logic.

[0087] Multi-round course distillation execution:

[0088] Round 1 (Basic Ability Development): Samples were collected from the validation set using a data ratio of "Easy:Medium:Difficult = 5:4:1" (2500 entries for "Easy" difficulty, 2000 entries for "Medium" difficulty, and 500 entries for "Difficult" difficulty), focusing on training clinical terminology matching skills. During training, the entity recognition accuracy of the student model was closely monitored, and evaluated every 5 rounds. Round 1 ended when the entity recognition accuracy stabilized above 95% (a total of 15 rounds of iterations).

[0089] Round 2 (Logical Ability Enhancement): The data ratio was adjusted to "Easy:Medium:Difficult = 3:4:3" (1500 entries for "Easy" difficulty, 2000 entries for "Medium" difficulty, and 1500 entries for "Difficult") to enhance diagnostic and treatment logical reasoning abilities. By increasing the L_align loss weight (from 0.1 to 0.15), the student model was guided to learn the complete "symptom → examination → diagnosis → treatment" reasoning chain. By the 20th round of training, the logical chain generated by the student model achieved a completeness of 92%, meeting clinical needs, and the second round of distillation was stopped.

[0090] Round 3 (Enhancing Complexity): A data ratio of "Easy:Medium:Difficult = 1:4:5" was adopted (500 records for "Easy" difficulty, 2000 for "Medium" difficulty, and 2500 for "Difficult") to improve the model's ability to handle complex scenarios. The focus was on optimizing the L_perturb loss by increasing the adversarial perturbation strength (ε = 1.2e-5) to enhance the model's robustness to difficult medical records and text perturbations. The third round of distillation iterated for 25 rounds. After training, the student model achieved an accuracy of over 90% on perturbed samples.

[0091] (iv) Dynamic optimization of closed-loop implementation

[0092] Clinically specific quantitative assessment operation: The assessment tool is developed according to the formula "Medical score = 0.6 × knowledge graph matching degree + 0.3 × diagnostic logic chain completeness + 0.1 × readability" to automatically calculate the medical score output by the model. Taking a sample of "coronary heart disease diagnosis and treatment" as an example: ① Knowledge graph matching accuracy: The model output relationships such as "coronary heart disease-treatment-aspirin" and "coronary heart disease-symptoms-chest pain" have a matching accuracy of 94% with the standard knowledge graph; ② Diagnostic logic chain completeness: The logic chain generated by the model (chest pain → electrocardiogram examination → coronary heart disease → aspirin treatment) has no missing nodes, and the completeness is 100%; ③ Readability: The treatment suggestions output by the model were rated by 3 clinicians as 4, 5, and 4 points respectively (out of 5), with an average of 4.3 points, and the readability is converted to 0.86; The final medical score = 0.6 × 0.94 + 0.3 × 1.0 + 0.1 × 0.86 = 0.954, which is higher than the passing threshold of 0.85, so the model's output in this round is qualified. If, in a certain round of evaluation, the model's knowledge graph matching degree for the "drug interaction" sample is only 78% (<0.8), a dynamic adjustment mechanism is triggered, reducing the temperature coefficient of the treatment plan text from 0.5 to 0.4, and the model is fine-tuned. In the next round of evaluation, the knowledge graph matching degree is improved to 89%.

[0093] Expert feedback is integrated into the implementation: A web-based expert feedback interface is built, allowing clinical experts to log in to the system via a browser, view the model output results, and annotate and score them. For example, if an expert discovers that the model recommends "aspirin" for "patients with gastric ulcers" (a contraindication conflict), the annotation module marks this case as an "omission of contraindication" error. The system automatically assigns it a "difficult" difficulty label with MFD=0.9 and includes it in the next round of incremental dataset. At the same time, the expert scores this output 2 points (≤3 points), triggering a targeted fine-tuning mechanism: 8 high-quality historical samples from the "medication recommendation for patients with gastric ulcers" scenario are extracted to construct a specialized fine-tuning dataset, and the model's domain adaptation layer is locally fine-tuned (10 iterations, learning rate 1e-5). Expert feedback data is compiled weekly to generate a "Model Optimization Requirement Report," which includes key information such as "Taboo omission error rate of 25%" and "Rare disease terminology error rate of 18%" to guide the adjustment of the next round of training strategies. The optimization effect is evaluated monthly. If the occurrence rate of "Taboo omission" errors decreases from 25% to 8% (a decrease of more than 40%), the sampling frequency of feedback samples in that scenario is reduced.

[0094] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for incremental training of large models and domain knowledge distillation for clinical diagnosis and treatment texts, characterized in that, It includes the following steps: S01. Construct a clinical diagnosis and treatment text dataset, process the data using a desensitization mechanism that combines differential privacy and federated learning, complete structured annotation based on a standard terminology system, and divide the data into levels through a model fitting difficulty score; S02. Based on the parameter hierarchical isolation and dynamic replay buffer strategy, inject clinical new knowledge at the concept layer, data layer, and task layer levels, and perform incremental training; S03. Configure a teacher-student model architecture with a clinical term enhancement layer, design a multi-objective joint loss function, and optimize the lightweight student model through task-aware multi-round curriculum distillation; S04. Based on a clinical-specific quantitative evaluation system and an expert feedback mechanism, construct a dynamic optimization closed-loop, and iteratively adjust the model parameters and training strategies.

2. The method according to claim 1, characterized in that, The specific desensitization mechanism described in step 1 is as follows: perform k-anonymization on patient identification information, where k≥10, and generate 768-dimensional feature vectors for the core content of diagnosis and treatment through local preprocessing by federal nodes; the structured annotation is based on the ICD-11 coding and SNOMED CT terminology system, and uses the BIOES specification to annotate 5 types of entities such as diseases and symptoms and extract 8 types of entity-relationship-entity triples; the formula for the model fitting difficulty score is: MFD = 0.3×Term matching degree + 0.4×Logical chain integrity + 0.3×Entity recall rate And accordingly, the data is divided into three levels: MFD≤0.4 (easy), 0.4<MFD<0.7 (medium), and MFD≥0.7 (difficult).

3. The method according to claim 1, characterized in that, The parameter hierarchical isolation described in step 2 divides the model into a general basic layer and a domain adaptation layer, the first 12 layers of Transformer and the basic word embedding matrix, the last 6 layers of Transformer and the medical-specific word embedding matrix. Incremental training only updates the parameters of the adaptation layer, and the learning rate is set to 2e-5; the initial capacity of the dynamic replay buffer pool is 50,000, sample historical data according to a 3:4:3 ratio, expand by 20% per round and preferentially supplement samples with MFD≥0.6, and mix the samples in the buffer pool and the newly added samples at a 1:1 ratio during training.

4. The method according to claim 1, characterized in that, The specific three-level incremental training described in step 2 includes: Concept layer: Convert the newly added guidelines into a knowledge graph, fuse the knowledge through an entity alignment module with a cosine similarity≥0.92, and update the domain word embedding matrix; Data layer: Use a dynamic temperature adjustment strategy for pre-training, with a temperature of 0.8 for the symptom description segment, 0.5 for the treatment plan segment, and 0.6 for the examination result segment; Task layer: For downstream tasks such as diagnostic classification, use a 5-shot few-shot learning strategy to fine-tune the model.

5. The method according to claim 1, characterized in that, The teacher-student model architecture described in step 3 is as follows: The teacher model selects LLaMA2-70B, the student model is based on the Qwen-7B modification, and the number of parameters is 1 / 10 of the teacher model. The clinical term enhancement layer integrates a medical knowledge graph embedding interface; the formula for the multi-objective joint loss function is: L total =0.4L KD +0.2L grad +0.1L perturb +0.2L CE +0.1L align The weights are dynamically adjusted with a step size of 0.05 per round through the validation set.

6. The method according to claim 5, characterized in that, In the multi-objective joint loss function: L KD The cross-entropy of the soft tag with a temperature coefficient T of 3 was used for calculation; L perturb Adversarial perturbations with L2 norm constraints added to the input embedding, ε=1e-5, ||δ||2≤ε; L align The formula for calculating the edit distance of the logical chain of symptoms, examinations, diagnosis, and treatment is as follows:

7. The method according to claim 1, characterized in that, The multi-round course distillation mentioned in step 3 includes 3 rounds of iteration: the first round samples easy, medium and difficult data in a 5:4:1 ratio to train terminology matching ability; the second round strengthens logical reasoning in a 3:4:3 ratio; and the third round improves the ability to handle complex scenarios in a 1:4:5 ratio. Each round samples diagnostic, treatment and question-and-answer task samples in a 4:3:3 ratio. Reasoning tasks are trained using a dual mode of "thinking chain + structured conclusion".

8. The method according to claim 1, characterized in that, The formula for the clinically specific quantitative assessment system mentioned in step 4 is as follows: The medical score is calculated as follows: 0.6 × knowledge graph matching degree + 0.3 × diagnostic logic chain completeness + 0.1 × readability. The knowledge graph matching degree is the average of the accuracy rates of entity link and relationship matching. The logic chain completeness requires a missing node rate of ≤5%. When the score is lower than 0.85, parameter adjustment is triggered. Error cases reported by experts are automatically assigned MFD = 0.9 and included in the incremental dataset, triggering the next round of fine-tuning.

Images