Medical data processing method and device based on knowledge distillation and passive data continuous learning, equipment and medium

Abstract

The invention discloses a medical data processing method and device based on knowledge distillation and passive data continuous learning, equipment and a medium, and relates to the technical field of deep learning, and the method comprises the steps: determining a copy of a preset teacher model as a preset student model, and changing a target weight matrix of the preset student model to obtain an initial student model; freezing a first model parameter in the initial student model to obtain a to-be-trained student model, and training the to-be-trained student model based on a soft label and a knowledge distillation technology to obtain an optimized rank parameter; and combining the optimized rank parameter with the first model parameter to obtain a corresponding trained student model, determining the trained student model as a new teacher model, and processing the current medical data by using the new teacher model. The low-rank matrix is only added to the weight of the key layer, so that catastrophic forgetting caused by global parameter updating is avoided, and the problem that the complexity of the model is sharply increased due to the fact that the number of adapters is dynamically increased is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a medical data processing method and device based on knowledge distillation and passive data continual learning, equipment and medium. BACKGROUND

[0002] In the medical field, the research of continual learning technology is crucial, and its core challenge comes from the natural characteristics of medical data and the dynamic evolution of clinical needs. First, the privacy and dispersion of medical data lead to strict restrictions on cross-institutional data sharing, which makes the model usually only face a specific hospital for training, and cannot access data sources from multiple hospitals at the same time. In addition, the multi-source heterogeneity (such as image, text signal) and distribution offset problem (such as noise difference of electrocardiogram signals collected by different devices) of medical data further aggravate the difficulty of model generalization. Traditional static models are difficult to adapt to this dynamic environment, and it is urgent to use continual learning technology to realize the adaptive evolution of the model.

[0003] The existing continual learning method is based on revisit and dynamic architecture, which can cause the model to have a dramatic increase in complexity. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a medical data processing method, device, equipment and medium based on knowledge distillation and passive data continual learning, which adds a low-rank matrix to only the key layer weight, avoids the catastrophic forgetting caused by global parameter update, and avoids the problem of dramatic increase in model complexity caused by dynamic increase in the number of adapters. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a medical data processing method based on knowledge distillation and passive data continual learning, comprising:

[0006] determining a copy of a preset teacher model as a preset student model, inserting a preset LoRA adapter into a target neural network layer of the preset student model, and adding a low-rank matrix to an original weight matrix of the preset student model using the preset LoRA adapter to obtain a corresponding initial student model; wherein the preset teacher model is a pre-trained neural network model;

[0007] freezing a first model parameter corresponding to the original weight matrix in the initial student model to obtain a corresponding to-be-trained student model, and processing current medical data in a current medical scene using the preset teacher model to obtain a probability distribution corresponding to a processing result of the current medical data;

[0008] determine the probability distribution as a soft label, and train the to-be-trained student model based on the soft label and a knowledge distillation technique to optimize an initial rank parameter corresponding to the low-rank matrix in the initial student model, and obtain an optimized rank parameter;

[0009] merge the optimized rank parameter and the first model parameter to obtain a corresponding trained student model, determine the trained student model as a new teacher model, and process the current medical data by using the new teacher model.

[0010] Optionally, the preset teacher model is a pre-trained neural network model for a general medical task or a pre-trained neural network model trained in a previous medical scene corresponding to the current medical scene.

[0011] Optionally, the target neural network layer includes at least one of an attention layer and a feedforward network layer of the initial student model.

[0012] Optionally, the training of the to-be-trained student model based on the soft label and the knowledge distillation technique includes:

[0013] processing the current medical data by using the to-be-trained student model to determine a target task loss, wherein the target task loss is a difference between an output result of the to-be-trained student model and labeled data corresponding to the current medical data;

[0014] determining a target distillation loss based on a knowledge distillation technique, the soft label, a preset KL divergence function, and an ROI technique, wherein the target distillation loss is a difference between the output result of the to-be-trained student model and the probability distribution;

[0015] jointly optimizing the target task loss and the target distillation loss to train the to-be-trained student model.

[0016] Optionally, after the merging of the optimized rank parameter and the first model parameter, the method further includes:

[0017] determining a copy of the new teacher model as a new preset student model, and inserting a preset LoRA adapter into a target neural network layer of the new preset student model to obtain a new initial student model;

[0018] freezing a second model parameter corresponding to an original weight matrix in the new initial student model to obtain a new to-be-trained student model, and processing new medical data in a new medical scene by using the new teacher model to obtain a new probability distribution corresponding to a processing result of the new medical data;

[0019] train the new student model to be trained based on the new probability distribution to optimize the initial rank parameter of the preset LoRA adapter, so as to obtain a new optimized rank parameter;

[0020] merge the new optimized rank parameter and the second model parameter to obtain a new trained student model, and process the new medical data by using the new trained student model.

[0021] In a second aspect, the present application provides a medical data processing device based on knowledge distillation and passive data continuous learning, comprising:

[0022] a weight matrix modification module configured to determine a copy of a preset teacher model as a preset student model, insert a preset LoRA adapter into a target neural network layer of the preset student model, and add a low-rank matrix to an original weight matrix of the preset student model by using the preset LoRA adapter, to obtain a corresponding initial student model; wherein the preset teacher model is a pre-trained neural network model;

[0023] a parameter freezing module configured to freeze a first model parameter corresponding to the original weight matrix in the initial student model, to obtain a corresponding student model to be trained, and process current medical data in a current medical scenario by using the preset teacher model, to obtain a probability distribution corresponding to a processing result of the current medical data;

[0024] a model training module configured to determine the probability distribution as a soft label, and train the student model to be trained based on the soft label and a knowledge distillation technique, to optimize an initial rank parameter corresponding to the low-rank matrix in the initial student model, and obtain an optimized rank parameter;

[0025] a medical data processing module configured to merge the optimized rank parameter and the first model parameter, to obtain a corresponding trained student model, determine the trained student model as a new teacher model, and process the current medical data by using the new teacher model.

[0026] Optionally, the model training module comprises:

[0027] a task loss determination unit configured to process the current medical data by using the student model to be trained, to determine a target task loss; wherein the target task loss is a difference between an output result of the student model to be trained and labeled data corresponding to the current medical data;

[0028] The distillation loss determination unit is configured to determine a target distillation loss based on a knowledge distillation technique, the soft label, a preset KL divergence function, and an ROI technique, wherein the target distillation loss is a difference between an output result of the to-be-trained student model and the probability distribution.

[0029] The first model training unit is configured to jointly optimize the target task loss and the target distillation loss to train the to-be-trained student model.

[0030] Optionally, the medical data processing module further includes:

[0031] The adapter insertion unit is configured to determine a copy of the new teacher model as a new preset student model, and insert a preset LoRA adapter into a target neural network layer of the new preset student model to obtain a new initial student model.

[0032] The parameter freezing unit is configured to freeze a second model parameter corresponding to an original weight matrix in the new initial student model to obtain a new to-be-trained student model, and process new medical data in a new medical scenario by using the new teacher model to obtain a new probability distribution corresponding to a processing result of the new medical data.

[0033] The second model training unit is configured to train the new to-be-trained student model based on the new probability distribution to optimize an initial rank parameter of the preset LoRA adapter, thereby obtaining a new optimized rank parameter.

[0034] The medical data processing unit is configured to merge the new optimized rank parameter and the second model parameter to obtain a new trained student model, and process the new medical data by using the new trained student model.

[0035] In a third aspect, the present application provides an electronic device, comprising:

[0036] A memory configured to save a computer program;

[0037] A processor configured to execute the computer program to implement the medical data processing method based on knowledge distillation and passive data continual learning.

[0038] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, wherein the computer program is executed by a processor to implement the medical data processing method based on knowledge distillation and passive data continual learning.

[0039] This application first determines a copy of a preset teacher model as a preset student model, inserts a preset LoRA adapter into the target neural network layer of the preset student model, and adds a low-rank matrix to the original weight matrix of the preset student model using the preset LoRA adapter to obtain a corresponding initial student model; wherein, the preset teacher model is a pre-trained neural network model. Then, the first model parameters corresponding to the original weight matrix in the initial student model are frozen to obtain a corresponding student model to be trained, and the preset teacher model is used to process the current medical data in the current medical scenario to obtain the probability distribution corresponding to the processing result of the current medical data. Then, the probability distribution is determined as a soft label, and the student model to be trained is trained based on the soft label and knowledge distillation technology to optimize the initial rank parameters corresponding to the low-rank matrix in the initial student model and obtain the optimized rank parameters. Finally, the optimized rank parameters are merged with the first model parameters to obtain a corresponding trained student model, and the trained student model is determined as a new teacher model, and the new teacher model is used to process the current medical data. Therefore, this application adapts to new tasks with a minimal number of parameters by adding only low-rank matrices to the weights of key layers, avoiding catastrophic forgetting caused by global parameter updates, and also avoiding the problem of drastically increasing model complexity due to dynamically increasing the number of adapters. By utilizing knowledge distillation techniques, cross-domain knowledge transfer can be achieved, that is, feature alignment loss is constructed between the teacher model and the student model applied to the new domain, thereby enabling the model to continuously learn and process multi-source heterogeneous data, thus improving the model's generalization ability. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 This is a flowchart of a medical data processing method based on knowledge distillation and passive data continuous learning disclosed in this application;

[0042] Figure 2 This is a schematic diagram of a specific medical data processing method based on knowledge distillation and passive data continuous learning disclosed in this application;

[0043] Figure 3 This is a schematic diagram of the structure of a medical data processing device based on knowledge distillation and passive data continuous learning disclosed in this application;

[0044] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Current continuous learning methods can lead to a surge in model complexity. To address this, this application provides a medical data processing method based on knowledge distillation and passive data continuous learning. By adding low-rank matrices only to the weights of key layers, it avoids catastrophic forgetting caused by global parameter updates, while also avoiding the problem of a surge in model complexity caused by dynamically increasing the number of adapters.

[0047] See Figure 1 As shown, this embodiment of the invention discloses a medical data processing method based on knowledge distillation and passive data continuous learning, including:

[0048] Step S11: Determine a copy of the preset teacher model as the preset student model, insert the preset LoRA (Low-Rank Adaptation, a large language model fine-tuning technique) adapter into the target neural network layer of the preset student model, and use the preset LoRA adapter to add a low-rank matrix to the original weight matrix of the preset student model to obtain the corresponding initial student model; wherein, the preset teacher model is a pre-trained neural network model.

[0049] The overall operation flowchart of the medical data processing method in this embodiment is as follows: Figure 2 As shown, firstly, the model trained and saved in the old scene is used as the teacher model, and the model to be trained in the new scene is used as the student model (i.e., the preset student model), thus achieving parameter replication and initialization. Next, the student model is fine-tuned using a LoRA-based training method, adding low-rank matrices only to key layers to adapt to the new task with a minimal number of parameters. Simultaneously, adapter learning is performed (LoRA fine-tuning can also be used when adapting to more than two scenes), and the final result is output.

[0050] It should be noted that the framework is supervised by two parts of loss: one part is the KL (Kullback-Leibler Divergence, a loss function) loss based on the ROI (Region of Interest), and the other part is the supervised loss based on the new task.

[0051] In this embodiment, the preset teacher model is a pre-trained neural network model for general medical tasks or a pre-trained neural network model trained in the previous medical scenario corresponding to the current medical scenario.

[0052] That is, there are two teacher model forms: (1) Pre-trained model: using the base model as the initial model, pre-training is completed on a general task (i.e., a pre-trained neural network model for a general medical task). (2) Old model storage: before learning in the t-th scenario, a copy of the current model is saved. (That is, the pre-trained neural network model that has been trained in the previous medical scenario) is used as the old model for subsequent distillation.

[0053] Additionally, in this embodiment, a preset LoRA adapter needs to be inserted into the target neural network layer of the preset student model. The target neural network layer in this embodiment includes at least one of the attention layer and feedforward network layer of the initial student model.

[0054] Specifically, a LoRA adapter is inserted into the key layers (i.e., the target neural network layers) of the pre-trained model, such as the attention layer and the feedforward network. The specific operation is as follows: For the weight moments W, decompose them into low-rank matrices. ,in , , is the rank parameter. Only adapter parameters A and B are trained. (The above...) These are the matrix parameters of the original weight matrix, and BA is the matrix parameter of the low-rank matrix. The target neural network layer in this embodiment can be determined according to actual application requirements and is not limited to attention layers or feedforward network layers.

[0055] By using LoRA fine-tuning to add low-rank matrices to the key neural network layers of the pre-defined student model, catastrophic forgetting caused by global parameter updates is avoided, while also avoiding the problem of a dramatic increase in model complexity caused by dynamically increasing the number of adapters.

[0056] Step S12: Freeze the first model parameters corresponding to the original weight matrix in the initial student model to obtain the corresponding student model to be trained, and use the preset teacher model to process the current medical data in the current medical scenario to obtain the probability distribution corresponding to the processing result of the current medical data.

[0057] In this embodiment, to avoid the catastrophic forgetting problem caused by global parameter updates, it is necessary to first update the first model parameters corresponding to the original weight matrix in the initial student model. Freeze the system so that subsequent actions can be taken. The frozen student model is then used for training.

[0058] By processing current medical data in a given medical scenario using a pre-defined teacher model, the system can output a prediction confidence vector (i.e., probability distribution) for the current medical data. For example, in an ECG signal classification task, the output might be: [Atrial fibrillation: 0.85, Ventricular tachycardia: 0.12, Normal: 0.03]. Essentially, this represents the teacher model's "cognitive judgment" of the current new scenario data based on historical knowledge, including: inter-category correlation (such as the similarity between atrial fibrillation and ventricular tachycardia); and prediction uncertainty (auxiliary information implied by the non-highest probability term).

[0059] Step S13: Determine the probability distribution as a soft label, and train the student model to be trained based on the soft label and knowledge distillation technology to optimize the initial rank parameter corresponding to the low-rank matrix in the initial student model, and obtain the optimized rank parameter.

[0060] In this embodiment, the aforementioned probability distribution needs to be used as a soft label for training the student model. The process of training the student model based on soft labeling and knowledge distillation techniques can specifically include: processing the current medical data using the student model to determine the target task loss; wherein, the target task loss is the difference between the output of the student model and the labeled data corresponding to the current medical data; determining the target distillation loss based on knowledge distillation techniques, soft labeling, KL divergence function, and ROI techniques; wherein, the target distillation loss is the difference between the output of the student model and the probability distribution; and jointly optimizing the target task loss and the target distillation loss to train the student model.

[0061] Specifically, (1) Task loss (hard label supervision): Standard loss functions such as cross-entropy are used to ensure that the model fits the current scenario. (2) ROI-weighted KL loss: Teacher model is used. Output As a soft label, where T is a temperature parameter (usually between 1 and 10), it guides the current model. (i.e., the student model to be trained) output distribution is close to Distillation losses were calculated using KL divergence and weighted by ROI.

[0062] ROI, or Region of Interest, refers to critical medical regions. ROI weighting for distillation loss means using predefined masks to increase the loss weight of critical medical regions, such as the Q, R, and S bands in an electrocardiogram or nodular areas in a lung CT scan.

[0063] The above training process only optimizes the LoRA adapter parameters A and B (i.e., the initial rank parameters), while the original model parameters... Keep the parameters frozen and significantly reduce the size of the training parameters (typically only update 0.1%-1% of the parameters).

[0064] By using ROI-weighted distillation, the "diagnostic experience" of the teacher model is transformed into a feature learning paradigm that can be absorbed by the student model, enabling safe transfer of medical knowledge with very little new labeled data. Furthermore, based on ROI-weighted KL loss, it can learn the transfer of key features, highlighting the value of important regions.

[0065] Step S14: Merge the optimized rank parameter with the first model parameter to obtain the corresponding trained student model, determine the trained student model as the new teacher model, and use the new teacher model to process the current medical data.

[0066] After the model training is completed, the optimized rank parameters need to be merged with the first model parameters. That is, the original model parameters and LoRA adapter parameters are merged and stored as a new teacher model.

[0067] In this embodiment, after merging the optimized rank parameters with the first model parameters, the method further includes: determining a copy of the new teacher model as a new preset student model, and inserting a preset LoRA adapter into the target neural network layer of the new preset student model to obtain a new initial student model; freezing the second model parameters corresponding to the original weight matrix in the new initial student model to obtain a new student model to be trained, and using the new teacher model to process new medical data in a new medical scenario to obtain a new probability distribution corresponding to the processing result of the new medical data; training the new student model to be trained based on the new probability distribution to optimize the initial rank parameters of the preset LoRA adapter, thereby obtaining new optimized rank parameters; merging the new optimized rank parameters with the second model parameters to obtain a new trained student model, and using the new trained student model to process the new medical data.

[0068] As can be seen from the above steps, the student model and teacher model in this embodiment can achieve continuous learning without source data. That is, the student model and teacher model in this embodiment do not need to use sensitive medical data collected from different data sources as training data for model training; they only need to be trained using the probability distribution of the current medical data processing results by the teacher model. Continuous learning without source data avoids the risk of leakage of sensitive medical data and improves data security during model training.

[0069] In other words, in this embodiment, the trained student model serves as the teacher model to train the student model in new medical scenarios, thereby enabling continuous learning of the model as the medical scenarios are updated. Weight changes are implemented by optimizing the low-rank matrix, avoiding the catastrophic forgetting problem caused by directly overwriting the original parameters.

[0070] Therefore, this application adapts to new tasks with a minimal number of parameters by adding only low-rank matrices to the weights of key layers, avoiding catastrophic forgetting caused by global parameter updates, and also avoiding the problem of drastically increasing model complexity due to dynamically increasing the number of adapters. By utilizing knowledge distillation techniques, cross-domain knowledge transfer can be achieved, that is, feature alignment loss is constructed between the teacher model and the student model applied to the new domain, thereby enabling the model to continuously learn and process multi-source heterogeneous data, thus improving the model's generalization ability.

[0071] See Figure 3 As shown, this embodiment of the invention discloses a medical data processing device based on knowledge distillation and passive data continuous learning, comprising:

[0072] The weight matrix modification module 11 is used to determine a copy of the preset teacher model as a preset student model, insert a preset LoRA adapter into the target neural network layer of the preset student model, and use the preset LoRA adapter to add a low-rank matrix to the original weight matrix of the preset student model to obtain the corresponding initial student model; wherein, the preset teacher model is a pre-trained neural network model.

[0073] The parameter freezing module 12 is used to freeze the first model parameters corresponding to the original weight matrix in the initial student model to obtain the corresponding student model to be trained, and to use the preset teacher model to process the current medical data in the current medical scenario to obtain the probability distribution corresponding to the processing result of the current medical data.

[0074] The model training module 13 is used to determine the probability distribution as a soft label, and to train the student model to be trained based on the soft label and knowledge distillation technology, so as to optimize the initial rank parameter corresponding to the low-rank matrix in the initial student model and obtain the optimized rank parameter.

[0075] The medical data processing module 14 is used to merge the optimized rank parameter with the first model parameter to obtain the corresponding trained student model, determine the trained student model as the new teacher model, and use the new teacher model to process the current medical data.

[0076] Therefore, this application adapts to new tasks with a minimal number of parameters by adding only low-rank matrices to the weights of key layers, avoiding catastrophic forgetting caused by global parameter updates, and also avoiding the problem of drastically increasing model complexity due to dynamically increasing the number of adapters. By utilizing knowledge distillation techniques, cross-domain knowledge transfer can be achieved, that is, feature alignment loss is constructed between the teacher model and the student model applied to the new domain, thereby enabling the model to continuously learn and process multi-source heterogeneous data, thus improving the model's generalization ability.

[0077] In some specific embodiments, the model training module 13 may specifically include:

[0078] The task loss determination unit is used to process the current medical data using the student model to be trained in order to determine the target task loss; wherein, the target task loss is the difference between the output of the student model to be trained and the labeled data corresponding to the current medical data;

[0079] The distillation loss determination unit is used to determine the target distillation loss based on knowledge distillation technology, the soft label, the preset KL divergence function, and ROI technology; wherein, the target distillation loss is the difference between the output of the student model to be trained and the probability distribution;

[0080] The first model training unit is used to jointly optimize the target task loss and the target distillation loss in order to train the student model to be trained.

[0081] In some specific embodiments, the medical data processing module 14 further includes:

[0082] An adapter insertion unit is used to identify a copy of the new teacher model as a new preset student model and insert a preset LoRA adapter into the target neural network layer of the new preset student model to obtain a new initial student model.

[0083] The parameter freezing unit is used to freeze the second model parameters corresponding to the original weight matrix in the new initial student model to obtain a new student model to be trained, and to use the new teacher model to process new medical data in the new medical scenario to obtain a new probability distribution corresponding to the processing result of the new medical data.

[0084] The second model training unit is used to train the new student model to be trained based on the new probability distribution, so as to optimize the initial rank parameters of the preset LoRA adapter and obtain new optimized rank parameters.

[0085] The medical data processing unit is used to merge the new optimized rank parameter with the second model parameter to obtain a new trained student model, and to process the new medical data using the new trained student model.

[0086] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0087] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the medical data processing method based on knowledge distillation and passive data continuous learning disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0088] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0089] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0090] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the medical data processing method based on knowledge distillation and passive data continuous learning disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0091] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed medical data processing method based on knowledge distillation and passive data continuous learning. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0092] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0093] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0094] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A medical data processing method based on knowledge distillation and passive data continuous learning, characterized in that, include: A copy of the preset teacher model is identified as the preset student model. A preset LoRA adapter is inserted into the target neural network layer of the preset student model, and a low-rank matrix is ​​added to the original weight matrix of the preset student model using the preset LoRA adapter to obtain the corresponding initial student model; wherein, the preset teacher model is a pre-trained neural network model. The first model parameters corresponding to the original weight matrix in the initial student model are frozen to obtain the corresponding student model to be trained, and the current medical data in the current medical scenario is processed using the preset teacher model to obtain the probability distribution corresponding to the processing result of the current medical data. The probability distribution is determined as a soft label, and the student model to be trained is trained based on the soft label and knowledge distillation technology to optimize the initial rank parameter corresponding to the low-rank matrix in the initial student model and obtain the optimized rank parameter. The optimized rank parameter is merged with the first model parameter to obtain the corresponding trained student model. The trained student model is then determined as the new teacher model, and the new teacher model is used to process the current medical data.

2. The medical data processing method based on knowledge distillation and passive data continuous learning according to claim 1, characterized in that, The preset teacher model is a pre-trained neural network model for general medical tasks or a pre-trained neural network model trained in the previous medical scenario corresponding to the current medical scenario.

3. The medical data processing method based on knowledge distillation and passive data continuous learning according to claim 1, characterized in that, The target neural network layer includes at least one of the attention layer and feedforward network layer of the initial student model.

4. The medical data processing method based on knowledge distillation and passive data continuous learning according to claim 1, characterized in that, The training of the student model based on the soft labeling and knowledge distillation technique includes: The current medical data is processed using the student model to be trained to determine the target task loss; wherein, the target task loss is the difference between the output of the student model to be trained and the labeled data corresponding to the current medical data; The target distillation loss is determined based on knowledge distillation technology, the soft labels, the preset KL divergence function, and ROI technology; wherein, the target distillation loss is the difference between the output of the student model to be trained and the probability distribution; The target task loss and the target distillation loss are jointly optimized to train the student model to be trained.

5. The medical data processing method based on knowledge distillation and passive data continuous learning according to any one of claims 1 to 4, characterized in that, After merging the optimized rank parameter with the first model parameter, the method further includes: The copy of the new teacher model is identified as the new preset student model, and the preset LoRA adapter is inserted into the target neural network layer of the new preset student model to obtain a new initial student model; The second model parameters corresponding to the original weight matrix in the new initial student model are frozen to obtain a new student model to be trained, and the new teacher model is used to process new medical data in a new medical scenario to obtain a new probability distribution corresponding to the processing result of the new medical data. The new student model to be trained is trained based on the new probability distribution to optimize the initial rank parameters of the preset LoRA adapter, thereby obtaining new optimized rank parameters. The new optimized rank parameter is merged with the second model parameter to obtain a new trained student model, and the new trained student model is used to process the new medical data.

6. A medical data processing device based on knowledge distillation and passive data continuous learning, characterized in that, include: The weight matrix modification module is used to determine a copy of the preset teacher model as a preset student model, insert a preset LoRA adapter into the target neural network layer of the preset student model, and use the preset LoRA adapter to add a low-rank matrix to the original weight matrix of the preset student model to obtain the corresponding initial student model; wherein, the preset teacher model is a pre-trained neural network model. The parameter freezing module is used to freeze the first model parameters corresponding to the original weight matrix in the initial student model to obtain the corresponding student model to be trained, and to use the preset teacher model to process the current medical data in the current medical scenario to obtain the probability distribution corresponding to the processing result of the current medical data. The model training module is used to determine the probability distribution as a soft label, and to train the student model to be trained based on the soft label and knowledge distillation technology, so as to optimize the initial rank parameter corresponding to the low-rank matrix in the initial student model and obtain the optimized rank parameter. The medical data processing module is used to merge the optimized rank parameter with the first model parameter to obtain the corresponding trained student model, determine the trained student model as the new teacher model, and use the new teacher model to process the current medical data.

7. The medical data processing device based on knowledge distillation and passive data continuous learning according to claim 6, characterized in that, The model training module includes: The task loss determination unit is used to process the current medical data using the student model to be trained in order to determine the target task loss; wherein, the target task loss is the difference between the output of the student model to be trained and the labeled data corresponding to the current medical data; The distillation loss determination unit is used to determine the target distillation loss based on knowledge distillation technology, the soft label, the preset KL divergence function, and ROI technology; wherein, the target distillation loss is the difference between the output of the student model to be trained and the probability distribution; The first model training unit is used to jointly optimize the target task loss and the target distillation loss in order to train the student model to be trained.

8. The medical data processing device based on knowledge distillation and passive data continuous learning according to claim 6, characterized in that, The medical data processing module also includes: An adapter insertion unit is used to identify a copy of the new teacher model as a new preset student model and insert a preset LoRA adapter into the target neural network layer of the new preset student model to obtain a new initial student model. The parameter freezing unit is used to freeze the second model parameters corresponding to the original weight matrix in the new initial student model to obtain a new student model to be trained, and to use the new teacher model to process new medical data in the new medical scenario to obtain a new probability distribution corresponding to the processing result of the new medical data. The second model training unit is used to train the new student model to be trained based on the new probability distribution, so as to optimize the initial rank parameters of the preset LoRA adapter and obtain new optimized rank parameters. The medical data processing unit is used to merge the new optimized rank parameter with the second model parameter to obtain a new trained student model, and to process the new medical data using the new trained student model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the medical data processing method based on knowledge distillation and passive data continuous learning as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the medical data processing method based on knowledge distillation and passive data continuous learning as described in any one of claims 1 to 5.

Images