Method, system and device for dynamic fusion of missing multimodal data based on knowledge distillation

Abstract

The present invention discloses a method, system and device for dynamic fusion of missing multimodal data based on knowledge distillation, wherein multiple single-modal teacher models are constructed based on a classification network, and the classification network is trained separately by each single-modal data to obtain a corresponding single-modal teacher model; a multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks, wherein the threshold network is used to determine which expert networks are activated, and a one-hot vector is output, and the length of the vector is the number of expert networks, and the data used by each expert network is a subset of multiple modes for feature fusion, and the multimodal dynamic fusion network after training with data containing complete modes is used as a student model; the student model is distilled and trained using the teacher model, and the multimodal data actually obtained is input into the multimodal dynamic fusion network to obtain a category prediction result. The present invention can improve the effective utilization rate of data and improve the prediction accuracy of the multimodal model.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge distillation, and in particular to a method, system and device for dynamic fusion of missing multimodal data based on knowledge distillation. Background Art

[0002] Multimodal data, such as different image data, text, and vector features, reflect different aspects of things and contain complementary or overlapping information. Multimodal learning can use information from various data modalities to improve the accuracy, generalization, and stability of the model. In the medical field, images, clinical examination data, demographic information, and genetic testing information are all multimodal data of patients, which can comprehensively reflect the comprehensive information of patients. One type of medical data can be regarded as one type of modal data. When doctors diagnose diseases, choose treatment plans, and predict prognosis, they usually use different information to make comprehensive considerations. However, medical data often has a large missing rate, such as multimodal imaging data and genetic testing data are prone to missing situations. For example, in the diagnosis of non-small cell lung cancer, CT, clinical data, and gene expression can be combined to achieve higher diagnostic accuracy, but genetic testing is relatively expensive, and general patients do not have gene expression data.

[0003] A common method for processing multimodal data is to find common information shared between different modalities for learning. However, most existing multimodal methods need to learn from samples of complete modalities, and a large amount of collected data often has modality missing. Directly deleting samples of incomplete modalities will cause a large amount of data waste, and the trainable sample data will be greatly reduced, limiting the training performance of the model. Some methods use data filling to complete the missing data, but this method is often based on generative models, or uses neural network methods to fit missing variables, which will introduce unnecessary noise and cannot handle the situation of missing modalities.

[0004] On the other hand, the current multimodal fusion methods generally adopt a static fusion method of early fusion, mid-term fusion and late fusion, that is, a fixed fusion strategy is adopted for different input data. In this way, in different application scenarios, the importance of information provided by different modal data is not considered, and the modal information that provides important information cannot be highlighted. Therefore, the static fusion method cannot maximize the value of multimodal data. In disease diagnosis, doctors often use certain medical data to make the main judgment, and then combine other data to make a determination.

[0005] Existing multimodal fusion methods generally extract features from data, and then fuse them in the early, mid-term or late-term manner to obtain prediction results. The so-called early fusion is to concatenate, add or multiply the feature vectors of each modality to obtain a fusion vector, and then construct a classifier. Mid-term fusion is generally implemented by a convolutional neural network, and the feature representations of each modality are concatenated, multiplied or added in the middle layer of the network. Late fusion is to construct a classifier for each modality, and finally integrate the results of each modality to obtain a fused prediction. These are all based on the assumption that the modalities are complete and fixed. In reality, there are often missing modalities and missing data in many application scenarios such as medical and financial. And different tasks have different requirements for modalities. Requiring data with complete modalities will greatly reduce the amount of available training data. Using a static fusion method cannot flexibly respond to different task requirements. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art and to propose a method, system and device for dynamic fusion of missing multimodal data based on knowledge distillation.

[0007] The objective of the present invention is achieved through the following technical solution: a method for dynamic fusion of missing multimodal data based on knowledge distillation, the method comprising the following steps:

[0008] (1) Obtain medical data containing multiple modalities and corresponding task labels, and divide them into medical data containing complete modalities and medical data containing only a single modality;

[0009] (2) Construct multiple single-modality teacher models based on the classification network, and train the classification network separately through each single-modality medical data to obtain the corresponding single-modality teacher model;

[0010] (3) A multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks. The threshold network inputs medical data containing complete modalities and outputs a one-hot vector whose length is the number of expert networks. The medical data used by each expert network is a subset of multiple modalities for feature fusion. The multimodal dynamic fusion network trained with medical data containing complete modalities is used as the student model.

[0011] (4) The teacher model is used to distill the student model. The teacher model is used to predict the category of each medical data containing the complete modality and calculate the classification loss and distillation loss.

[0012] (5) The multimodal medical data actually obtained is input into the multimodal dynamic fusion network to obtain the category prediction results.

[0013] Furthermore, a 2-layer multilayer perceptron is used as the structure of the threshold network. After each modality of medical data is encoded separately through multiple feature encoders, it is input into the threshold network. The threshold network is used to decide which expert networks are activated.

[0014] Furthermore, the output of the multimodal dynamic fusion network is a synthesis of multiple expert networks, using a one-hot vector representation so that only one expert network is used for each instance.

[0015] Furthermore, the architecture of the expert network selects a 2-layer multi-layer perceptron, which plays the role of modality selection and feature fusion. Feature fusion includes concatenation, addition and multiplication operations to achieve the combination of different modal data.

[0016] Furthermore, the trained teacher model and student model are combined to form a teacher-student multimodal dynamic fusion network. The student model is distilled and trained using the teacher model, and the parameters of the teacher-student multimodal dynamic fusion network are updated using the stochastic gradient descent algorithm until convergence.

[0017] Furthermore, the classification loss uses non-negative cross entropy loss, and the distillation loss uses KL divergence.

[0018] In a second aspect, the present invention further provides a missing multimodal data dynamic fusion system based on knowledge distillation, the system comprising:

[0019] A multimodal data acquisition module is used to acquire medical data containing multiple modalities and corresponding task labels, and divide the medical data containing complete modalities and the medical data containing only a single modality;

[0020] The teacher network model training module is used to construct multiple single-modality teacher models based on the classification network, and the classification network is trained separately through each single-modality medical data to obtain the corresponding single-modality teacher model;

[0021] The multimodal dynamic fusion network training module is used to build a multimodal dynamic fusion network based on a threshold network and a series of expert networks. The threshold network inputs medical data containing complete modalities and outputs a one-hot vector whose length is the number of expert networks. The data used by each expert network is a subset of multiple modalities for feature fusion. The multimodal dynamic fusion network trained with medical data containing complete modalities is used as a student model.

[0022] The teacher-student distillation module is used to distill the student model using the teacher model. The teacher model is used to predict the category of each medical data containing the complete modality and calculate the classification loss and distillation loss.

[0023] The multimodal data category prediction module is used to input the actually acquired multimodal medical data into the multimodal dynamic fusion network to obtain the category prediction result.

[0024] In the third aspect, the present invention also provides a device for dynamic fusion of missing multimodal data based on knowledge distillation, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the method for dynamic fusion of missing multimodal data based on knowledge distillation is implemented.

[0025] In a fourth aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0026] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0027] Beneficial effects of the present invention: The present invention designs a dynamic multimodal fusion network and proposes a method for adaptively dynamically fusing input data of different modalities, which can improve the representation ability and effective fusion of multimodal data and overcome the problem of high missing rate of multimodal data. In addition, a knowledge distillation framework is proposed to make full use of the existing incomplete unimodal data, train the teacher models of each modality, use the multimodal dynamic fusion model of the complete multimodal data as the student model, and let the teacher model guide the training of the student model, so that the supplementary information of a large amount of unimodal data improves the performance of the full modal model of a small amount of data, improves the effective utilization of data and improves the prediction accuracy of the multimodal model. At the same time, dynamic fusion of data of different modalities is achieved for different task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a schematic flow chart of the method for dynamic fusion of missing multimodal data based on knowledge distillation of the present invention.

[0030] Figure 2 It is a schematic diagram of the multi-modal dynamic fusion network process of the present invention.

[0031] Figure 3It is a structural diagram of a missing multimodal data dynamic fusion device based on knowledge distillation provided by the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] like Figure 1 As shown, the present invention provides a method for dynamic fusion of missing multimodal data based on knowledge distillation, which constructs a series of expert models containing different modal combinations and different fusion methods, and uses a threshold network to adaptively activate different expert models to dynamically fuse different modal data. The specific implementation process is as follows:

[0034] 1. Obtain multimodal data

[0035] 1) Obtain the patient's medical imaging data, clinical data, gene expression data, and task labels. The task is a classification task, such as disease diagnosis results and prognosis prediction results. Medical imaging data is generally X-ray, CT or MRI data.

[0036] 2) Preprocess or extract features for each type of extracted data. For imaging data, extract imaging features based on the delineated lesion area or region of interest. Based on the Pyradiomics toolkit, first-order statistical features, shape features, and texture features (GLCM, GLRLM, NGTDM, GLDM) were calculated, and a total of 85 imaging features were calculated.

[0037] For the acquired clinical data, several variables related to the task disease are determined, such as the patient's demographic information such as age, gender, medical history, lifestyle, etc., and the test results such as tumor markers such as CA125. These variables are digitized, for example, for gender, male is set to 1 and female is set to 0. Then the clinical information is normalized to obtain clinical characteristics.

[0038] The acquired gene expression data were normalized by z-score.

[0039] 3) The obtained imaging features, clinical features and gene expression features are used as the multimodal data of the patient. It is not required that the three modal data of each patient are complete, and missing data are allowed.

[0040] 2. Use each unimodal data to train the teacher model

[0041] Assume that the obtained data is expressed as: , where M is the number of modalities, here M=3, and the modalities include imaging, clinical data, and gene expression data. represents the data vector length of the i-th mode, i=1,...,M, and their corresponding task labels are The data containing the complete modality is defined as ,in is the number of data. The data containing only one mode is defined as , ,..., , ,..., ,in Indicates the number of data containing only the i-th mode.

[0042] 3. Construct a unimodal model as the teacher model.

[0043] Build a single-modal classification network. Select ResNet18 as the network. You can also select other classification network structures. Initialize ResNet18 with the pre-trained network parameters on ImageNet. Then, select all data containing the first modality. , used to train the ResNet18 network. Set the loss function of the first single-mode network to , trained to obtain a unimodal model , as the first unimodal teacher model. Among them, are network parameters, is a non-negative cross entropy loss, It is a softmax function, specifically defined as , represents the forward propagation process of the network, represents the true label of the i-th data, Represents the amount of data with the first modality.

[0044] For other modes, the teacher model is obtained in the same way as above .

[0045] 4. Construction of multi-modal dynamic fusion network

[0046] 1) Build a multimodal dynamic fusion network through a hybrid expert approach, such as Figure 2As shown in the figure, the network includes multiple feature encoders, a threshold network and a series of expert networks. Mixture-of-Experts (MoE) is a machine learning technology that divides a single task space into multiple subtasks through a gating model, and then multiple expert networks (sub-models) process specific subtasks respectively, and finally obtain the overall prediction result. The feature encoder can use the encoder part of ResNet18.

[0047] 2) Design a threshold network denoted as G(X). A 2-layer multilayer perceptron can be used as the structure of the threshold network. The threshold network determines which expert networks are activated. The input of G(X) is the multimodal data encoded by the feature encoder for each modal data, and the output is a vector g with a length of B, where B is the number of expert networks. Then, the output of the multimodal dynamic fusion network is a synthesis of B expert models, that is, . g is a one-hot vector, that is, for each instance, only one expert model is used.

[0048] 3) Design a series of expert network models. The multimodal data used by each expert network model is a subset of M modes, and different feature fusion methods are considered, including splicing, addition, and multiplication operations as feature fusion strategies. , multiplication is , adding . The number of feature types can be extended to multiple.

[0049] Set the number of expert networks to B. If the number of modes M = 3, and only consider the combination of different modes, B = 7, the expert network set contains If three fusion methods are considered, then B=7×3-3=18. The number of expert networks can also be selected according to actual conditions. The architecture of the expert network selects a 2-layer multi-layer perceptron, which mainly plays the role of modality selection and feature fusion.

[0050] 4) Utilize all data containing complete modalities , train the multimodal dynamic fusion network, and the obtained model is used as the student model S.

[0051] 5) Use the teacher model obtained above to predict the category of each data containing complete modality. Get the predicted value

[0052]

[0053] Among them, i represents the i-th data, Represents the prediction result obtained by the first teacher model for the i-th data.

[0054] 5. Teacher-Student Distillation

[0055] Using the information of the teacher model, the student model is improved under the knowledge distillation learning framework. The teacher model trained previously and the multimodal dynamic fusion student model are combined to form a teacher-student multimodal dynamic fusion network. Specifically,

[0056] 1) Set the loss function of the teacher-student multimodal dynamic fusion network to

[0057]

[0058] in is the classification loss, is the distillation loss corresponding to the Mth modal teacher model. To adjust the parameters, we control the influence of the teacher model on the student model. The classification loss is specifically defined as:

[0059]

[0060] in represents the forward propagation function of the multimodal dynamic fusion network, Represents the parameters of the teacher-student multimodal dynamic fusion network. is a non-negative cross entropy loss, is a softmax function. The distillation loss is specifically defined as:

[0061]

[0062] Where KL is the KL (Kullback-Leibler divergence) divergence, which is a measure of the asymmetry of the difference between two probability distributions P and Q. KL divergence is used to measure the average number of additional bits required to encode samples that follow the distribution of P using a distribution based on Q. Typically, P represents the true distribution of the data, and Q represents the theoretical distribution of the data, the estimated model distribution, or the approximate distribution of P. The KL divergence formula is defined as:

[0063] It is a softmax function with temperature T, which plays a role of stretching and smoothing. The larger T is, the smoother the output probability value will be. The output probabilities are the smoothed soft labels output by the teacher model. Defined as

[0064]

[0065] 2) Use the stochastic gradient descent algorithm to dynamically fusion the parameters of the teacher-student multimodal network Update until convergence. In this way, the original multimodal dynamic fusion network learns the information of the teacher model and obtains a training-enhanced multimodal dynamic fusion network model.

[0066] 6. Use the trained multimodal dynamic fusion network to predict the classification results.

[0067] For the patients being tested, medical data such as imaging, clinical or gene expression data will be collected. Then, after preprocessing or feature extraction of each modality data, multimodal test data about these test patients is obtained. The test data is input into the enhanced multimodal dynamic fusion network to obtain the predicted value, i.e., the task label:

[0068]

[0069] On the other hand, corresponding to the aforementioned embodiment of a method for dynamic fusion of missing multimodal data based on knowledge distillation, the present invention also provides an embodiment of a system for dynamic fusion of missing multimodal data based on knowledge distillation. It includes a multimodal data acquisition module, a teacher network model training module, a multimodal dynamic fusion network training module, a teacher-student distillation module and a multimodal data category prediction module; for the specific implementation process of each module, please refer to the implementation steps of the aforementioned embodiment of a method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0070] The multimodal data acquisition module is used to acquire data containing multiple modalities and corresponding task labels, and divide the data containing the complete modality and the data containing only a single modality;

[0071] The teacher network model training module is used to construct multiple single-modal teacher models based on the classification network, and the classification network is trained separately through each single-modal data to obtain the corresponding single-modal teacher model;

[0072] The multimodal dynamic fusion network training module is used to train a multimodal dynamic fusion network using data from the complete modality; in the network, different modal combinations and different fusion methods are set, and a threshold network is used to dynamically adjust the modality and fusion method used. Specifically: a multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks. The threshold network inputs data containing the complete modality and outputs a one-hot vector. The length of the vector is the number of expert networks. The data used by each expert network is a subset of multiple modalities for feature fusion. The multimodal dynamic fusion network trained with data containing the complete modality is used as a student model.

[0073] The teacher-student distillation module is used to distill the student model using the teacher model, calculate the classification loss and the distillation loss by predicting the category of each data containing the complete modality through the teacher model, and guide the performance of the student model by using the predicted soft label of the teacher model;

[0074] The multimodal data category prediction module is used to input the actually acquired multimodal data into the multimodal dynamic fusion network to obtain a category prediction result.

[0075] Corresponding to the aforementioned embodiment of a method for dynamic fusion of missing multimodal data based on knowledge distillation, the present invention also provides an embodiment of a device for dynamic fusion of missing multimodal data based on knowledge distillation.

[0076] See also Figure 3 An embodiment of the present invention provides a device for dynamic fusion of missing multimodal data based on knowledge distillation, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a method for dynamic fusion of missing multimodal data based on knowledge distillation in the above embodiment.

[0077] An embodiment of a device for dynamic fusion of missing multimodal data based on knowledge distillation provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 3 As shown, it is a hardware structure diagram of any device with data processing capability where a missing multimodal data dynamic fusion device based on knowledge distillation provided by the present invention is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0078] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0079] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0080] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for dynamic fusion of missing multimodal data based on knowledge distillation in the above embodiment is implemented.

[0081] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0082] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0083] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for dynamic fusion of missing multimodal data based on knowledge distillation, characterized in that: The method comprises the following steps: (1) Obtain medical data containing multiple modalities and corresponding task labels, and divide them into medical data containing complete modalities and medical data containing only a single modality; (2) Construct multiple single-modality teacher models based on the classification network, and train the classification network separately through each single-modality medical data to obtain the corresponding single-modality teacher model; (3) A multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks. The threshold network inputs medical data containing complete modalities and outputs a one-hot vector whose length is the number of expert networks. The threshold network is used to determine which expert networks are activated. The medical data used by each expert network is a subset of multiple modalities for feature fusion. The output of the multimodal dynamic fusion network is a synthesis of multiple expert networks. A one-hot vector is used to indicate that only one expert network is used for each instance. The multimodal dynamic fusion network trained with medical data containing complete modalities is used as a student model. (4) The trained teacher model and student model are combined to form a teacher-student multimodal dynamic fusion network. The teacher model is used to distill and train the student model. The teacher model is used to predict the category of each medical data containing complete modalities to calculate the classification loss and distillation loss. The parameters of the teacher-student multimodal dynamic fusion network are updated using the stochastic gradient descent algorithm until convergence. The original multimodal dynamic fusion network learns the information of the teacher model and obtains a training-enhanced multimodal dynamic fusion network model. (5) The multimodal medical data actually obtained is input into the training enhanced multimodal dynamic fusion network model to obtain the category prediction results.

2. According to claim 1, a method for dynamic fusion of missing multimodal data based on knowledge distillation is characterized in that: A 2-layer multilayer perceptron is used as the structure of the threshold network. Each modality of medical data is encoded separately through multiple feature encoders and then input into the threshold network.

3. According to claim 1, a method for dynamic fusion of missing multimodal data based on knowledge distillation is characterized in that: The architecture of the expert network selects a 2-layer multi-layer perceptron, which plays the role of modality selection and feature fusion. Feature fusion includes concatenation, addition and multiplication operations to realize the combination of different modal data.

4. According to claim 1, a method for dynamic fusion of missing multimodal data based on knowledge distillation is characterized in that: The classification loss uses non-negative cross entropy loss, and the distillation loss uses KL divergence.

5. A missing multimodal data dynamic fusion system based on knowledge distillation, characterized in that: The system includes: A multimodal data acquisition module is used to acquire medical data containing multiple modalities and corresponding task labels, and divide the medical data containing complete modalities and the medical data containing only a single modality; The teacher network model training module is used to construct multiple single-modality teacher models based on the classification network, and the classification network is trained separately through each single-modality medical data to obtain the corresponding single-modality teacher model; The multimodal dynamic fusion network training module is used to build a multimodal dynamic fusion network based on a threshold network and a series of expert networks. The threshold network inputs medical data containing complete modalities and outputs a one-hot vector whose length is the number of expert networks. The threshold network is used to determine which expert networks are activated. The data used by each expert network is a subset of multiple modalities for feature fusion. The output of the multimodal dynamic fusion network is a synthesis of multiple expert networks. A one-hot vector is used to indicate that only one expert network is used for each instance. The multimodal dynamic fusion network trained with medical data containing complete modalities is used as a student model. The teacher-student distillation module is used to distill the student model using the teacher model. The teacher model is used to predict the category of each medical data containing the complete modality and calculate the classification loss and distillation loss. The multimodal data category prediction module is used to input the actually acquired multimodal medical data into the multimodal dynamic fusion network to obtain the category prediction result.

6. A device for dynamic fusion of missing multimodal data based on knowledge distillation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a missing multimodal data dynamic fusion method based on knowledge distillation as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a method for dynamic fusion of missing multimodal data based on knowledge distillation as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a method for dynamic fusion of missing multimodal data based on knowledge distillation as described in any one of claims 1 to 4 is implemented.

Images