Knowledge-distillation-based dynamic fusion method, system and apparatus for missing multi-modal data

Abstract

Disclosed in the present invention are a knowledge-distillation-based dynamic fusion method, system and apparatus for missing multi-modal data. The method comprises: on the basis of a classification network, constructing a plurality of single-modal teacher models, and respectively preforming training on the classification network by means of each type of single-modal data to obtain corresponding single-modal teacher models; on the basis of a threshold network and a series of expert networks, constructing a multi-modal dynamic fusion network, wherein the threshold network is used for determining which expert networks are activated and outputting one one-hot vector, the length of the vector is the number of expert networks, the data used by each expert network is a subset of a plurality of modalities for feature fusion, and the multi-modal dynamic fusion network, on which training performed using data including complete modalities has been completed, is used as a student model; and using the teacher models to perform distillation training on the student model, and inputting actually acquired multi-modal data into the multi-modal dynamic fusion network to obtain a category prediction result. The present invention can increase the effective utilization rate of data and improve the prediction accuracy of multi-modal models.

Description

A method, system, and apparatus for dynamic fusion of missing multimodal data based on knowledge distillation. Technical Field

[0001] This invention relates to the field of knowledge distillation technology, and in particular to a method, system and apparatus for dynamic fusion of missing multimodal data based on knowledge distillation. Background Technology

[0002] Multimodal data, such as different image data, text, and vector features, reflects different aspects of information about things, containing complementary or overlapping information. Multimodal learning can utilize information from various data modalities to improve the accuracy, generalization, and stability of models. In the medical field, imaging, clinical examination data, demographic information, and genetic testing information all belong to multimodal patient data, which can comprehensively reflect the patient's overall information. Each type of medical data can be considered a modality of data. Doctors typically use different information to make comprehensive considerations when making disease diagnoses, choosing treatment plans, and predicting prognosis. However, medical data often has a high rate of missing data; for example, multimodal imaging data and genetic testing data are prone to missing information. For instance, in the diagnosis of non-small cell lung cancer, CT scans, clinical data, and gene expression data can be combined to achieve higher diagnostic accuracy, but gene testing is expensive, and most patients do not have gene expression data.

[0003] A common approach to multimodal data processing is to learn from shared information among different modalities. However, most existing multimodal methods require learning from samples of complete modalities, while large amounts of collected data often contain missing modalities. Directly deleting samples of incomplete modalities results in a significant waste of data, greatly reducing the amount of trainable data and limiting the model's training performance. Some methods use data imputation to fill in the missing data, but this approach is often based on generative models or uses neural networks to fit the missing variables, introducing unnecessary noise and failing to handle cases of missing modalities.

[0004] On the other hand, current multimodal fusion methods generally employ a static fusion approach, consisting of early-stage fusion, mid-stage fusion, and late-stage fusion. This means a fixed fusion strategy is used for different input data. This approach fails to consider the varying importance of information provided by different modalities in different application scenarios, and thus cannot highlight the modalities that provide crucial information. Therefore, static fusion methods cannot maximize the value of multimodal data. In disease diagnosis, however, doctors often make a primary judgment based on certain medical data and then combine it with other data to confirm the diagnosis.

[0005] Existing multimodal fusion methods typically extract features from the data and then fuse them using early, mid, or late-stage approaches to obtain a prediction result. Early-stage fusion involves concatenating, adding, or multiplying the feature vectors of each modality to obtain a fused vector, which is then used to build a classifier. Mid-stage fusion is generally implemented using convolutional neural networks, where the feature representations of each modality are concatenated, multiplied, or added in the middle layers of the network. Late-stage fusion involves building classifiers for each modality and finally combining the results from all modalities to obtain a fused prediction. These methods are all based on the assumption that all modalities are complete and fixed. However, in reality, many applications, such as healthcare and finance, often suffer from missing modalities and data gaps. Furthermore, different tasks have different modality requirements. Requiring complete modality data significantly reduces the amount of usable training data. Static fusion methods cannot flexibly address the diverse task requirements. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a dynamic fusion method, system, and device for missing multimodal data based on knowledge distillation.

[0007] The objective of this invention is achieved through the following technical solution: a dynamic fusion method for missing multimodal data based on knowledge distillation, the method comprising the following steps:

[0008] (1) Obtain medical data containing multiple modalities and corresponding task labels, and divide the medical data containing complete modalities into medical data containing only a single modality;

[0009] (2) Construct multiple unimodal teacher models based on the classification network, and train the classification network separately using each individual modality of medical data to obtain the corresponding unimodal teacher model;

[0010] (3) A multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks. The threshold network takes in medical data containing complete modalities as input and outputs a one-hot vector. The length of the vector is the number of expert networks. Each expert network uses a subset of medical data containing multiple modalities for feature fusion. The multimodal dynamic fusion network trained with medical data containing complete modalities is used as the student model.

[0011] (4) Distill the student model using the teacher model. The classification loss and distillation loss are calculated for each medical data containing a complete modality by performing category prediction using the teacher model.

[0012] (5) Input the actual acquired multimodal medical data into the multimodal dynamic fusion network to obtain the category prediction results.

[0013] Furthermore, a two-layer multilayer perceptron is used as the structure of the threshold network. Each modality of medical data is encoded by multiple feature encoders and then input into the threshold network. The threshold network is used to determine which expert networks are activated.

[0014] Furthermore, the output of the multimodal dynamic fusion network is a synthesis of multiple expert networks, using a one-hot vector representation where only one expert network is used for each instance.

[0015] Furthermore, the expert network architecture is designed with a two-layer multilayer perceptron, which plays a role in modality selection and feature fusion. Feature fusion includes concatenation, addition, and multiplication operations to achieve the combination of data from different modalities.

[0016] Furthermore, the trained teacher and student models are combined to form a teacher-student multimodal dynamic fusion network. The student model is trained by distillation using the teacher model, and the parameters of the teacher-student multimodal dynamic fusion network are updated using the stochastic gradient descent algorithm until convergence.

[0017] Furthermore, the classification loss uses non-negative cross-entropy loss, and the distillation loss uses KL divergence.

[0018] Secondly, the present invention also provides a dynamic fusion system for missing multimodal data based on knowledge distillation, the system comprising:

[0019] The multimodal data acquisition module is used to acquire medical data containing multiple modalities and corresponding task labels, and to divide the medical data into those containing complete modalities and those containing only a single modality.

[0020] The teacher network model training module is used to construct multiple unimodal teacher models based on the classification network. The classification network is trained separately using medical data of each individual modality to obtain the corresponding unimodal teacher model.

[0021] The multimodal dynamic fusion network training module is used to construct a multimodal dynamic fusion network based on a threshold network and a series of expert networks. The threshold network takes medical data containing complete modalities as input and outputs a one-hot vector. The length of the vector is the number of expert networks. Each expert network uses a subset of data from multiple modalities for feature fusion. The multimodal dynamic fusion network trained with medical data containing complete modalities serves as the student model.

[0022] The teacher-student distillation module is used to distill the student model using the teacher model. It calculates the classification loss and distillation loss for each medical data point containing a complete modality using the teacher model.

[0023] The multimodal data category prediction module is used to input the actual acquired multimodal medical data into the multimodal dynamic fusion network to obtain the category prediction results.

[0024] Thirdly, the present invention also provides a device for dynamic fusion of missing multimodal data based on knowledge distillation, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0025] Fourthly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0026] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0027] The beneficial effects of this invention are as follows: This invention designs a dynamic multimodal fusion network and proposes an adaptive method for dynamically fusing input data from different modalities. This improves the representational ability and effective fusion of multimodal data, overcoming the problem of high missing data rates in multimodal data. Furthermore, it proposes a knowledge distillation framework, fully utilizing existing incomplete single-modal data to train teacher models for each modality. The dynamic multimodal fusion model with complete multimodal data is used as a student model, allowing the teacher model to guide the training of the student model. This enables the supplementary information from a large amount of single-modal data to improve the performance of the full-modal model with limited data, increasing the effective utilization of data and improving the predictive accuracy of the multimodal model. Simultaneously, it achieves dynamic fusion of different modalities for different task requirements. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 is a schematic diagram of the process of the dynamic fusion method for missing multimodal data based on knowledge distillation according to the present invention.

[0030] Figure 2 is a schematic diagram of the multimodal dynamic fusion network process of the present invention.

[0031] Figure 3 is a structural diagram of a dynamic fusion device for missing multimodal data based on knowledge distillation provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention.

[0033] As shown in Figure 1, this invention provides a dynamic fusion method for missing multimodal data based on knowledge distillation. It constructs a series of expert models containing different modal combinations and fusion methods, and uses a threshold network to adaptively activate different expert models, dynamically fusing data from different modalities. The specific implementation process is as follows:

[0034] 1. Acquire multimodal data

[0035] 1) Acquire the patient's medical imaging data, clinical data, and gene expression data, as well as task labels. The task is a classification task, such as disease diagnosis results or prognostic prediction results. Medical imaging data generally includes X-ray films, CT scans, or MRI data.

[0036] 2) Preprocessing or feature extraction was performed on each type of extracted data. For image data, radiomics features were extracted based on the delineated lesion area or region of interest. First-order statistical features, shape features, and texture features (GLCM, GLRLM, NGTDM, GLDM) were calculated using the Pyradiomics toolkit, resulting in a total of 85 image features.

[0037] For the acquired clinical data, several variables related to the task disease are identified, such as patient demographic information like age, sex, medical history, and lifestyle habits, and laboratory test results like tumor markers such as CA125. These variables are then numerically represented; for example, for sex, male is set to 1 and female to 0. The clinical information is then normalized to obtain the clinical characteristics.

[0038] The acquired gene expression data were normalized using z-score.

[0039] 3) The obtained imaging features, clinical features, and gene expression features are used as multimodal data for patients. It is not required that all three modalities of data for each patient be complete; missing data are allowed.

[0040] 2. Train the teacher model using data from each single modality.

[0041] Assume the obtained data is expressed as follows: Where M represents the number of modalities, here M=3, and modalities include imaging, clinical data, and gene expression data. iLet represent the length of the data vector for the i-th modality, i = 1, ..., M, and let M be the corresponding task labels. Define data containing complete modes as Where n c The number of data points. Data containing only one modality is defined as... n1 = n c +n 1u ,...,n M =n c +n Mu , where n iu This represents the number of data items that contain only the i-th modality.

[0042] 3. Construct a single-modal model as the teacher model.

[0043] Construct a unimodal classification network, using ResNet18 as the initial choice, but other classification network architectures can also be selected. Initialize ResNet18 using the parameters of a pre-trained network on ImageNet. Then, select all data containing the first modality. Used to train the ResNet18 network. The loss function for the first unimodal network is set to... The trained unimodal model T1(φ1) is used as the first unimodal teacher model. Here, φ1 represents the network parameters, H is a non-negative cross-entropy loss, and σ() is a softmax function, specifically defined as... g1 represents the forward propagation process of the network, y i Let n represent the true label of the i-th data point, and n1 represent the amount of data with the first modality.

[0044] For other modalities, the teacher models T2(φ2),...,T are obtained in the same way as above. M (φ M ).

[0045] 4. Construction of Multimodal Dynamic Fusion Network

[0046] 1) A multimodal dynamic fusion network is constructed using a hybrid expert approach, as shown in Figure 2. This network includes multiple feature encoders, a gate network, and a series of expert networks. Mixture-of-Experts (MoE) is a machine learning technique that divides a single task space into multiple sub-tasks using a gating model, and then multiple expert networks (sub-models) process specific sub-tasks respectively, ultimately obtaining the overall prediction result. The feature encoders can use the encoder part of ResNet18.

[0047] 2) Design a threshold network denoted as G(X), which can be structured using a two-layer multilayer perceptron. The threshold network determines which expert networks are activated. The input of G(X) is multimodal data encoded by a feature encoder for each modality, and the output is a vector g of length B, where B is the number of expert networks. Therefore, the output of the multimodal dynamic fusion network is a synthesis of B expert models, i.e. g is a one-hot vector, meaning that for each instance, only one expert model is used.

[0048] 3) Design a series of expert network models. Each expert network model uses a subset of M modalities of multimodal data and considers different feature fusion methods, including concatenation, addition, and multiplication operations, as feature fusion strategies. Specifically, concatenation is f = concate[f1, f2], multiplication is f = f1·f2, and addition is f = f1 + f2. The number of feature types can be generalized to multiple.

[0049] Let the number of expert networks be B. If the number of modes M = 3, and only considering combinations of different modes, B = 7, then the expert network set includes E1(X). 1 ),E2(X 2 ),E3(X 3 E4(X) 1 ,X 2 ),E5(X 1 ,X 3 ),E6(X 2 ,X 3 ),E7(X 1 ,X 2 ,X 3 If we consider three fusion methods, then B = 7 × 3 - 3 = 18. The number of expert networks can also be selected according to the actual situation. The expert network architecture is chosen to be a two-layer multilayer perceptron, which mainly plays the role of modality selection and feature fusion.

[0050] 4) Utilize all data containing complete modalities The multimodal dynamic fusion network is trained, and the resulting model is used as the student model S.

[0051] 5) Using the teacher model obtained earlier, perform category prediction for each data point containing a complete modality. Obtain the predicted values.

[0052] Where i represents the i-th data. This represents the prediction result obtained by the first teacher model for the i-th data.

[0053] 5. Teacher-Student Distillation

[0054] By leveraging information from the teacher model within a knowledge distillation learning framework, the capabilities of the student model are enhanced. The previously trained teacher model and the multimodal dynamic fusion student model are combined to form a teacher-student multimodal dynamic fusion network. Specifically,

[0055] 1) Set the loss function of the teacher-student multimodal dynamic fusion network as follows:

[0056] in It is classification loss. This represents the distillation loss corresponding to the Mth modal teacher model. α m To adjust the parameters, the influence of the teacher model on the student model was controlled. The classification loss was specifically defined as:

[0057] Where f multi Let H represent the forward propagation function of the multimodal dynamic fusion network, and 0 represent the parameters of the teacher-student multimodal dynamic fusion network. H is a non-negative cross-entropy loss, and σ() is a softmax function. The distillation loss is specifically defined as:

[0058] Where KL stands for Kullback-Leibler divergence, it is a measure of the asymmetry of the difference between two probability distributions P and Q. KL divergence measures the additional average number of bits required to encode samples following a distribution P using a Q-based distribution. Typically, P represents the true distribution of the data, and Q represents the theoretical distribution, estimated model distribution, or approximate distribution of P. The KL divergence formula is defined as follows:

[0059] σ T (x;T) is a softmax function with temperature T applied, where temperature T acts as a stretching and smoothing agent. The larger T is, the smoother the output probability values ​​will be. σ T The probability output (x;T) is a smoothed soft label from the teacher model's output. σ T (x;T) is defined as

[0060] 2) The parameters θ of the teacher-student multimodal dynamic fusion network are updated using the stochastic gradient descent algorithm until convergence. In this way, the original multimodal dynamic fusion network learns information from the teacher model, resulting in a trained and enhanced multimodal dynamic fusion network model.

[0061] 6. Use the trained multimodal dynamic fusion network to predict the classification results.

[0062] For the test patients, available medical data such as imaging, clinical data, or gene expression data will be collected. Then, after preprocessing or feature extraction of each modality of data, multimodal test data for these test patients will be obtained. The test data will be input into an enhanced multimodal dynamic fusion network to obtain predicted values, i.e., task labels.

[0063] On the other hand, corresponding to the aforementioned embodiment of a dynamic fusion method for missing multimodal data based on knowledge distillation, this invention also provides an embodiment of a dynamic fusion system for missing multimodal data based on knowledge distillation. This system includes a multimodal data acquisition module, a teacher network model training module, a multimodal dynamic fusion network training module, a teacher-student distillation module, and a multimodal data category prediction module. For the specific implementation process of each module, please refer to the implementation steps of the aforementioned embodiment of the dynamic fusion method for missing multimodal data based on knowledge distillation.

[0064] The multimodal data acquisition module is used to acquire data containing multiple modalities and corresponding task labels, and to divide the data into data containing complete modalities and data containing only a single modality;

[0065] The teacher network model training module is used to construct multiple unimodal teacher models based on the classification network. The classification network is trained separately using data of each individual modality to obtain the corresponding unimodal teacher model.

[0066] The multimodal dynamic fusion network training module is used to train a multimodal dynamic fusion network using data from all modalities. This network incorporates different modality combinations and fusion methods, and uses a threshold network to dynamically adjust the modalities and fusion methods used. Specifically, a multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks. The threshold network takes data containing all modalities as input and outputs a one-hot vector, the length of which is the number of expert networks. Each expert network uses a subset of data from multiple modalities for feature fusion. The multimodal dynamic fusion network trained using data containing all modalities serves as the student model.

[0067] The teacher-student distillation module is used to distill the student model using the teacher model. The teacher model performs category prediction and calculates classification loss and distillation loss for each data point containing a complete modality. The predicted soft labels of the teacher model are used to guide the performance of the student model.

[0068] The multimodal data category prediction module is used to input the actual acquired multimodal data into the multimodal dynamic fusion network to obtain the category prediction result.

[0069] Corresponding to the aforementioned embodiment of a dynamic fusion method for missing multimodal data based on knowledge distillation, the present invention also provides an embodiment of a dynamic fusion device for missing multimodal data based on knowledge distillation.

[0070] Referring to Figure 3, an embodiment of the present invention provides a dynamic fusion device for missing multimodal data based on knowledge distillation, which includes a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a dynamic fusion method for missing multimodal data based on knowledge distillation in the above embodiment.

[0071] The embodiment of the knowledge distillation-based missing multimodal data dynamic fusion device provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the device with data processing capabilities reading the corresponding computer program instructions from non-volatile memory into memory and running them. From a hardware perspective, as shown in Figure 3, it is a hardware structure diagram of any device with data processing capabilities where the knowledge distillation-based missing multimodal data dynamic fusion device provided by this invention is located. Besides the processor, memory, network interface, and non-volatile memory shown in Figure 3, the device with data processing capabilities in the embodiment may also include other hardware depending on the actual function of the device, which will not be elaborated further.

[0072] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0073] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0074] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a method for dynamic fusion of missing multimodal data based on knowledge distillation as described in the above embodiments.

[0075] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0076] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for dynamic fusion of missing multimodal data based on knowledge distillation.

[0077] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A dynamic fusion method for missing multimodal data based on knowledge distillation, characterized in that, The method includes the following steps: (1) Obtain medical data containing multiple modalities and corresponding task labels, and divide the medical data containing complete modalities into medical data containing only a single modality; (2) Construct multiple unimodal teacher models based on the classification network, and train the classification network separately using each individual modality of medical data to obtain the corresponding unimodal teacher model; (3) A multimodal dynamic fusion network is constructed based on a threshold network and a series of expert networks. The threshold network takes in medical data containing complete modalities as input and outputs a one-hot vector with the length of the vector being the number of expert networks. The threshold network is used to determine which expert networks are activated. Each expert network uses a subset of multiple modalities of medical data for feature fusion. The output of the multimodal dynamic fusion network is a synthesis of multiple expert networks. The one-hot vector indicates that only one expert network is used for each instance. The multimodal dynamic fusion network trained with medical data containing complete modalities is used as the student model. (4) The trained teacher model and student model are combined to form a teacher-student multimodal dynamic fusion network. The student model is trained by distillation using the teacher model. The classification loss and distillation loss are calculated by class prediction for each medical data containing complete modalities using the teacher model. The parameters of the teacher-student multimodal dynamic fusion network are updated using the stochastic gradient descent algorithm until convergence. The original multimodal dynamic fusion network learns the information of the teacher model and obtains a trained and enhanced multimodal dynamic fusion network model. (5) Input the actual acquired multimodal medical data into the trained and enhanced multimodal dynamic fusion network model to obtain the category prediction results.

2. The method for dynamic fusion of missing multimodal data based on knowledge distillation according to claim 1, characterized in that, A two-layer multilayer perceptron is used as the structure of the threshold network. Each modality of medical data is encoded by multiple feature encoders and then input into the threshold network.

3. The method for dynamic fusion of missing multimodal data based on knowledge distillation according to claim 1, characterized in that, The expert network architecture uses a two-layer multilayer perceptron to perform modality selection and feature fusion. Feature fusion includes concatenation, addition, and multiplication operations to combine data from different modalities.

4. The method for dynamic fusion of missing multimodal data based on knowledge distillation according to claim 1, characterized in that, The classification loss uses non-negative cross-entropy loss, and the distillation loss uses KL divergence.

5. A dynamic fusion system for missing multimodal data based on knowledge distillation, characterized in that, The system includes: The multimodal data acquisition module is used to acquire medical data containing multiple modalities and corresponding task labels, and to divide the medical data into those containing complete modalities and those containing only a single modality. The teacher network model training module is used to construct multiple unimodal teacher models based on the classification network. The classification network is trained separately using medical data of each individual modality to obtain the corresponding unimodal teacher model. The multimodal dynamic fusion network training module is used to construct a multimodal dynamic fusion network based on a threshold network and a series of expert networks. The threshold network takes medical data containing complete modalities as input and outputs a one-hot vector with a length equal to the number of expert networks. The threshold network determines which expert networks are activated. Each expert network uses a subset of data from multiple modalities for feature fusion. The output of the multimodal dynamic fusion network is a synthesis of multiple expert networks, represented by a one-hot vector. Only one expert network is used for each instance. The multimodal dynamic fusion network trained with medical data containing complete modalities serves as the student model. The teacher-student distillation module is used to distill the student model using the teacher model. It calculates the classification loss and distillation loss for each medical data point containing a complete modality using the teacher model. The multimodal data category prediction module is used to input the actual acquired multimodal medical data into the multimodal dynamic fusion network to obtain the category prediction results.

6. A dynamic fusion device for missing multimodal data based on knowledge distillation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a method for dynamic fusion of missing multimodal data based on knowledge distillation as described in any one of claims 1-4.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a method for dynamic fusion of missing multimodal data based on knowledge distillation as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a method for dynamic fusion of missing multimodal data based on knowledge distillation as described in any one of claims 1-4.

Images