Fuzzy knowledge distillation method for incomplete multimodal data

Abstract

The present invention provides a fuzzy knowledge distillation method for incomplete multimodal data, which belongs to the field of intelligent diagnosis technology of medical information, and solves the technical problem of model capability mismatch between teacher model and student model in the process of knowledge distillation. Its technical scheme is: including the following steps: S1, using edge-to-edge convolution E2E, edge-to-node convolution E2N, and node-to-graph convolution N2G proposed by Kawahara to obtain the feature graph of each step; S2, obtaining summarized and selected information; S3, designing three types of loss functions, namely representation loss, prediction distribution loss and cross entropy loss; S4, training the student network by minimizing the three types of loss functions. The beneficial effects of the present invention are: the present invention can significantly improve the accuracy of the model in identifying mental illness in the case of modality loss, assist doctors in diagnostic analysis, and bring better medical services to patients.

Description

Technical Field

[0001] The present invention relates to the technical field of medical information intelligent diagnosis, and in particular to a fuzzy knowledge distillation method for incomplete multimodal data. Background Art

[0002] In medicine, mental illness can cause changes in thoughts, feelings, emotions and behaviors, which in turn can cause patients to be unable to work and study normally, seriously affecting their social functions, and thus increasing the burden on the patient's family and society. Mental illness is quite common in my country.

[0003] Existing psychiatric disease recognition models based on multimodal brain networks only consider the situation where the modality is complete. However, in actual clinical applications, some patients often lack information on a certain modality due to factors such as patient non-cooperation, site restrictions, and low data quality. This problem is not considered in the existing decision-level fusion methods and pre-fusion methods. At present, the methods for solving the problem of modality missing in multimodal learning can be roughly divided into traditional methods and deep learning methods. Filling methods and tensor decomposition-based methods are two commonly used traditional techniques. However, they are mainly applicable to scenarios involving scattered missing data, and are not sufficient to handle the situation where the entire modality is completely missing. Deep learning methods can solve this problem by using generative adversarial networks (GANs). However, the GAN model has certain limitations: first, it cannot retain discriminative information during image generation; second, when the data set is insufficient, it encounters challenges in training a stable model.

[0004] The knowledge distillation framework is a powerful and widely applicable technique for transferring knowledge from complex models to simpler models. It can be used to address the missing modality problem. However, existing methods transfer all information in a point-to-point manner without considering the ability mismatch between the teacher model and the student model. Recent studies have shown that the correlation between teacher quality and student performance may not be that strong. In other words, the teacher model does not need to provide all information to the student model. Summary of the invention

[0005] The purpose of the present invention is to provide a fuzzy knowledge distillation method for incomplete multimodal data to improve the accuracy of mental illness identification; by combining the structural brain network and the functional network and drawing on the knowledge distillation technology to carry out research, it can assist doctors in diagnosing mental illness, thereby improving the accuracy of disease diagnosis under incomplete multimodal data.

[0006] The inventive concept of the present invention is as follows: drawing on the classic teacher-student network, the teacher network is a network that uses the features of the structural brain network and the functional brain network in depth, and the student network is a network that only uses single-modal information for feature learning; in the feature learning process, a fuzzy knowledge distillation module is designed to process the teacher representation, mainly obtaining generalized and selected information through fuzzy processing and feature selection; the distance between the processed teacher representation and the student representation in the feature learning process is calculated as the representation loss; the difference between the final predicted distribution of the teacher and the student is calculated as the predicted distribution loss; the difference between the predicted label and the true label of the student network is calculated as the cross entropy loss; and the student network is trained by minimizing these three loss functions.

[0007] In order to better achieve the above-mentioned invention object, the present invention is implemented by the following technical solution: a fuzzy knowledge distillation method for incomplete multimodal data, the specific steps are as follows:

[0008] A. To obtain the deep features of the brain network from the connection matrix of the input data of the teacher network and the student network, the edge-to-edge convolution E2E, edge-to-node convolution E2N, and node-to-graph convolution N2G proposed by Kawahara are used to obtain the feature map of each step;

[0009] B. During the feature learning process, the teacher representation is processed using the fuzzy knowledge distillation module to obtain summarized and selected information. This module has four main steps: fuzzification, modeling feature importance, reconstructing feature maps, and feature selection. The teacher representation is aligned with the student representation using 1×1 convolution.

[0010] C. Three types of loss functions are designed, namely representation loss, prediction distribution loss and cross entropy loss. Representation loss is the difference between the feature representation learned by the student network and the teacher network; prediction distribution loss is the difference between the final prediction distribution of the student network and the teacher network; cross entropy loss is the difference between the predicted label of the student network and the true label;

[0011] D. Finally, the student network is trained by minimizing three types of loss functions, so that the student network can master the basic knowledge of the teacher network and have better performance in disease diagnosis.

[0012] As the fuzzy knowledge distillation method for incomplete multimodal data provided by the present invention, the specific steps of step B are as follows:

[0013] a. Fuzzy processing aims to learn the inherent complex rules between feature maps and corresponding segmentation results, thereby improving the generalization of the original representation. For a specific channel c, M membership functions are applied to each feature point on the channel. Each membership function assigns a fuzzy language term label to the feature point, and all membership functions are represented by Gaussian functions, as follows:

[0014]

[0015] in, represents the feature point at channel c with coordinates (x, y), μ k,c and is the mean and standard deviation of the kth Gaussian membership function, Z x,y,k,c Represents the kth output fuzzy language label of the feature point (x, y) in channel c;

[0016] The membership of all feature points is obtained using "AND" fuzzy logic The fuzziness. The fuzziness can be obtained by the following formula:

[0017]

[0018] The feature representation of the fuzzy processing module output is the fuzziness tensor Z = {Z x,y,c}, its feature dimension is the same as the input feature T E Compared with the original features, the fuzziness tensor can effectively reduce information noise, reduce the integrity and complexity of the data after Gaussian blur processing, thereby enhancing the generalization ability of the model;

[0019] b. Modeling feature importance. The blurred feature map uses the global average pooling operation avg to extract global information, and then the feature importance modeling layer f m (consisting of 1×1 convolution and sigmoid activation function) to obtain the importance vector u. The feature importance modeling process can be expressed as:

[0020] u=f m (avg(Z)) (12)

[0021] Among them, f m =sigmoid(W m *X+b m ),W m is the weight matrix of the 1×1 convolution, X is the input tensor of the feature importance modeling layer, and b m is the deviation value;

[0022] c. Reconstructed feature map, original input feature map T E After the feature importance modeling layer fm The obtained importance vector u is scaled, and then the scaled feature map is added to the original feature map to obtain the reconstructed feature map T E′ The operation of reconstructing the feature map can enhance the important features in the feature map. The process of reconstructing the feature map is shown in the formula:

[0023] t E′ =t E +u*T E (13)

[0024] d. Feature selection: The reconstructed feature representation passes through the feature selection layer f s (i.e., 1×1 transformation layer) to retain important feature maps and remove useless feature maps, thereby reducing channels. Finally, the feature representation after feature selection is obtained. The feature selection process is shown in the formula:

[0025]

[0026] Among them, f s =W s *Y+b s , W s is the weight matrix of the 1×1 convolution, Y is the input tensor of the feature selection layer, and b s is the bias value.

[0027] As the fuzzy knowledge distillation method for incomplete multimodal data provided by the present invention, the specific steps of step C are as follows:

[0028] a. Indicates loss L rep It measures the difference between the feature representations learned by the student model and the teacher model. The teacher and the student transform the feature representations into T and S through their respective transformation modules, and calculate the distance between the feature representations of the teacher and the student after each transformation, and take the sum of the distances after three transformations as the representation loss L. rep . Distance function d p is defined as follows:

[0029]

[0030] Among them, T i and S i Represent the i-th component of tensors T and S respectively;

[0031] After each convolution transformation, the distance function is used to calculate the distance between the teacher model and the student model representation. The representation loss is calculated as:

[0032] L rep =d1+d2+d3 (16)

[0033] Among them, d1, d2, and d3 represent the distance between the teacher representation and the student representation after feature transformation after E2E, E2N, and N2G convolution, respectively;

[0034] b. Prediction distribution loss L pred Measures the difference in the final prediction distribution between the teacher model and the student model, where the prediction distributions of the teacher model and the student model are p T =[p T 1 ,p T 2 ,…,p T m ] and p s =[p S 1 ,p S 2 ,…,p S m ], where m is the category type, L pred It is given by the following formula:

[0035] L pred =d KL (p T ,p s ) (17)

[0036] Among them, d KL is the KL divergence, which is used to measure the distribution distance;

[0037] c. Cross entropy loss L ce , is the most common loss, which is a measure of the difference between the student model's predicted label and the true label, and its formula is as follows:

[0038]

[0039] Among them, y i is the true label of sample i, is the predicted label, and |y| is the number of all samples.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The fuzzy knowledge distillation method proposed in the present invention uses the modality-complete structural-functional brain network information to assist the feature learning of the modality-missing brain network data. Compared with most previous fusion models, it takes the modality-missing situation into consideration and makes full use of the correlation information between the functional brain network and the structural brain network to solve the modality-missing problem in clinical applications and improve the accuracy of mental illness identification based on modality-missing data.

[0042] 2. Compared with the classic knowledge distillation module that transmits complete information, the fuzzy knowledge distillation module transmits summarized and selected information. The fuzzy knowledge distillation module designed by the present invention solves the problem of mismatch between the teacher model and the student model in knowledge distillation. The module combines the fuzzy processing of teacher information and feature selection operations to enhance the learning ability of the student model.

[0043] 3. The present invention designs three loss functions: representation loss, prediction distribution loss and cross entropy loss. End-to-end training is achieved by minimizing the loss function, so that the student model can more effectively master the basic knowledge of the teacher model to improve the student model's ability to diagnose mental illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention but do not constitute a limitation of the present invention.

[0045] Figure 1 The overall framework diagram of the fuzzy knowledge distillation method for incomplete multimodal data of the present invention is as follows

[0046] Figure 2 This is a model diagram of the fuzzy knowledge distillation method for incomplete multimodal data of the present invention. DETAILED DESCRIPTION

[0047] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with embodiments and drawings. The embodiments are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0048] like Figure 1 and Figure 2 As shown, this embodiment is specifically implemented by the following technical solution: a fuzzy knowledge distillation method for incomplete multimodal data, the specific steps are as follows:

[0049] A. To obtain the deep features of the brain network from the connection matrix of the input data of the teacher network and the student network, the edge-to-edge convolution E2E, edge-to-node convolution E2N, and node-to-graph convolution N2G proposed by Kawahara are used to obtain the feature map of each step;

[0050] The above model was tested using a mental illness brain map dataset, with 217 samples and decision attributes divided into two categories, namely patients with confirmed mental illness and normal people. The connection matrix of the functional brain network and the structural brain network is 90×90.

[0051] B. During the feature learning process, the teacher representation is processed using the fuzzy knowledge distillation module to obtain summarized and selected information. This module has four main steps: fuzzy processing, modeling feature importance, reconstructing feature maps and feature selection, and using 1×1 convolution to align the teacher representation with the student representation; the specific steps are as follows:

[0052] a. Fuzzy processing aims to learn the inherent complex rules between feature maps and corresponding segmentation results, thereby improving the generalization of the original representation. For a specific channel c, M membership functions are applied to each feature point on the channel. Each membership function assigns a fuzzy language term label to the feature point, and all membership functions are represented by Gaussian functions, as follows:

[0053]

[0054] in, represents the feature point at channel c with coordinates (x, y), μ k,c and is the mean and standard deviation of the kth Gaussian membership function, Z x,y,k,c Represents the kth output fuzzy language label of the feature point (x, y) in channel c;

[0055] The membership of all feature points is obtained using "AND" fuzzy logic The fuzziness. The fuzziness can be obtained by the following formula:

[0056]

[0057] The feature representation of the fuzzy processing module output is the fuzziness tensor Z = {Z x,y,c}, its feature dimension is the same as the input feature T E Compared with the original features, the fuzziness tensor can effectively reduce information noise, reduce the integrity and complexity of the data after Gaussian blur processing, thereby enhancing the generalization ability of the model;

[0058] b. Modeling feature importance. The blurred feature map uses the global average pooling operation avg to extract global information, and then the feature importance modeling layer f m (consisting of 1×1 convolution and sigmoid activation function) to obtain the importance vector u. The feature importance modeling process can be expressed as:

[0059] u=f m (avg(Z)) (21)

[0060] Among them, f m =sigmoid(W m *X+bm ),W m is the weight matrix of the 1×1 convolution, X is the input tensor of the feature importance modeling layer, and b m is the deviation value;

[0061] c. Reconstructed feature map, original input feature map T E After the feature importance modeling layer f m The obtained importance vector u is scaled, and then the scaled feature map is added to the original feature map to obtain the reconstructed feature map T E′ The operation of reconstructing the feature map can enhance the important features in the feature map. The process of reconstructing the feature map is shown in the formula:

[0062] T E′ =T E +u*T e (twenty two)

[0063] d. Feature selection: The reconstructed feature representation passes through the feature selection layer f s (i.e., 1×1 transformation layer) to retain important feature maps and remove useless feature maps, thereby reducing channels. Finally, the feature representation after feature selection is obtained. The feature selection process is shown in the formula:

[0064]

[0065] Among them, f s =W s *Y+b s , W s is the weight matrix of the 1×1 convolution, Y is the input tensor of the feature selection layer, and b s is the bias value.

[0066] C. Three types of loss functions are designed, namely representation loss, prediction distribution loss and cross entropy loss. Representation loss is the difference between the feature representation learned by the student network and the teacher network; prediction distribution loss is the difference between the final prediction distribution of the student network and the teacher network; cross entropy loss is the difference between the predicted label of the student network and the true label; the specific steps are as follows:

[0067] a. Indicates loss L rep It measures the difference between the feature representations learned by the student model and the teacher model. The teacher and the student transform the feature representations into T and S through their respective transformation modules, and calculate the distance between the feature representations of the teacher and the student after each transformation, and take the sum of the distances after three transformations as the representation loss L. rep . Distance function d p is defined as follows:

[0068]

[0069] Among them, T i and S i Represent the i-th component of tensors T and S respectively;

[0070] After each convolution transformation, the distance function is used to calculate the distance between the teacher model and the student model representation. The representation loss is calculated as:

[0071] L rep =d1+d2+d3 (25)

[0072] Among them, d1, d2, and d3 represent the distance between the teacher representation and the student representation after feature transformation after E2E, E2N, and N2G convolution, respectively;

[0073] b. Prediction distribution loss L pred Measures the difference in the final prediction distribution between the teacher model and the student model, where the prediction distributions of the teacher model and the student model are p T =[p T 1 ,p T 2 ,…,p T m ] and p s =[p S 1 ,p S 2 ,…,p S m ], where m is the category type, L pred It is given by the following formula:

[0074] L pred =d KL (p T ,p s ) (26)

[0075] Among them, d KL is the KL divergence, which is used to measure the distribution distance;

[0076] c. Cross entropy loss L ce , is the most common loss, which is a measure of the difference between the student model's predicted label and the true label, and its formula is as follows:

[0077]

[0078] Among them, y i is the true label of sample i, is the predicted label, and |y| is the number of all samples.

[0079] D. Finally, the student network is trained by minimizing three types of loss functions, so that the student network can master the basic knowledge of the teacher network and have better performance in disease diagnosis.

[0080] The model was trained by dividing it into different training sets, test sets and validation sets for 5 times to obtain different accuracies, among which the maximum accuracy was 92.9% and the average accuracy was 85.3%. Therefore, it can be verified that the model can improve the accuracy of mental illness recognition in the absence of modality.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fuzzy knowledge distillation method for incomplete multimodal data, characterized in that: The following steps are involved: S1. To obtain the deep features of the brain network from the connection matrix of the input data of the teacher network and the student network, the edge-to-edge convolution E2E, edge-to-node convolution E2N, and node-to-graph convolution N2G proposed by Kawahara are used to obtain the feature graph of each step; S2. During the feature learning process, the teacher representation is processed using the fuzzy knowledge distillation module to obtain summarized and selected information; The specific steps of fuzzy knowledge distillation in the feature learning process in step S2 are as follows: S21, fuzzy processing, for a specific channel c, M membership functions are applied to each feature point on the channel, each membership function assigns a fuzzy language term label to the feature point, and all membership functions are represented by Gaussian functions. The formula is as follows: in, represents the feature point at channel c with coordinates (x, y), μ k,c and is the mean and standard deviation of the kth Gaussian membership function, Z x,y,k,c Represents the kth output fuzzy language label of the feature point (x, y) in channel c; The membership of all feature points is obtained using "AND" fuzzy logic The fuzziness, The fuzziness of is obtained by the following formula: The feature representation of the fuzzy processing module output is the fuzziness tensor Z = {Z x,y,c }, its feature dimension is the same as the input feature T E same; S22, the blurred feature map uses the global average pooling operation avg to extract global information, and then passes the feature importance modeling layer f m Obtain the importance vector u, and the feature importance modeling process is expressed as: u=f m (Aug(Z)) (3) Among them, f m =sigmoid(W m *X+b m ),W m is the weight matrix of the 1×1 convolution, X is the input tensor of the feature importance modeling layer, and b m is the deviation value; S23, reconstructed feature map, original input feature map T E After the feature importance modeling layer f m The obtained importance vector u is scaled, and then the scaled feature map is added to the original feature map to obtain the reconstructed feature map T E′ , the operation of reconstructing the feature map enhances the important features in the feature map, and the process of reconstructing the feature map is shown in the formula: T E′ =T E +u*T E (4); S24, feature selection, the reconstructed feature representation passes through the feature selection layer f s , retain important feature maps, remove useless feature maps, and obtain the feature representation after feature selection The feature selection process is shown in the formula: Among them, f s =W s *Y+b s , W s is the weight matrix of the 1×1 convolution, Y is the input tensor of the feature selection layer, and b s is the bias value; S3. Three types of loss functions are designed, namely representation loss, prediction distribution loss and cross entropy loss. The representation loss is the difference between the feature representation learned by the student network and the teacher network; the prediction distribution loss is the difference between the final prediction distribution of the student network and the teacher network; the cross entropy loss is the difference between the predicted label of the student network and the true label; The specific steps of designing three types of loss functions in step S3 are as follows: S31, indicates loss L rep It measures the difference between the feature representations learned by the student model and the teacher model. The teacher and the student transform the feature representations into T and S through their respective transformation modules. The distance between the feature representations of the teacher and the student after each transformation is calculated, and the sum of the distances after three transformations is used as the representation loss L rep , distance function d p is defined as follows: Among them, T i and S i Represent the i-th component of tensors T and S respectively; After each convolution transformation, the distance function is used to calculate the distance between the teacher model and the student model representation. The formula for calculating the representation loss is: L rep =d1+d2+d3 (7) Among them, d1, d2, and d3 represent the distance between the teacher representation and the student representation after feature transformation after E2E, E2N, and N2G convolution, respectively; S32, prediction distribution loss L pred Measures the difference in the final prediction distribution between the teacher model and the student model, where the prediction distributions of the teacher model and the student model are p T =[p T 1 ,p T 2 ,...,p T m ] and p s =[p S 1 ,p S 2 ,…,p S m ], where m is the category type, L pred It is given by the following formula: L pred =d KL (p T ,p s ) (8) Among them, d KL is the KL divergence, which is used to measure the distribution distance; S33, cross entropy loss L ce It is a measure of the difference between the student model's predicted label and the true label, and its formula is as follows: Among them, y i is the true label of sample i, is the predicted label, |y| is the number of all samples; S4. Train the student network by minimizing three types of loss functions.

Images