A hepatocellular carcinoma prediction method based on modality-aware distillation network

Abstract

The present invention discloses a hepatocellular carcinoma prediction method based on a modality-aware distillation network, comprising the following steps: S1, obtaining a data set of hepatocellular carcinoma patients, and dividing the entire data set into five folds according to a five-fold cross-validation scheme. In each round of cross-validation, one fold of the data is used as a test set, and the other four folds of data are used as training sets; S2, preprocessing the data to find the largest circumscribed cube for the tumor of all patients, and then removing other non-tumor areas except the cube; S3, establishing a modality-aware distillation network, and training the modality-aware distillation network. The modality-aware distillation network is used to transfer the knowledge learned by the teacher network through the fusion of clinical data modality and image modality to the student network with only image modality; S4, predicting hepatocellular carcinoma through the trained modality-aware distillation network.

Description

Technical Field

[0001] The present invention relates to the field of biological technology, and in particular to a hepatocellular carcinoma prediction method based on a modality-aware distillation network. Background Art

[0002] Hepatocellular carcinoma (HCC) is a malignant tumor that develops in liver cells and is a common pathological type of primary liver cancer. Currently, the following methods are used to predict microvascular invasion in HCC: 1. Predicting preoperative MVI using extreme gradient enhancement and deep learning of CT images; 2. Utilizing a 3D CNN prediction model to fuse features from multiple MRI sequences; 3. Embedding long short-term memory (LSTM) into a CNN to fuse multimodal MR volumes to predict MVI in HCC patients. These three methods rely solely on MR images to predict MVI status, resulting in low accuracy. In addition, the following methods use knowledge distillation for prediction: 1. KD (knowledge distillation) is used to effectively segment neuronal structure microscope images from 3D optical images; 2. The concept of KD is cited to use soft labels to segment brain injuries by expanding the mask boundary; 3. KD is used for multi-source transfer learning lung pattern analysis tasks; 4. A category-guided contrast distillation module is developed to pull positive image pairs from the same category in the teacher and student models, while pushing away negative image pairs from different categories; the distillation networks used in the above four methods only consider different image data and transmit information from the input image data, with poor classification accuracy and low prediction accuracy. Summary of the Invention

[0003] The purpose of the present invention is to provide a hepatocellular carcinoma prediction method based on a modality-aware distillation network. The hepatocellular carcinoma prediction method based on a modality-aware distillation network transfers the knowledge of a teacher network with image modalities and non-image clinical data to a student network with only image modalities, and proposes a modality-aware distillation network (MD-Net) for HCC MVI prediction, which can effectively improve the classification accuracy and prediction accuracy.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A hepatocellular carcinoma prediction method based on a modality-aware distillation network comprises the following steps:

[0006] S1. Obtain a dataset of hepatocellular carcinoma patients and divide the entire dataset into five folds according to a five-fold cross-validation scheme. In each round of cross-validation, use one fold of the data as the test set and the other four folds as the training set.

[0007] S2. Preprocess the data to find the largest circumscribed cube for all patients' tumors, and then remove all non-tumor areas outside the cube.

[0008] S3. Establish a modality-aware distillation network and train it. The modality-aware distillation network is used to transfer the knowledge learned by the teacher network through the fusion of clinical data modality and image modality to the student network with only image modality.

[0009] S4. Hepatocellular carcinoma prediction using trained modality-aware distillation network.

[0010] Preferably, the dataset in step S1 consists of data from 270 pathologically confirmed HCC patients, including 128 M0 patients, 93 M1 patients, and 49 M2 patients; wherein M0 indicates no microvascular invasion, M1 indicates no more than 5 invading blood vessels or located within 1 cm of the tumor surface, and M2 indicates more than 5 invading blood vessels or more than 1 cm from the tumor surface.

[0011] Preferably, in step S2, the size of the cube is set to 80*80*20 pixels.

[0012] Preferably, the training process of the modality-aware distillation network in step S3 is specifically as follows:

[0013] S31. The teacher network passes the HBP image and clinical data to the MRI-clinical fusion module to extract a 512-dimensional vector feature. The PRE image and clinical data are then input into another MRI-clinical fusion module to obtain another 512-dimensional vector feature. The two obtained 512-dimensional vector features are input into the SA module to obtain a new feature that fuses each other's information. and Finally, the new features and Splice them together to generate Z t , and Z t Passed to two fully connected layers to predict the classification result P t ;

[0014] S32, the student network takes 3DHBPMRI images and 3DPREMRI images as input, passes the HBP data to the MRI-only module to obtain features, and then passes the PRE data to another MRI-only module to obtain features; the two obtained features are input into the SA module to obtain new features that fuse each other's information. and Among them, the new features and are two feature vectors containing 512 dimensions; finally, the new features and Connect them to generate Z s, and Z s Input into the fully connected layer to predict the MVI classification result P s ;

[0015] S33. In the student network, a regression task is introduced to connect the feature Z of the input HBP image and the input PRE image. s Sent to two fully connected layers to predict a 52-dimensional vector P c , used to estimate potential clinical information and then use the input clinical data as the prediction P c The true label of

[0016] S34. Classification-level distillation loss and feature-level distillation loss are used to distill the features fused from the teacher network clinical data and MRI images into features extracted from MRI images, and the knowledge distillation strategy is used to transfer the clinical information of the teacher network to the student network.

[0017] Preferably, the MRI-clinical fusion module in step S31 integrates MRI data and non-imaging clinical data, and takes 3D MRI data and vectorized clinical data as input. The MRI-clinical fusion module applies four fully connected layers to the input clinical data to obtain four feature maps, and these feature channels are 64, 128, 256 and 256 respectively; four convolution blocks on the input MRI image are used to obtain another 3D feature map, and the feature channels are also set to 64, 128, 256 and 256, and each convolution block consists of two 3×3 convolution layers; the four feature maps in the clinical data and the corresponding four features in the MRI data are channel-wise multiplied to integrate them together, and then a 3×3 convolution layer and a fully connected layer are applied to output a feature vector with 512 dimensions.

[0018] Preferably, the MRI-only module in step S32 extracts a 512-dimensional feature vector from the 3D MRI image; the MRI-only module consists of nine convolutional blocks and a fully connected layer, each convolutional block contains a batch normalization layer, a ReLU activation layer and a 3×3 convolutional layer, which is used to improve the robustness of the network; the number of channels of the output features of the nine convolutional blocks is set to different, the feature channels of the first five layers are 32, 32, 64, 64 and 128, and the feature channels of the last four layers are 256, 128, 256 and 256, which are used to balance efficiency and computational burden.

[0019] Preferably, the SA module in steps S31 and S32 is a symmetric attention module, X and Y represent two input feature maps of the SA module, and the SA module applies a linear transformation layer on X to obtain three feature maps, including the query vector Q x , Key vector Kx and Value vector V x ; SA module applies a transformation layer on Y to generate the Key feature map K y and Value feature map V y ; by multiplying by Q x and K x The transpose of is used to generate the score feature vector S x , by multiplying Q y and K y The transpose of to generate another score feature vector S y ; The obtained score feature vector S x and the Value feature vector V x Multiply and add S y With V y Multiply them to generate two result feature vectors, and then add them together to finally generate the output refined feature vector

[0020] The SA module applies another linear transformation layer on Y to obtain the feature vector Query vector Q y , by adding Q y and K y Multiply the transpose of Q y and K y The two score eigenvectors are calculated by multiplying the transpose of , and then the refined eigenvector is calculated by the following formula

[0021]

[0022] Preferably, the modality-aware distillation in step S3 includes classification-level distillation and special-level distillation;

[0023] In classification-level distillation, let Represents the MRI image data x generated from the student network i The category probability of the class to which it belongs, and Represents the MRI image data x generated from the teacher network i Class probability of the class to which it belongs; define the classification level distillation loss In order to make the class probabilities from the teacher network the target for training the student network, the Kullback-Leibler divergence is used to measure the difference between the two distributions:

[0024]

[0025] Where N and M represent the number of training samples and the total number of categories, respectively, and DKL(·) represents the Kullback-Leibler divergence between two probabilities. represents the student network prediction sample, represents the teacher network prediction sample;

[0026] In characteristic distillation, the characteristic distillation loss Calculated as and The Kullback-Leibler divergence between and The combination of the Kulback-Leibleer divergence between

[0027]

[0028] Among them, β is used to weight the Kullback-Leibler divergence term, with weight β1=1, Represents two features and The Kullback-Leibler divergence between Represents two features and The Kullback-Leibler divergence between represents the student network prediction sample, represents the teacher network prediction sample;

[0029] The final loss function includes two supervised losses on the teacher network and the student network, the self-supervised loss for clinical data prediction, and the distillation loss between the student network and the teacher network. The loss function is defined as follows:

[0030]

[0031] in, and They represent the supervision loss of teacher network prediction and the supervision loss of student network prediction, respectively, and are calculated using Focalloss loss. and The prediction loss, L clinical represents the self-supervised loss for clinical data prediction, using cross entropy loss to calculate P C prediction errors and the underlying facts of clinical data; represents the classification-level distillation loss, Denotes the feature-level distillation loss between the teacher network and the student network, using equation L total The loss function is used to train the modality-aware distillation network for MIV prediction.

[0032] After adopting the above technical solution, the present invention has the following beneficial effects:

[0033] 1. This paper proposes a modality-aware distillation network (MD-Net) for HCCMVI prediction by transferring the knowledge of a teacher network with image modality and non-image clinical data to a student network with only image modality, which can effectively improve the classification accuracy and prediction accuracy.

[0034] 2. The student network of the modality-aware distillation network (MD-Net) of the present invention includes two MRI-only modules for extracting MRI features and a symmetric attention (SA) module for refining features from two MRI images, while the teacher network includes two MRI-clinical fusion modules for fusing MRI data and clinical data with 52-dimensional vectors and an SA module for refining the two fused features.

[0035] 3. In addition to the original classification-level distillation results, the Modality-Aware Distillation Network (MD-Net) also designs a feature-level distillation to better transfer clinical data from the teacher network to the student network. In addition, a new self-supervised task is designed to predict clinical data from image data to further enhance MVI prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the present invention;

[0037] Figure 2 is an example diagram of a tumor in an HCC MRI image of the present invention;

[0038] Figure 3 This is a framework flow chart of the modality-aware distillation network of the present invention;

[0039] Figure 4 Figures showing examples of the MRI-clinical fusion module, MRI-only module, and Channel-wise multiplication of the present invention, where (a) the MRI-clinical fusion module fuses MRI data with non-imaging clinical data; (b) the MRI-only module uses only MRI imaging data; (c) an example of Channel-wise multiplication;

[0040] Figure 5 This is a framework flow chart of the SA module of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] Example

[0043] like Figures 1 to 5 As shown, a hepatocellular carcinoma prediction method based on modality-aware distillation network includes the following steps:

[0044] S1. Obtain a dataset of hepatocellular carcinoma patients and divide the entire dataset into five folds according to a five-fold cross-validation scheme. In each round of cross-validation, use one fold of the data as the test set and the other four folds as the training set.

[0045] The dataset described in step S1 consists of data from 270 pathologically confirmed HCC patients, including 128 M0 patients, 93 M1 patients, and 49 M2 patients. M0 indicates no microvascular invasion, M1 indicates no more than five invading vessels or the vessel is located within 1 cm of the tumor surface, and M2 indicates more than five invading vessels or the vessel is more than 1 cm from the tumor surface.

[0046] S2. Preprocess the data to find the largest circumscribed cube for all patients' tumors, and then remove all non-tumor areas outside the cube.

[0047] In step S2, the size of the cube is set to 80*80*20 pixels;

[0048] S3. Establish a modality-aware distillation network and train it. The modality-aware distillation network is used to transfer the knowledge learned by the teacher network through the fusion of clinical data modality and image modality to the student network with only image modality.

[0049] The training process of the modality-aware distillation network in step S3 is specifically as follows:

[0050] S31. The teacher network passes the HBP image and clinical data to the MRI-clinical fusion module to extract a 512-dimensional vector feature. The PRE image and clinical data are then input into another MRI-clinical fusion module to obtain another 512-dimensional vector feature. The two obtained 512-dimensional vector features are input into the SA module to obtain a new feature that fuses each other's information. and Finally, the new features and Splice them together to generate Z t , and Z t Passed to two fully connected layers to predict the classification result P t ;

[0051] S32, the student network takes 3DHBPMRI images and 3DPREMRI images as input, passes the HBP data to the MRI-only module to obtain features, and then passes the PRE data to another MRI-only module to obtain features; the two obtained features are input into the SA module to obtain new features that fuse each other's information. and Among them, the new features and are two feature vectors containing 512 dimensions; finally, the new features and Connect them to generate Z s , and Z s Input into the fully connected layer to predict the MVI classification result P s ;

[0052] S33. In the student network, a regression task is introduced to connect the feature Z of the input HBP image and the input PRE image. s Sent to two fully connected layers to predict a 52-dimensional vector P c , used to estimate potential clinical information and then use the input clinical data as the prediction P c The true label of

[0053] S34, using classification-level distillation loss and feature-level distillation loss, distill the features fused from the teacher network clinical data and MRI images into features extracted from MRI images, and use the knowledge distillation strategy to transfer the clinical information of the teacher network to the student network;

[0054] The MRI-clinical fusion module in step S31 integrates MRI data and non-imaging clinical data. Taking 3D MRI data and vectorized clinical data as input, the MRI-clinical fusion module applies four fully connected layers to the input clinical data to obtain four feature maps, where the feature channels are 64, 128, 256, and 256, respectively. Another 3D feature map is obtained using four convolutional blocks on the input MRI image, and the feature channels are also set to 64, 128, 256, and 256. Each convolutional block consists of two 3×3 convolutional layers. The four feature maps in the clinical data are channel-wise multiplied with the corresponding four features in the MRI data to integrate them together, and then a 3×3 convolutional layer and a fully connected layer are applied to output a feature vector with 512 dimensions.

[0055] In step S32, the MRI-only module extracts a 512-dimensional feature vector from the 3D MRI image. The MRI-only module consists of nine convolutional blocks and a fully connected layer. Each convolutional block contains a batch norm layer, a ReLU activation layer, and a 3×3 convolutional layer to improve the robustness of the network. The number of channels of the output features of the nine convolutional blocks is set to different values. The feature channels of the first five layers are 32, 32, 64, 64, and 128, and the feature channels of the last four layers are 256, 128, 256, and 256, respectively, to balance efficiency and computational burden.

[0056] The SA module in steps S31 and S32 is a symmetric attention module. X and Y represent the two input feature maps of the SA module. The SA module applies a linear transformation layer on X to obtain three feature maps, including the query vector Q x , Key vector K x and Value vector V x ; SA module applies a transformation layer on Y to generate the Key feature map K y and Value feature map V y ; by multiplying by Q x and K x The transpose of is used to generate the score feature vector S x , by multiplying Q y and K y The transpose of to generate another score feature vector S y ; The obtained score feature vector S x and the Value feature vector V x Multiply and S y With V y Multiply them to generate two result feature vectors, and then add them together to finally generate the output refined feature vector

[0057] The SA module applies another linear transformation layer on Y to obtain the feature vector Query vector Q y , by adding Q y and K y Multiply the transpose of Q y and K y The two score eigenvectors are calculated by multiplying the transpose of , and then the refined eigenvector is calculated by the following formula

[0058]

[0059] In step S3, the modality-aware distillation includes classification-level distillation and special-level distillation;

[0060] In classification-level distillation, let Represents the MRI image data x generated from the student network i The category probability of the class to which it belongs, and Represents the MRI image data x generated from the teacher network i Class probability of the class to which it belongs; define the classification level distillation loss In order to make the class probabilities from the teacher network the target for training the student network, the Kullback-Leibler divergence is used to measure the difference between the two distributions:

[0061]

[0062] Where N and M represent the number of training samples and the total number of categories, respectively, and DKL(·) represents the Kullback-Leibler divergence between two probabilities. represents the student network prediction sample, represents the teacher network prediction sample;

[0063] In characteristic distillation, the characteristic distillation loss Calculated as and The Kullback-Leibler divergence between and The combination of the Kulback-Leibleer divergence between

[0064]

[0065] Among them, β is used to weight the Kullback-Leibler divergence term, with weight β1=1, Represents two features and The Kullback-Leibler divergence between Represents two features and The Kullback-Leibler divergence between represents the student network prediction sample, represents the teacher network prediction sample;

[0066] The final loss function includes two supervised losses on the teacher network and the student network, the self-supervised loss for clinical data prediction, and the distillation loss between the student network and the teacher network. The loss function is defined as follows:

[0067]

[0068] in, and They represent the supervision loss of teacher network prediction and the supervision loss of student network prediction, respectively, and are calculated using Focalloss loss. and The prediction loss, L clinical represents the self-supervised loss for clinical data prediction, using cross entropy loss to calculate P C prediction errors and the underlying facts of clinical data; represents the classification-level distillation loss, Denotes the feature-level distillation loss between the teacher network and the student network, using equation L total The loss function is used to train the modality-aware distillation network for MIV prediction;

[0069] S4. Hepatocellular carcinoma prediction using trained modality-aware distillation network.

[0070] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A hepatocellular carcinoma prediction method based on modality-aware distillation network, characterized in that: The following steps are involved: S1. Obtain a dataset of hepatocellular carcinoma patients and divide the entire dataset into five folds according to a five-fold cross-validation scheme. In each round of cross-validation, use one fold of the data as the test set and the other four folds as the training set. Calculate the five-round mean and variance values ​​of the evaluation indicators. S2. Preprocess the data to find the largest circumscribed cube for all patients' tumors, and then remove all non-tumor areas outside the cube. S3. Establish a modality-aware distillation network and train it. The modality-aware distillation network is used to transfer the knowledge learned by the teacher network through the fusion of clinical data modality and image modality to the student network with only image modality. In step S3, the modality-aware distillation includes classification-level distillation and special-level distillation; In classification-level distillation, let Represents the MRI image data x generated from the student network i The category probability of the class to which it belongs, and Represents the MRI image data x generated from the teacher network i Class probability of the class to which it belongs; define the classification level distillation loss In order to make the class probabilities from the teacher network the target for training the student network, the Kullback-Leibler divergence is used to measure the difference between the two distributions: Where N and M represent the number of training samples and the total number of categories, respectively, and DKL(·) represents the Kullback-Leibler divergence between two probabilities. represents the student network prediction sample, represents the teacher network prediction sample; In characteristic distillation, the characteristic distillation loss Calculated as and The Kullback-Leibler divergence between and The combination of the Kulback-Leibleer divergence between Among them, β is used to weight the Kullback-Leibler divergence term, with weight β1=1, Represents two features and The Kullback-Leibler divergence between Represents two features and The Kullback-Leibler divergence between represents the student network prediction sample, represents the teacher network prediction sample; The final loss function includes two supervised losses on the teacher network and the student network, the self-supervised loss for clinical data prediction, and the distillation loss between the student network and the teacher network. The loss function is defined as follows: in, and They represent the supervision loss of teacher network prediction and the supervision loss of student network prediction, respectively, and are calculated using Focal loss. and The prediction loss, L clinical represents the self-supervised loss for clinical data prediction, using cross entropy loss to calculate P C prediction errors and the underlying facts of clinical data; represents the classification-level distillation loss, Denotes the feature-level distillation loss between the teacher network and the student network, using equation L total The loss function is used to train the modality-aware distillation network for MIV prediction; S4. Hepatocellular carcinoma prediction using trained modality-aware distillation network.

2. The method for predicting hepatocellular carcinoma based on a modality-aware distillation network according to claim 1, wherein: The dataset described in step S1 consists of data from 270 pathologically confirmed HCC patients, including 128 M0 patients, 93 M1 patients, and 49 M2 patients; among them, M0 indicates no microvascular invasion, M1 indicates no more than 5 invading blood vessels or the invasion is located within 1 cm of the tumor surface, and M2 indicates more than 5 invading blood vessels or the invasion is more than 1 cm away from the tumor surface.

3. The method for predicting hepatocellular carcinoma based on modality-aware distillation network according to claim 1, wherein: In step S2, the size of the cube is set to 80*80*20 pixels.

4. The method for predicting hepatocellular carcinoma based on a modality-aware distillation network according to claim 1, wherein: The training process of the modality-aware distillation network in step S3 is specifically as follows: S31. The teacher network passes the HBP image and clinical data to the MRI-clinical fusion module to extract a 512-dimensional vector feature. The PRE image and clinical data are then input into another MRI-clinical fusion module to obtain another 512-dimensional vector feature. The two obtained 512-dimensional vector features are input into the SA module to obtain a new feature that fuses each other's information. and Finally, the new features and Splice them together to generate Z t , and Z t Passed to two fully connected layers to predict the classification result P t ; S32, the student network takes 3D HBP MRI images and 3D PRE MRI images as input, passes the HBP data to the MRI-only module to obtain features, and then passes the PRE data to another MRI-only module to obtain features; the two obtained features are input into the SA module to obtain new features that fuse each other's information and Among them, the new features and are two feature vectors containing 512 dimensions; finally, the new features and Connect them to generate Z s , and Z s Input into the fully connected layer to predict the MVI classification result P s ; S33. In the student network, a regression task is introduced to connect the feature Z of the input HBP image and the input PRE image. s Sent to two fully connected layers to predict a 52-dimensional vector P c , used to estimate potential clinical information and then use the input clinical data as the prediction P c The true label of S34. Classification-level distillation loss and feature-level distillation loss are used to distill the features fused from the teacher network clinical data and MRI images into features extracted from MRI images, and the knowledge distillation strategy is used to transfer the clinical information of the teacher network to the student network.

5. The method for predicting hepatocellular carcinoma based on modality-aware distillation network according to claim 4, characterized in that: The MRI-clinical fusion module in step S31 integrates MRI data and non-imaging clinical data. Taking 3D MRI data and vectorized clinical data as input, the MRI-clinical fusion module applies four fully connected layers to the input clinical data to obtain four feature maps, and these feature channels are 64, 128, 256, and 256, respectively; four convolution blocks on the input MRI image are used to obtain another 3D feature map, and the feature channels are also set to 64, 128, 256, and 256, and each convolution block consists of two 3×3 convolution layers; the four feature maps in the clinical data and the corresponding four features in the MRI data are channel-wise multiplied to integrate them together, and then a 3×3 convolution layer and a fully connected layer are applied to output a feature vector with 512 dimensions.

6. The method for predicting hepatocellular carcinoma based on a modality-aware distillation network according to claim 4, wherein: In step S32, the MRI-only module extracts a 512-dimensional feature vector from the 3D MRI image. The MRI-only module consists of nine convolutional blocks and a fully connected layer. Each convolutional block contains a batch norm layer, a ReLU activation layer, and a 3×3 convolutional layer to improve the robustness of the network. The number of channels of the output features of the nine convolutional blocks is set to different values. The feature channels of the first five layers are 32, 32, 64, 64, and 128, and the feature channels of the last four layers are 256, 128, 256, and 256 to balance efficiency and computational burden.

7. The method for predicting hepatocellular carcinoma based on a modality-aware distillation network according to claim 4, wherein: The SA module in steps S31 and S32 is a symmetric attention module. X and Y represent the two input feature maps of the SA module. The SA module applies a linear transformation layer on X to obtain three feature maps, including the query vector Q x , Key vector K x and Value vector V x ; SA module applies a transformation layer on Y to generate the Key feature map K y and Value feature map V y ; by multiplying by Q x and K x The transpose of is used to generate the score feature vector S x , by multiplying Q y and K y The transpose of to generate another score feature vector S y ; The obtained score feature vector S x and the Value feature vector V x Multiply and S y With V y Multiply them to generate two result feature vectors, and then add them together to finally generate the output refined feature vector The SA module applies another linear transformation layer on Y to obtain the feature vector Query vector Q y , by adding Q y and K y Multiply the transpose of Q y and K y The two score eigenvectors are calculated by multiplying the transpose of , and then the refined eigenvector is calculated by the following formula

Images