A lung cancer risk prediction method based on cross-modal knowledge distillation

Abstract

The present invention belongs to the field of medical health big data, and specifically relates to a lung cancer risk prediction method based on cross-modal knowledge distillation, comprising selecting a trained I3D model based on CT image input as a teacher model; using a multi-layer perceptron model to construct a student model; inputting a CT image data set into the teacher model to obtain a first intermediate feature vector set; using structured data as a student input to train the student model, and defining the features extracted from the structured data during the student model training process as second intermediate features; calculating the teacher model spatial distribution and the student model spatial distribution based on the first intermediate feature vector set and the second intermediate feature vector set; using KL divergence to calculate the divergence loss between the teacher model spatial distribution and the student model spatial distribution; calculating the weighted sum of the divergence loss and the cross entropy loss, and back-propagating the student model to train the student model until the student model parameters converge; using the trained student model as a lung cancer risk prediction model to predict the patient's lung cancer risk data to assist doctors in diagnosis.

Description

Technical Field

[0001] The present invention belongs to the field of medical health big data, and specifically relates to a lung cancer risk prediction method based on cross-modal knowledge distillation. Background Art

[0002] Lung cancer is one of the most common malignant tumors in the world. According to the GLOBOCAN 2018 database, the global incidence of lung cancer is 11.6% and the mortality rate is 18.4%. Lung cancer is the leading cause of cancer death worldwide. The earlier it is detected, the better. Therefore, risk prediction for lung cancer is also crucial. The risk prediction model can initially identify high-risk individuals who meet the screening criteria for lung cancer, provide certain medical decision-making support for doctors' accurate diagnosis, and also provide early warning of individual living habits.

[0003] From the current model training methods, whether it is a model trained with multimodal data or a model trained with unimodal data, the model needs to provide data of the same modality as its training data when it is used. This results in that when using multimodal technology, although it can ensure that data of different modalities in the data platform can be used for model training, so that the data in the platform can be fully utilized, the cost of obtaining data for prediction will increase when the model is used; in contrast, when using unimodal technology, the cost of obtaining data for prediction will not be too high, but the platform data cannot be fully utilized when the model is trained. Summary of the invention

[0004] To solve the above problems, the present invention provides a lung cancer risk prediction method based on cross-modal knowledge distillation, which is characterized by comprising the following steps:

[0005] S1. Select the trained I3D model based on CT image input as the teacher model; use the multi-layer perceptron model to build the student model, and use the structured questionnaire data as input;

[0006] S2. Input the CT image data set into the teacher model and extract the intermediate feature map set, and convert each intermediate feature map into a corresponding first intermediate feature vector;

[0007] S3. Input the structured questionnaire data set corresponding to the CT image data set into the student model, output the prediction result and extract the second intermediate feature vector set, and calculate the cross entropy loss according to the prediction result;

[0008] S4. Calculate the teacher model spatial distribution and the student model spatial distribution based on the first intermediate feature vector set and the second intermediate feature vector set;

[0009] S5. Use KL divergence to calculate the divergence loss between the teacher model space distribution and the student model space distribution;

[0010] S6. Calculate the weighted sum of the divergence loss and the cross entropy loss, and train the student model by backpropagation until the student model parameters converge;

[0011] S7. The trained student model is used as a lung cancer risk prediction model to predict the patient’s lung cancer risk data to assist doctors in diagnosis.

[0012] Further, step S2 inputs any CT image into the teacher model to extract the intermediate feature map, and converts the intermediate feature map into the corresponding first intermediate feature vector; including:

[0013] S21. A CT image CT volume Input the teacher model and extract the corresponding n×m dimensional intermediate feature map featuremap; where n represents the length of the intermediate feature map and m represents the width of the intermediate feature map;

[0014] S22. Calculate the row average value for each row element value of the n×m dimensional intermediate feature map featuremap, and obtain the first intermediate feature vector of n×1 dimension according to the row average value.

[0015] Furthermore, in step S4, calculating the teacher model spatial distribution according to the first intermediate feature vector set includes:

[0016] S41. Process the first intermediate feature vector of each two CT images using a kernel function to obtain the corresponding vector dot product, expressed as:

[0017]

[0018] Among them, x i , x j ∈X represent the i-th CT image and the j-th CT image in the CT image dataset X, K() represents the kernel function, represents the first intermediate feature vector corresponding to the i-th CT image, and T represents the transposition operation;

[0019] S42. The kernel density estimation method is used to process all vector dot products to obtain the teacher model space distribution. The conditional probability density of each CT image relative to the other different CT images in the teacher model space distribution is expressed as:

[0020]

[0021] Among them, p(x i |x j ) represents the conditional probability density of the i-th CT image relative to the j-th CT image in the CT image dataset, and N represents the number of CT images in the CT image dataset.

[0022] Furthermore, step S5 uses KL divergence to calculate the divergence loss between the teacher model space distribution and the student model space distribution, which is expressed as:

[0023]

[0024] Among them, q(y i |y j ) represents the conditional probability density of the i-th structured questionnaire data relative to the j-th structured questionnaire data in the structured questionnaire data set, p(x i |x j ) represents the conditional probability density of the i-th CT image relative to the j-th CT image in the CT image dataset, N represents the number of samples, and KL(·||·) is used to measure the KL divergence representing the distance between two distributions.

[0025] Furthermore, step S6 calculates the weighted sum of the divergence loss and the cross entropy loss, expressed as:

[0026] loss = αloss CE +(1-α)loss KD ×grad_ratio

[0027] Among them, α is the weighting coefficient, loss CE is the cross entropy loss, loss KD is the divergence loss, and grad_ratio is the gradient correction parameter.

[0028] Furthermore, the calculation formula of the gradient correction parameter grad_ratio is:

[0029]

[0030]

[0031]

[0032] Among them, θ represents the student model parameter, grad1_norm represents the second norm of the gradient provided by the cross entropy loss, grad2_norm represents the second norm of the gradient provided by the divergence loss, and ||·||2 represents the second norm calculation.

[0033] Beneficial effects of the present invention:

[0034] 1) The existing lung cancer risk prediction model can be trained and tested using various data, and the input data required by the model during testing is consistent with the training data. However, in actual applications after the model is trained, some data acquisition costs are high, such as the CT image data mentioned in the present invention. In order to obtain CT image data, you need to go to the hospital to register for examination. After obtaining the data, you can input the data into the model for prediction. However, the risk prediction model itself provides decision support for whether to go to the hospital for examination. Going to the hospital for examination in order to use the model is contrary to the function of the risk prediction model. 2) Therefore, the constructed model should select easily accessible data as input, such as the structured questionnaire data in the present invention. However, the existing risk prediction model requires the input data to be consistent with the training data during testing, which makes it impossible to use diagnosis-related data such as CT data to improve model performance. In order to solve problems 1) and 2), the present invention applies cross-modal knowledge distillation technology to the lung cancer risk prediction model. The ultimate beneficial effect is that it removes the restriction that the input data of the model must be consistent with the training data during testing. Through cross-modal knowledge distillation, data from more modalities and more views can be used to train the model, but when the model is trained, only easily accessible data needs to be used for prediction when it is used, ensuring a low cost of use for the model while making more effective use of multi-source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 The figure is a schematic block diagram of the method structure of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The present invention considers different modal data and distillation technology based on feature knowledge, and proposes a lung cancer risk prediction method based on cross-modal knowledge distillation. In common practice, distilling feature knowledge from a fine-grained level can achieve good results, but this is limited to the modalities across which the samples are similar, such as: image-to-image types such as RGB images to infrared images, or sequence-to-sequence types such as video to audio. Since there are certain similarities between these modalities, similar model structures can be used to process these modal data. When performing modal pairing, additional network modules can be introduced or the mean square error can be directly applied to process the output of the intermediate network to achieve a more ideal effect. However, in terms of unstructured image data and structured tabular data, there are greater differences between their modalities. In addition, the information content contained in these two data types is also different, which further increases the complexity and challenge of student model training. Therefore, considering this point, the present invention migrates feature knowledge from a coarse-grained level, and uses KL divergence to align the conditional probability distribution of the feature space samples of the student model and the teacher model, thereby performing feature knowledge distillation.

[0039] Specific plans such as Figure 1 , Figure 2 As shown, the following steps are included:

[0040] S1. The trained I3D model based on CT image input is selected as the teacher model; the student model is constructed using a multi-layer perceptron model, and the student model takes structured questionnaire data as input.

[0041] Specifically, the present invention obtains a multimodal training data set for model training, wherein the multimodal training data set includes multiple training samples, each training sample includes CT image data of the same participant and its corresponding structured questionnaire data. During the training process, the multimodal training data set is divided into multiple batches of training sets for training.

[0042] S2. Input the CT image data set in a batch of training data sets into the teacher model, extract the corresponding intermediate feature map set, and convert each intermediate feature map into a corresponding first intermediate feature vector.

[0043] Specifically, step S2 inputs any CT image into the teacher model to extract the intermediate feature map, and converts the intermediate feature map into the corresponding first intermediate feature vector; including:

[0044] S21. A CT image CT volume Input the teacher model and extract the corresponding n×m dimensional intermediate feature map featuremap; where n represents the length of the intermediate feature map and m represents the width of the intermediate feature map;

[0045] The process can be expressed as:

[0046] featuremap=Feature_extractor teacher (CT volume )(1)

[0047] v = mean(featuremap,1)(2)

[0048] Among them, Feature_extractor teacher represents the teacher model feature extractor, v represents the first intermediate feature vector, mean() is the row-wise averaging function, mean(x,y) represents averaging the tensor x according to the y-th dimension (y=0,1,2,…,n). The present invention specifically refers to averaging the first dimension of the intermediate feature map, i.e., the row.

[0049] S22. Calculate the row average value for each row element value of the n×m dimensional intermediate feature map featuremap, and obtain the first intermediate feature vector of n×1 dimension according to the row average value.

[0050] S3. Use the structured questionnaire data set in the same batch training set as the input of the student model to train the student model. During this training process, calculate the corresponding cross entropy loss according to the prediction results of each structured questionnaire data, and define the features extracted from the structured questionnaire data during the student model training process as the second intermediate features, and convert the second intermediate features into corresponding second intermediate feature vectors.

[0051] S4. Based on the first set of intermediate feature vectors and the second set of intermediate feature vectors, calculate the teacher model space distribution and the student model space distribution.

[0052] Specifically, in step S4, calculating the teacher model spatial distribution according to the first intermediate feature vector set includes:

[0053] S41. Process the first intermediate feature vector of each two CT images using a kernel function to obtain the corresponding vector dot product, expressed as:

[0054]

[0055] Among them, x i , x j ∈X represent the i-th CT image and the j-th CT image in the CT image dataset X, K() represents the kernel function, represents the first intermediate feature vector corresponding to the i-th CT image, and T represents the transposition operation;

[0056] S42. The kernel density estimation method is used to process all vector dot products to obtain the teacher model space distribution. The conditional probability density of each CT image relative to the other different CT images in the classroom model space distribution is expressed as:

[0057]

[0058] Among them, p(x i |x j ) represents the conditional probability density of the i-th CT image relative to the j-th CT image in the CT image dataset, and N represents the number of CT images in the CT image dataset.

[0059] S5. KL divergence is used to calculate the divergence loss between the teacher model space distribution and the student model space distribution.

[0060] Specifically, KL divergence is used as the loss function to measure and bring the distribution of the two spaces closer. The specific calculation formula is:

[0061]

[0062] Among them, q(y i |y j ) represents the conditional probability density of the i-th structured questionnaire data relative to the j-th structured questionnaire data in the structured questionnaire data set, p(x i |x j ) represents the conditional probability density of the i-th CT image relative to the j-th CT image in the CT image dataset, n represents the number of samples, and KL(·||·) is used to measure the KL divergence representing the distance between two distributions.

[0063] S6. Calculate the weighted sum of the divergence loss and the cross entropy loss, and back-propagate the student model until the student model parameters converge.

[0064] Specifically, the student model is constructed by a multi-layer perceptron and trained by the back propagation algorithm. When it is trained with CT images and structured questionnaire data, the loss function consists of two parts: one is the loss of distilled feature knowledge. KD , and the other part is the cross entropy loss with the correct label CE , and finally the complete loss can be obtained by weighted combination of these two parts, which is expressed as:

[0065] loss = αloss CE +(1-α)loss KD (6)

[0066] Among them, α is the weighting coefficient, loss CE is the cross entropy loss, loss KDis the divergence loss.

[0067] In the practice of student model training, we found that in the early stage of student model training, the gradient provided by the distillation loss is much smaller than the gradient provided by the cross entropy (about 20,000 to 30,000 times). In the gradient descent algorithm, the direction of gradient descent is a key point. Among the two losses with large gradient differences, the loss that provides a larger gradient plays a dominant role in model training, which affects the effect of knowledge distillation. Although the gradient gap between the two will narrow in the later stage of training, the model has been overfitted at this time, so the loss of formula (6) is corrected to:

[0068] loss = αloss CE +(1-α)loss KD ×grad_ratio(7)

[0069] Among them, grad_ratio is the gradient correction parameter, and its calculation formula is:

[0070]

[0071]

[0072]

[0073] Among them, θ represents the student model parameter, grad1_norm represents the second norm of the gradient provided by the cross entropy loss, grad2_norm represents the second norm of the gradient provided by the divergence loss, and ||·||2 represents the second norm calculation.

[0074] Through such correction, the gradients provided by the two losses will not differ too much during the training process, which is conducive to promoting the distillation effect.

[0075] S7. The trained student model is used as a lung cancer risk prediction model to predict the patient’s lung cancer risk data to assist doctors in diagnosis.

[0076] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lung cancer risk prediction method based on cross-modal knowledge distillation, It is characterized in that The following steps are involved: S1. Select the trained I3D model based on CT image input as the teacher model; use the multi-layer perceptron model to build the student model, and use the structured questionnaire data as input; S2. Input the CT image data set into the teacher model and extract the intermediate feature map set, and convert each intermediate feature map into a corresponding first intermediate feature vector; S3. Input the structured questionnaire data set corresponding to the CT image data set into the student model, output the prediction result and extract the second intermediate feature vector set, and calculate the cross entropy loss according to the prediction result; S4. Calculate the teacher model spatial distribution and the student model spatial distribution based on the first intermediate feature vector set and the second intermediate feature vector set; In step S4, the teacher model spatial distribution is calculated according to the first intermediate feature vector set, including: S41. Process the first intermediate feature vector of each two CT images using a kernel function to obtain the corresponding vector dot product, expressed as: Among them, x i , x j ∈X represent the i-th CT image and the j-th CT image in the CT image dataset X, K() represents the kernel function, represents the first intermediate feature vector corresponding to the i-th CT image, and T represents the transposition operation; S42. The kernel density estimation method is used to process all vector dot products to obtain the teacher model space distribution. The conditional probability density of each CT image relative to the other different CT images in the teacher model space distribution is expressed as: Among them, p(x i |x j ) represents the conditional probability density of the i-th CT image relative to the j-th CT image in the CT image dataset, and N represents the number of CT images in the CT image dataset; S5. Use KL divergence to calculate the divergence loss between the teacher model space distribution and the student model space distribution; S6. Calculate the weighted sum of the divergence loss and the cross entropy loss, and train the student model by backpropagation until the student model parameters converge; Step S6 calculates the weighted sum of the divergence loss and the cross entropy loss, expressed as: loss=αloss CE +(1-α)loss KD ×grad_ratio Among them, α is the weighting coefficient, loss CE is the cross entropy loss, loss KD is the divergence loss, grad_ratio is the gradient correction parameter; The calculation formula of the gradient correction parameter grad_ratio is: Among them, θ represents the student model parameters, grad1_norm represents the second norm of the gradient provided by the cross entropy loss, grad2_norm represents the second norm of the gradient provided by the divergence loss, ||·|| 2 represents the calculation of the second norm; S7. The trained student model is used as a lung cancer risk prediction model to predict the patient’s lung cancer risk data to assist doctors in diagnosis.

2. According to claim 1, a lung cancer risk prediction method based on cross-modal knowledge distillation, It is characterized in that Step S2: any CT image is input into the teacher model to extract the intermediate feature map, and the intermediate feature map is converted into the corresponding first intermediate feature vector; include: S21. A CT image CT volume Input the teacher model and extract the corresponding n×m dimensional intermediate feature map featuremap; where n represents the length of the intermediate feature map and m represents the width of the intermediate feature map; S22. Calculate the row average value for each row element value of the n×m dimensional intermediate feature map featuremap, and obtain the first intermediate feature vector of n×1 dimension according to the row average value.

3. According to claim 1, a lung cancer risk prediction method based on cross-modal knowledge distillation, It is characterized in that Step S5 uses KL divergence to calculate the divergence loss between the teacher model space distribution and the student model space distribution, expressed as: Among them, q(y i |y j ) represents the conditional probability density of the i-th structured questionnaire data relative to the j-th structured questionnaire data in the structured questionnaire data set, p(x i |x j ) represents the conditional probability density of the i-th CT image relative to the j-th CT image in the CT image dataset, N represents the number of samples, and KL(·||·) is used to measure the KL divergence representing the distance between two distributions.

Images