A lung nodule diagnosis model effectively fusing clinical knowledge and AI knowledge

Abstract

A lung nodule diagnosis model effectively fusing clinical knowledge and AI knowledge comprises a detection module, an initial diagnosis module, a semantic attribute optimization module and a final diagnosis result prediction module; the detection module is used for extracting a region of interest of a nodule from a CT image; the initial diagnosis module performs shared feature extraction and multi-task prediction according to the region of interest of the nodule, predicts a nodule semantic attribute score and an initial benign-malignant diagnosis result, and completes initialization prediction; the semantic attribute optimization module selectively optimizes the semantic score according to the nodule semantic attribute obtained by the initial diagnosis module, reassigns weights to corresponding semantic features, and obtains modified and optimized semantic features; the final diagnosis result prediction module maps the optimized semantic features into a feature vector, and finally obtains a nodule malignancy prediction probability; the application effectively fuses semantic features (clinical knowledge) and AI knowledge, and increases the accuracy of lung nodule benign-malignant prediction.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence algorithm technology, and specifically relates to a lung nodule diagnostic model that effectively integrates clinical knowledge and AI knowledge. Background Technology

[0002] Clinicians typically use semantic features to determine the malignancy of lung nodules, while artificial intelligence (AI) systems tend to extract unknown features to diagnose nodules. The former relies on clinical knowledge, while the latter explores AI knowledge. Although numerous studies have shown that integrating clinical and AI knowledge can help computer-aided diagnostic systems improve diagnostic accuracy and gain clinician acceptance, how to effectively integrate clinical and AI knowledge remains an unresolved issue.

[0003] Early diagnosis and treatment are considered the most effective way to improve the survival rate of lung cancer patients. Studies have shown that if early diagnosis and treatment are initiated when the primary tumor is small and has not yet spread, the 5-year survival rate of stage IV lung cancer patients can be increased from 5% to 60%. In clinical practice, lung CT scans, which can visualize nodules, are the primary tool for screening potentially malignant nodules. Typically, radiologists visually screen CT images layer by layer to detect lung nodules, and then determine whether the nodule is benign or malignant based on clinically interpretable nodule characteristics. Commonly used clinically interpretable characteristics include: subtlety, internal structure, calcification, sphericity, margin, lobulation, spiculation, and texture; these eight characteristics are also known as the semantic attributes of nodules.

[0004] Given the time-consuming and labor-intensive nature of relying solely on visual screening for diagnosing lung nodules, and its susceptibility to subjective factors, researchers have dedicated decades to developing computer-aided diagnosis (CAD) systems to supplement or even replace traditional manual diagnostic methods. Generally, these CAD methods can be categorized into classical machine learning and current deep learning approaches. Classical methods rely on carefully selected handcrafted features and rigorous image preprocessing, resulting in poor diagnostic accuracy in complex application scenarios. In contrast, deep learning models can adaptively learn high-dimensional features with minimal human intervention, achieving higher accuracy than traditional machine learning methods. Consequently, recently proposed lung nodule diagnostic methods are primarily based on deep neural networks. While these deep learning methods have accelerated the automation of lung nodule diagnosis, they still suffer from a significant drawback: they fail to effectively integrate clinical knowledge into the deep learning models, leading to poor model reliability and interpretability.

[0005] Although many studies have shown that predicting the benign or malignant nature of nodules and analyzing their semantic attributes are highly correlated and mutually reinforcing, how to effectively improve the reliability and interpretability of nodule diagnosis using semantic attributes remains an open question. One researcher (Chen et al.) used a parallel neural network to study the intrinsic relationship between eight semantic features of nodules and their benign or malignant nature (Strategy 1). While this method considers both benign / malignant nature and other attributes, the relationship between these attributes is not explored causally because the output attribute scores fail to serve as direct evidence for benign / malignant prediction. Another researcher (Hussein et al.) proposed a multi-task learning framework to explore the intrinsic relationship between nodule benign / malignant nature and attribute scores in a causal manner (Strategy 2). However, this structure has a drawback: each subtask is used to predict a specific semantic attribute, ignoring some potential features that clinicians have not yet discovered but are related to nodule benignity or malignancy. In light of this, some scholars (Liu et al. and Xie et al.) designed a separate branch in the neural network to extract unknown features of nodules. This branch is not supervised by semantic labels, and the features obtained are called AI features. Liu et al. fused semantic attribute features and AI features to obtain more reliable benign and malignant predictions; this method is called feature-level knowledge fusion (strategy 3). Xie et al. fused diagnostic results based on semantic features and diagnostic results based on AI features to obtain more reliable benign and malignant predictions; this method is called decision-level knowledge fusion (strategy 4). The invention, with application number CN201910790034.8 and titled "An Intelligent Diagnostic Method for Lung Nodules Based on Hybrid Features," utilizes a 3D residual-tightly connected network to learn high-dimensional depth features from 3D CT images, obtaining LBP-based texture features and HOG-based shape features to describe the characterization features of lung nodules; and employs a GBM gradient booster to classify the benign and malignant nature of lung nodules based on the high-dimensional depth features, LBP-based texture features, and HOG-based shape features. However, fusion that only occurs at the feature level weakens the interpretability of the model, leading to distrust from clinicians; fusion that only occurs at the decision level loses the rich information in high-dimensional features, resulting in a decrease in the model's diagnostic accuracy. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to provide a lung nodule diagnostic model that effectively integrates clinical knowledge and AI knowledge. This model can effectively integrate semantic knowledge (clinical knowledge) and AI knowledge. In the initial prediction stage (IPS), a series of semantic features are first extracted, and semantic attribute scores and preliminary diagnostic results of lung nodules are provided. These results are then sent to the subsequent prediction refinement stage (PRS) to finally obtain a diagnostic result that can be used for reference.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A lung nodule diagnostic model that effectively integrates clinical knowledge and AI knowledge includes a detection module, an initial diagnosis module, a semantic attribute optimization module, and a final diagnosis result prediction module.

[0009] The detection module is used to extract the region of interest (ROI) of a nodule from CT images.

[0010] The initial diagnosis module performs shared feature extraction and multi-task prediction based on the region of interest of the nodule output by the detection module, predicts the semantic attribute score of the nodule and the initial benign or malignant diagnosis result, and completes the initial prediction.

[0011] The semantic attribute optimization module selectively optimizes the semantic score based on the nodule semantic attributes obtained from the initial diagnosis module, and re-weights the corresponding semantic features to obtain the modified optimized semantic features.

[0012] The final diagnosis result prediction module maps the semantic features optimized by the semantic attribute optimization module to feature vectors to obtain the predicted probability of nodule malignancy.

[0013] The extraction of the region of interest of the nodule specifically refers to:

[0014] (1) In a publicly available pulmonary nodule dataset LIDC-IDRI, heterogeneous CT images acquired from different devices and reconstruction parameters are included. These CT images are first resampled and normalized, that is, all CT images are resampled to a voxel spacing of [1, 1, 1] using the nearest neighbor interpolation algorithm. Then, the resampled CT images are converted to [-1000, 500HU] and finally normalized to [0, 1].

[0015] (2) Based on the nodule location provided in the LIDC-IDRI annotation file of the lung nodule dataset, a nodule cube of 80×80×60 mm is cut out and the cube is used as the region of interest of the nodule. The region of interest of the nodule is obtained by using object detection algorithm or segmentation algorithm.

[0016] The completion of the initial prediction, as described above, specifically includes:

[0017] (1) 3D-ResNet18 is used as the shared feature extraction in the initial diagnosis module. The input of the shared feature extraction is the nodule cube of 80×80×60 mm output by the detection module, and the output of the shared feature extraction is a 512-dimensional feature.

[0018] (2) The 512-dimensional shared features are used to predict the benignity or malignancy of nodules and eight semantic scores. Therefore, there are a total of nine parallel branches. Each branch predicts a different task. In the task prediction branch, the 512-dimensional shared features are first compressed into 128-dimensional features using a fully connected layer, which represents the specific features of a particular task. Among these nine parallel branches, eight branches are recorded as eight semantic features and one branch is recorded as an AI feature. Then, the 128-dimensional features are compressed into 1-dimensional features using a fully connected layer. Finally, the 1-dimensional feature is mapped to the benignity or malignancy probability or semantic score through the activation function Sigmoid.

[0019] The semantic attribute optimization module consists of two steps: (a) modifying the semantic score that is inconsistent with its judgment; and (b) using the modified semantic score as the weight coefficient of the corresponding semantic feature to obtain the modified semantic feature.

[0020] The final diagnostic result prediction module works as follows: First, it concatenates eight 128-dimensional semantic features and one 128-dimensional AI feature to obtain a 9×128-dimensional feature vector. Second, this 9×128-dimensional feature vector is processed through a self-attention mechanism to learn the intrinsic relationships between the nine features. Then, a linear connection layer maps the 9×128-dimensional feature vector back to a 128-dimensional feature vector, referred to as the malignancy-related feature. Finally, another linear connection layer maps the malignancy-related feature back to a 1-dimensional vector, which serves as the final probability prediction of nodule malignancy. If this probability is greater than or equal to 0.5, the nodule is considered malignant; if the probability is less than 0.5, the nodule is considered benign.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects:

[0022] 1. When predicting the benign or malignant nature of nodules, this invention provides predictions for eight semantic attributes of the nodules. These semantic attributes are features that clinicians are familiar with and can intuitively perceive. Therefore, our model has good interpretability, which is conducive to the promotion and application of artificial intelligence models.

[0023] 2. This invention effectively integrates semantic features (clinical knowledge) and AI knowledge, increasing the accuracy of predicting the benign or malignant nature of lung nodules.

[0024] 3. This invention provides accurate prediction of benign and malignant pulmonary nodules and semantic attribute representation, which can meet the urgent clinical need for an accurate and reliable computer-aided diagnostic system. Attached Figure Description

[0025] Figure 1 This is a structural diagram of the model of the present invention.

[0026] Figure 2 This is an example of a branch of the multi-task prediction module.

[0027] Figure 3 This is an optional semantic attribute optimization module.

[0028] Figure 4 This is a module for predicting the final diagnostic results.

[0029] Figure 5 This invention presents the prediction results for the benign or malignant nature of nodules and eight semantic attributes. Detailed Implementation

[0030] The invention will now be described in further detail with reference to the accompanying drawings.

[0031] A diagnostic model for lung nodules that effectively integrates clinical knowledge and AI knowledge, referring to Figure 1 It includes a detection module, an initial diagnosis module, a semantic attribute optimization module, and a final diagnosis result prediction module.

[0032] The detection module is used to extract the region of interest (ROI) of a nodule from CT images.

[0033] The initial diagnosis module, based on the region of interest of the nodule output by the detection module, performs shared feature extraction and multi-task prediction to predict the semantic attribute score of the nodule and an initial benign / malignant diagnosis result, thus completing the initial prediction. The initial prediction stage is essentially a multi-task neural network, whose input is a nodule cube and whose output includes eight semantic scores and an initial benign / malignant prediction result.

[0034] The semantic attribute optimization module selectively optimizes the semantic score and reweights the corresponding semantic features based on the nodule semantic attributes (semantic score, semantic features) obtained from the initial diagnosis module, thereby obtaining the modified optimized semantic features and influencing the final diagnosis result.

[0035] The final diagnosis result prediction module maps the semantic features optimized by the semantic attribute optimization module to feature vectors to obtain the predicted probability of nodule malignancy.

[0036] The extraction of the region of interest of the nodule specifically refers to:

[0037] (1) In a publicly available pulmonary nodule dataset LIDC-IDRI, there are 1018 heterogeneous CT images acquired from different devices and reconstruction parameters. To address the heterogeneity, we resampled and normalized these CT images before use. Specifically, we used the nearest neighbor interpolation algorithm to resample all CT images to a voxel spacing of [1, 1, 1]. Then, we converted the resampled CT images to [-1000, 500HU] and finally normalized them to [0, 1].

[0038] (2) Since the maximum length of the entire nodule in the length, width, and height dimensions are 67, 68, and 51 mm, respectively, we cropped an 80×80×60 mm nodule cube based on the nodule location provided in the LIDC-IDRI annotation file of the lung nodule dataset, and used this cube as the region of interest (ROI) of the nodule. It should be noted that if the CT image dataset we obtained does not contain the location information of the nodule, then existing object detection algorithms (Yolo series) or segmentation algorithms (U-Net series) can be used to obtain the ROI of the nodule.

[0039] The completion of the initial prediction, as described above, specifically includes:

[0040] (1) Considering the model performance and computational complexity, 3D-ResNet18 is used as the shared feature extraction in the initial diagnosis module. The input of the shared feature extraction is the nodule cube of 80×80×60 mm output by the detection module, and the output of the shared feature extraction is a 512-dimensional feature.

[0041] (2) The benign or malignant nodules and eight semantic scores are predicted using 512-dimensional shared features, thus containing a total of nine parallel branches, each predicting different tasks. Figure 2The structural details of one branch are shown. In the task prediction branch, a fully connected layer is first used to compress the 512-dimensional shared features into 128-dimensional features, representing task-specific features. Of these nine parallel branches, eight branches are recorded as eight semantic features, and one branch is recorded as an AI feature. Then, a fully connected layer is used to compress the 128-dimensional AI features into 1-dimensional features. Finally, the 1-dimensional feature is mapped to a benign or malignant probability or a semantic score using the sigmoid activation function.

[0042] The semantic attribute optimization module described above faces a challenging task for both clinicians and deep learning modules in diagnosing nodules with similar appearances. To combine the strengths of both clinicians and deep learning models, we designed an optional semantic attribute optimization module, the structure of which is as follows: Figure 3 As shown in the diagram. In this module, clinicians selectively optimize semantic scores. The optimized semantic scores reweight the corresponding semantic features, thereby affecting the final diagnostic results. The specific operation consists of two steps: (a) Clinicians modify semantic scores that are inconsistent with their judgment. For example, if the network predicts the lobulation attribute score of a nodule to be 0.8, but the clinician believes it to be 0.3, the clinician can modify the lobulation score to 0.3; (b) The modified semantic scores are used as weight coefficients for the corresponding semantic features to obtain the modified semantic features, which can help improve the final diagnostic accuracy.

[0043] The final diagnosis result prediction module, such as Figure 4 As shown, the specific steps are as follows: First, eight 128-dimensional semantic features and one 128-dimensional AI feature are concatenated to obtain a 9×128-dimensional feature vector. Second, this 9×128-dimensional feature vector is processed through a self-attention mechanism to learn the intrinsic relationships between the nine features. Then, a linear connection layer maps the 9×128-dimensional feature vector back to a 128-dimensional feature vector, called the malignancy-related feature. Finally, another linear connection layer maps the malignancy-related feature back to a 1-dimensional vector, which serves as the final prediction probability of nodule malignancy. If this probability is greater than 0.5, the nodule is considered malignant; if the probability is less than 0.5, the nodule is considered benign. Figure 5 This section displays the prediction results for nodule malignancy and eight semantic attributes. The number in the upper right corner of each image indicates the degree of malignancy: less than 3 indicates benign, and greater than 3 indicates malignancy. The MP at the bottom of the image represents the CKAK predicted probability of nodule malignancy: greater than 0.5 predicts malignancy, and less than 0.5 predicts benignity.

[0044] Experimental results

[0045] To verify the effectiveness of the proposed method (CKAK), it was compared with eight similar methods. The comparative experiments were conducted on the LIDC-IDRI dataset; for convenience, these eight methods are referred to as M1 to M8. The details are as follows:

[0046] M1 uses only texture features to predict the benign or malignant nature of nodules;

[0047] M2 integrates shape, edge sharpness, and texture features to predict the benign or malignant nature of nodules;

[0048] M3 uses 2D convolutional neural networks (CNNs) to predict the benign or malignant nature of nodules;

[0049] M4 uses six semantic attributes and a 3D-CNN to predict the benign or malignant nature of nodules;

[0050] M5, M6, and M7 fuse shape features, texture features, and convolutional neural network features to predict the benignity or malignancy of nodules. The difference between these three models lies in their methods for extracting deep learning features: M5 uses a single 3D-CNN to extract deep features; M6 uses three parallel 2D-CNNs to extract deep features; and M7 uses 27 2D-CNNs with different viewpoints to extract deep features.

[0051] M8 uses three parallel 3D-CNNs to extract multi-scale features of lung nodules and fuses nodule size and texture features to predict the benign or malignant nature of nodules.

[0052] The results of the comparative experiment are shown in the table below.

[0053] The diagnostic performance of different methods for pulmonary nodules.

[0054]

[0055] As can be seen from the table, this invention is significantly superior to the eight comparative methods. This is mainly because we are able to fully and effectively integrate clinical knowledge and AI knowledge at the feature and decision levels, thus achieving better diagnostic accuracy.

[0056] Figure 5 This demonstrates the prediction performance of the CKAK method proposed in this invention for eight nodules. From... Figure 5 As can be seen, our proposed CKAK can accurately predict the benign or malignant nature of nodules and eight semantic attributes. These semantic attributes can help clinicians understand why neural networks make benign or malignant predictions, thus improving the interpretability of neural networks.

[0057] Experiments have shown that the model of this invention can meet the urgent clinical need for an accurate and reliable computer-aided diagnostic system for pulmonary nodules.

Claims

1. A lung nodule diagnostic system that effectively integrates clinical knowledge and AI knowledge, characterized in that, It includes a detection module, an initial diagnosis module, a semantic attribute optimization module, and a final diagnosis result prediction module; The detection module is used to extract the region of interest of the nodule from the CT image; The initial diagnosis module, based on the region of interest of the nodule output by the detection module, performs shared feature extraction and multi-task prediction to predict the semantic attribute score of the nodule and the initial benign or malignant diagnosis result, thus completing the initialization prediction; The semantic attribute optimization module selectively optimizes the semantic score based on the nodule semantic attributes obtained by the initial diagnosis module, and re-assigns weights to the corresponding semantic features to obtain the modified optimized semantic features. The final diagnosis result prediction module maps the semantic features optimized by the semantic attribute optimization module to feature vectors to finally obtain the predicted probability of nodule malignancy. The semantic attribute optimization module consists of two steps: (a) the clinician modifies the semantic score that is inconsistent with his / her judgment; (b) the modified semantic score is used as the weight coefficient of the corresponding semantic feature to obtain the modified semantic feature. The initial diagnosis module completes the initial prediction, specifically: (1) 3D-ResNet18 is used as the shared feature extraction in the initial diagnosis module. The input of the shared feature extraction is the nodule cube of 80×80×60 mm output by the detection module, and the output of the shared feature extraction is a 512-dimensional feature. (2) The 512-dimensional shared features are used to predict the benignity or malignancy of nodules and eight semantic scores. Therefore, there are a total of nine parallel branches. Each branch predicts a different task. In the task prediction branch, the 512-dimensional shared features are first compressed into 128-dimensional features using a fully connected layer, which represents the specific features of a particular task. Among these nine parallel branches, eight branches are recorded as eight semantic features and one branch is recorded as AI features. Then, the 128-dimensional features are compressed into 1-dimensional features using a fully connected layer. Finally, the 1-dimensional features are mapped to benignity or malignancy probability or semantic score through the activation function Sigmoid. The final diagnostic result prediction module is specifically structured as follows: First, eight 128-dimensional semantic features and one 128-dimensional AI feature are concatenated to obtain a 9×128-dimensional feature vector. Second, this 9×128-dimensional feature vector is processed through a self-attention mechanism to learn the intrinsic relationship between the nine features. Then, a linear connection layer maps the 9×128-dimensional feature vector to a 128-dimensional feature vector, which is called the malignancy-related feature. Finally, another linear connection layer maps the malignancy-related feature to a 1-dimensional vector, which serves as the final prediction probability of nodule malignancy. If the probability is greater than or equal to 0.5, the nodule is considered malignant; if the probability is less than 0.5, the nodule is considered benign.

2. The lung nodule diagnostic system that effectively integrates clinical knowledge and AI knowledge according to claim 1, characterized in that, The detection module extracts the region of interest (ROI) of the nodule, specifically as follows: (1) In a publicly available pulmonary nodule dataset LIDC-IDRI, heterogeneous CT images acquired from different devices and reconstruction parameters are included. These CT images are first resampled and normalized, that is, all CT images are resampled to a voxel spacing of [1, 1, 1] using the nearest neighbor interpolation algorithm. Then, the resampled CT images are converted to [-1000HU, 500HU] and finally normalized to [0, 1]. (2) Based on the nodule location provided in the LIDC-IDRI annotation file of the lung nodule dataset, a nodule cube of 80×80×60 mm is cut out and the cube is used as the region of interest of the nodule. The region of interest of the nodule is obtained by using object detection algorithm or segmentation algorithm.

Images