A CT image automatic classification method based on fusion time attention mechanism
By constructing a 3D CNN that integrates a temporal attention mechanism and a noise correction network, the problems of physician experience dependence and high data annotation costs in CT image classification are solved, and a CT image classification method with high accuracy and generalization ability is achieved, which is suitable for applications on multi-center datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2022-09-01
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, disease classification methods based on CT images are limited by differences in doctors' experience and high data annotation costs, resulting in insufficient classification accuracy and generalization performance. In particular, the misclassification rate is high when the lesions and background are highly similar in CT images of chronic diseases, and the model performance is unstable across different datasets.
A 3D CNN classification network based on a fusion temporal attention mechanism is adopted. By constructing a temporal attention network to replace the global average pooling layer and combining it with a noise correction network, the temporal features and noise distribution characteristics of CT images are utilized to perform image classification without slice annotation.
It improves the accuracy and generalization ability of CT image classification, reduces false positive samples, alleviates the multicenter effect, adapts to the recognition differences of different datasets, enhances lesion information and reduces background interference, and improves the robustness and accuracy of the model.
Smart Images
Figure CN115409812B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, as well as computer vision, deep learning, and more specifically, to a disease classification method based on CT images. Background Technology
[0002] Computed tomography (CT) scans are an important imaging modality used by radiologists and imaging specialists to assess lesions. In clinical diagnostic practice, doctors and specialists interpret given CT images based on their medical knowledge and daily diagnostic experience to provide lesion analysis results. However, the accuracy of diagnosis is limited by the differences in clinical experience and knowledge levels among different doctors and specialists. Furthermore, interpreting a large number of CT images requires doctors to invest a significant amount of time, and this high-intensity work may affect their mental state, leading to misdiagnosis. In recent years, with the continuous development of computer technology, the potential of artificial intelligence in the medical field has become undeniable, especially deep learning, which has made breakthroughs in computer vision and has been widely applied in image classification, image localization and detection, and medical image segmentation. However, most commonly used medical image diagnostic methods are based on supervised learning, which requires a large amount of labeled data. In reality, however, only a small number of labeled samples are available in the medical field. This is because obtaining data with complete lesion annotations requires a significant amount of time and effort from professional doctors, making the acquisition of medical images with complete lesion annotations extremely costly.
[0003] Currently, some scholars are using datasets without lesion annotations for classification research. However, in many studies, the high similarity between lesions and their surrounding background in CT images of chronic diseases leads to many non-lesion features being misclassified as lesions, resulting in low accuracy of classification models. Furthermore, most models only perform well on specific datasets; their performance often falls short when applied to different datasets, indicating poor generalization ability. Summary of the Invention
[0004] This invention aims to at least solve the technical problems existing in the prior art, and proposes an automatic classification method for CT images based on a 3D CNN with a fusion temporal attention mechanism.
[0005] To achieve the above-mentioned objectives of the present invention, the technical solution adopted by the method of the present invention includes the following steps:
[0006] Step S1: Collect medical imaging information related to suspected patients, including CT images, and establish the relationship between CT images and category labels;
[0007] Step S2: Preprocess the acquired raw CT images to obtain the processed dataset, and divide the preprocessed CT image dataset into training and test sets;
[0008] Step S3: Construct block inputs for CT images and build a 3D CNN classification network with a fused temporal attention mechanism. The specific steps are as follows:
[0009] Step S31: Define X = [x1, x2, ..., x n Let x be a CT image with n slices, where x i This represents a 2D CT slice. The CT image is divided into *s* disjoint 3D image blocks according to the slice order. Each 3D image block is considered as a set containing an equal number of consecutive CT slices.
[0010] Step S32: Construct a 3D CNN classification network with fused temporal attention mechanism:
[0011] First, a 3D CNN feedforward neural network is constructed, replacing the global average pooling layer in the general architecture with a temporal attention network. This leverages the temporal pooling effect of temporal attention to aggregate the correlations between CT scan slices and correctly learn the temporal features contained in the CT images, aiding in classification. Simultaneously, this network utilizes 3D convolutions to recognize CT images, requiring only knowledge of the entire CT image label, without needing the annotation of individual CT slices.
[0012] Step S33: Transfer each 3D image block S i The data are fed into a 3D CNN classification network with a fused temporal attention mechanism to obtain the classification prediction probability P(c|S) for each 3D image patch. i ), where c represents the category. The predicted probabilities of all 3D image patches are combined to obtain the predicted value for the entire CT image:
[0013] Step S34: Define y c For the true label of category c, the classification loss is calculated as follows:
[0014]
[0015] Step S4: Construct a noise correction network by adding a noise channel with unknown parameters to the real labels, and using a neural network to learn the noise distribution parameters to improve the model's recognition and generalization performance; the specific steps are as follows:
[0016] Step S41: When performing CT image classification, it is assumed that the classification network cannot observe the correct label y for each slice, but can only obtain a noisy label z. There is a hidden distribution relationship P(z) between the noisy label and the true label.c =i|y c =j,X); define this distribution relationship as a noise transformation function. First calculate
[0017]
[0018] Where i,j∈{0,1} are the label values; It is a nonlinear mapping function; and For learnable parameters, It can be viewed as the transformation score between the true label i and the noise label j for category c.
[0019] Step S42: Calculate the estimated probability of the noise label based on the noise transformation function:
[0020]
[0021] Step S43: Calculate the noise classification loss for the entire CT image based on the noise probability estimated for each CT slice.
[0022]
[0023] Step S5: Calculate the final loss: L total =L cls +λL noisy , where λ is the loss balance parameter.
[0024] Preferably, the CT images are desensitized data processed using desensitization technology.
[0025] Preferably, the segmentation algorithm is a deep learning method.
[0026] Preferably, the preprocessing of the acquired raw CT images specifically includes:
[0027] Step S21: Given an independent dataset, use it to pre-train a U-Net segmentation network;
[0028] Step S22: Apply a pre-trained U-Net network to segment each original CT image; a CT image consists of several single-channel CT slices, assuming one of the CT slices is I. ori When applying the segmentation algorithm, the segmentation mask I is first obtained. mask Then, a segmentation mask is used to extract the corresponding region from the original CT image. That is, the area covered by the mask remains unchanged, while other areas are filled with 0. The segmented image is denoted as I. seg ;
[0029] Step S23: Place Iori and the obtained I mask and I seg The original single-channel CT slices are overlaid and combined to form three-channel images. All CT slices are subjected to this operation.
[0030] Step S24: Resample the CT images to a uniform size, specifically: using segmentation mask I mask The center of the mask region is calculated, and the CT image is cropped around the center of the region with a uniform volume. Finally, the size of each CT image is uniformly h*w*n, where h and w are the length and width of the slice, and n is the number of slices.
[0031] The methods for obtaining class labels for CT images include: obtaining them through diagnosis by experienced clinicians based on the patient's condition, and judging them based on other medical examination indicators of the patient.
[0032] Preferably, all CT images are resampled using voxels and processed using windowing techniques before segmentation and preprocessing.
[0033] Preferably, the dataset used for pre-training the segmentation network is a dataset with annotations of specific segmentation parts.
[0034] As a preferred option, the dataset used for network training contains only patient-level labels and does not contain specific lesion information.
[0035] Preferably, the time attention mechanism is one of LSTM, Transformer, or BERT.
[0036] Preferably, the 3D CNN is replaced by one of ResNeXt, I3D, or R(2+1)D networks.
[0037] Compared with the prior art, the present invention has the following advantages:
[0038] (1) This invention can be applied to various classification and recognition tasks based on medical CT images and has universality.
[0039] (2) This invention proposes a method for converting single-channel CT slices into three-channel images, which not only makes the processed images adaptable to the channel input requirements of most networks, but also improves the richness of category information of the processed CT images, which is beneficial to subsequent feature extraction and image classification.
[0040] (3) The present invention uses a deep learning classification method based on an improved 3D CNN, which can train the network without knowing the real labels of the slices in the CT image, and can alleviate the problem of insufficient medical image data with specific lesion information annotation to a certain extent.
[0041] (4) This invention uses a temporal attention network to replace the global average pooling layer in the original 3D network, which can learn the temporal distribution features between CT slices, making the obtained classification features more discriminative, thereby better promoting the classifier to move in the correct decision direction and improving the accuracy of classification.
[0042] (5) The noise correction network constructed in this invention can enhance and highlight lesion-related information to a great extent, while filtering out non-lesion-related interference features. This reduces the number of false positive samples to a certain extent. At the same time, through this noise correction network, the differences in CT image recognition in different datasets can be minimized, which helps to alleviate the multicenter effect problem and thus improve the accuracy and generalization of the model. Attached Figure Description
[0043] Figure 1 This is an overall flowchart of a robust BERT-based 3D CNN-based CT image classification method according to the present invention.
[0044] Figure 2 This is a schematic diagram of the structure of a robust BERT-based 3D CNN-based CT image classification method according to the present invention.
[0045] Figure 3 This is a schematic diagram of the BERT-based 3D CNN classification network of the present invention.
[0046] Figure 4 This is a schematic diagram of the noise correction network of the present invention. Detailed Implementation
[0047] The present invention will now be described in further detail with reference to specific examples and accompanying drawings.
[0048] like Figure 1 , Figure 2 As shown, this example focuses on the classification of COPD and non-COPD patients using chest CT images. A total of 1060 patient medical imaging data were collected from two different large public hospitals, including 557 COPD samples and 503 non-COPD samples. All imaging data only indicated the patient category, without specific lesion information. A proposed automatic CT image classification method based on a 3D CNN fusion temporal attention mechanism was used to classify COPD CT images, including the following steps:
[0049] Step S1: Read all relevant CT image information and lung function test report information of all subjects, generate patient-level labels for each subject based on the test reports, and establish the relationship between CT images and category labels.
[0050] Step S2: Preprocess the acquired raw CT images to obtain the processed dataset, and then divide the preprocessed CT image dataset into training and test sets using a 10x cross-validation method. The specific steps are as follows:
[0051] Step S21: Select several fully labeled data from the Kaggle and CNBC datasets and combine them into an independent dataset, and use it to pre-train a lung segmentation network based on U-Net;
[0052] Step S22: Apply the pre-trained U-Net segmentation network to segment each original CT image. A CT image consists of several single-channel CT slices; assume one of these slices is I... ori When applying the segmentation algorithm, the lung segmentation mask I can be obtained first. mask Subsequently, a lung segmentation mask was used to extract the corresponding lung regions from the original CT image. Specifically, the lung regions covered by the mask remained unchanged, while other regions were filled with zeros. The segmented image is denoted as I. seg ;
[0053] Step S23: Place I ori and the obtained I mask and I seg The original single-channel CT slices are overlaid and combined to form three-channel images. All CT slices are subjected to this operation.
[0054] Step S24: Resample the CT images to a uniform size, specifically: using segmentation mask I mask The center of the lung mask region is calculated, and the CT image is cropped around the center of the lung region with a uniform volume. The final size of each CT image is uniformly h*w*n, where h and w are the length and width of the slice, and n is the number of slices. In this embodiment, the final size of each CT image is 288*320*448.
[0055] Step S3: As Figure 3 As shown, the block input of CT images is constructed, and a BERT-based 3D CNN classification network is built. The specific steps are as follows:
[0056] Step S31: Define X = [x1, x2, ..., x n Let x be a CT image with n slices, where x i This represents a 2D CT slice. The CT image is divided into *s* disjoint 3D image blocks according to the slice order. Each 3D image block can be considered as a set containing an equal number of consecutive CT slices, i.e. In this embodiment, s = 16, so each 3D image block contains 28 consecutive CT slices;
[0057] Step S32: Construct a BERT-based 3D CNN classification network. First, a 3D CNN feedforward neural network is constructed. The difference is that the global average pooling layer in the general architecture is replaced with the temporal attention mechanism BERT. This utilizes BERT's temporal pooling to aggregate the correlations between CT scan slices and correctly learn the temporal features contained in the CT images, which aids in classification. Simultaneously, this network uses 3D convolutions to recognize CT images, requiring only the label of the entire CT image, without needing the annotation of individual CT slices.
[0058] Step S33: Transfer each 3D image block S i The data are fed into a BERT-based 3D CNN classification network to obtain the classification prediction probability P(c|S) for each 3D image patch. i ), where c represents the category, in this embodiment c∈{0,1}, c=0 represents a non-COPD patient, and c=1 represents a COPD patient. Then, by combining the predicted probabilities of all 3D image patches, the predicted value of the entire CT image can be obtained:
[0059] Step S34: Define y c If the patient's true label is for category c, then the classification loss can be calculated as:
[0060]
[0061] Step S4: As Figure 4 As shown, a noise correction network is constructed by adding a noise channel with unknown parameters to the real labels and using a neural network to learn the noise distribution parameters, thereby improving the model's recognition and generalization performance. The specific steps are as follows:
[0062] Step S41: When performing CT image classification, assume that the classification network cannot observe the correct label y for each slice, but can only obtain a noisy label z, and there is a hidden distribution relationship P (zc = i | yc = j, X) between the noisy label and the true label; define this distribution relationship as the noise transformation function. First calculate
[0063]
[0064] Where i,j∈{0,1} are the label values; It is a nonlinear mapping function; and For learnable parameters, It can be viewed as the transformation score between the true label i and the noise label j for category c.
[0065] Step S42: Calculate the estimated probability of the noise label based on the noise transformation function:
[0066]
[0067] Step S43: Calculate the noise classification loss for the entire CT image based on the noise probability estimated for each CT slice.
[0068]
[0069] Step S5: Calculate the final loss: L total =L cls +λL noisy , where λ is the loss balance parameter, and in this embodiment λ=0.0001.
[0070] The above embodiments are merely exemplary preferred embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.
[0071] The contents not described in detail in this specification are existing technologies known to those skilled in the art.
Claims
1. An automatic classification method for CT images based on a fusion temporal attention mechanism, characterized in that, The method specifically includes the following steps: Step S1: Collect medical imaging information related to suspected patients, including CT images, and establish the relationship between CT images and category labels; Step S2: Preprocess the acquired raw CT images to obtain the processed dataset, and divide the preprocessed CT image dataset into training and test sets; Step S3: Construct block inputs for CT images and build a 3D CNN classification network with a fused temporal attention mechanism. The specific steps are as follows: Step S31: Definition For having CT images of slices, among which This represents a 2D CT slice; the CT image is divided into segments according to the slice order. If there are disjoint 3D image patches, then each 3D image patch is considered as a set containing an equal number of consecutive CT slices, i.e. ; Step S32: Construct a 3D CNN classification network with fused temporal attention mechanism: First, a 3D CNN feedforward neural network is constructed, and then the global average pooling layer of the general architecture is replaced with a temporal attention network. Step S33: Extract each 3D image block The data are fed into a 3D CNN classification network with a fused temporal attention mechanism to obtain the classification prediction probability for each 3D image patch. ,in Represent the category; combine the predicted probabilities of all 3D image patches to obtain the predicted value for the entire CT image: ; Step S34: Definition For categories If the true label is obtained, the classification loss is calculated as follows: ； Step S4: Construct a noise correction network by adding a noise channel with unknown parameters to the real labels, and using a neural network to learn the noise distribution parameters to improve the model's recognition and generalization performance; the specific steps are as follows: Step S41: When performing CT image classification, it is assumed that the classification network cannot observe the correct label for each slice. However, only noisy labels can be obtained. There is a hidden distribution relationship between noise labels and real labels. Define this distribution relationship as a noise conversion function. First calculate : in For tag values; It is a nonlinear mapping function; and For learnable parameters, Consider it as the true label for category c. To noise label Conversion score between; Step S42: Calculate the estimated probability of the noise label based on the noise transformation function: Step S43: Calculate the noise classification loss for the entire CT image based on the noise probability estimated for each CT slice. Step S5: Calculate the final loss: ,in This is the loss balance parameter.
2. The automatic CT image classification method based on fusion temporal attention mechanism according to claim 1, characterized in that: in, The CT images mentioned are desensitized data processed using desensitization technology.
3. The automatic CT image classification method based on fusion temporal attention mechanism according to claim 1, characterized in that: The preprocessing of the acquired raw CT images specifically includes: Step S21: Given an independent dataset, use it to pre-train a U-Net segmentation network; Step S22: Apply a pre-trained U-Net network to segment each original CT image; a CT image consists of several single-channel CT slices, assuming one of the CT slices is... When applying the segmentation algorithm, the segmentation mask is first obtained. Then, a segmentation mask is used to extract the corresponding regions from the original CT image. That is, the area covered by the mask remains unchanged, while other areas are filled with 0. The segmented image is denoted as [image name missing]. ; Step S23: and what was obtained as well as The original single-channel CT slices are overlaid and combined to form three-channel images. All CT slices are subjected to this operation. Step S24: Resample the CT images to a uniform size, specifically: using a segmentation mask. The center of the mask region is calculated, and the CT images are cropped around the center of the region with a uniform volume, so that the final size of each CT image is uniform: ,in and For the length and width of the slice, The number of slices.
4. The automatic CT image classification method based on fusion temporal attention mechanism according to claim 1, characterized in that: Before segmenting and preprocessing the CT images, all CT images are resampled using voxels and processed using windowing techniques.
5. The automatic CT image classification method based on fusion temporal attention mechanism according to claim 1, characterized in that: The dataset used for pre-training the segmentation network is a dataset with annotations of specific segmentation parts.
6. The automatic CT image classification method based on fusion temporal attention mechanism according to claim 1, characterized in that: The dataset used for network training only contains patient-level labels and does not contain specific lesion information.
7. The automatic CT image classification method based on fusion temporal attention mechanism according to claim 1, characterized in that: in, The time attention mechanism mentioned is one of LSTM, Transformer, or BERT.
8. The automatic classification method for CT images based on a fusion temporal attention mechanism according to claim 1, characterized in that: The 3D CNN is replaced by one of the ResNeXt, I3D, or R(2+1)D networks.