Multi-task CT image classification method based on deep learning
Through deep learning technology, ResNet152 and Transformer encoder are used to perform multi-task CT image classification, which solves the problem of automatic 3D image classification and multiple conditions in the diagnosis of portal hypertension patients, and realizes efficient multi-label automatic diagnosis, reducing manpower Cost and diagnostic complexity.
Patent Information
- Application Number
- CN202411142137.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-12
AI Technical Summary
The existing technology lacks a deep learning-based automatic classification method for 3D images in the diagnosis of portal hypertension patients. Traditional imaging microscopy feature extraction is cumbersome and relies on human experience. The single-task model cannot complete the classification of multiple conditions at the same time, and the lesion positioning in 3D images difficulty.
Design a multi-task CT image classification method based on deep learning, using ResNet152 to extract features, combined with Transformer encoder and feedforward network, realize multi-label automatic classification of CT images with abdominal contrast enhancement, reduce the complexity of manual annotation and model training, and support End-to-end classification of multiple conditions.
It realizes multi-label automatic classification of 3D images, improves diagnostic efficiency and saves labor costs, and a model can complete multiple condition judgments at the same time, adapt to clinical needs, and has the potential to expand to other medical image modalities.
Smart Images

Figure CN118941876A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a multi-task CT image classification method based on deep learning. Background Art
[0002] In the past, most diagnostic models for conditions related to portal hypertension (such as significant portal hypertension (CSPH), risk of esophageal varicose vein rupture, etc.) were based on traditional radiomics, that is, machine learning classification was performed after extracting lesion-related radiomics features. The features extracted by traditional radiomics are manually set. Although they involve features such as shape, texture, and grayscale, due to experience limitations, some features are still not taken into account, such as location, which is limited by personal experience and requires data preprocessing, region of interest delineation, radiomics feature extraction, and machine learning model establishment. The process is relatively cumbersome. Delineating relevant regions of interest on medical images takes a long time and requires experts in related fields, and the cost of delineation is high.
[0003] Methods based on deep learning are data-driven and are not limited by experience. They can use big data to extract deeper and more hidden features. However, previous deep learning-based image classification methods have mostly focused on chest X-rays, fundus, skin and other parts, and there have been basically no studies on portal hypertension. In addition, previous deep learning-based image classification methods are mostly single-task, that is, a model can only complete the classification of one disease condition. To complete the classification of multiple diseases, multiple models need to be trained. In addition, previous deep learning-based image classification methods are mostly used for 2D images. Since lesions or related targets account for a small proportion in 3D images, it is difficult for the network to locate them. There is basically no research on automatic classification methods for 3D images. Summary of the invention
[0004] The purpose of the technical solution of the present invention is to design a model for end-to-end classification of various conditions of abdominal contrast-enhanced CT in patients with portal hypertension.
[0005] The technical solution of the present invention provides a multi-task CT image classification method based on deep learning, comprising the following steps:
[0006] Step 1: Obtain abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension, and perform normalized preprocessing to obtain preprocessed abdominal contrast-enhanced CT images;
[0007] Step 2: According to the pathological status, image-level annotation of the pre-processed abdominal contrast-enhanced CT is performed based on clinical guidelines to obtain annotated abdominal contrast-enhanced CT. The pathological status includes esophageal varices, gastric varices, portal vein thrombosis, and portosystemic shunt.
[0008] Step 3: Given the preprocessed abdominal contrast-enhanced CT image x∈R^(H×W×D), where H is the image height, W is the image width, and D is the image depth, extract the image features Z∈R by using the feature extractor set to ResNet152 H×W×D×c , h is the height of the output feature, w is the width of the output feature, d is the depth of the output feature, c is the number of channels of the output feature, and the output encoder feature is z i ∈R c , i ranges from [1, P], P = H × W × D;
[0009] Step 4: Establish a label set L = {L1, L2, L3, L4} for the labeled abdominal contrast-enhanced CT image corresponding to the preprocessed abdominal contrast-enhanced CT image, and embed the label to obtain the label feature l i ={l1,l2,l3,l4},l i ∈R c ;
[0010] The result of step 3 and step 4 is H = {z1,z2,…,z H×W×D ,l1,l2,l3,l4}, input to the Transformer encoder, obtain the dependency relationship between the abdominal contrast-enhanced CT image features and labels and model them;
[0011] The classifier is used to predict the final pathological state label of the modeled data, using an independent feedforward network FFN i The probability value corresponding to the final pathological state label is obtained, and a multi-label classification model is generated according to the modeled data, the final pathological state label and the probability value to realize multi-task CT image classification.
[0012] Preferably, the attention weight α in the Transformer encoder ij The calculation formula is as follows:
[0013]
[0014] Among them, W k is the key matrix, W q is the query matrix, d is the depth of the output feature, and h is the height of the output feature.
[0015] Preferably, the feed-forward network FFN i The formula is as follows:
[0016]
[0017] y i is the probability value. If the actual calculation result is greater than 0.5, the probability value output is 1, and if the calculation result is less than 0.5, the probability value output is 0.
[0018] The technical solution of the present invention proposes a multi-task CT image classification method based on deep learning, which is a 3D image multi-label automatic classification method, which is used for end-to-end classification of various conditions of abdominal contrast-enhanced CT in patients with portal hypertension, reducing the burden on radiologists and intelligent auxiliary diagnosis. The advantages of this method are as follows:
[0019] 1) Based on the imaging features of deep learning, there is no need to go through the lengthy process of traditional imaging, and there is no need to spend a lot of manpower on manual labeling. End-to-end image classification can improve efficiency and save labor costs.
[0020] 2) Regardless of whether it is a binary classification task or a multi-classification task, a model can only complete the judgment of one indicator. For a multi-label classification task, a model can be used to simultaneously complete the judgment of multiple indicators.
[0021] 3) Multi-label classification tasks, disease indicators can be added according to clinical needs in the future, and the clinical application prospects are huge.
[0022] 4) Transformer is the basic architecture of the large language model. Later, modalities such as pathology and report text can be added to build a large medical model. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A flowchart of a multi-task CT image classification method based on deep learning provided by an embodiment of the present invention;
[0024] Figure 2 An embodiment of the present invention provides a flowchart of generating esophageal varices, gastric varices, portal vein thrombosis and portosystemic shunt using a multi-task CT image classification method based on deep learning. DETAILED DESCRIPTION
[0025] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0026] The embodiment of the present invention provides a multi-label automatic classification system for abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension based on deep learning, which can perform end-to-end CT image classification and assist diagnosis. Figure 1 As shown, it includes two stages: 1) training, which is used to train the multi-label classification model. 2) diagnosis, which is used to automatically classify patients with portal hypertension and obtain results to assist in diagnosis.
[0027] In the above training phase, we first retrospectively obtained abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension, and then normalized and preprocessed them, resampling all CT images to a uniform Spacing. Experienced radiologists and gastroenterologists performed image-level annotation to determine the state of each CT image, such as whether there were esophageal varices, gastric varices, portal vein thrombosis, portosystemic shunts, etc. After obtaining the preprocessed CT images and the corresponding image-level labels, we performed network training.
[0028] In the above-mentioned diagnosis stage, contrast-enhanced CT images of the patient's abdomen are obtained and input into the classification network to output the results to assist clinical diagnosis.
[0029] 1) Image feature embedding.
[0030] For a given image x∈R H×W×D , H is the image height, W is the image width, and D is the image depth. The image features Z∈R are extracted by the feature extractor H×W×D×c , h is the height of the output feature, w is the width of the output feature, d is the depth of the output feature, and c is the number of channels of the output feature. The feature extractor is set to ResNet152. The final input Transformer encoder feature is z i ∈R c , i ranges from [1,P], P = H×W×D.
[0031] 2) Tag embedding.
[0032] For each given input image, there is a label set L = {L1, L2, L3, L4}. After label embedding, the label feature l is obtained. i ={l1,l2,l3,l4},l i ∈R c , representing the probability of the image belonging to each label under positive and negative results.
[0033] 3) Transformer training.
[0034] We use Transformer to model the interaction between image feature embedding and label embedding. Previous work has shown that Transformer is a powerful mechanism for capturing rich dependency information between variables. This paper can jointly input image feature embedding and labels into the Transformer encoder to learn the dependencies between all features and labels.
[0035] Let H = {z1,z2,…,z H×W×D ,l1,l2,l3,l4},H i ∈R cIt is input into the Transformer encoder as the joint feature of the image and the label. i ∈H and h j The weights between ∈H are learned through the self-attention mechanism, and the attention weight α ij The calculation method is as follows. After calculating the attention weights of all i and j, a nonlinear ReLU layer is used to transform each h i Update to h′ i .
[0036]
[0037] W k is the key matrix, W q is the query matrix, W v is the value matrix, W r and W o is the transformation matrix, b1 and b2 are bias matrices. This update process is repeated in N layers of the Transformer, and the weight matrices {W k ,W q ,W v ,W r ,W o} are not shared. The final output after the N-layer encoder is represented as H′={z′1,z′2,…,z′ H×W×D ,l′1,l′2,l′3,l′4}.
[0038] 4) Multi-label reasoning.
[0039] Finally, after modeling the image feature embedding and label embedding dependencies through the Transformer encoder, the classifier makes the final label prediction. i To obtain the probability value corresponding to the final label, the feedforward network can be expressed as follows:
[0040]
[0041] y i is the probability value, and the overall process is as follows Figure 2 shown.
Claims
1. A multi-task CT image classification method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension, and perform normalized preprocessing to obtain preprocessed abdominal contrast-enhanced CT images; Step 2: According to the pathological status, image-level annotation of the pre-processed abdominal contrast-enhanced CT is performed based on clinical guidelines to obtain annotated abdominal contrast-enhanced CT. The pathological status includes esophageal varices, gastric varices, portal vein thrombosis, and portosystemic shunt. Step 3: Given the preprocessed abdominal contrast-enhanced CT image x∈R^(H×W×D), where H is the image height, W is the image width, and D is the image depth, extract the image features Z∈R by using the feature extractor set to ResNet152 H×W×D×c , h is the height of the output feature, w is the width of the output feature, d is the depth of the output feature, c is the number of channels of the output feature, and the output encoder feature is z i ∈R c , i ranges from [1, P], P = H × W × D; Step 4: Establish a label set L = {L1, L2, L3, L4} for the labeled abdominal contrast-enhanced CT image corresponding to the preprocessed abdominal contrast-enhanced CT image, and embed the label to obtain the label feature l i ={l1,l2,l3,l4},l i ∈R c ; The result of step 3 and step 4 is H = {z1, z2, ..., z H×W×D , l1, l2, l3, l4}, input to the Transformer encoder, obtain the dependency relationship between the abdominal contrast-enhanced CT image features and labels and model them; The classifier is used to predict the final pathological state label of the modeled data, using an independent feedforward network FFN i The probability value corresponding to the final pathological state label is obtained, and a multi-label classification model is generated according to the modeled data, the final pathological state label and the probability value to realize multi-task CT image classification.
2. A multi-task CT image classification method based on deep learning as claimed in claim 1, characterized in that: The attention weight α in the Transformer encoder ij The calculation formula is as follows: Among them, W k is the key matrix, W q is the query matrix, d is the depth of the output feature, and h is the height of the output feature.
3. The multi-task CT image classification method based on deep learning according to claim 1, characterized in that: The feed-forward network FFN i The formula is as follows: y i is the probability value.
Citation Information
Patent Citations
Vision Transform-based multi-label pelvic organ prolapse detection method
CN118071717A
Chest medical image multi-label intelligent diagnosis algorithm based on multi-modal comparative learning
CN118136239A
