Multi-task CT image classification method based on deep learning

Through deep learning technology, ResNet152 and Transformer encoder are used to perform multi-task CT image classification, which solves the problem of automatic 3D image classification and multiple conditions in the diagnosis of portal hypertension patients, and realizes efficient multi-label automatic diagnosis, reducing manpower Cost and diagnostic complexity.

CN118941876APending Publication Date: 2024-11-12ZHONGSHAN HOSPITAL FUDAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411142137.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing technology lacks a deep learning-based automatic classification method for 3D images in the diagnosis of portal hypertension patients. Traditional imaging microscopy feature extraction is cumbersome and relies on human experience. The single-task model cannot complete the classification of multiple conditions at the same time, and the lesion positioning in 3D images difficulty.

Method used

Design a multi-task CT image classification method based on deep learning, using ResNet152 to extract features, combined with Transformer encoder and feedforward network, realize multi-label automatic classification of CT images with abdominal contrast enhancement, reduce the complexity of manual annotation and model training, and support End-to-end classification of multiple conditions.

Benefits of technology

It realizes multi-label automatic classification of 3D images, improves diagnostic efficiency and saves labor costs, and a model can complete multiple condition judgments at the same time, adapt to clinical needs, and has the potential to expand to other medical image modalities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941876A_ABST
    Figure CN118941876A_ABST
Patent Text Reader

Abstract

The invention provides a multi-task CT image classification method based on deep learning, which is used for carrying out end-to-end classification on various abdominal contrast enhanced CT conditions of patients with portal hypertension, reducing the burden of imaging department doctors and intelligently assisting diagnosis. The method is based on image omics characteristics of deep learning, does not need to experience a tedious process of traditional image omics, and does not need to spend a large amount of manpower for manual marking. The end-to-end image classification can improve the efficiency and save the labor cost. According to the method, only one index can be judged by one model regardless of a binary-classification task or a multi-classification task. According to the multi-label classification task, judgment of multiple indexes can be completed at the same time through one model. According to the method, a multi-label classification task is established, disease condition indexes can be added subsequently according to clinical requirements, and the clinical application prospect is huge. According to the method, based on the fact that Transform is a basic framework of a large language model, modalities such as pathology and report texts can be added subsequently, and a medical large model is constructed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-task CT image classification method based on deep learning. Background Art

[0002] In the past, most diagnostic models for conditions related to portal hypertension (such as significant portal hypertension (CSPH), risk of esophageal varicose vein rupture, etc.) were based on traditional radiomics, that is, machine learning classification was performed after extracting lesion-related radiomics features. The features extracted by traditional radiomics are manually set. Although they involve features such as shape, texture, and grayscale, due to experience limitations, some features are still not taken into account, such as location, which is limited by personal experience and requires data preprocessing, region of interest delineation, radiomics feature extraction, and machine learning model establishment. The process is relatively cumbersome. Delineating relevant regions of interest on medical images takes a long time and requires experts in related fields, and the cost of delineation is high.

[0003] Methods based on deep learning are data-driven and are not limited by experience. They can use big data to extract deeper and more hidden features. However, previous deep learning-based image classification methods have mostly focused on chest X-rays, fundus, skin and other parts, and there have been basically no studies on portal hypertension. In addition, previous deep learning-based image classification methods are mostly single-task, that is, a model can only complete the classification of one disease condition. To complete the classification of multiple diseases, multiple models need to be trained. In addition, previous deep learning-based image classification methods are mostly used for 2D images. Since lesions or related targets account for a small proportion in 3D images, it is difficult for the network to locate them. There is basically no research on automatic classification methods for 3D images. Summary of the invention

[0004] The purpose of the technical solution of the present invention is to design a model for end-to-end classification of various conditions of abdominal contrast-enhanced CT in patients with portal hypertension.

[0005] The technical solution of the present invention provides a multi-task CT image classification method based on deep learning, comprising the following steps:

[0006] Step 1: Obtain abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension, and perform normalized preprocessing to obtain preprocessed abdominal contrast-enhanced CT images;

[0007] Step 2: According to the pathological status, image-level annotation of the pre-processed abdominal contrast-enhanced CT is performed based on clinical guidelines to obtain annotated abdominal contrast-enhanced CT. The pathological status includes esophageal varices, gastric varices, portal vein thrombosis, and portosystemic shunt.

[0008] Step 3: Given the preprocessed abdominal contrast-enhanced CT image x∈R^(H×W×D), where H is the image height, W is the image width, and D is the image depth, extract the image features Z∈R by using the feature extractor set to ResNet152 H×W×D×c , h is the height of the output feature, w is the width of the output feature, d is the depth of the output feature, c is the number of channels of the output feature, and the output encoder feature is z i ∈R c , i ranges from [1, P], P = H × W × D;

[0009] Step 4: Establish a label set L = {L1, L2, L3, L4} for the labeled abdominal contrast-enhanced CT image corresponding to the preprocessed abdominal contrast-enhanced CT image, and embed the label to obtain the label feature l i ={l1,l2,l3,l4},l i ∈R c ;

[0010] The result of step 3 and step 4 is H = {z1,z2,…,z H×W×D ,l1,l2,l3,l4}, input to the Transformer encoder, obtain the dependency relationship between the abdominal contrast-enhanced CT image features and labels and model them;

[0011] The classifier is used to predict the final pathological state label of the modeled data, using an independent feedforward network FFN i The probability value corresponding to the final pathological state label is obtained, and a multi-label classification model is generated according to the modeled data, the final pathological state label and the probability value to realize multi-task CT image classification.

[0012] Preferably, the attention weight α in the Transformer encoder ij The calculation formula is as follows:

[0013]

[0014] Among them, W k is the key matrix, W q is the query matrix, d is the depth of the output feature, and h is the height of the output feature.

[0015] Preferably, the feed-forward network FFN i The formula is as follows:

[0016]

[0017] y i is the probability value. If the actual calculation result is greater than 0.5, the probability value output is 1, and if the calculation result is less than 0.5, the probability value output is 0.

[0018] The technical solution of the present invention proposes a multi-task CT image classification method based on deep learning, which is a 3D image multi-label automatic classification method, which is used for end-to-end classification of various conditions of abdominal contrast-enhanced CT in patients with portal hypertension, reducing the burden on radiologists and intelligent auxiliary diagnosis. The advantages of this method are as follows:

[0019] 1) Based on the imaging features of deep learning, there is no need to go through the lengthy process of traditional imaging, and there is no need to spend a lot of manpower on manual labeling. End-to-end image classification can improve efficiency and save labor costs.

[0020] 2) Regardless of whether it is a binary classification task or a multi-classification task, a model can only complete the judgment of one indicator. For a multi-label classification task, a model can be used to simultaneously complete the judgment of multiple indicators.

[0021] 3) Multi-label classification tasks, disease indicators can be added according to clinical needs in the future, and the clinical application prospects are huge.

[0022] 4) Transformer is the basic architecture of the large language model. Later, modalities such as pathology and report text can be added to build a large medical model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a multi-task CT image classification method based on deep learning provided by an embodiment of the present invention;

[0024] Figure 2 An embodiment of the present invention provides a flowchart of generating esophageal varices, gastric varices, portal vein thrombosis and portosystemic shunt using a multi-task CT image classification method based on deep learning. DETAILED DESCRIPTION

[0025] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0026] The embodiment of the present invention provides a multi-label automatic classification system for abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension based on deep learning, which can perform end-to-end CT image classification and assist diagnosis. Figure 1 As shown, it includes two stages: 1) training, which is used to train the multi-label classification model. 2) diagnosis, which is used to automatically classify patients with portal hypertension and obtain results to assist in diagnosis.

[0027] In the above training phase, we first retrospectively obtained abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension, and then normalized and preprocessed them, resampling all CT images to a uniform Spacing. Experienced radiologists and gastroenterologists performed image-level annotation to determine the state of each CT image, such as whether there were esophageal varices, gastric varices, portal vein thrombosis, portosystemic shunts, etc. After obtaining the preprocessed CT images and the corresponding image-level labels, we performed network training.

[0028] In the above-mentioned diagnosis stage, contrast-enhanced CT images of the patient's abdomen are obtained and input into the classification network to output the results to assist clinical diagnosis.

[0029] 1) Image feature embedding.

[0030] For a given image x∈R H×W×D , H is the image height, W is the image width, and D is the image depth. The image features Z∈R are extracted by the feature extractor H×W×D×c , h is the height of the output feature, w is the width of the output feature, d is the depth of the output feature, and c is the number of channels of the output feature. The feature extractor is set to ResNet152. The final input Transformer encoder feature is z i ∈R c , i ranges from [1,P], P = H×W×D.

[0031] 2) Tag embedding.

[0032] For each given input image, there is a label set L = {L1, L2, L3, L4}. After label embedding, the label feature l is obtained. i ={l1,l2,l3,l4},l i ∈R c , representing the probability of the image belonging to each label under positive and negative results.

[0033] 3) Transformer training.

[0034] We use Transformer to model the interaction between image feature embedding and label embedding. Previous work has shown that Transformer is a powerful mechanism for capturing rich dependency information between variables. This paper can jointly input image feature embedding and labels into the Transformer encoder to learn the dependencies between all features and labels.

[0035] Let H = {z1,z2,…,z H×W×D ,l1,l2,l3,l4},H i ∈R cIt is input into the Transformer encoder as the joint feature of the image and the label. i ∈H and h j The weights between ∈H are learned through the self-attention mechanism, and the attention weight α ij The calculation method is as follows. After calculating the attention weights of all i and j, a nonlinear ReLU layer is used to transform each h i Update to h′ i .

[0036]

[0037] W k is the key matrix, W q is the query matrix, W v is the value matrix, W r and W o is the transformation matrix, b1 and b2 are bias matrices. This update process is repeated in N layers of the Transformer, and the weight matrices {W k ,W q ,W v ,W r ,W o} are not shared. The final output after the N-layer encoder is represented as H′={z′1,z′2,…,z′ H×W×D ,l′1,l′2,l′3,l′4}.

[0038] 4) Multi-label reasoning.

[0039] Finally, after modeling the image feature embedding and label embedding dependencies through the Transformer encoder, the classifier makes the final label prediction. i To obtain the probability value corresponding to the final label, the feedforward network can be expressed as follows:

[0040]

[0041] y i is the probability value, and the overall process is as follows Figure 2 shown.

Claims

1. A multi-task CT image classification method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain abdominal contrast-enhanced CT images of patients with cirrhosis and portal hypertension, and perform normalized preprocessing to obtain preprocessed abdominal contrast-enhanced CT images; Step 2: According to the pathological status, image-level annotation of the pre-processed abdominal contrast-enhanced CT is performed based on clinical guidelines to obtain annotated abdominal contrast-enhanced CT. The pathological status includes esophageal varices, gastric varices, portal vein thrombosis, and portosystemic shunt. Step 3: Given the preprocessed abdominal contrast-enhanced CT image x∈R^(H×W×D), where H is the image height, W is the image width, and D is the image depth, extract the image features Z∈R by using the feature extractor set to ResNet152 H×W×D×c , h is the height of the output feature, w is the width of the output feature, d is the depth of the output feature, c is the number of channels of the output feature, and the output encoder feature is z i ∈R c , i ranges from [1, P], P = H × W × D; Step 4: Establish a label set L = {L1, L2, L3, L4} for the labeled abdominal contrast-enhanced CT image corresponding to the preprocessed abdominal contrast-enhanced CT image, and embed the label to obtain the label feature l i ={l1,l2,l3,l4},l i ∈R c ; The result of step 3 and step 4 is H = {z1, z2, ..., z H×W×D , l1, l2, l3, l4}, input to the Transformer encoder, obtain the dependency relationship between the abdominal contrast-enhanced CT image features and labels and model them; The classifier is used to predict the final pathological state label of the modeled data, using an independent feedforward network FFN i The probability value corresponding to the final pathological state label is obtained, and a multi-label classification model is generated according to the modeled data, the final pathological state label and the probability value to realize multi-task CT image classification.

2. A multi-task CT image classification method based on deep learning as claimed in claim 1, characterized in that: The attention weight α in the Transformer encoder ij The calculation formula is as follows: Among them, W k is the key matrix, W q is the query matrix, d is the depth of the output feature, and h is the height of the output feature.

3. The multi-task CT image classification method based on deep learning according to claim 1, characterized in that: The feed-forward network FFN i The formula is as follows: y i is the probability value.

Citation Information

Patent Citations

  • Vision Transform-based multi-label pelvic organ prolapse detection method

    CN118071717A

  • Chest medical image multi-label intelligent diagnosis algorithm based on multi-modal comparative learning

    CN118136239A