A four-phase CT image classification method based on multidimensional feature interaction

Abstract

The present invention discloses a four-phase CT image classification method based on multi-dimensional feature interaction. The method first collects four-phase CT images, performs preprocessing and annotation, and constructs a training set. Secondly, a feature extraction network is constructed to extract features from the four-phase CT images respectively to obtain a one-dimensional feature representation of each phase. Then, a spatiotemporal prediction module is constructed to obtain a prediction result P1 based on the one-dimensional feature representation of each phase; the one-dimensional features of each phase are passed through an average pooling layer and a classifier to obtain a classification result of a single phase, and the classification results of the four single phases are added and averaged to obtain an average prediction result P2. Finally, P1 and P2 are weighted averaged to output the classification result, and training is performed. The present invention improves the accuracy of CT image recognition and can better adapt to different data and scenarios. It injects new ideas and methods into the development of medical imaging technology.

Description

Technical Field

[0001] The present invention belongs to the fields of medical image processing, computer vision and medical image analysis, and more specifically relates to a classification method for four-phase CT images based on multi-dimensional feature interaction. Background Art

[0002] In the field of medical imaging, computed tomography (CT) images are widely used for disease diagnosis and treatment monitoring. However, traditional lesion recognition methods in CT images often face numerous challenges. For example, lesions may appear differently between different phases, making identification difficult. Furthermore, CT images are highly complex and noisy, making it difficult for traditional feature extraction and classification methods to fully utilize their information.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image processing. However, relatively little research has focused on lesion recognition and classification in multi-phase CT images, particularly on the interaction between features in different phases. Existing studies have mostly focused on lesion recognition in a single phase, neglecting the correlation between different phases and the importance of spatiotemporal information for lesion recognition. Therefore, it is necessary to develop new methods to overcome these challenges and improve the accuracy and robustness of multi-phase CT image classification. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a four-phase CT image classification method based on multi-dimensional feature interaction, which considers the information exchange between phases and improves the accuracy of CT image classification.

[0005] A four-phase CT image classification method based on multi-dimensional feature interaction specifically includes the following steps:

[0006] Step 1: Collect four-phase CT images including plain phase, arterial phase, portal venous phase and delayed phase, preprocess and label the sample labels, use channel expansion technology to expand these images into three-channel pseudo-color images, and mark the type as the sample label to construct a training set.

[0007] Preferably, the original four-phase CT images are windowed using two different combinations of window width (WW) and window level (WL) parameters, resulting in two different CT images. These two windowed CT images are then fused with the original four-phase CT images in the channel dimension to form a three-channel pseudo-color image that is added to the training set.

[0008] Preferably, data augmentation operations are performed on the original four-phase CT images, including but not limited to random transformation or expansion. These augmentation operations can generate new images that are different from the original images, and then these new images are added to the training set to expand the size of the training set.

[0009] Step 2: Build a feature extraction network based on ResNet18, extract features from the four-phase CT images respectively, and obtain a one-dimensional feature representation of each phase.

[0010] Four feature extraction networks are used to extract features from four-phase CT images. Each feature extraction network is responsible for processing the image data of a single phase to obtain the feature representation of each phase.

[0011] The feature extraction network is specifically as follows: In the ResNet18 network, each block outputs a two-dimensional feature image. In order to achieve the interactive weighting of channel-spatial information between the two-dimensional features of adjacent phases, a CBAM (Channel Attention and Spatial Attention) module is introduced between each convolution block of ResNet18. The CBAM module is used to extract the channel-spatial weights of the two-dimensional features of adjacent phases. Subsequently, the channel attention weights and spatial attention weights extracted by the CBAM module are applied to the two-dimensional feature map corresponding to this phase, achieving interactive weighting of channel-spatial information and performing feature extraction on the four-phase CT image, thereby better characterizing the information association between adjacent phases. The convolution block is composed of four convolutional layers.

[0012] Step 3: Build a spatiotemporal prediction module and obtain the prediction result P1 based on the one-dimensional feature representation of each phase;

[0013] The one-dimensional features of each phase are concatenated along the feature dimension to form a one-dimensional feature vector that integrates information from multiple phases. This integrated one-dimensional feature vector is then fed into a one-dimensional deep residual convolutional network for processing. The one-dimensional deep residual convolutional network effectively learns and extracts features, further enhancing the ability to represent features. Finally, a gated recurrent unit (GRU) is used to perform a preliminary prediction, yielding the prediction result P1.

[0014] Step 4: The one-dimensional features of each phase are passed through the average pooling layer and the classifier to obtain the classification result of the single phase. The classification results of the four single phases are added and averaged to obtain the average prediction result P2.

[0015] Step 5: Take the weighted average of P1 and P2 and output the classification result. Use the multi-task learning framework to train the network. The spatiotemporal prediction module predicts the four-phase CT image to obtain P1 as the main task, and the auxiliary task is the average result P2 obtained from the four phases. The λ factor is used to adjust the importance between the two tasks. The network parameters are optimized through the weighted sum of the loss function. The loss function of the multi-task framework can be expressed as:

[0016] L=λ*L main +(1-λ)*L aux

[0017] Where L represents the overall loss function, L main Represents the loss function of the main task, L aux represents the loss function of the auxiliary task; λ represents the weight factor, which is greater than 0.5 and less than or equal to 1, and is used to balance the importance of the two tasks.

[0018] The present invention has the following beneficial effects:

[0019] This invention utilizes ResNet18 to extract features from CT images. By introducing a multi-phase information fusion module and a spatiotemporal prediction module, it can fully leverage the information interaction and spatiotemporal relationships between different phases, significantly improving the accuracy of CT image classification and recognition. Secondly, a multi-task learning framework is used to comprehensively optimize the network, improving the robustness of the system and enabling better adaptation to diverse data and scenarios. Most importantly, this invention has injected new ideas and methods into the development of medical imaging technology, promoting progress in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a network structure diagram of the present invention;

[0021] Figure 2 This is a schematic diagram of CT image channel expansion;

[0022] Figure 3 Flowchart for training the lesion recognition network;

[0023] Figure 4 This is a schematic diagram of the multi-phase information fusion module;

[0024] Figure 5 Schematic diagram of the spatiotemporal prediction module. DETAILED DESCRIPTION

[0025] The present invention will be further explained below with reference to the accompanying drawings.

[0026] A four-phase CT image classification method based on multi-dimensional feature interaction is only used to assist doctors in determining the type of multi-phase CT images corresponding to liver cancer patients, such as Figure 1As shown, the specific steps include:

[0027] Step 1: Collect four-phase liver cancer CT image data and create corresponding classification labels.

[0028] S1.1. We collected four-phase CT images of patients with liver cancer, including plain, arterial, portal venous, and delayed phases. Using data desensitization techniques, we removed sensitive personal information. Ultimately, we obtained 398 liver CT images from patients with hepatocellular carcinoma (HCC) and intrahepatic cholangiocarcinoma (ICC).

[0029] s1.2. Use ITK-SNAP software to annotate CT images, forming 3D bounding boxes and obtaining voxel-level segmentation labels. CT images with HCC are labeled as 0, and CT images with ICC are labeled as 1. A total of 202 CT images with HCC types and 196 CT images with ICC types were annotated.

[0030] s1.3. Resample the voxels in the CT images to a uniform size of 1 mm to eliminate differences between different CT scans. Then, use a 3D bounding box to extract the pixels within the lesion and its surrounding 10 mm area. The 3D image resolution of the lesion was adjusted to 64*128*128, where 64 represents the number of slices in each phase of the liver CT image and 128 represents the length and width of each liver CT image.

[0031] s1.4, such as Figure 2 As shown in FIG, two different combinations of WW (window width) and WL (window level) are used to perform windowing on the CT image, resulting in two different CT images. The two windowed images are then fused in the channel dimension to form a three-channel CT image.

[0032] Step 2: Build a feature extraction network, such as Figure 3 shown.

[0033] s2.1 builds ResNet18 as the basic feature extraction network. The network structure is shown in Table 1:

[0034] Table 1. ResNet18 network structure

[0035]

[0036] Among them, the output dimension of the fully connected layer is 2, which is the number of target categories for classification.

[0037] s2.2. The 5D (B, T, C, H, W) CT image is resized to a 4D (B × T, C, H, W) image and fed into the ResNet18 network for feature extraction. Each residual block obtains a two-dimensional feature map of the slice at this phase, where B is the batch size, T is the slice dimension, C is the channel dimension, and H and W are the height and width of the image.

[0038] The feature extraction network includes a multi-phase information fusion framework, such as Figure 4 The details are as follows (taking the NC and ART phases as examples):

[0039] First, the CBAM module is used to extract the channel and spatial dependencies in the two-dimensional feature map obtained in step 2.

[0040] Next, the channel-spatial weights of the NC and PV phase image features are applied to the ART phase image features. Similarly, the channel-spatial weights of the ART and DL phase image features are applied to the NC phase image features. It is worth emphasizing that this weight calculation and channel-spatial information interaction weighting are not limited to a single block in the ResNet18 network; rather, the weight calculation and channel-spatial information interaction weighting are performed on the output features of all four blocks. This can be expressed as the following formula:

[0041]

[0042] in The two-dimensional features of NC and ART phases extracted from each layer of ResNet18, and are the channel attention weights and spatiotemporal attention weights of the ART phase, and They are the channel attention weight and spatiotemporal attention weight of the NC phase, and So too, Represents the matrix multiplication operation between feature maps. This interactive perception mechanism is not only applied to the NC and ART phases, but also involves four phases, in which channel-spatial information interaction weighting exists between adjacent phases.

[0043] Step 3: Construct a spatiotemporal prediction module, such as Figure 5 The specific steps are as follows:

[0044] S3.1. Concatenate the one-dimensional features of each phase extracted by ResNet18 in the feature dimension to form a one-dimensional feature vector that integrates multi-phase information.

[0045] S3.2. Next, the integrated one-dimensional feature vector (B×T, C) is converted to (B, T, C) and sent to the one-dimensional residual block based on the deep residual network for processing, which effectively learns and extracts features, further enhancing the feature representation ability.

[0046] S3.3. The one-dimensional features in step S4.2 are fed into the GRU architecture to achieve preliminary prediction of the four-phase CT image and obtain the prediction result P1.

[0047] Step 4: The one-dimensional features of each phase are passed through the average pooling layer and the classifier to obtain the classification result of the single phase. The classification results of the four single phases are added and averaged to obtain the average prediction result P2.

[0048] The classifier consists of a fully connected layer and a softmax function.

[0049] Step 5: Build a multi-task learning framework. The spatiotemporal prediction module predicts the four-phase CT images to obtain P1 as the main task, and the auxiliary task is the average result P2 obtained from the four phases. The λ factor is used to adjust the importance between the two tasks. The loss function is:

[0050]

[0051] Where E(x,y) is the cross entropy loss, λ is the weight balance factor for multi-task learning, and is greater than 0.5 and less than or equal to 1, X f is the output of the prediction module, X n 、X a 、X p 、X d Represent the outputs of the four phase unimodal networks respectively.

[0052] The model demonstrated significant performance on the four-phase CT dataset, achieving an accuracy of 85.5% and an area under the curve (AUC) of 89.73 in the test cohort, outperforming several 3D models. These results demonstrate that the proposed method offers superior CT image classification performance and demonstrates high reliability in distinguishing between the two types of liver cancer in CT images.

Claims

1. A four-phase CT image classification method based on multi-dimensional feature interaction, characterized in that: The following steps are involved: Step 1: Collect four-phase CT images including plain phase, arterial phase, portal venous phase, and delayed phase, perform preprocessing and label the samples to construct a training set; Step 2: Build a feature extraction network based on ResNet18 to extract features from the four-phase CT images and obtain a one-dimensional feature representation for each phase; The feature extraction network is as follows: a channel-spatial attention module (CBAM) is introduced between each convolutional block of ResNet18, and the channel attention weight and spatial attention weight extracted by the CBAM module are applied to the two-dimensional feature map corresponding to this phase, thereby realizing the interactive weighting of channel-spatial information and performing feature extraction on the four-phase CT image; The convolution block consists of four convolutional layers; Step 3: Build a spatiotemporal prediction module and obtain the prediction result P1 based on the one-dimensional feature representation of each phase. The specific process is as follows: The one-dimensional features of each phase are spliced ​​in the feature dimension to form a one-dimensional feature vector that integrates the information of the four phases; the integrated one-dimensional feature vector is then sent to a one-dimensional deep residual convolutional network for processing; finally, a preliminary prediction is made using the gated recurrent unit (GRU) to obtain the prediction result P1; Step 4: The one-dimensional features of each phase are passed through the average pooling layer and the classifier to obtain the classification result of the single phase. The classification results of the four single phases are added and averaged to obtain the average prediction result P2; Step 5: Perform weighted average of P1 and P2 to output the classification result, and use the multi-task learning framework for training. The training is as follows: The spatiotemporal prediction module predicts the four-phase CT image to obtain P1 as the main task, and the auxiliary task is the average result P2 obtained from the four phases. The loss function of the multi-task framework is expressed as: L=λ*L main +(1-λ)*L aux Where L represents the overall loss function, L main Represents the loss function of the main task, L aux represents the loss function of the auxiliary task; λ represents the weight factor, which is greater than 0.5 and less than or equal to 1.

2. The four-phase CT image classification method based on multidimensional feature interaction according to claim 1 is characterized in that: The pretreatment in step 1 is as follows: The original four-phase CT image was windowed using two different combinations of window width WW and window level WL parameters to obtain two different CT images. The two windowed CT images were then fused with the original four-phase CT image in the channel dimension to form a three-channel pseudo-color image that was added to the training set. Data augmentation operations are performed on the original four-phase CT images, including random transformation or expansion, and then the new CT images are added to the training set.

3. The four-phase CT image classification method based on multidimensional feature interaction according to claim 1 is characterized in that: The classifier described in step 4 is composed of a fully connected layer and a softmax function.

Images