Lung CT image segmentation method based on global-local feature correlation fusion

Abstract

The present invention discloses a global-local feature association fusion lung CT image segmentation method, which mainly solves the problem of inaccurate segmentation of infected areas in lung CT images in the existing technology. Its implementation scheme is: establish an encoder composed of Transformer layers; establish an encoder composed of convolutional neural networks; establish a local feature association fusion module composed of Transformer layers; establish a decoder composed of convolutional neural networks and symmetrical with the second encoder; connect the first encoder and the second encoder in parallel, and then cascade with the local feature association fusion module and the decoder to form a segmentation network model; input the training set data into the two encoders in parallel, and iteratively train the segmentation network model using the backpropagation method; input the test image into the trained segmentation network to segment the infected area of ​​the lung CT image. The present invention has high segmentation accuracy and low resource consumption, and can be used for the recognition of CT images.

Description

Technical Field

[0001] The present invention belongs to the field of bioinformatics technology, and particularly relates to a lung CT image segmentation method, which can be used to segment the infected area of ​​the lungs of infected patients, and can also be used to accurately determine whether the patient is infected and judge the severity of the disease. Background Art

[0002] Image segmentation is the first step in analyzing a given image data. It is an important part of image understanding and one of the most difficult problems in image processing. It plays an important role in computer vision. Image segmentation refers to dividing an image into several non-overlapping regions based on grayscale, color, spatial texture, and geometric shape, so that these features show consistency or similarity within the same region and then show obvious differences between different regions. Simply put, it is to separate the target from the background in an image and label each pixel in the image, where pixels with the same label have the same characteristics. The segmentation task of patient computed tomography (CT) images plays an important role in the medical field. For lung CT images of patients with viral pneumonia, the lesions in their CT images have complex and variable shapes, large scale spans, blurred lesion boundaries, and are easily confused with background tissues. In addition, due to the difficulty in labeling medical imaging data represented by some viral pneumonia and the small number of samples with high-quality labels, these problems need to be solved. More and more researchers have proposed various segmentation methods to obtain models with superior segmentation performance.

[0003] In recent years, computer vision-based segmentation methods have primarily utilized deep learning to construct deep networks for medical CT image segmentation. These deep learning-based methods can be broadly divided into two categories. The first category involves deep segmentation networks based on convolutional neural networks. Convolutional neural networks apply convolutional layers to the input image, effectively applying the same filter repeatedly to various parts of the image. This generates feature maps representing the intensity of specific locations within the image. These deep feature maps are then repeatedly upsampled to their original pixel size and compared with the true labels to complete the segmentation task. The second category builds on convolutional neural networks with an attention mechanism. This mechanism selectively focuses on the more important feature map information in the segmentation task while ignoring less important information. This mechanism achieves superior performance in both feature map extraction and restoration. The third category applies the Transformer concept to image segmentation. Leveraging its unique network structure and self-attention mechanism, these methods have achieved impressive performance. These three approaches are not independent of each other; their combination and complementarity often lead to superior performance.

[0004] Patent publication number CN115457021A discloses a skin disease image segmentation method based on a joint attention convolutional neural network. This method combines a convolutional neural network, a spatial attention module, and a channel attention module to construct a segmentation network. This method integrates multi-scale feature information to achieve good segmentation results.

[0005] Patent publication number CN111915612A discloses an image segmentation method based on a multi-encoder convolutional neural network. It utilizes multiple encoders to extract image feature information. Different encoders have different parameter settings for their convolutional and pooling layers. The multiple encoders extract features of different dimensions, which are then combined and fed into a decoder to restore the image's original resolution, resulting in the final segmentation result.

[0006] Since the above two methods do not consider the correlation between global features and local features, the extraction of image feature information is not comprehensive enough, resulting in insufficient feature information when the decoder restores the original resolution of the image, affecting the model segmentation accuracy. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned existing technologies and propose a lung CT image segmentation method based on global-local feature association fusion, so as to obtain stronger feature representation and structural information of lung CT images through network modeling, expand the amount of feature information, and improve the segmentation accuracy of the model by obtaining and integrating the feature information of the image from both global and local scales.

[0008] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0009] (1) Download CT scan images of pneumonia patients from two public datasets to obtain training and test sets respectively;

[0010] (2) The training set and test set are preprocessed by image enhancement, pixel value normalization, dimension conversion and segmentation mask;

[0011] (3) Construct a segmentation model based on global-local feature correlation fusion:

[0012] (3a) Build an encoder consisting of N Transformer layers, where N is greater than or equal to 1;

[0013] (3b) Building an encoder consisting of a convolutional neural network;

[0014] (3c) Establishing a local feature correlation fusion module composed of Transformer layers to extract local correlation features;

[0015] (3d) Establish a decoder composed of a convolutional neural network and symmetrical to (3b) to restore the image resolution and output the segmentation result;

[0016] (3e) Connecting the first encoder and the second encoder in parallel, and then cascading them with the local feature association fusion module and the decoder to form a segmentation network model, and using the focalloss function as the loss function of the network model;

[0017] (4) The training set data is input into two encoders in parallel, and the segmentation network model is iteratively trained using the back-propagation method until the loss function converges, thereby obtaining the entire segmentation network after training;

[0018] (5) Predicting infected areas in lung CT images based on the trained segmentation model;

[0019] 5a) Input the test set into the trained segmentation model to obtain the predicted probability value p for each sample pixel;

[0020] 5b) Set the probability threshold d in the range of 0.3-0.6, compare the predicted probability value with the probability threshold, and obtain the infected area segmentation results of the lung CT images of the test set;

[0021] If p is greater than d, it indicates an infected area.

[0022] If p is less than or equal to d, it indicates an uninfected area.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] 1) The present invention constructs a segmentation network model, and uses two decoders therein to extract global image features and local edge features respectively, and uses the local feature association module therein to extract local association features with global properties. The advantages of these three modules are integrated to obtain a stronger image feature representation and expand the amount of feature information.

[0025] 2) Since the present invention migrates the pre-trained model of a large image vision library, that is, the encoder composed of a convolutional neural network, it facilitates the training of data sets with smaller data sizes and improves the accuracy of the segmentation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an implementation flow chart of the present invention;

[0027] Figure 2 It is a structural diagram of the segmentation model constructed by the present invention;

[0028] Figure 3 This is a visualization diagram of the prediction results of the test set using the present invention. DETAILED DESCRIPTION

[0029] The specific embodiments and effects of the present invention are further described in detail below with reference to the accompanying drawings.

[0030] This invention is a lung CT image segmentation method based on the COVID-19 dataset. Computed tomography (CT) scans the lungs using precise X-rays, producing clear, visible images of affected lung areas. However, CT scans require professional diagnosis by radiologists. When widespread disease outbreaks occur and human resources are insufficient, computer-aided diagnosis (CAD) provides a significant boost, using computational methods to segment and identify large numbers of patient CT images. This invention is a method for segmenting lung infection areas during CAD diagnosis.

[0031] Reference Figure 1 , the implementation steps of this example are as follows:

[0032] In step 1, data will be collected from datasets from public sources and preprocessed.

[0033] 1.1) Collect CT scan images of patients with different pneumonia infections from the COVID-19-CT-Seg and MosMed datasets, respectively, to obtain two datasets D and S. All images in D are used as the training set, and all images in S are used as the independent test set. In this example, a total of 2049 CT scan images and their corresponding true annotated segmentation masks are collected in dataset D, and a total of 3520 CT scan images and their corresponding segmentation masks are collected in dataset S;

[0034] 1.2) All CT image data are preprocessed as follows to obtain two preprocessed data sets D1 and S1:

[0035] 1.2.1) Normalize the pixel values ​​of all images to their maximum and minimum values ​​to prevent pixels from being too large or too small. The formula for normalization is:

[0036]

[0037] Among them, x min Indicates the maximum pixel value in an image, x max Indicates the minimum pixel value;

[0038] 1.2.2) Perform image enhancement processing by sequential translation, scaling, horizontal and vertical rotation on the training and test sets using the CV2 software package;

[0039] 1.2.3) Resize all images to 224×224 pixels.

[0040] 1.2.4) Mark the segmentation masks of all CT images. Mark all findings related to pneumonia infection as white in the segmentation mask and treat them as '1' pixels. Mark all areas of the segmentation mask except the lesion area as black and treat them as '0' pixels. The segmentation mask is processed into a one-hot encoding format.

[0041] Step 2: Construct a segmentation network that associates and fuses global and local features.

[0042] Reference Figure 2 , this step is implemented as follows:

[0043] 2.1) Construct a Transformer encoder consisting of 12 Transformer layers to extract global features from the image:

[0044] The input image dimension is processed into (3, 224, 224), and then the input image is evenly divided into multiple small blocks Pathes, and Pathes is set to 16;

[0045] Multiple small blocks are concatenated into a long vector L1, and then the position code is added to form the input vector;

[0046] Input the input vector to the encoder for encoding, and obtain the output vector Q1, whose dimension is (196, 768);

[0047] 2.2) Construct an encoder composed of a convolutional neural network to extract local edge features from the image:

[0048] The ResNet18 model pre-trained on ImageNet, a large visual object recognition image library, is selected as the encoder. ResNet18 has 18 layers, divided into 4 blocks. Each block consists of multiple layers of 3=3 convolutional layers, 1×1 convolutional layers, and pooling layers.

[0049] 2.3) Construct a local feature association fusion module to extract local association features from images:

[0050] (2.3.1) Take the hidden layer vectors of the first three modules of the convolutional neural network encoder and split them into small blocks of different sizes using the Pathes method. Set the Pathes size to 7, and split the three hidden layer vectors into 64, 16, and 4 blocks respectively. Then, concatenate these three blocks to obtain a long vector L2.

[0051] (2.3.2) Input the long vector L2 into a Transformer layer to obtain the output vector;

[0052] (2.3.3) Split the output vector in the ratio of 64:16:4 to obtain the local correlation feature vectors Q2, Q3, Q4

[0053] 2.4) Construct a decoder composed of a convolutional neural network and symmetrical with the encoder in 2.2) to restore the image resolution and output the segmentation result:

[0054] (2.4.1) The encoder output vector Q1 is used as the decoder input layer vector, and the local correlation fusion module output vectors Q2, Q3, and Q4 are used as the concatenation vector;

[0055] (2.4.2) Upsample and convolve the Q1 vector, allowing it to propagate forward from bottom to top. Each time it passes through a network layer, it is concatenated with Q2, Q3, and Q4 in the channel dimension to restore the image resolution.

[0056] (2.4.3) Repeat (2.4.2) until the forward propagation is completed and the segmentation result of the original resolution is obtained.

[0057] 2.5) The Transformer encoder and the convolutional neural network encoder are connected in parallel, and then cascaded with the local feature association fusion module and the decoder to form the entire segmentation network.

[0058] Step 3: Selection of loss function and segmentation network training.

[0059] 3.1) Select the loss function of the segmentation network:

[0060] The 0 and 1 pixels in the input image labels are considered as a binary classification problem. By visualizing the label data, it is found that the number of 1s in the label image, i.e., the infected area of ​​pneumonia, is much less than the number of 0s, i.e., the background area. Some CT images are even located at the edge of the scan and are all 0 images. It can be seen that the ratio of positive and negative samples is very unbalanced. The traditional positive and negative sample imbalance problem can be solved by data sampling, but it is not suitable for the entire image. For this reason, this step uses the existing focalloss function as the loss function of the segmentation network to solve the problem of pixel imbalance in the image. The formula is as follows:

[0061]

[0062] Where y is the sample value, p is the probability of judging y=1, and γ>0 is an adjustable factor. Focalloss reduces the loss for samples that are accurately classified, while the loss remains unchanged for samples that are incorrectly classified. It gives higher weight to samples that are more difficult to classify.

[0063] 3.2) Segmentation network training:

[0064] 3.2.1) Input the training samples in parallel to the encoder in the segmentation network, and obtain the probability value matrix of the pixel value binary classification through the decoder calculate The loss function value of the segmentation mask y corresponding to the training sample

[0065]

[0066] Among them, γ>0 is an adjustable factor;

[0067] 3.2.2) Set the learning rate to 0.001 and the total number of iterations to 15;

[0068] 3.2.3) Use the Adam optimization algorithm to backpropagate the segmentation network to optimize the loss function and update all parameters in the network until the loss function converges or the maximum number of iterations is reached. Stop network training and obtain the trained segmentation model.

[0069] Step 4: Predict the infected area of ​​the lung CT image based on the trained segmentation model.

[0070] The images in the test set D1 are input into the trained prediction model, and the probability threshold d is set in the range of 0.3-0.6. The predicted probability value is compared with the probability threshold to obtain the infected area segmentation results of the lung CT images in the test set;

[0071] If p is greater than d, it indicates an infected area.

[0072] If p is less than or equal to d, it indicates an uninfected area;

[0073] Output a visual segmentation result image, which can show the background area and the infected lesion area in the lungs.

[0074] The effect of the present invention can be further illustrated by the following simulation:

[0075] 1. Simulation conditions

[0076] The simulation experiments are performed on an Intel(R) Core(TM) i7-8700k CPU with a main frequency of 3.70GHz, 48G memory, and Python3.7.1 combined with Pytorch on the Ubuntu platform.

[0077] A total of 2049 CT scan images and their corresponding ground-truth annotated segmentation masks are used in the prediction examples.

[0078] 2. Simulation content:

[0079] In simulation 1, on the test instance, the five segmentation methods of the present invention and the existing Unet, Unet++, DeepLabV3, DMDF-NET and FPN were used to predict the infected area of ​​the CT scan images of pneumonia patients, and the relevant evaluation indicators, namely, Dice coefficient, intersection over union (IoU), pixel accuracy (PA), sensitivity (SEN) and specificity (SPE), were calculated. The results are shown in Table 1.

[0080] Table 1 Performance evaluation comparison of the present invention and other segmentation methods

[0081]

[0082] As can be seen from Table 1, the present invention has achieved the best results in both the Dice coefficient and the intersection-over-union (IoU) index. Compared with the traditional Unet image segmentation network, the performance has been greatly improved. The technology of image feature pyramid is used in both DeepLabV3 and FPN, that is, the original image is converted into images of different sizes and then sent to the convolutional neural network, and finally the segmentation results are fused. They all fuse deep features with shallow features, making the expression ability of features more comprehensive, so their performance is also better than the traditional Unet segmentation network. The present invention is similar to DeepLabV3 and FPN ideas, and utilizes Transformer to combine with convolutional neural networks. The features trained have more comprehensive information representation, so the best segmentation performance is achieved. Although the present invention is slightly lower than DMDF-NET and DeepLabV3 methods in specificity SPE index, it has the best effect on the visual segmentation result graph.

[0083] Simulation 2: The segmentation results of the four selected test cases Case#1, Case#2, Case#3, and Case#4 are visualized using the present invention and compared with the Unet segmentation model. The results are as follows: Figure 3 shown.

[0084] from Figure 3 It can be seen that the visualization segmentation result graph of the present invention has the best effect.

[0085] The existing method used in Simulation 1 is from:

[0086] Unet:U-net:Convolutional networks for biomedical image segmentation[C].Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015:18th International Conference,Munich,Germany,October 5-9,2015,Proceedings,Part III 18.Springer International Publishing,2015:234-241。

[0087] Unet++:Unet++:A nested u-net architecture for medical imagesegmentation.Deep Learning in Medical Image Analysis and Multimodal Learningfor Clinical Decision Support:4th International Workshop,DLMIA 2018,and 8thInternational Workshop,ML-CDS2018,Held in Conjunction with MICCAI 2018,Granada,Spain,September 20,2018,Proceedings 4,2018:3-11。

[0088] DeepLabV3:Rethinking atrous convolution for semantic imagesegmentation.arXiv preprint arXiv:1706.05587,2017。

[0089] DMDF-NET:DMDF-Net:Dual multiscale dilated fusion network for accuratesegmentation of lesions related to COVID-19 in lung radiographic scans[J].Expert Systems with Applications,2022,202:117360。

[0090] FPN:Feature pyramid networks for object detection.Proceedings of theIEEE conference on computer vision and pattern recognition,2017:2117-2125。

Claims

1. A lung CT image segmentation method based on global-local feature association fusion, characterized in that: include: (1) Download CT scan images of pneumonia patients from two public datasets to obtain training and test sets respectively; (2) The training set and test set are preprocessed by image enhancement, pixel value normalization, dimension conversion and segmentation mask; (3) Construct a segmentation model based on global-local feature correlation fusion: (3a) Build an encoder consisting of N Transformer layers, where N is greater than or equal to 1; (3b) Building an encoder consisting of a convolutional neural network; (3c) Establishing a local feature correlation fusion module composed of Transformer layers to extract local correlation features; (3d) Establish a decoder composed of a convolutional neural network and symmetrical to (3b) to restore the image resolution and output the segmentation result; (3e) Connecting the first encoder and the second encoder in parallel, and then cascading them with the local feature association fusion module and the decoder to form a segmentation network model, and using the focal loss function as the loss function of the network model; (4) The training set data is input into two encoders in parallel, and the segmentation network model is iteratively trained using the back-propagation method until the loss function converges, thereby obtaining the entire segmentation network after training; (5) Predicting infected areas in lung CT images based on the trained segmentation model; 5a) Input the test set into the trained segmentation model to obtain the predicted probability value p for each sample pixel; 5b) Set the probability threshold d in the range of 0.3-0.6, compare the predicted probability value with the probability threshold, and obtain the infected area segmentation results of the lung CT images of the test set; If p is greater than d, it indicates an infected area. If p is less than or equal to d, it indicates an uninfected area.

2. The method according to claim 1, characterized in that In step (2), the training set and test set are preprocessed by image enhancement, pixel value normalization, dimension conversion and segmentation mask in sequence, which is implemented as follows: 2a) Data augmentation was performed on all CT images using the CV2 software package; 2b) Perform maximum and minimum normalization on the pixel values ​​x of all CT images using the following formula: Among them, x min Indicates the maximum pixel value in an image, x max Indicates the minimum pixel value; 2c) Convert the image dimensions, i.e. adjust the pixel values ​​of all images to a size of 224×224; 2d) Preprocess the segmentation masks corresponding to all images: All findings related to pneumonia infection were marked as white in the segmentation mask, represented as ‘1’ pixels; All remaining areas except the lesion area are marked in black and represented as '0' pixels.

3. The method according to claim 1, characterized in that The encoder constructed in step (3a) has a long vector as input and a Q1 as output vector. The long vector L1 is formed by evenly dividing the input image into multiple small blocks Pathes, then splicing the multiple small blocks and adding position codes with the same dimension as the long vector.

4. The method according to claim 1, wherein The convolutional neural network that constitutes the encoder in step (3b) is composed of a ResNet model pre-trained on a visual object recognition image library. The ResNet model is composed of a cascade of four Block modules, each of which consists of a multi-layer 3×3 convolutional layer, a 1×1 convolutional layer, and a pooling layer.

5. The method according to claim 1, characterized in that In step (3c), the local feature association fusion module composed of the Transformer layer extracts local correlation features, which is implemented as follows: (3c1) Take out the hidden layer vectors of the first three modules of the convolutional neural network in (3b), and use the Pathes processing method to split the three vectors into small blocks of different sizes, and then splice the three small blocks to obtain a long vector L2; (3c2) Input the long vector L2 into a Transformer layer to obtain the output vector; (3c3) Split the output vector in the ratio of 64:16:4 to obtain local correlation feature vectors Q2, Q3, and Q4.

6. The method according to claim 1, characterized in that In step (3d), the decoder composed of a convolutional neural network restores the image resolution and outputs the segmentation result. The process is as follows: (3d1) The encoder output vector Q1 is used as the decoder input layer vector, and the output vectors Q2, Q3, and Q4 of the local correlation fusion module are used as the connection vector; (3d2) Upsample and convolve the Q1 vector so that it propagates forward from bottom to top. Each time it passes through a layer of the network, it is concatenated with Q2, Q3, and Q4 in the channel dimension to restore the image resolution. (3d3) Repeat (3d2) until the forward propagation is completed and the segmentation result of the original resolution is obtained.

7. The method according to claim 1, characterized in that The loss function of the segmentation network model set in step (3e) is expressed as follows: FL(p t )=-(1-p t ) γ log(p t ), Where y is the sample value, p is the probability of judging y=1, and γ>0 is an adjustable factor.

8. The method according to claim 1, characterized in that Step (4) inputs the training set data into two encoders in parallel, and uses the back-propagation method to iteratively train the segmentation network model, as follows: (4a) The training samples are input into the encoder of the segmentation network in parallel, and the probability value matrix of the pixel value binary classification is obtained through the decoder. calculate The loss function value of the segmentation mask y corresponding to the training sample Among them, γ>0 is an adjustable factor; (4b) Set the learning rate to 0.001 and the total number of iterations to 15; (4c) Use the Adam optimization algorithm to backpropagate the segmentation network to optimize the loss function and update all parameters in the network until the loss function converges or the maximum number of iterations is reached. Stop network training and obtain a trained segmentation model.

Images