A patient ventricle image segmentation method based on improved U-net

Abstract

The application discloses a patient ventricle image segmentation method based on an improved U-net, improves a classical U-net segmentation model, adopts an encoder-decoder structure on the basis of the U-net segmentation model, replaces a deconvolution operation subpixel convolution with a decoder part, integrates information of surrounding points of interpolation points of the subpixel convolution, realizes small-matrix-to-large-matrix mapping while maximizing utilization and enriching image information. Secondly, after the model is constructed, in order to solve the problems that thick-layer image segmentation is difficult, accuracy is not high, and clinical application is difficult, a loss function for constraining a thick-layer image with an unmarked thin-layer image is designed, the difference between the thick-layer image and the unmarked thin-layer image is minimized, and the thick-layer image segmentation precision is improved.

Description

Technical Field

[0001] This invention is a hydrocephalus image segmentation method based on an improved U-Net segmentation model. The aim is to automatically and accurately segment hydrocephalus regions in CT and MRI images, assist doctors in diagnosis, improve diagnostic efficiency, and save medical resources. Background Technology

[0002] Studies have shown that ventricular volume is closely related to degenerative brain diseases and traumatic brain injury. Researchers have also pointed out that changes in ventricular volume are closely related to diseases such as schizophrenia, Parkinson's disease, Alzheimer's disease, hydrocephalus, and brain trauma. Currently, ventricular volume assessment mainly involves obtaining ventricular images through ventricular segmentation and then manually measuring them. This method is not only time-consuming and labor-intensive, but also highly subjective, has poor repeatability, and relies on expert experience. Therefore, researching automated and accurate ventricular image segmentation methods remains a challenging problem. Thus, developing a new automated segmentation method is urgent and necessary. Currently, research on automated segmentation methods based on machine learning and deep learning has become a major trend in the new era.

[0003] In previous studies on automated ventricle segmentation, researchers typically used single-modality images, namely CT or MRI images. Therefore, automated ventricle segmentation methods are often incompatible, and different algorithms may perform differently when segmenting different parts of the ventricle. Previous machine learning-based (including deep learning) studies have used images with slice thicknesses less than 3 mm. This is because, at the same scanning distance, smaller image thickness allows researchers to obtain more images, which is beneficial for the machine to extract more image features for deep learning algorithms.

[0004] However, in clinical practice, due to limitations (including instrumental and manual conditions), thicker slices are more common, thus limiting the clinical effectiveness of such methods. Ventricular image segmentation still faces challenges in clinical application and accurate segmentation of complex brain lesions, such as those in pathology. Currently, few methods comprehensively address these issues. Therefore, there is still room for improvement in automated and precise ventricular image segmentation methods. Summary of the Invention

[0005] To address the challenges of ventricle image segmentation, including difficulties in extracting effective information during model feature extraction and the complexity of clinical applications due to the difficulty of segmenting thick images, this paper proposes a ventricle segmentation method based on an improved U-net. It also designs an image dataset acquisition method and a training strategy based on the contrast learning between thick and thin images to improve segmentation accuracy. The improved U-net uses a pre-trained ResNet-34 as the encoding layer and sub-pixel convolution as the decoding layer, effectively expanding the image information during the decoding process. Furthermore, a novel loss function is proposed, constraining the thick image by using unlabeled thin images, further improving the segmentation accuracy of thick images.

[0006] A method for segmenting patient ventricles images based on an improved U-net, characterized by the following steps:

[0007] Step 1: Acquire the raw images of the patient's brain CT scan, obtaining thin-slice CT images, thick-slice CT images, thin-slice MRI images, and thick-slice MRI images, and annotate them. Images with a slice thickness >3mm are defined as thick-slice images; otherwise, they are defined as thin-slice images. This includes automatic machine annotation and expert review.

[0008] The specific steps are as follows:

[0009] First, head CT scans were performed on two CT scanners: a Siemens SOMATOM DefinitionFlash and a Siemens SOMATOM Emotion16. Next, MRI scans were performed using a 1.5T TMR scanner (Avanto, Siemens, Erlangen, Germany) and a 3.0T MRI scanner (Prisma, Siemens, Erlangen, Germany). All images were stored in a Picture Archiving and Communication System (PACS). The ventricles were then manually delineated. For the MRI images, T1WI was selected for manual labeling.

[0010] The specific annotation process is as follows:

[0011] The ventricles were labeled using ITK software; the ventricle delineation results were evaluated, and adjustments were made if the labels were inaccurate; when the scan layer thickness was >3mm, it was defined as a thick-slice image, otherwise it was defined as a thin-slice image.

[0012] In summary, all ventricular images were divided into four groups: thin-slice CT images, thick-slice CT images, thin-slice MRI images, and thick-slice MRI images.

[0013] Step 2: Using the data processed in Step 1, construct a ventricle segmentation model based on the improved U-net. Input the data into the model and obtain the segmentation results. The specific operations are as follows:

[0014] Step 2-1: Pre-train a ResNet-34 model on the ImageNet public dataset using the encoder;

[0015] Step 2-2: Use subpixel recursion instead of deconvolution to augment image information, as follows:

[0016] Subpixel convolution, based on the pixels between two actual pixels in an image, reconstructs their representation using information from surrounding pixels. This allows for mapping from small to large matrices while adding more dimensional and detailed image information, resulting in higher resolution and faster processing efficiency. Therefore, this paper uses subpixel convolution to construct the segmentation results. Subpixel convolution is represented as:

[0017] F L =SP(W L *F L-1 +b L )⑴

[0018] The SP(.) operator defines the shape H×W×C×r. 2 The tensor transformation arranges the tensors into tensors of shape rH×rW×C, F L-1 and F L These are input features and output features, W L and b L These are the parameters of the subpixel convolution operator, which takes thick and thin slice images as input; the thick slice images are labeled, while the thin slice images are unlabeled.

[0019] Step 3: For the feature image obtained in Step 2, calculate the loss function for the model output feature layer from the perspectives of minimizing the difference between thick and thin images and the segmentation difference, and then backpropagate to adjust the model. The specific operation is as follows:

[0020] To minimize the performance difference between thick and thin slices using unlabeled thin slice images, the model loss is divided into two parts: one part minimizes the difference between the model output and the true segmentation for thick slices, and the other part minimizes the difference between thick and thin slices using the model output for thin slices. The following objective function is used to optimize the project's model:

[0021] L(x s ,x t ) = L s (p s ,y s )+λL T (p t (2)

[0022] Where λ is the weighted L T The hyperparameters that affect this, P s and P tIt is the model's prediction of H×W×C, P s It is a prediction of thick-layer images, P t It is a prediction of thin-layer images, y s This represents the actual segmentation. L s It is the cross-entropy loss, defined as follows:

[0023]

[0024] L T The probability distribution p considered as the target domain t The objective domain, or thin-slice domain, is the distance between the two distributions and the uniform distribution; therefore, minimizing the distance between the two distributions makes the inter-class discrepancy larger, making them easier to separate. This is because this approach implicitly pushes image features away from the decision boundary and aligns them between the two distributions. Mathematically, the objective function for a thin-slice image is defined as follows:

[0025]

[0026] Where f(x) is selected from Personχ 2 That is, f(x) = x 2 -1, as the loss function, therefore L T The gradient is defined as follows:

[0027]

[0028] The above formula balances the gradient between simple and complex samples. During model training, the loss function is iteratively optimized until the model converges. To facilitate the comparison of model performance, slices of each image and the prediction results are used as output.

[0029] Preferably, the encoder is used to pre-train a ResNet-34 model on the ImageNet public dataset; specifically, the encoder extracts high semantic features of the input image data as the basis for the decoder segmentation; including upsampling, downsampling and skip connections; first, downsampling is performed through convolution to extract image features, and then upsampling is performed through deconvolution to gradually repair and increase the details and dimensions of the image; for modules with the same pixels during upsampling and downsampling, skip connections are used to expand information, which can better reflect the details of the image.

[0030] As a preferred approach, the U-Net skip connection layer is improved by introducing a dual attention module (DAM) to assign attention weights to features in deep and shallow layers. The specific operation is as follows:

[0031] Unlike U-net, which directly concatenates deep and shallow features, the Dual Attention Module (DAM) receives shallow features from the encoding region and deep features from the decoding region, respectively. These features are then processed by Channel Attention (CA) and Pixel Attention (PA) to perform weighted fusion from both channel and pixel perspectives. Finally, the weighted fused image is processed by normalization to enrich the image information.

[0032] The beneficial effects of this invention are:

[0033] 1. This invention improves upon the classic U-net segmentation model by employing an encoder-decoder structure. The decoder uses sub-pixel convolution instead of deconvolution, and the interpolation points of the sub-pixel convolution integrate information from surrounding points, maximizing and enriching image information while achieving a small-matrix to large-matrix mapping. Secondly, after constructing the model, to address the difficulty and low accuracy of thick image segmentation, which hinders clinical application, a loss function is designed that constrains the thick image using an unlabeled thin image. By minimizing the difference between the two, the segmentation accuracy of thick images is improved. Attached Figure Description

[0034] Figure 1 This article proposes a methodology;

[0035] Figure 2 Patient inclusion and exclusion flowchart;

[0036] Figure 3 Model building process;

[0037] Figure 4 Encoder-decoder internal network structure;

[0038] Figure 5 Visualization of segmentation results from thin-slice images on MRI and CT. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings. The overall structure of the method of the present invention is shown in the accompanying drawings. Figure 1 As shown, the specific steps are divided into two steps: acquisition and annotation of ventricle images, and construction and training of ventricle segmentation models.

[0040] In the acquisition and annotation of brain ventricles images, the image dataset construction process is as follows: Figure 4 As shown.

[0041] First, the project team selected images of patients aged 60 and above who underwent brain CT or MRI examinations at the Zhejiang Provincial Key Laboratory of Geriatrics between January 1, 2018, and December 31, 2020, as the dataset. Second, to delineate the ventricles and conduct comprehensive analysis, the team selected healthy elderly patients, elderly patients with cerebral atrophy, elderly patients with idiopathic normal pressure hydrocephalus, and elderly patients with acquired hydrocephalus. This is because the ventricular shape and size of these four categories of patients are highly representative, showing a trend from normal to severe, which can help the project team systematically and comprehensively analyze the ventricles. Third, the diagnostic results for these patients were approved by two radiologists with over 10 years of experience and strictly followed disease diagnostic guidelines. Finally, the dataset contains a large number of healthy elderly patients and patients with cerebral atrophy. Since their ventricular structures are not completely deformed, automated segmentation and annotation are more advantageous than manual annotation. The patient inclusion and exclusion flowchart is shown below. Figure 2 The basic study population is shown in Table 1.

[0042] Table 1. Statistical information of the subjects

[0043]

[0044] In Table 1, age reports are mean ± standard deviation. CT-1 represents the Siemens SOMATOM DefinitionFlash CT scanner (Germany). CT-2 represents the Siemens SOMATOM Emotion 16 CT scanner (Germany). MRI-1 represents the 1.5T TMR scanner (Siemens Avanto, Erlangen, Germany). MRI-2 represents the 3.0T MRI scanner (Prisma, Siemens, Erlangen, Germany). CT image slice thicknesses include: 0.5, 1.0, 1.5, 2.0, 4.8, 5.0 mm. MRI image slice thicknesses include: 1.0, 7.8, 8.0 mm.

[0045] Table 2. Number of thin-slice and thick-slice images in this experiment

[0046]

[0047] Table 2 divides the dataset into two groups for MRI and CT images. The training set is used only for model training and optimization, while the test set is used only for validation.

[0048] This study was conducted based on the recommendations of the Ethics Committee of Zhejiang Hospital and the Key Laboratory of Geriatric Medicine of Zhejiang Province, and all participants signed written informed consent forms in accordance with the Declaration of Helsinki.

[0049] In the construction and training of the ventricle segmentation model, the overall algorithm flow is as follows: Figure 3 As shown.

[0050] In real-world scenarios, thick slice images are more readily available, while thin slice images are rarer, making accurate annotation more challenging for clinicians. Furthermore, the distribution of different image thicknesses can lead to domain shift issues, thus obfuscating deep learning models. Therefore, the project team proposed a segmentation model for images with unknown thickness. This model only requires annotation of thick slices for training, aiming to minimize the performance difference between thick and thin slice images using unlabeled thin slice images. After data refinement and training, thin and thick slice scans (CT or MRI images) are input, and the segmentation results for each ventricle and the whole brain are extracted. Feature extraction is performed on the input data, and the model utilizes an encoder and decoder for high-precision resampling algorithms. Then, whole-brain 3D structural reconstruction and ventricle 3D reconstruction are performed, and these two methods are combined to automatically calculate the relative ventricular volume. The calculated relative ventricular volume is compared with the manually annotated ventricular volume for error calculation. After error analysis, the generalization and utility of the automated ventricular volume measurement model can be measured and evaluated.

[0051] This invention is based on the traditional U-net network and improves the encoder and decoder layers, with the structure as follows: Figure 4 As shown, the encoder uses a ResNet-34 model pre-trained on the ImageNet public dataset, and the decoder uses multiple sub-pixel convolutional layers to decode the encoded features. The encoder can extract high semantic features from the input image data as the basis for segmentation by the decoder.

[0052] After completing the model construction work of this invention, a series of model performance analyses were conducted.

[0053] To investigate the effectiveness and advancement of our proposed method, we compared it with U-net and U-net++ models on thin-slice, thick-slice, and mixed images. Our team collected thick and thin slice samples from iNPH patients using different imaging modalities (MRI and CT), as shown in Tables 4 and 5. In this study, we only used the annotations of the thick-slice images to supervise the training of the deep learning network. Both U-net and U-net++ employ encoder-decoder structures and utilize intermediate features to maintain image information. Experimental results are shown in Tables 4 and 5. Bold text indicates the best performance.

[0054] Table 4. Comparison of the effectiveness of our method with other advanced methods (Dice)

[0055]

[0056] Table 5. Comparison of the effectiveness of our method with other advanced methods (IOU)

[0057]

[0058] As shown in the table, U-net++ significantly improves the accuracy of thin-slice MRI images and thick-slice CT images compared to U-net. U-net++ is an improvement on U-net based on skip connection layers, indicating that improving skip connection layers helps enrich image information and improve accuracy. However, U-net only fuses information from the next layer, which can easily lose positional information, resulting in less refined segmentation results. Compared with traditional and state-of-the-art models, our proposed method achieves significant improvements in both thick and thin slices, especially in thin-slice images, where it improves accuracy by 5.8% and 5.3% compared to U-net++ in CT and MRI, respectively, significantly outperforming the commonly used U-net and U-net++ networks in medical segmentation. This demonstrates that our proposed method can extract richer image information and simultaneously improve the segmentation accuracy of both thin and thick slice images, making it applicable to various images obtained from various instruments and of great significance for clinical applications. Furthermore, with the help of a pre-trained ResNet-34 encoder, our model can achieve an accuracy improvement of up to 10% in thick-slice images.

[0059] To demonstrate the superiority of the visualization method, this paper selected the U-net and U-net++ models for comparative experiments. Figure 5 Example segmentation results are shown for thin slice images randomly selected from the test sets of MRI and CT modalities. In the second column of each modality, it can be observed that U-net performs poorly on MRI images. Meanwhile, U-net++, an upgraded version of U-net, shows better predictions compared to the U-net model, but is still not accurate enough. Compared to U-net and U-net++, our proposed method employs subpixel convolution in the decoder and improves the loss function. Experimental visualizations show that our method extracts richer edge information, achieves accurate results, and can segment MRI and CT images with high precision. Especially in the last row of the CT examples, although the original image has low contrast, our method can still accurately identify and segment each part, demonstrating the robustness of our approach.

[0060] Figure 5 From left to right, the images show the original CT image, the U-net network result, the U-net++ network result, the result of the new network designed by our project team, and the labeled target image. Different colors in the image indicate the volumetric space of the patient's brain after segmentation.

[0061] pass Figure 5As shown in Tables 4 and 5, the automated ventricle segmentation method proposed by the project team can be applied to CT and MRI images of varying thicknesses. Furthermore, compared to the widely used U-net and later versions of U-net++, the project team's method achieves better segmentation results (Dice > 0.9). Therefore, the method proposed by the project team has practical application value in various clinical scenarios.

[0062] To examine the differences in training models using different loss functions under the influence of each dataset component (labeled thick slices and unlabeled thin slices) in our proposed method, the project team conducted comparative experiments. These experiments included three different variant structures: using only unlabeled thin slice images with the thin slice objective function; using only labeled thick slice images with the thick slice cross-entropy function; and using our novel loss function that fuses unlabeled thin slices and labeled thick slices. These variations validated the effectiveness of our proposed method, and the experimental results are shown in Tables 6 and 7. In these tables, bold text indicates the best performance. "Thin" indicates the model was trained using unlabeled thin slice images, and "thick" indicates the model was trained using labeled thick slice images.

[0063] Table 6. Comparison of coefficients (Dice) in the comparative experiments presented in this paper.

[0064]

[0065] Table 7. Comparison of coefficients (Dice) in the comparative experiments presented in this paper.

[0066]

[0067] Experimental results show that while the proposed model performs well on thick slice images when trained only on them, it performs poorly on thin slice images, as shown in the first row of Tables 6 and 7. However, training on unannotated thin slice images results in highly unreliable predictions because the objective function reaches a global minimum when the probability of each class is assigned the same value, as shown in the second row of Tables 6 and 7. Furthermore, when both thick and thin slice images are included in the semi-supervised training paradigm, Dice and IOU show significant improvements compared to training with only labeled thick slice images. Improvements of 8.8% and 8.4% are observed on thin slices and mixed slices of MRI, respectively, while improvements of 3.8% and 2.1% are observed on CT. This demonstrates that labeled thick slice images ensure high accuracy on thick slice images, and combining this with unlabeled thin slice images yields superior performance on both thin slices and mixed slices, achieving an overall optimal solution.

Claims

1. An improved U-net based method for ventricle segmentation of patient brain images, characterized in that, The method specifically comprises the following steps: Step 1, collect the original image of the patient's brain CT, get thin layer CT image, thick layer CT image, thin layer MRI image and thick layer MRI image and label them, wherein when the scanning layer thickness is > 3mm, it is defined as thick layer image, otherwise it is defined as thin layer image; Step 2, using the data processed in step 1, constructing a ventricle segmentation model based on improved U-net, inputting the data into the model, and obtaining the segmentation result, the specific operation is as follows: Step 2-1: using an encoder to pretrain a ResNet-34 model on an ImageNet public dataset; Step 2-2: using subpixel instead of deconvolution operation to expand image information, specifically as follows: Subpixel convolution is based on the pixels between two actual pixels on the image, which is restored by the information of its surrounding pixels; Subpixel convolution is expressed as: F L = SP(W L *F L-1 +b L ) ⑴ The SP(.) operator defines the shape H×W×C×r. 2 The tensor transformation arranges the tensors into tensors of shape rH×rW×C, F L-1 and F L These are input features and output features, W L and b L These are the parameters of the subpixel convolution operator, which takes thick and thin slice images as input; the thick slice images are labeled, while the thin slice images are unlabeled. Step 3, for the feature image obtained in step 2, the loss function of the model output feature layer is calculated from the aspects of minimizing the difference between thick and thin images and the difference between segmentation, and the model is adjusted by back propagation, the specific operation is as follows: In order to utilize unlabelled thin slice images to minimize the performance difference between thick and thin slice images, the loss of the model is divided into two parts, one part is to minimize the difference between the output of the thick slice model and the real segmentation, and the other part is to minimize the difference between the thick and thin slice images using the thin slice model output, the following objective function is used to optimize the model of the project group: L(x s ,y t ) = L s (p s ,y s )+ λL T (p t ) (2) where λ is the weighted L T hyperparameters, P s and P t is the prediction of the model for H x W x C, P s is the prediction for the thick layer image, P t is the prediction for the thin layer image, y s is the true segmentation; L s is the cross-entropy loss, defined as follows: L T the probability distribution p t distance between the uniform distribution, the target domain, i.e., the thin slice domain; the objective function of the thin slice image is defined as follows: where f(x) is chosen to be Personχ 2 i.e. f(x) = x 2 -1, as the loss function, so L T The gradient of L The above formula balances the gradient between simple samples and complex samples, and the above loss function is iteratively optimized to the convergence of the model during the model training process; In order to facilitate the comparison of model performance, the slice and prediction result of each image are taken as output.

2. The improved U-net based method for ventricle segmentation of brain images of a patient according to claim 1, characterized in that: The ResNet-34 model pre-trained by the encoder on the ImageNet public dataset; Specifically, the encoder extracts the high semantic features of the input image data as the basis for the decoder segmentation; Including upsampling, downsampling and jump connection; First, the image features are extracted by convolution downsampling operation, and then the details and dimensions of the image are gradually repaired by deconvolution operation for upsampling; For the pixel modules in the upsampling and downsampling process, the information is expanded through jump connection.

3. The improved U-net based method for ventricle segmentation of brain images of a patient according to claim 2, characterized in that: The U-Net jump connection layer is improved, and a double attention module DAM is introduced to assign attention weights to deep and shallow features, and the specific operation is as follows: Unlike U-net directly splicing deep and shallow features, the double attention module DAM receives shallow features from the encoding area and deep features from the decoding area, respectively, and respectively passes through channel attention CA and pixel attention PA for weighted fusion from the aspects of channel and pixel, and then processes the weighted fused image through normalization.

Images