A liver and liver tumor segmentation method based on parallel residual attention

Abstract

The application discloses a liver and liver tumor segmentation method based on a parallel residual attention network, relates to the fields of deep learning and medical image processing, and comprises data processing and preparation and a segmentation prediction visualization network.The data processing firstly processes a disclosed 3D abdominal medical image into a 2D slice image containing a liver and a liver tumor; the 2D data set is preprocessed and data enhanced; the segmentation prediction network uses a feature fusion network based on an encoding-decoding structure, after loading common pre-training weights through migration learning, the data-enhanced 2D image data is loaded into the segmentation model for training, and after the training is completed, special optimal weights suitable for the liver and the liver tumor are generated, which are used for liver and liver tumor special segmentation.The parallel residual attention convolution network structure proposed by the application for liver and liver tumor segmentation solves the problem that due to the complexity of the liver and the liver tumor, the segmentation of the liver and the liver tumor is prone to be interfered by other parts such as the kidney and the like, improves the segmentation efficiency and the segmentation precision, and has certain advancement compared with the prior art.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and medical image processing, specifically to a method for converting liver and liver tumor data formats and segmentation using a residual attention network based on parallel feature extraction. Background Technology

[0002] Malignant liver tumors (liver cancer) have become a serious threat to human health. However, the appearance, shape, and spatial location of the liver and liver tumors are unpredictable and vary from person to person and from stage to stage. Early-stage malignant liver tumors often do not present obvious symptoms and are therefore somewhat insidious. Thus, early screening for malignant liver tumors is crucial.

[0003] The location and morphological characteristics of liver tumors are important indicators for the early diagnosis of liver cancer. The diagnostic process for specialists is very time-consuming, labor-intensive, and relatively subjective. Therefore, designing a liver and liver tumor segmentation network and segmentation visualization method is of great significance in assisting doctors to improve diagnostic speed and effectiveness.

[0004] Currently, segmentation techniques for the liver and liver tumors in abdominal medical images mainly include visual recognition techniques based on human-defined features and deep learning techniques based on automatic features. Techniques based on human-defined features include those based on color, texture, contrast thresholding, edge detection, segmentation models, and region segmentation. Due to the complexity of abdominal medical images, the segmentation of the liver and liver tumors is easily affected by interference from normal physiological structures such as the kidneys, thus impacting segmentation accuracy. Traditional liver medical image feature extraction techniques rely on manual extraction, which can lead to decreased segmentation accuracy when small livers or discontinuous liver structures are present in the medical image. Therefore, automatic segmentation of the liver and liver tumors in abdominal medical images is crucial. Summary of the Invention

[0005] To address the issues of accuracy and sensitivity in existing liver and liver tumor segmentation methods, and the insufficient robustness of many models to noise, which makes liver and liver tumor segmentation results susceptible to interference from noise factors in abdominal medical images, particularly the poor quality of liver and liver tumor edge segmentation, leading to oversegmentation and undersegmentation of the segmented liver and liver tumor boundaries, this invention designs corresponding solutions and methods. It proposes an efficient data processing method and a U-Net model based on a parallel residual attention network: Res-ECA-UneXt (RECA-U-NeXt) for liver and liver tumor segmentation. The Backbone module is improved for feature extraction; to enhance model accuracy, an EA module is added to the Backbone section. The EA module is described in detail below. Figure 1As shown, the module was redesigned into a dual-channel structure. This module first redesigns the ConvNeXt Backbone convolutional module for medical image features and then connects it in parallel with a residual attention network. Verification showed that this module effectively improves segmentation accuracy. In the downsampling part, the number of block stackings was adjusted from (3, 4, 6, 3) to (3, 3, 9, 3), achieving similar FLOPs to Swin-T. Improvements were made to address the low sensitivity and insufficient robustness of the classic U-Net model. A feature extraction module with a parallel structure and channel attention mechanism was designed, and the introduction of a residual structure accelerated the network's convergence speed, preventing gradient explosion and gradient vanishing problems. This improved the network's segmentation ability for liver and liver tumors, enhancing segmentation sensitivity and robustness.

[0006] The technical solution of this invention: a residual attention mechanism segmentation network based on a parallel structure and various segmentation prediction visualization methods, the flowchart of which is shown below. Figure 1 Includes the following steps:

[0007] S1. Process publicly available abdominal medical images to create a 2D abdominal medical image containing the liver and liver tumors;

[0008] S2. Preprocess the prepared 2D abdominal medical image;

[0009] S3. Perform data augmentation on the preprocessed image;

[0010] S4. Build a parallel residual channel attention network model, and input the processed dataset into the built network for training. Save the optimal training weights to obtain the optimal training model.

[0011] S5. After preprocessing and data enhancement in step 3, the segmented test set data is loaded into the model for testing. The segmentation results are obtained and compared with the gold standard of manually annotated liver and liver tumors to obtain comparison data and segmentation results.

[0012] To further summarize S1, it includes the following two steps:

[0013] Step 1: Process the gold standard data by slicing the existing publicly available 3D abdominal medical dataset along the z-axis. Mirroring the slices vertically ensures their orientation conforms to requirements. Valueless slices without liver tissue or liver tumors are removed based on their size. However, the 2D labels on the gold standard slices (i.e., the 2D labels) become formatted incorrectly after 2D conversion: in labels only containing liver tissue, the liver pixel value is (255, 255, 255), while in labels containing both liver tissue and liver tumors, the liver pixel value is (127, 127, 127). Therefore, a Python program is designed to uniformly set the liver pixel value in the labels to (127, 127, 127). Step 2: Process the original image by adjusting the Hu value of the medical image to grayscale based on the liver window value. Slice the image along the z-axis and mirror it vertically. Valueless slices without liver tissue or liver tumors are then filtered out based on the 2D label filenames, ensuring the original image's 2D slice name order matches the 2D label order.

[0014] To further summarize S2, it includes the following three steps:

[0015] The first step is image grayscale conversion, which transforms the original image into a single-channel image by converting it to grayscale.

[0016] The second step is: after obtaining the grayscale image, the present invention uses the Limiting Contrast Adaptive Histogram Equalization (CLAHE) method to enhance the contrast of the obtained grayscale image, increase the distinction between the visual disc and the background. This method can also suppress some background noise and highlight the liver and liver tumor structures.

[0017] Finally, a nonlinear Gamma transformation is used to adjust the illumination intensity of the input abdominal medical image, and a nonlinear operation is performed on the intensity values ​​to create an exponential relationship between the intensity values ​​of the input and output images. The goal is to enhance the darker parts of the image while ensuring that the brighter parts are not affected. This can further increase the image contrast, reduce the interference of noise factors, and improve the network's segmentation ability and the model's generalization ability.

[0018] Data changes during the above processing are as follows Figure 2 As shown, by comparing (b) and (c), it can be found that compared with the results of weighted grayscale and histogram equalization in (c), the result of histogram equalization method (d) can effectively improve the distinction between liver and liver tumor and background pixels in grayscale image. By comparing (d) and (e), it can be found that after using nonlinear Gamma transformation on the basis of CLAHE result (d), the liver and liver tumor areas become clearer. This shows that (e) further increases the contrast difference between liver and liver tumor and background on the basis of (d). Figure 1The results demonstrate that this preprocessing method effectively addresses the issue of indistinct liver and liver tumor features in the original image, increasing the contrast between pixels in the liver and liver tumor regions and other areas in abdominal medical images. The preprocessed images will help subsequent deep learning models learn more expressive and robust features during feature extraction, improving the model's ability to distinguish between liver and liver tumor features and background pixels.

[0019] Further, S3 includes:

[0020] Data augmentation is performed on the original images, including mirror flipping, center rotation, horizontal flipping, affine transformation, and elastic deformation. To increase the number of samples, reduce the impact of overfitting, and improve the accuracy of pixel segmentation in local edge regions, this invention performs random cropping on the dataset used. During cropping, to facilitate feature extraction by the deep network model, the cropping size is uniformly set to 512×512 dpi.

[0021] Further, S4 includes:

[0022] In the training phase of the proposed RECA-U-NeXt model, the abdominal medical images are first preprocessed and data augmented, and the images are cropped to the same size of 512×512. Then, the processed abdominal medical images are fed into the REA-U-Net model for training. After training, the specific optimal weights for liver and liver tumor segmentation will be generated.

[0023] Further, step 3 includes:

[0024] For network model training, two publicly available abdominal medical image datasets, Refuge and Drishti-GS1, were selected. The preprocessed and enhanced datasets were imported into the model for training. The datasets were divided into training and validation sets at an 8:2 ratio to improve network generalization. The training results were validated using the validation set, and the optimal weights were saved. The trained model was then tested on the test set to obtain the optimal segmentation results for the liver and liver tumors.

[0025] Experimental parameters:

[0026] In the retinal fundus vessel segmentation experiment, the hardware environment was: NVIDIA GeForce RTX2080Ti.

[0027] Graphics card with 11GB VRAM; Operating system: Ubuntu 18.04; PyTorch deep learning framework. The Adam optimizer is used to update parameters. The initial learning rate (lr) is set to (1e-4)*3, the betas parameter is set to (0.9, 0.999), the batch size is set to 14, and the total number of training epochs is set to 500. During training, a linear decay strategy for the learning rate is used. The learning rate at the t-th epoch can be expressed as:

[0028] Attached Figure Description

[0029] Figure 1 Create a flowchart for the dataset;

[0030] Figure 2 To segment the flowchart;

[0031] Figure 3 Diagram of ECA attention module;

[0032] Figure 4 This is the feature extraction module (RECA Block).

[0033] Figure 5 Image showing the preprocessing results of the liver dataset;

[0034] Figure 6 The diagram shows the RECA-U-NeXt network structure.

[0035] Figure 7 Comparison of segmentation results

[0036] Figure 8 Comparison chart of generalized segmentation results Detailed Implementation

[0037] The following is a clear and complete description of the technical solutions of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. The description is only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without creative effort are within the protection scope of the present invention.

[0038] The distribution and processing flowchart of this invention is as follows: Figure 1 As shown, the entire process is divided into two phases: training and testing.

[0039] I. Training Phase

[0040] like Figure 1 As shown

[0041] During the training phase, the images are first converted from a 3D NII format abdominal medical dataset to a 2D image dataset containing liver and liver tumors. Preprocessing is then performed, followed by data augmentation. The augmented data is then imported into RECA-U-NeXt for model training, and the training results are obtained and the weights are saved. Details are as follows:

[0042] (1) Dataset transformation

[0043] The 3D abdominal medical image data was sliced ​​on the Z-axis, and the labeled datasets without liver and liver tumors were filtered out based on the size of the image pixel values. The original image datasets containing liver and liver tumors were then selected based on the corresponding labeled datasets.

[0044] (2) Data preprocessing

[0045] Windowing is performed before data conversion, defining a window function and setting the window center and width to convert the medical image with Hu values ​​into a grayscale image with pixel values ​​of (0-255). After obtaining the grayscale image, a contrast-limited adaptive histogram equalization method is used to enhance the contrast of the resulting grayscale image, increasing the distinction between the liver and the background. A nonlinear Gamma transform is used to adjust the illumination intensity of the input abdominal medical image, and a nonlinear operation is performed on the intensity values ​​to ensure an exponential relationship between the intensity values ​​of the input and output images. The main purpose is to enhance the darker parts of the image while ensuring that the brighter parts are not affected. It can further increase the contrast of the image, reduce the interference of noise factors, and improve the network's segmentation ability and model generalization ability.

[0046] (3) Data Augmentation

[0047] Using transformers makes data augmentation more flexible. PyTorch not only allows you to set operations on datasets, but also to randomly combine and select these operations.

[0048] (4) Network construction

[0049] Res-ECA-U-NeXt (RECA-U-NeXt), an improvement on the traditional U-Net network, is an end-to-end multi-label deep neural network. The overall network structure is as follows: Figure 4 As shown. A channel attention module is added during the encoding process, and a feature extraction module with a parallel structure is used, as shown in the diagram. Figure 5As shown, a 7×7 large convolutional kernel is connected in parallel with a 3×3 small convolutional kernel. This effectively extracts the main features without losing detailed features, enhancing the network's feature extraction capability, reducing information loss during feature extraction, and fusing low-resolution and high-resolution features to extract features within a wider receptive field while retaining a large amount of detailed information. Then, an ECA module is introduced to assign different weights to each feature map, enabling the network to focus on more useful feature information. Simultaneously, a residual module is added to the BottleNeck module to avoid gradient vanishing. The BottleNeck module structure is as follows: Figure 6 As shown.

[0050] The decoder includes an ECA module with channel and spatial attention mechanisms. The main idea of ​​this module is to learn the weights of each channel using the correlations between different channels, thereby achieving effective attention focusing. Specifically, the module computes channel attention in three steps. First, for each channel in the input feature map, average pooling and max pooling are used to extract its global contextual information. Then, this information is concatenated, and a multilayer perceptron is used to generate an attention vector. Finally, this vector is dot-productted with the input feature map and scaled to obtain a weighted output feature map.

[0051] Compared to other attention mechanisms, the ECA attention module boasts superior performance and efficient computation, achieving higher accuracy with fewer parameters and computational resources. It compresses the input feature map spatially, uses global average pooling to obtain a 1*1*C feature map, and then learns channel features through 1*1 convolution. Specifically, this method employs a channel attention mechanism, learning the importance of different channels to achieve effective feature extraction. Finally, the channel-attention feature map is multiplied by the original input feature map to obtain a channel-attention-enabled feature map. Furthermore, this method utilizes skip connections, enabling the fusion of semantic information at different levels on the same scale. This allows the encoder to extract more useful features and find the algorithm's solution within an iteration, preserving the end-to-end nature of U-Net. While this results in finer segmentation boundaries, the abstraction and compression of information can negatively impact the segmentation results. To address this, this invention retains the upsampling skip connections in U-Net while concatenating the output of the current layer with the output of the corresponding downsampling layer. This preserves more original detailed information, which helps improve the accuracy and robustness of the model, thereby improving the accuracy of segmentation.

[0052] Network testing:

[0053] On the test set, RECA-U-NeXt outperformed many other deep neural network models from recent years across multiple metrics. It achieved 96.87%, 92.64%, 93.89%, and 94.97% on four metrics: AUC, Dice, MIoU, and Accuracy, respectively, outperforming other models in all four evaluation metrics. The proposed RECA-U-NeXt shows an AUC improvement of approximately 3.6% compared to U-Net, approximately 3.08% compared to ResU-Net, approximately 2.11% compared to SE-ResU-Net, approximately 2.00% compared to ECA-ResU-Net, and approximately 0.64% compared to ConvUNeXt on the test dataset. Dice improvements were approximately 1.30% compared to U-Net, approximately 1.25% compared to ResU-Net, approximately 0.87% compared to SE-ResU-Net, approximately 0.75% compared to ECA-ResU-Net, and approximately 0.48% compared to ConvUNeXt. MIoU achieves approximately 2.85% improvement over U-Net, 2.54% over ResU-Net, 1.27% over SE-ResU-Net, 1.02% over ECA-ResU-Net, and 0.78% over ConvUNeXt. Accuracy is approximately 13.24% higher than U-Net, 12.22% higher than ResU-Net, 8.84% higher than SE-ResU-Net, 8.20% higher than ECA-ResU-Net, and 4.99% higher than ConvUNeXt.

[0054] Table 1. Comparison of modules on the Refuge and Drishti-GS1 datasets.

[0055]

[0056] The Internet Figure 7 The segmentation results are closer to the expert-annotated images and contain less noise. Furthermore, upon closer inspection, the segmentation output of the FE-U-Net network exhibits fewer irregular edge shapes and a smoother effect.

[0057] The comparison of experimental results shows that the RECA-U-NeXt segmentation model presented in this paper outperforms many classic methods and models in recent years on the LiTS17 abdominal medical image dataset, demonstrating strong competitiveness across multiple metrics. Compared with other classic liver and liver tumor segmentation methods, it not only significantly improves evaluation metrics such as Acc, AUC, Dice, and MIoU, but also greatly enhances the visualization quality of liver and liver tumor segmentation, particularly for liver and liver tumors with blurred boundaries. The RECA-U-NeXt segmentation model produces clearer liver and liver tumor segments compared to other methods, with more defined and complete liver and liver tumor contours. The model in this invention is also more sensitive to blurred contours and liver and liver tumor boundary regions with background noise interference, successfully segmenting subtle liver and liver tumor contours that are difficult for other models to segment.

[0058] Based on the comparison of experimental results, the RECA-U-NeXt model of this invention performs excellently on the LiTS17 abdominal medical image dataset, surpassing many classic methods and models in recent years, demonstrating strong competitiveness. Compared to other classic liver and liver tumor segmentation methods, it not only improves evaluation metrics such as Acc, AUC, Dice, and MIoU, but also significantly enhances the visualization quality of the segmentation, especially when segmenting liver and liver tumors with blurred boundaries. Compared to other methods, the RECA-U-NeXt model of this invention produces clearer liver and liver tumor segments with more complete contour edges. Furthermore, the model of this invention is more sensitive to some blurred contours and liver and liver tumor boundary regions affected by background noise, demonstrating outstanding performance in segmenting small contours. To verify the generalization performance of the RECA-U-NeXt model, the same experiments as before were conducted on the 3DIRCADb dataset. The experimental results are as follows: Figure 8 As shown, the RECA-U-NeXt model of this invention is more sensitive to blurred boundaries, retains more detailed information about the edge contours of the liver and liver tumors, and produces clearer segmented liver and liver tumors with fewer boundary discontinuities compared to other methods. Furthermore, the morphological structure of the segmented liver and liver tumors is closer to the expert-annotated labels.

[0059] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A liver and liver tumor segmentation method based on parallel residual attention mechanism, characterized in that, Comprise: S11: processing 3D abdominal medical images into 2D gray images containing liver and liver tumors, including window processing; S12: preprocessing the cropped picture, including gray scale transformation and histogram equalization; S13: data enhancement of the preprocessed image, in order to improve the generalization ability of the model, so the data set needs to be transformed to enhance the image and expand the data set; S14: a new network framework is proposed: Res-ECA-U-NeXt, which is composed of U-shaped network structure of encoding and decoding structure, in the encoding process, a lot of information features are lost, in order to solve this problem, a parallel feature extraction module is proposed, large convolution kernel is used to extract main features, small convolution kernel is used to extract detailed features, main features and detailed features are fused, more scale receptive field features are extracted to retain a lot of detailed information, it contains attention module and residual structure, spatial channel attention module ECA is added in the encoder module, the attention module is to use full connection layer FC in SE to learn channel structure, and 1*1 convolution is used to learn channel to obtain attention information, 1*1 convolution is used to capture information between different channels, avoid channel dimension reduction when learning channel attention information, use a non-dimensional local cross-channel interaction strategy, effectively avoid the influence of dimension reduction on channel attention learning effect, so as to solve the algorithm in an iteration, keep the end-to-end characteristics of U-Net, at the same time, get more detailed liver and liver tumor segmentation boundary, on this basis, through the jump type connection, different levels of semantic information can be fused in the same scale, so that the encoder can extract more useful features, the shallow level feature can make the original structure information of the image be saved to the greatest extent, and the deeper information contains more abstract structure, the combination of the two can help to effectively restore the spatial dimension and structure details of the object in the final convolution channel, so as to improve the segmentation accuracy; S15: transmitting the liver and liver tumor segmentation image to be segmented to the segmentation model for segmentation to obtain the liver and liver tumor segmentation result.

2. The method of claim 1, wherein the method is based on a parallel residual attention mechanism. The S12 comprises: S21: first, format conversion of the data set, including cropping 3D abdominal medical images in Z-axis direction to 2D data set, and removing the slice without data value, and processing the 3D Hu value image in nii format into gray scale image; S22: after getting the gray scale image, the method of limited contrast adaptive histogram equalization is used to enhance the contrast of the obtained gray scale image, increase the differentiation degree of liver and liver tumor and background blood vessels, this method can also suppress part of the background noise and enhance the local contrast to highlight the liver and liver tumor structure; S23: Adjust the light intensity of the input 2D image by using a nonlinear Gamma transformation, and perform nonlinear operation on the intensity value, so that the intensity values of the input and output images form an exponential relationship: The purpose is to enhance the darker part of the image while ensuring that the brighter part is not affected, which can further increase the contrast of the image, reduce the interference of noise factors, improve the network segmentation capability and model generalization capability, in addition, the Gamma transformation can also reduce the interference of noise factors.

3. The method of claim 1, wherein the parallel residual attention mechanism based liver and liver tumor segmentation method is characterized by, The S13 comprises: S31: Data augmentation is performed on the original image, and various rigid and elastic transformations are adopted, including scaling, rotation, horizontal and vertical flipping, and B-spline elastic deformation. In order to reduce the influence of overfitting and prevent some local optic disc edge region pixels from deviating, random cropping is performed on the data set. In order to facilitate feature extraction of the deep network model, the cropping size is uniformly set to 512*512 dpi.

4. The method of claim 1, wherein the parallel residual attention mechanism based liver and liver tumor segmentation method is characterized by, The S14 comprises: S41: Network building 1. Res-ECA-U-NeXt is an improved network based on traditional U-Net network, which is an end-to-end multi-label deep neural network. In the encoding process, residual modules and attention mechanisms are added, and multi-scale feature fusion modules are added to the first and second layers of the network for feature fusion, reducing information loss in the feature extraction process and fusing low-resolution features with high-resolution features to extract more features under a large scale receptive field and retain a large amount of detailed information. First, the low-resolution feature mapping is restored to high-resolution size by deconvolution, and then added to the high-resolution feature mapping, as shown in equation (4.1): …………… (4.1) Then, the integrated feature map is input into the BottleNeck module to improve the feature extraction capability and avoid gradient disappearance. An ECA attention module is embedded in the encoding, which has excellent performance and efficient computing overhead because it only needs fewer parameters and computing resources to achieve higher accuracy results. By compressing the spatial features of the input feature map and using global average pooling to obtain a 1*1*C feature map, 1*1 convolution is used to learn the channel features. Specifically, this method uses channel attention mechanism to learn the importance between different channels to achieve effective feature extraction. Finally, the channel attention feature map is multiplied by the original input feature map to obtain a feature map with channel attention. In addition, this method also uses a skip connection to fuse semantic information of different levels at the same scale, so that the encoder can better extract more useful features, and the solution of the algorithm is obtained in an iteration to maintain the end-to-end characteristics of U-Net. This can get more detailed segmentation boundaries. Due to the abstraction and compression of information, it may have some impact on the segmentation results. In order to solve this problem, the skip connection of the up-sampling in U-Net is retained, and the output of the current layer is spliced with the output of the corresponding down-sampling layer. This can retain more original detail information, which is beneficial to improve the accuracy and robustness of the model, thereby improving the accuracy of segmentation. S42: Network model data training According to the network built in S41, the data set after preprocessing and data augmentation is selected for training on the segmentation network. The loss between the network segmentation result and the validation set is used to guide the training of the network learning, and the best liver and liver tumor segmentation effect diagram is obtained.

Images