A liver tumor segmentation method

Abstract

The application provides a liver tumor segmentation method, and belongs to the field of medical image segmentation. In view of the starting point of how to improve the global context feature extraction ability and how to efficiently combine with local information, two parallel encoders are used, wherein a VGG convolutional neural network branch is introduced to extract local features, and an axial decomposition self-attention branch is designed to extract global features, and then a shared residual fusion decoder is used to effectively integrate the information of the two branches. While reducing the model calculation complexity and reducing the training parameter amount, the application can significantly improve the segmentation effect of liver tumors and reduce the false segmentation or misclassification phenomenon.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for segmenting liver tumors. Background Technology

[0002] The liver is a vital metabolic organ and a high-risk area for cancer. Computed tomography (CT) scans are frequently used for the examination, clinical diagnosis, and surgical reference of liver diseases. Traditional clinical imaging diagnosis relies heavily on manual annotation by physicians. With the increasing frequency and volume of imaging scans in recent years, the workload of clinical experts has also increased significantly. Subjective diagnosis under conditions of overwork carries the potential risk of missed diagnoses and misdiagnoses.

[0003] Current main techniques still primarily rely on U-shaped encoder-decoder structures and their improved variants based on convolutional neural networks (CNNs). However, due to the limited receptive field size of CNNs, their inductive bias restricts the ability of convolutional kernels to integrate long-range dependencies in feature maps. Although numerous improvements have been made, such as introducing dilated convolutions to maximize the receptive field, adding dense connections to enhance information interaction between different levels, and adding spatial pyramid structures or multiple attention mechanisms to model global information at higher-level feature outputs, the improvements achieved by these techniques remain limited. In actual liver tumor segmentation, undersegmentation or oversegmentation / undersegmentation still occur. Furthermore, excessive insertion of additional modules or the introduction of too many dense connections can lead to excessive parameters or the generation of redundant information and invalid features.

[0004] In 2017, the self-attention mechanism, or Transformer, achieved revolutionary success in the field of natural language processing. In 2020, this technology was first transferred to visual classification models, resulting in significant performance improvements. Since then, hybrid segmentation models based on convolutional neural networks and self-attention, as well as pure visual self-attention models, have developed rapidly. However, they also suffer from drawbacks such as high computational complexity, huge training overhead, and difficulty in convergence. Especially for medical images, whose data volume is much smaller than that of natural image datasets, applying them to some large-scale self-attention models may lead to problems such as overfitting and reliance on pre-trained weight loading. Summary of the Invention

[0005] To address the limitations of existing technologies in long-range relationship modeling, high computational complexity, and difficulty in convergence, this invention proposes a liver tumor segmentation method based on bi-branch parallel coding and residual fusion decoding.

[0006] First, given the inherent inductive bias of convolutional neural networks, this invention proposes a dual-branch encoding structure that combines global and local features to improve the ability to model long-range relational dependencies. Second, to effectively combine global and local features and accelerate training convergence, a residual fusion encoding strategy is proposed. Then, addressing the computational overhead of Transformers, the global feature encoding branch uses an axial decomposition self-attention mechanism, decomposing the single self-attention calculation within the entire 2D feature map into two cascaded self-attention calculations along the vertical / horizontal axes of the feature map, i.e., a 1D range.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0008] (1) Obtain the original format liver CT image slices and the gold standard corresponding to the liver CT image slices, and preprocess the liver CT slice data and the gold standard data respectively; wherein, the preprocessing of the liver CT slice data includes numerical adjustment and normalization of the liver CT slice data; the preprocessing of the gold standard data includes segmentation label ground truth extraction and encoding conversion of the gold standard data.

[0009] (1a) The system reads the original nii format CT image volume, acquires each CT slice along the cross-sectional direction of the tomographic scan, and saves the slice array as an npy image format;

[0010] (1b) Use the window truncation method to set the CT values ​​of npy format slices. Set the CT values ​​less than or equal to the first threshold to the first threshold, set the CT values ​​greater than or equal to the second threshold to the second threshold, and keep the CT values ​​greater than the first threshold and less than the second threshold unchanged.

[0011] (1c) Normalize the CT slice data using the minimum-maximum normalization method, mapping all data values ​​to the range [0, 1]:

[0012]

[0013] (1d) The system reads the gold standard in the original nii format, i.e., the label mask data, obtains the true value of the segmentation label corresponding to each CT slice along the cross-sectional dimension of the tomographic scan, and converts it into npy format; then, it uses one-hot encoding to convert the single-channel multi-class label map into a multi-channel binary label map.

[0014] (2) Randomly sample the preprocessed data and divide it into training set, validation set and test set according to a preset ratio;

[0015] (3) Construct a liver tumor segmentation model based on dual-branch parallel coding and residual fusion decoding. The model includes a convolutional neural network branch for extracting local features and an axial decomposition self-attention calculation branch for extracting global context features, as well as a residual fusion decoder shared by the two branches.

[0016] (3a) The convolutional neural network branch for extracting local features is composed of convolutional neural network modules, wherein the first layer is composed of VGG convolutional modules, and the subsequent other layers are input features that are first downsampled by max pooling layers and then input into VGG convolutional modules to extract features.

[0017] (3b) The axial decomposition self-attention calculation branch for extracting global context features consists of a convolutional neural network module and an axial decomposition self-attention module. The first layer consists of cascaded convolutional modules, and the remaining layers are constructed using axial decomposition self-attention calculation modules. The axial decomposition self-attention module consists of cascaded vertical axis self-attention calculations and horizontal axis self-attention calculations. For a feature map of size m×m, the self-attention calculation is performed along the vertical axis of the feature map.

[0018]

[0019] Self-attention calculation along the horizontal axis of the feature map:

[0020]

[0021] in and The self-attention calculation ranges are the vertical and horizontal axes containing the feature pixel o; q o k i v i It is the query vector, key vector, and value vector obtained by linear embedding the channel vector where the feature pixel o is located; , , These correspond to q. o k i v i The location encoding vector is used to enhance location information;

[0022] (3c) The residual fusion decoder with dual branches shared by the two coding branches performs a concatenation operation on the channel dimension between the feature map transmitted from the current stacked layer of the two coding branches through skip connections and the feature map output by the previous level decoder and upsampled. The concatenation is then input into a convolutional layer with residual connections to decode semantic information. The output of the decoder is a 1×1 convolutional layer that compresses the number of channels in the feature map. The compressed channels correspond to liver segmentation prediction and tumor segmentation prediction, respectively.

[0023] (4) Construct the loss function, determine the optimizer and hyperparameters, input the training set and validation set, and train the weight parameters of the liver tumor segmentation model;

[0024] (4a) Use a weighted combination of the Dice similarity coefficient loss function and the binary cross-entropy loss function to construct the joint loss function L=α*L BCE +β*L Dice Where α and β are weights, L BCE It is a binary cross-entropy loss function, L Dice The Dice loss function is:

[0025]

[0026]

[0027] Where N is the total number of pixels in the predicted image / label image, X is the predicted image, Y is the label image, and x is the number of pixels in the label image. i To predict the pixel value of the i-th pixel for the model, y i This is the pixel value corresponding to the i-th pixel of the actual label;

[0028] (4b) Using the Adam optimizer, set the initial learning rate, learning rate decay strategy, batch size, weight decay, total number of training rounds, and initial value of the early stopping strategy;

[0029] (4c) Train the liver tumor segmentation model and optimize the hyperparameters on the training set and validation set using the backpropagation algorithm until the loss function basically converges, thus completing the training;

[0030] (5) Load the weight parameters obtained from the above training onto the constructed segmentation model, input the liver CT test set data into the model, and perform thresholding on the output confidence map to obtain the segmented liver and liver tumor results.

[0031] (5a) The output confidence map is a two-channel confidence map. A third threshold is set. Pixel values ​​greater than the third threshold are set to 1, and those less than or equal to the third threshold are set to 0.

[0032] (5b) Given an empty three-channel color image, index all pixels with a value of 1 in the first channel and record their positions. Assign specific values ​​to all corresponding (R,G,B) values ​​in the empty color image to color the corresponding pixels. Index all pixels with a value of 1 in the second channel and record their positions. Assign specific values ​​to all corresponding (R,G,B) values ​​in the empty color image to color the corresponding pixels.

[0033] (5c) Save and output the assigned color image to obtain the final segmentation prediction.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] Compared with existing U-shaped encoder-decoder structures and their various variant models, this invention can better integrate global contextual information of feature maps and effectively combine it with local information;

[0036] Compared with Transformer-based medical image segmentation models proposed in recent years, the model of this invention achieves parameter convergence stability without loading pre-trained weights during training, making it easier to train and achieving more accurate liver tumor segmentation with fewer trainable parameters and lower computational overhead. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0038] Figure 2 This is a schematic diagram of the segmentation model framework of the present invention;

[0039] Figure 3 These are example diagrams of self-attention calculation across the entire feature map and example diagrams of self-attention calculation along the vertical / horizontal axes of the feature map.

[0040] Figure 4 This is a block diagram of the computational unit for the axial self-attention mechanism;

[0041] Figure 5 This is a schematic diagram of the liver tumor segmentation network structure proposed in this invention;

[0042] Figure 6 This is an example diagram showing the segmentation effect of the method of the present invention on a test set divided based on the LiTS dataset. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the following embodiments and accompanying drawings are provided for further detailed explanation of the present invention.

[0044] Figure 1 This is a flowchart illustrating the implementation of an embodiment of the present invention, such as... Figure 1 As shown, the liver tumor segmentation method in this example includes the following steps:

[0045] (1) Obtain the original format liver CT image slices and the gold standard corresponding to the liver CT image slices, and preprocess the liver CT slice data and the gold standard data respectively; wherein, the preprocessing of the liver CT slice data includes numerical adjustment and normalization of the liver CT slice data; the preprocessing of the gold standard data includes segmentation label ground truth extraction and encoding conversion of the gold standard data.

[0046] (1a) The system reads the original nii format CT image volume, uses a loop function to iterate along the cross-sectional direction of the tomographic scan to obtain each CT slice, and uses the SimpleItk function based on Python 3.8 to save the slice array as an npy image format; In this embodiment, the data used are all from the public dataset LiTS2017, each slice has a resolution of 512×512 and occupies one channel, that is, each slice is a single-channel 512×512 grayscale image;

[0047] (1b) Using the window truncation method, the first threshold is the minimum threshold taken by the window truncation method, preferably -160, and the second threshold is the maximum threshold taken by the window truncation method, preferably 240; negative values ​​of CT values ​​outside the range of [-160, 240] in the npy format slice are set to -160, positive values ​​outside the range are set to 240, and the remaining values ​​are left unchanged;

[0048] (1c) Normalize the CT slice data using the minimum-maximum normalization method, mapping all data values ​​to the range [0, 1]:

[0049]

[0050] The current value is the pixel value of the CT slice obtained after the window truncation operation;

[0051] (1d) The system reads the gold standard in the original nii format, i.e., the label mask data, and uses a loop function to iteratively obtain the ground truth value of the segmentation label corresponding to each CT slice along the cross-sectional direction of the tomographic scan. Then, it uses the SimpleItk library function based on Python 3.8 to save the slice array as an npy image format. Then, it uses one-hot encoding to convert the single-channel multi-class label image into a multi-channel binary label image. Specifically, the original multi-class label mask of the LiTS2017 dataset is a 512×512 single-channel image G with values ​​[0,1,2], where "0" represents the pixel set corresponding to the background and "1" represents the pixel set corresponding to the liver. The pixel set, where "2" represents the pixel set corresponding to the liver tumor, is implemented as follows: An empty two-channel 512×512 array M (i.e., an array of all "0"s) is created. The positions of all pixels with a value of "1" in the original multi-class label mask [x1, y1] are indexed, and the corresponding pixels in the first channel of the two-channel empty array are all assigned the value 1, i.e., M[0, x1, y1] = 1. Similarly, the positions of all pixels with a value of "2" in the original multi-class label mask [x2, y2] are indexed, and the corresponding pixels in the second channel of the two-channel empty array are all assigned the value 1, i.e., M[1, x2, y2] = 1.

[0052] (2) Randomly sample the preprocessed data and divide it into training set, validation set and test set according to a preset ratio;

[0053] The preprocessed data was randomly divided in an 8:1:1 ratio using the train_test_split() function of the sklearn library based on Python 3.8, resulting in 15364 images as the training set, 1921 images as the validation set, and 1921 images as the test set.

[0054] (3) Construct a liver tumor segmentation model based on dual-branch parallel coding and residual fusion decoding. The model includes a convolutional neural network branch for extracting local features and an axial decomposition self-attention calculation branch for extracting global context features, as well as a residual fusion decoder shared by the two branches.

[0055] A brief framework of the segmentation model is attached. Figure 2 As shown, the specific model structure used in this embodiment is as follows:

[0056] (3a) A convolutional neural network branch for extracting local features is stacked with 5 convolutional neural network layers. The first layer consists of VGG convolutional modules. The second to fifth layers are where the input features are first downsampled by a max pooling layer (pooling window size 2×2) and then input into the VGG convolutional module to extract features; see attached. Figure 5 As shown, the input data batch size is 2, i.e., B=2, the image resolution is 512×512, i.e., H=512, and the output feature map dimensions of each layer of the local branch are (2,32,512,512), (2,64,256,256), (2,128,128,128), (2,256,64,64), (2,512,32,32);

[0057] (3b) The axial decomposition self-attention calculation branch for extracting global context features is stacked with 5 network layers. The first layer consists of three cascaded convolutional layers with kernel sizes of 7×7 (stride 2, padding 3), 3×3 (stride 1, padding 1), and 3×3 (stride 1, padding 1). The second to fifth layers are self-attention calculation modules using axial decomposition. The third to fifth layers each contain a downsampling layer based on average pooling (pooling window size 2×2). The axial decomposition self-attention layer consists of cascaded vertical and horizontal self-attention calculations. A comparison of the schematic diagrams of self-attention calculations in the feature map global range and those in the axial range is attached. Figure 3 As shown; where, for a feature map of size m×m, the self-attention is calculated along the range of the feature map's vertical axis:

[0058]

[0059] Self-attention calculation along the horizontal axis of the feature map:

[0060]

[0061] A more intuitive diagram of the axial self-attention calculation mechanism is attached. Figure 4 As shown; where and The self-attention calculation ranges are the vertical and horizontal axes containing the feature pixel o; q o k i v i It is the query vector, key vector, and value vector obtained by linear embedding the channel vector where the feature pixel o is located; , , These correspond to q. o k i v i The location encoding vector is used to enhance location information; such as Figure 5 As shown, the dimensions of the output feature maps of each layer of the global branch are (2,8,256,256), (2,32,256,256), (2,64,128,128), (2,128,64,64), and (2,256,32,32).

[0062] (3c) The residual fusion decoder concatenates the feature maps from the current stacked layers of the two coding branches via skip connections with the upsampled feature map output from the previous decoder in the channel dimension, and inputs it into the convolutional layer with residual connections to decode semantic information; the intuitive residual fusion decoder module and fusion concatenation steps are attached. Figure 5 As shown;

[0063] (3d) Perform a 1×1 convolution on the output of the 5-layer stacked decoder to compress the number of feature map channels to 2, with the 2 channels corresponding to liver segmentation prediction and tumor segmentation prediction, respectively;

[0064] (4) Construct the loss function, determine the optimizer and hyperparameters, input the training set and validation set, and train the weight parameters of the liver tumor segmentation model;

[0065] (4a) Using a weighted combination of the Dice similarity coefficient loss function and the binary cross-entropy loss function, with α and β set to 0.5 and 1 respectively, a joint loss function L=0.5*L is constructed. BCE +L Dice ; where L BCE It is a binary cross-entropy loss function, L Dice The Dice loss function is:

[0066]

[0067]

[0068] N is the total number of pixels in the predicted image / label image, X is the predicted image, and Y is the label image. For the 512×512 resolution image used in this embodiment, N is 262144; x i The pixel value predicted by the model, ranging from [0,1], y i This is the pixel value corresponding to the actual label after one-hot encoding, and its value can be 0 or 1;

[0069] (4b) The Adam optimizer was used, with an initial learning rate of 2e-4, i.e., 0.0002. A fixed step learning rate decay strategy was adopted, with the step size set to [60, 80, 100, 120] and the decay factor of 0.5, i.e., the step learning rates were [0.0002, 0.0001, 0.00005, 0.000025, 0.000013]. The batch size was set to 2, the weight decay was set to 5e-5, i.e., 0.00005, the total number of training epochs was set to 150, and the early stopping strategy was set to 30 epochs. The software framework used to train the network in this experiment was PyTorch 1.7.1 based on Python 3.8, the operating system was Ubuntu 20.0.4, the CUDA version was 11.3, and the hardware computing platform was a server equipped with an Nvidia RTX A40 graphics card.

[0070] (4c) Train the liver tumor segmentation model and optimize the hyperparameters on the training set and validation set using the backpropagation algorithm until the loss function basically converges, thus completing the training;

[0071] (5) Load the weight parameters obtained from the above training onto the constructed segmentation model, input the liver CT test set data into the model, and perform thresholding on the output confidence map to obtain the segmented liver and liver tumor results.

[0072] (5a) Load the trained parameters into the model, input the test set, obtain the two-channel confidence map, and set the threshold to 0.5. Pixel values ​​greater than 0.5 are set to 1, and those less than or equal to 0.5 are set to 0.

[0073] (5b) Given an empty three-channel color image Y, index all pixels with a value of 1 in the first channel and record their positions [x1, y1]. Set the corresponding (R, G, B) values ​​of all positions in the empty color image to (0, 128, 0), i.e., Y[0, x1, y1] = 0, Y[1, x1, y1] = 128, Y[2, x1, y1] = 0, to color the corresponding pixels green. Index all pixels with a value of 1 in the second channel and record their positions [x2, y2]. Set the corresponding (R, G, B) values ​​of all positions in the empty color image to (255, 0, 0), i.e., Y[0, x2, y2] = 255, Y[1, x2, y2] = 0, Y[2, x2, y2] = 0, to color the corresponding pixels red.

[0074] (5c) Save and output the assigned color image to obtain the final segmentation prediction; the prediction effect of this invention is shown in the appendix. Figure 6 As shown.

[0075] The technical means disclosed in this invention are not limited to those disclosed in the above-described embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that any variations or substitutions, improvements, and modifications that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention.

Claims

1. A method for segmenting liver tumors, characterized in that, The method includes the following steps: S1. Obtain the original format liver CT image slices and the corresponding gold standard for the liver CT image slices, and preprocess the liver CT slice data and the gold standard data respectively; wherein, the preprocessing of the liver CT slice data includes numerical adjustment and normalization of the liver CT slice data; the preprocessing of the gold standard data includes segmentation label ground truth extraction and encoding conversion of the gold standard data. S2. Randomly sample the preprocessed data and divide it into training set, validation set and test set according to a preset ratio; S3. Construct a liver tumor segmentation model based on dual-branch parallel coding and residual fusion decoding. The model includes a convolutional neural network branch for extracting local features and an axial decomposition self-attention calculation branch for extracting global context features, as well as a residual fusion decoder shared by the two branches. S4. Construct the loss function, determine the optimizer and hyperparameters, input the training set and validation set, and train the weight parameters of the liver tumor segmentation model; S5. Load the weight parameters obtained from the above training onto the constructed segmentation model, input the liver CT test set data into the model, and perform thresholding on the output confidence map to obtain the segmented liver and liver tumor results.

2. The liver tumor segmentation method according to claim 1, characterized in that, S1 involves acquiring raw format liver CT image slices and the corresponding gold standard for the liver CT image slices, and performing data preprocessing, including: The system reads the original nii format CT image volume, acquires each CT slice along the cross-sectional direction of the tomographic scan, and saves the slice array as an npy image format; The CT values ​​of .npy format slices are set using the window truncation method. CT values ​​less than or equal to the first threshold are uniformly set to the first threshold, CT values ​​greater than or equal to the second threshold are uniformly set to the second threshold, and CT values ​​greater than the first threshold and less than the second threshold remain unchanged. The CT slice data were normalized using the minimum-maximum normalization method, mapping all data values ​​to the range [0, 1]. The system reads the gold standard in the original nii format, i.e., the label mask data, obtains the ground truth value of the segmentation label corresponding to each CT slice along the cross-sectional dimension of the tomographic scan, and converts it into .npy format; then, it uses one-hot encoding to convert the single-channel multi-class label map into a multi-channel binary label map.

3. The liver tumor segmentation method according to claim 1, characterized in that, The construction of S3 is based on a liver tumor segmentation model with dual-branch parallel encoding and residual fusion decoding. The model includes a convolutional neural network branch for extracting local features and an axial decomposition self-attention calculation branch for extracting global context features, as well as a residual fusion decoder shared by the two branches. The specific components are as follows: The convolutional neural network branch for extracting local features is composed of convolutional neural network modules. The first layer is composed of VGG convolutional modules, and the subsequent layers are formed by inputting features through a max pooling layer for downsampling before inputting them into the VGG convolutional modules to extract features. The axial decomposition self-attention calculation branch for extracting global context features consists of a convolutional neural network module and an axial decomposition self-attention module. The first layer comprises cascaded convolutional modules, while the remaining layers use axial decomposition self-attention calculation modules. These modules consist of cascaded vertical and horizontal self-attention calculations. For an m×m feature map, the self-attention calculation is performed along the vertical axis of the feature map. Self-attention calculation along the horizontal axis of the feature map: in and The self-attention calculation ranges are the vertical and horizontal axes containing the feature pixel o; q o k i v i It is the query vector, key vector, and value vector obtained by linear embedding the channel vector where the feature pixel o is located; , , These correspond to q. o k i v i The location encoding vector is used to enhance location information; The dual-branch shared residual fusion decoder concatenates the feature maps from the current stacked layers of the two coding branches via skip connections with the upsampled feature maps output from the previous decoder in the channel dimension, and inputs them into a convolutional layer with residual connections to decode semantic information. The decoder output is a 1×1 convolutional layer that compresses the number of channels in the feature map. The compressed channels correspond to liver segmentation prediction and tumor segmentation prediction, respectively.

4. The liver tumor segmentation method according to claim 1, characterized in that, The process of constructing the loss function in S4, determining the optimizer and hyperparameters, inputting the training set and validation set, and training the weight parameters of the liver tumor segmentation model includes the following: A joint loss function L=α*L is constructed by weighting the Dice similarity coefficient loss function and the binary cross-entropy loss function. BCE + β*L Dice Where α and β are weights, L BCE It is a binary cross-entropy loss function, L Dice The Dice loss function is: Where N is the total number of pixels in the predicted image / label image, X is the predicted image, Y is the label image, and x is the number of pixels in the label image. i To predict the pixel value of the i-th pixel for the model, y i This is the pixel value corresponding to the i-th pixel of the actual label; The Adam optimizer is used, and the initial learning rate, learning rate decay strategy, batch size, weight decay, total number of training epochs, and initial value of early stopping strategy are set. The liver tumor segmentation model was trained and hyperparameters were optimized using the backpropagation algorithm on the training and validation sets until the loss function basically converged, thus completing the training.

5. The liver tumor segmentation method according to claim 1, characterized in that, The thresholding process of the output confidence graph in S5 includes: The output confidence map is a two-channel confidence map. A third threshold is set, with pixel values ​​greater than the third threshold set to 1 and those less than or equal to the third threshold set to 0. It also includes: given an empty three-channel color image, indexing all pixels with a value of 1 in the first channel and recording their positions, assigning specific values ​​to all corresponding (R, G, B) values ​​in the empty color image to color the corresponding pixels; indexing all pixels with a value of 1 in the second channel and recording their positions, assigning specific values ​​to all corresponding (R, G, B) values ​​in the empty color image to color the corresponding pixels. The assigned color image is saved and output to obtain the final segmentation prediction.

Images