A liver and tumor three-dimensional image segmentation method and system
By introducing multi-scale feature fusion module, deformable attention module and dynamic channel attention module into the network model, the problems of structural complexity and ambiguity in automatic segmentation of liver and tumors are solved, and more efficient and accurate liver is achieved Tumor segmentation improves the robustness and accuracy of segmentation tasks.
Patent Information
- Application Number
- CN202411230405.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The prior art has structural complexity and ambiguity in the automatic segmentation of liver and tumors, resulting in poor generalization performance and the inability to accurately segment different tumor cases. The fixed receptive field of ordinary convolutional nuclei limits feature extraction and global information. expressive ability.
A three-dimensional image segmentation method for liver and tumors is processed through network models, and a multi-scale feature fusion module, a deformable attention module and a dynamic channel attention module are introduced to enhance the network's ability to capture features of different scales and attention mechanisms. Effectiveness, adaptive treatment of irregular morphology and complex boundaries of tumors.
It significantly improves the accuracy and robustness of liver tumor segmentation, reduces feature redundancy, improves the limitations of traditional methods, and improves the accuracy of liver tumor segmentation tasks and the generalization ability of the model.
Smart Images

Figure CN118736235B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and system for segmenting three-dimensional images of liver and tumor. Background Art
[0002] Liver cancer has a high incidence rate and a low survival rate, which requires early detection and effective treatment. Current diagnostic methods rely heavily on manual evaluation, which not only consumes valuable time, but also has individual differences due to the expertise of medical professionals. Therefore, advanced automated tumor segmentation methods are urgently needed in the clinic. Accurate segmentation of liver tumors from CT scans can avoid subjective differences, improve diagnostic efficiency, reduce clinical workload, and promote the diagnosis and treatment of liver cancer. However, the automatic segmentation of the liver and tumors is challenging due to the following problems: 1) Complex structure. Due to the possibility of metastasis and spread, the number, size, and shape of liver tumors in patients vary, and the location of liver tumors at different stages also varies. This characteristic of liver tumors makes it difficult for traditional segmentation methods to extract common features, resulting in poor generalization performance and inability to accurately segment different tumor cases; 2) Blurred boundaries. The low contrast between the tumor area and the surrounding tissues leads to blurred tumor boundaries in CT images. Therefore, it is difficult to accurately identify the tumor area from normal liver tissue.
[0003] To alleviate the above problems, some studies have used convolutional neural networks (CNNs) to automatically extract tumor features from CT scans, thereby improving the accuracy of tumor segmentation compared to earlier methods. Although these methods have achieved promising results, there are still some problems that need to be addressed. On the one hand, the fixed receptive field of ordinary convolution kernels hinders its ability to effectively represent features and extract global information. Therefore, it still has great limitations in dealing with liver tumors of different sizes, numbers, and shapes. On the other hand, the morphological knowledge of the segmented objects, such as shape and boundaries, is still not fully utilized, which makes it difficult to match the extracted features with fuzzy boundaries, resulting in unsatisfactory segmentation boundaries.
[0004] Therefore, the present invention proposes a liver and tumor three-dimensional image segmentation method and system to solve the above problems. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention develops a method and system for liver and tumor three-dimensional image segmentation. The invention processes the input CT images through a network model, which can better adapt to the irregular morphology and complex boundaries of the tumor, achieve efficient and accurate segmentation effects, and improve the accuracy and robustness of the liver tumor segmentation task.
[0006] On the one hand, the technical solution to the technical problem of the present invention is a method for segmenting liver and tumor three-dimensional images, comprising the following steps:
[0007] S1. Collect CT images of liver-related cases to create a data set, preprocess the image data in the data set, and then divide the preprocessed data set into a training set and a test set;
[0008] S2. Create a network model for image segmentation, input the image data in the training set into the network module for processing, and obtain the predicted segmented image ;
[0009] S3. Optimize and train the network model, and then input the test set into the optimized and trained network model to obtain the final segmented image. .
[0010] Step S1 is specifically as follows:
[0011] The dataset was preprocessed for data enhancement. The preprocessing operations included rotation, scaling, adding Gaussian noise, Gaussian blur, brightness adjustment, contrast adjustment, gamma adjustment, up and down and left and right mirroring. The CT images in the dataset were cut into fixed sizes to make the format of the image data in the dataset consistent. The preprocessed dataset was then randomly divided into training sets at a ratio of 4:1. and test set ,in, , , represents any CT image in the training set, represents any CT image in the test set, Indicates the training set CT images, Indicates the test set CT images, and .
[0012] S2.1. Create an end-to-end network model to jointly segment the liver and tumor in CT images. The network model includes an encoder, a decoder, a deformable attention module DeAtt, a dynamic channel attention module DyAtt, and five multi-scale feature fusion modules MSFF.
[0013] S2.2. The training set Images in Input to the encoder for feature extraction. The encoder has six layers. First, each input image is extracted through the first layer of the encoder. Downsampling is performed to obtain the feature map of the first layer output, and then the feature map of the first layer output is used as the input of the second layer, and downsampling operation is performed again to obtain the feature map of the second layer output. The above operation is repeated many times, and the feature map of the previous layer output is used as the input of the next layer, and downsampling operation is performed on it to obtain the feature map of the corresponding output of this layer, and finally each image is obtained. The feature map corresponding to the output of each layer of the encoder , , Indicates the encoder Feature map of layer output;
[0014] S2.3, the five multi-scale feature fusion modules MSFF are respectively the first multi-scale feature fusion module MSFF1, the second multi-scale feature fusion module MSFF2, the third multi-scale feature fusion module MSFF3, the fourth multi-scale feature fusion module MSFF4 and the fifth multi-scale feature fusion module MSFF5. and Input to the first multi-scale feature fusion module, the feature map and Input to the second multi-scale feature fusion module, the feature map and Input to the third multi-scale feature fusion module, the feature map and Input to the fourth multi-scale feature fusion module, the feature map and Input to the fifth multi-scale feature fusion module;
[0015] The feature maps of the two layers and feature map Input to the multi-scale feature fusion module MSFF to enhance channel dependency and establish channel attention. , the feature map of the previous layer After average pooling, maximum pooling, Pooling and fully connected layers are used to enhance channel dependency and establish channel attention, and the output is the feature map , the calculation process is as follows:
[0016] ,
[0017] in, represents the average pooling operation, represents the maximum pooling operation, Indicates based on Weighted pooling operation, Represents the fully connected layer operation;
[0018] Then, the feature map of the next layer After upsampling, it is refined by multiplication and compared with the feature map of the previous layer. and feature map Concatenate to get the feature map , the calculation process is as follows:
[0019] ,
[0020] in, Represents a splicing operation, represents the upsampling operation, Indicates The feature map output by the multi-scale feature fusion module.
[0021] S2.4. The feature map output by the fourth multi-scale feature fusion module Input to the deformable attention module DeAtt;
[0022] S2.4.1. Feature Map First, the query of the attention mechanism is obtained through the convolution operation , and then the query of the attention mechanism Input to the offset generation network to get the offset ;
[0023] Among them, the offset generation network includes the first 3D convolution layer, Activation function, second 3D convolution layer, Activation function and Layer, query of attention mechanism First, the convolution operation is performed through the first 3D convolution layer, and then through The activation function increases nonlinearity, and then passes through a second 3D convolutional layer to reduce the number of channels to 3 to correspond to the offset in the x, y, and z directions in 3D space, and then uses The activation function limits it to [-1, 1], and finally passes The layer scales the offset to adapt to different scale changes. The specific calculation formula is as follows:
[0024] ,
[0025] ,
[0026] in, Express The weight matrix for linear transformation, represents the generation offset network, represents the convolution operation of the first 3D convolutional layer, Represents the activation function Operation, represents the convolution operation of the second 3D convolutional layer, Represents the activation function Operation, express The scaling operation performed by the layer;
[0027] S2.4.2. Feature Map and offset Perform a biphase interpolation operation to obtain the offset feature map , the calculation formula is as follows:
[0028] ,
[0029] in, Represents a biphase interpolation operation;
[0030] S2.4.3. Offset feature map Perform convolution operation to get the key of attention mechanism Sum , the calculation formula is as follows:
[0031] ,
[0032] ,
[0033] in, Express The weight matrix for linear transformation, Express The weight matrix for linear transformation;
[0034] S2.4.4. Query by Computational Attention Mechanism and key The relative position between them is the relative position offset , and then offset by relative position Get the attention score , and finally the attention score With feature map Multiply them together to get the final output feature map of the DeAtt module , the calculation formula is as follows:
[0035] ,
[0036] ,
[0037] ,
[0038] in, represents the operation of a multilayer perceptron, represents the operation of the sine function, Representation feature map The coordinate grid of Represents the offset feature map The coordinate grid of Represents the activation function Operation, represents the attention score, Represents a transpose operation.
[0039] S2.5. The feature map output by the fifth multi-scale feature fusion module Input to the dynamic channel attention module DyAtt;
[0040] S2.5.1. Feature map of the fifth MSFF module output Perform global maximum pooling and global average pooling operations in the spatial dimension respectively to obtain feature maps and feature map , the calculation formula is as follows:
[0041] ,
[0042] ,
[0043] in, represents the maximum pooling operation, represents the global average pooling operation;
[0044] S2.5.2. Feature Map and feature map They are input into the shared multi-layer perceptron Shared MLP for learning, and feature maps are obtained respectively. and feature map , and then the feature map and feature map Add after activation function Processing to obtain the channel attention weight matrix , and then the channel attention weight matrix and Multiply to get the enhanced feature map , the specific calculation formula is as follows:
[0045] ,
[0046] ,
[0047] ,
[0048] ,
[0049] in, Represents the activation function Operation, and Represent the first layer weight matrix and the second layer weight matrix respectively, and represents two different bias vectors, Represents the activation function Operation, Represents element-wise multiplication;
[0050] S2.5.3. Enhanced feature map The 3D attention module Attention3D in the DyAtt module is input to extract features in the spatial dimension. Attention3D includes an average pooling layer, a first fully connected layer, an activation function ReLU, a second fully connected layer, and a Softmax activation function.
[0051] Enhanced feature map First, it is downsampled through the average pooling layer, and then through the first fully connected layer and activation function , maps the input feature map to a higher dimensional space, and then further extracts features through the second fully connected layer, and finally The activation function converts the output of the second fully connected layer into the convolution kernel attention weights corresponding to K convolutions. The calculation process is as follows:
[0052] ,
[0053] in, represents the average pooling layer, represents the first fully connected layer, Represents the activation function Operation, represents the second fully connected layer, Represents the activation function Operation, Indicates The convolution kernel attention weight value of the convolution, ;
[0054] The convolution kernels are then weighted and summed using the attention weights, and then the weighted convolution kernels are used to enhance the feature map. Perform convolution operation and finally get the feature map , the calculation formula is as follows:
[0055] ,
[0056] in, represents a three-dimensional convolution operation, Indicates convolution kernels, represents the number of convolutions, Indicates The weight of the convolution kernel, Represents the feature map of the final output of the DyAtt module.
[0057] S2.6, input the final feature map of each part into the decoder;
[0058] S2.6.1. Feature map output by the first multi-scale feature fusion module , the feature map output by the second multi-scale feature fusion module , the feature map output by the third multi-scale feature fusion module , feature map output by the deformable attention module DeAtt , feature map output by the dynamic channel attention module DyAtt And the feature map output by the sixth layer of the encoder Perform denoising, normalization, and resizing operations;
[0059] S2.6.2. The processed feature map is skipped and input into the decoder. The decoder has six layers. Input to the sixth layer of the decoder to get the feature map , and then starting from the fifth layer to the first layer, each layer of the decoder accepts the jump connection output from the encoder and the decoded output of the previous layer for fusion and upsampling:
[0060] The feature map and feature map The fifth layer of the decoder is input and fused and upsampled to obtain the feature map ;
[0061] Then the feature map and feature map The input is sent to the fourth layer of the decoder, and the feature map is obtained after fusion and upsampling. ;
[0062] Then the feature map and feature map The input is sent to the third layer of the decoder and the feature map is obtained after fusion and upsampling. ;
[0063] Then the feature map and feature map The input is sent to the second layer of the decoder and the feature map is obtained after fusion and upsampling. ;
[0064] Then the feature map and feature map The first layer of the input decoder is fused and upsampled to obtain the feature map ;
[0065] The calculation process is as follows:
[0066] ,
[0067] ,
[0068] ,
[0069] ,
[0070] ,
[0071] ,
[0072] in, represents the operation of the fifth layer of the decoder, represents the operation of the fourth layer of the decoder, represents the operation of the third layer of the decoder, represents the operation of the second layer of the decoder, Represents the operation of the first layer of the decoder;
[0073] S2.6.3. Feature Map Through a 1×1×1 convolutional layer The number of feature channels is reduced to 3, which refer to the background, liver and tumor categories respectively, and then The activation function generates the predicted segmentation image , the calculation process is as follows:
[0074] .
[0075] On the other hand, the present invention also provides a liver and tumor three-dimensional image segmentation system, including a module for executing processing instructions for each step in a liver and tumor three-dimensional image segmentation method.
[0076] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:
[0077] The present invention discloses a method and system for segmenting three-dimensional images of liver and tumors, which significantly improves the segmentation performance by introducing a multi-scale feature fusion module, a deformable attention module and a dynamic attention module into the network structure. The multi-scale feature fusion module is added to the jump connections of the first, second and third layers to enhance the network's ability to capture features of different scales; the multi-scale feature fusion module and the deformable attention module are sequentially added to the jump connections of the fourth layer, so that the network can adaptively focus on important areas of the tumor; the multi-scale feature fusion module and the dynamic attention module are sequentially added to the jump connections of the fifth layer, further enhancing the effectiveness of feature expression and attention mechanisms. Therefore, the present invention can flexibly adjust the shape and position of the convolution kernel to better adapt to the irregular morphology and complex boundaries of the tumor, and combine channel information with spatial information. By dynamically adjusting the attention weight and integrating feature information from different channels, the representation and generalization capabilities of the model are enhanced, feature redundancy is reduced, and feature information of different scales is integrated, which improves the limitations of traditional methods and improves the accuracy and robustness of liver tumor segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0079] Figure 1 It is a schematic diagram of the method flow of the present invention.
[0080] Figure 2 It is a structural diagram of the deformable attention module DeAtt in the present invention.
[0081] Figure 3 This is a structural diagram of the dynamic channel attention module DyAtt in the present invention.
[0082] Figure 4 It is a structural diagram of the multi-scale feature fusion module MSFF in the present invention.
[0083] Figure 5 Three segmentation examples of liver and tumor on the LiTS dataset of the present invention. DETAILED DESCRIPTION
[0084] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below.
[0085] Example 1
[0086] like Figures 1 to 4 As shown, a liver and tumor three-dimensional image segmentation method comprises the following steps:
[0087] S1. Collect CT images of liver-related cases to create a data set, preprocess the image data in the data set, and then divide the preprocessed data set into a training set and a test set;
[0088] S2. Create a network model for image segmentation, input the image data in the training set into the network module for processing, and obtain the predicted segmented image ;
[0089] S3. Optimize and train the network model, and then input the test set into the optimized and trained network model to obtain the final segmented image. .
[0090] Step S1 is specifically as follows:
[0091] The dataset was preprocessed for data enhancement. The preprocessing operations included rotation, scaling, adding Gaussian noise, Gaussian blur, brightness adjustment, contrast adjustment, gamma adjustment, up and down and left and right mirroring. The CT images in the dataset were cut into fixed sizes to make the format of the image data in the dataset consistent. The preprocessed dataset was then randomly divided into training sets at a ratio of 4:1. and test set ,in, , , represents any CT image in the training set, represents any CT image in the test set, Indicates the training set CT images, Indicates the test set CT images, and .
[0092] S2.1. Create an end-to-end network model to jointly segment the liver and tumor in CT images. The network model includes an encoder, a decoder, a deformable attention module DeAtt, a dynamic channel attention module DyAtt, and five multi-scale feature fusion modules MSFF.
[0093] S2.2. The training set Images in Input to the encoder for feature extraction. The encoder has six layers. First, each input image is extracted through the first layer of the encoder. Downsampling is performed to obtain the feature map of the first layer output, and then the feature map of the first layer output is used as the input of the second layer, and downsampling operation is performed again to obtain the feature map of the second layer output. The above operation is repeated many times, and the feature map of the previous layer output is used as the input of the next layer, and downsampling operation is performed on it to obtain the feature map of the corresponding output of this layer, and finally each image is obtained. The feature map corresponding to the output of each layer of the encoder , , Indicates the encoder Feature map of layer output;
[0094] S2.3, the five multi-scale feature fusion modules MSFF are respectively the first multi-scale feature fusion module MSFF1, the second multi-scale feature fusion module MSFF2, the third multi-scale feature fusion module MSFF3, the fourth multi-scale feature fusion module MSFF4 and the fifth multi-scale feature fusion module MSFF5. and Input to the first multi-scale feature fusion module, the feature map and Input to the second multi-scale feature fusion module, the feature map and Input to the third multi-scale feature fusion module, the feature map and Input to the fourth multi-scale feature fusion module, the feature map and Input to the fifth multi-scale feature fusion module;
[0095] The feature maps of the two layers and feature map Input to the multi-scale feature fusion module MSFF to enhance channel dependency and establish channel attention. , the feature map of the previous layer After average pooling, maximum pooling, Pooling and fully connected layers are used to enhance channel dependency and establish channel attention, and the output is the feature map , the calculation process is as follows:
[0096] ,
[0097] in, represents the average pooling operation, represents the maximum pooling operation, Indicates based on Weighted pooling operation, Represents the fully connected layer operation;
[0098] Then, the feature map of the next layer After upsampling, it is refined by multiplication and compared with the feature map of the previous layer. and feature map Concatenate to get the feature map , the calculation process is as follows:
[0099] ,
[0100] in, Represents a splicing operation, represents the upsampling operation, Indicates The feature map output by the multi-scale feature fusion module.
[0101] S2.4. The feature map output by the fourth multi-scale feature fusion module Input to the deformable attention module DeAtt;
[0102] S2.4.1. Feature Map First, the query of the attention mechanism is obtained through the convolution operation , and then the query of the attention mechanism Input to the offset generation network to get the offset ;
[0103] Among them, the offset generation network includes the first 3D convolution layer, Activation function, second 3D convolution layer, Activation function and Layer, query of attention mechanism First, the convolution operation is performed through the first 3D convolution layer, and then through The activation function increases nonlinearity, and then passes through a second 3D convolutional layer to reduce the number of channels to 3 to correspond to the offset in the x, y, and z directions in 3D space, and then uses The activation function limits it to [-1, 1], and finally passes The layer scales the offset to adapt to different scale changes. The specific calculation formula is as follows:
[0104] ,
[0105] ,
[0106] in, Express The weight matrix for linear transformation, represents the generation offset network, represents the convolution operation of the first 3D convolutional layer, Represents the activation function Operation, represents the convolution operation of the second 3D convolutional layer, Represents the activation function Operation, express The scaling operation performed by the layer;
[0107] S2.4.2. Feature Map and offset Perform a biphase interpolation operation to obtain the offset feature map , the calculation formula is as follows:
[0108] ,
[0109] in, Represents a biphase interpolation operation;
[0110] S2.4.3. Offset feature map Perform convolution operation to get the key of attention mechanism Sum , the calculation formula is as follows:
[0111] ,
[0112] ,
[0113] in, Express The weight matrix for linear transformation, Express The weight matrix for linear transformation;
[0114] S2.4.4. Query by Computational Attention Mechanism and key The relative position between them is the relative position offset , and then offset by relative position Get the attention score , and finally the attention score With feature map Multiply them together to get the final output feature map of the DeAtt module , the calculation formula is as follows:
[0115] ,
[0116] ,
[0117] ,
[0118] in, represents the operation of a multilayer perceptron, represents the operation of the sine function, Representation feature map The coordinate grid of Represents the offset feature map The coordinate grid of Represents the activation function Operation, represents the attention score, Represents a transpose operation.
[0119] S2.5. The feature map output by the fifth multi-scale feature fusion module Input to the dynamic channel attention module DyAtt;
[0120] S2.5.1. Feature map of the fifth MSFF module output Perform global maximum pooling and global average pooling operations in the spatial dimension respectively to obtain feature maps and feature map , the calculation formula is as follows:
[0121] ,
[0122] ,
[0123] in, represents the maximum pooling operation, represents the global average pooling operation;
[0124] S2.5.2. Feature Map and feature map They are input into the shared multi-layer perceptron Shared MLP for learning, and feature maps are obtained respectively. and feature map , and then the feature map and feature map Add after activation function Processing to obtain the channel attention weight matrix , and then the channel attention weight matrix and Multiply to get the enhanced feature map , the specific calculation formula is as follows:
[0125] ,
[0126] ,
[0127] ,
[0128] ,
[0129] in, Represents the activation function Operation, and Represent the first layer weight matrix and the second layer weight matrix respectively, and represents two different bias vectors, Represents the activation function Operation, Represents element-wise multiplication;
[0130] S2.5.3. Enhanced feature map The 3D attention module Attention3D in the DyAtt module is input to extract features in the spatial dimension. Attention3D includes an average pooling layer, a first fully connected layer, an activation function ReLU, a second fully connected layer, and a Softmax activation function.
[0131] Enhanced feature map First, it is downsampled through the average pooling layer, and then through the first fully connected layer and activation function , maps the input feature map to a higher dimensional space, and then further extracts features through the second fully connected layer, and finally The activation function converts the output of the second fully connected layer into the convolution kernel attention weights corresponding to K convolutions. The calculation process is as follows:
[0132] ,
[0133] in, represents the average pooling layer, represents the first fully connected layer, Represents the activation function Operation, represents the second fully connected layer, Represents the activation function Operation, Indicates The convolution kernel attention weight value of the convolution, ;
[0134] The convolution kernels are then weighted and summed using the attention weights, and then the weighted convolution kernels are used to enhance the feature map. Perform convolution operation and finally get the feature map , the calculation formula is as follows:
[0135] ,
[0136] in, represents a three-dimensional convolution operation, Indicates convolution kernels, represents the number of convolutions, Indicates The weight of the convolution kernel, Represents the feature map of the final output of the DyAtt module.
[0137] S2.6, input the final feature map of each part into the decoder;
[0138] S2.6.1. Feature map output by the first multi-scale feature fusion module , the feature map output by the second multi-scale feature fusion module , the feature map output by the third multi-scale feature fusion module , feature map output by the deformable attention module DeAtt , feature map output by the dynamic channel attention module DyAtt And the feature map output by the sixth layer of the encoder Perform denoising, normalization, and resizing operations;
[0139] S2.6.2. The processed feature map is skipped and input into the decoder. The decoder has six layers. Input to the sixth layer of the decoder to get the feature map , and then starting from the fifth layer to the first layer, each layer of the decoder accepts the jump connection output from the encoder and the decoded output of the previous layer for fusion and upsampling:
[0140] The feature map and feature map The fifth layer of the decoder is input and fused and upsampled to obtain the feature map ;
[0141] Then the feature map and feature map The input is sent to the fourth layer of the decoder, and the feature map is obtained after fusion and upsampling. ;
[0142] Then the feature map and feature map The input is sent to the third layer of the decoder and the feature map is obtained after fusion and upsampling. ;
[0143] Then the feature map and feature map The input is sent to the second layer of the decoder and the feature map is obtained after fusion and upsampling. ;
[0144] Then the feature map and feature map The first layer of the input decoder is fused and upsampled to obtain the feature map ;
[0145] The calculation process is as follows:
[0146] ,
[0147] ,
[0148] ,
[0149] ,
[0150] ,
[0151] ,
[0152] in, represents the operation of the fifth layer of the decoder, represents the operation of the fourth layer of the decoder, represents the operation of the third layer of the decoder, represents the operation of the second layer of the decoder, Represents the operation of the first layer of the decoder;
[0153] S2.6.3. Feature Map Through a 1×1×1 convolutional layer The number of feature channels is reduced to 3, which refer to the background, liver and tumor categories respectively, and then The activation function generates the predicted segmentation image , the calculation process is as follows:
[0154] .
[0155] Example 2
[0156] A liver and tumor three-dimensional image segmentation system comprises a module for executing processing instructions of each step in a liver and tumor three-dimensional image segmentation method.
[0157] Example 3
[0158] Figure 5 Figure 1 shows three examples of joint segmentation of liver and tumor on the LiTS dataset (Example 1, Example 2, Example 3). From left to right, they are the original CT image, CT image label, and segmentation result. The liver area is marked in red, and the tumor area is marked in green. DSC1 and DSC2 represent the Dice coefficients of the liver and tumor, respectively. Figure 5 Judging from the presented results, the network model of the present invention performs satisfactorily in automatic liver segmentation, showing strong stability and consistency, and a low mis-segmentation rate. The network model of the present invention has high segmentation accuracy for the liver and can accurately identify and segment the liver area. For the automatic segmentation of tumors, the model also shows good performance and can effectively detect and segment tumor areas. This shows that although the segmentation of tumors is more challenging, the network model of the present invention can still provide reliable segmentation results and show high sensitivity and accuracy when detecting tumor areas.
[0159] Example 4
[0160] The present invention achieved a Dice value of 0.961 in the liver segmentation task and a Dice value of 0.690 in the liver tumor segmentation task, proving that the technical solution can effectively improve the segmentation accuracy and robustness of three-dimensional medical images.
[0161] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.
Claims
1. A method for segmenting liver and tumor three-dimensional images, characterized in that: The following steps are involved: S1. Collect CT images of liver-related cases to create a dataset, preprocess the image data in the dataset, and then divide the preprocessed dataset into a training set and test set , , represents any CT image in the training set, Indicates the training set CT images; S2. Create a network model for image segmentation, input the image data in the training set into the network module for processing, and obtain the predicted segmented image ; Step S2 is specifically as follows: S2.
1. Create an end-to-end network model to jointly segment the liver and tumor in CT images. The network model includes an encoder, a decoder, a deformable attention module DeAtt, a dynamic channel attention module DyAtt, and five multi-scale feature fusion modules MSFF. S2.
2. The training set Images in Input to the encoder for feature extraction. The encoder has six layers. First, each input image is extracted through the first layer of the encoder. Downsampling is performed to obtain the feature map of the first layer output, and then the feature map of the first layer output is used as the input of the second layer, and downsampling operation is performed again to obtain the feature map of the second layer output. The above operation is repeated many times, and the feature map of the previous layer output is used as the input of the next layer, and downsampling operation is performed on it to obtain the feature map of the corresponding output of this layer, and finally each image is obtained. The feature map corresponding to the output of each layer of the encoder , , Indicates the encoder Feature map of layer output; S2.3, the five multi-scale feature fusion modules MSFF are respectively the first multi-scale feature fusion module MSFF1, the second multi-scale feature fusion module MSFF2, the third multi-scale feature fusion module MSFF3, the fourth multi-scale feature fusion module MSFF4 and the fifth multi-scale feature fusion module MSFF5. and Input to the first multi-scale feature fusion module, the feature map and Input to the second multi-scale feature fusion module, the feature map and Input to the third multi-scale feature fusion module, the feature map and Input to the fourth multi-scale feature fusion module, the feature map and Input to the fifth multi-scale feature fusion module; The feature maps of the two layers and feature map Input to the multi-scale feature fusion module MSFF to enhance channel dependency and establish channel attention. , the feature map of the previous layer After average pooling, maximum pooling, Pooling and fully connected layers are used to enhance channel dependency and establish channel attention, and the output is the feature map , the calculation process is as follows: , in, represents the average pooling operation, represents the maximum pooling operation, Indicates based on Weighted pooling operation, Represents the fully connected layer operation; Then, the feature map of the next layer After upsampling, it is refined by multiplication and compared with the feature map of the previous layer. and feature map Concatenate to get the feature map , the calculation process is as follows: , in, Represents a splicing operation, represents the upsampling operation, Indicates The feature map output by the multi-scale feature fusion module; S2.
4. The feature map output by the fourth multi-scale feature fusion module Input to the deformable attention module DeAtt; S2.4.
1. Feature Map First, the query of the attention mechanism is obtained through the convolution operation , and then the query of the attention mechanism Input to the offset generation network to get the offset ; Among them, the offset generation network includes the first 3D convolution layer, Activation function, second 3D convolution layer, Activation function and Layer, query of attention mechanism First, the convolution operation is performed through the first 3D convolution layer, and then through The activation function increases nonlinearity, and then passes through a second 3D convolutional layer to reduce the number of channels to 3 to correspond to the offset in the x, y, and z directions in 3D space, and then uses The activation function limits it to [-1, 1], and finally passes The layer scales the offset to adapt to different scale changes. The specific calculation formula is as follows: , , in, Express The weight matrix for linear transformation, represents the generation offset network, represents the convolution operation of the first 3D convolutional layer, Represents the activation function Operation, represents the convolution operation of the second 3D convolutional layer, Represents the activation function Operation, express The scaling operation performed by the layer; S2.4.
2. Feature Map and offset Perform a biphase interpolation operation to obtain the offset feature map , the calculation formula is as follows: , in, express Biphase interpolation operation; S2.4.
3. Offset feature map Perform convolution operation to get the key of attention mechanism Sum , the calculation formula is as follows: , , in, Express The weight matrix for linear transformation, Express The weight matrix for linear transformation; S2.4.
4. Query by Computational Attention Mechanism and key The relative position between them is the relative position offset , and then offset by relative position Get the attention score , and finally the attention score With feature map Multiply them together to get the final output feature map of the DeAtt module , the calculation formula is as follows: , , , in, represents the operation of a multilayer perceptron, represents the operation of the sine function, Representation feature map The coordinate grid of Represents the offset feature map The coordinate grid of Represents the activation function Operation, represents the attention score, Represents a transpose operation; S2.
5. The feature map output by the fifth multi-scale feature fusion module Input to the dynamic channel attention module DyAtt; S2.5.
1. Feature map of the fifth MSFF module output Perform global maximum pooling and global average pooling operations in the spatial dimension respectively to obtain feature maps and feature map , the calculation formula is as follows: , , in, represents the maximum pooling operation, represents the global average pooling operation; S2.5.
2. Feature Map and feature map They are input into the shared multi-layer perceptron Shared MLP for learning, and feature maps are obtained respectively. and feature map , and then the feature map and feature map Add after activation function Processing to obtain the channel attention weight matrix , and then the channel attention weight matrix and Multiply to get the enhanced feature map , the specific calculation formula is as follows: , , , , in, Represents the activation function Operation, and Represent the first layer weight matrix and the second layer weight matrix respectively, and represents two different bias vectors, Represents the activation function Operation, Represents element-wise multiplication; S2.5.
3. Enhanced feature map The 3D attention module Attention3D in the DyAtt module is input to extract features in the spatial dimension. Attention3D includes an average pooling layer, a first fully connected layer, an activation function ReLU, a second fully connected layer, and a Softmax activation function. Enhanced feature map First, it is downsampled through the average pooling layer, and then through the first fully connected layer and activation function , maps the input feature map to a higher dimensional space, and then further extracts features through the second fully connected layer, and finally The activation function converts the output of the second fully connected layer into the convolution kernel attention weights corresponding to K convolutions. The calculation process is as follows: , in, represents the average pooling layer, represents the first fully connected layer, Represents the activation function Operation, represents the second fully connected layer, Represents the activation function Operation, Indicates The convolution kernel attention weight value of the convolution, ; The convolution kernels are then weighted and summed using the attention weights, and then the weighted convolution kernels are used to enhance the feature map. Perform convolution operation and finally get the feature map , the calculation formula is as follows: , in, represents a three-dimensional convolution operation, Indicates convolution kernels, represents the number of convolutions, Indicates The weight of the convolution kernel, Represents the feature map of the final output of the DyAtt module; S3. Optimize and train the network model, and then input the test set into the optimized and trained network model to obtain the final segmented image. .
2. The method for segmenting liver and tumor three-dimensional images according to claim 1, characterized in that: Step S1 is specifically as follows: The dataset was preprocessed for data enhancement. The preprocessing operations included rotation, scaling, adding Gaussian noise, Gaussian blur, brightness adjustment, contrast adjustment, gamma adjustment, up and down and left and right mirroring. The CT images in the dataset were cut into fixed sizes to make the format of the image data in the dataset consistent. The preprocessed dataset was then randomly divided into training sets at a ratio of 4:
1. and test set ,in, , , represents any CT image in the test set, Indicates the test set CT images, and .
3. The method for segmenting liver and tumor three-dimensional images according to claim 2, characterized in that: S2.6, input the final feature map of each part into the decoder; S2.6.
1. Feature map output by the first multi-scale feature fusion module , the feature map output by the second multi-scale feature fusion module , the feature map output by the third multi-scale feature fusion module , feature map output by the deformable attention module DeAtt , feature map output by the dynamic channel attention module DyAtt And the feature map output by the sixth layer of the encoder Perform denoising, normalization, and resizing operations; S2.6.
2. The processed feature map is skipped and input into the decoder. The decoder has six layers. Input to the sixth layer of the decoder to get the feature map , and then starting from the fifth layer to the first layer, each layer of the decoder accepts the jump connection output from the encoder and the decoded output of the previous layer for fusion and upsampling: The feature map and feature map The fifth layer of the decoder is input and fused and upsampled to obtain the feature map ; Then the feature map and feature map The input is sent to the fourth layer of the decoder, and the feature map is obtained after fusion and upsampling. ; Then the feature map and feature map The input is sent to the third layer of the decoder and the feature map is obtained after fusion and upsampling. ; Then the feature map and feature map The input is sent to the second layer of the decoder and the feature map is obtained after fusion and upsampling. ; Then the feature map and feature map The first layer of the input decoder is fused and upsampled to obtain the feature map ; The calculation process is as follows: , , , , , , in, represents the operation of the fifth layer of the decoder, represents the operation of the fourth layer of the decoder, represents the operation of the third layer of the decoder, represents the operation of the second layer of the decoder, Represents the operation of the first layer of the decoder; S2.6.
3. Feature Map Through a 1×1×1 convolutional layer The number of feature channels is reduced to 3, which refer to the background, liver and tumor categories respectively, and then The activation function generates the predicted segmentation image , the calculation process is as follows: 。 4. A liver and tumor three-dimensional image segmentation system, characterized in that: The invention comprises a module for executing processing instructions of each step in a method for segmenting three-dimensional images of liver and tumor as described in any one of claims 1 to 3.
