Multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path

Abstract

The invention relates to the field of liver cancer computed tomography images, and discloses a multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path, comprising the following steps: S1, inputting 3D CT image data, and preprocessing the input 3D CT image data; s2, performing feature extraction by using 3D convolutional layers, performing maximum pooling operation after each down-sampling layer to reduce the spatial dimension of the feature map, each down-sampling layer including a series of 3D convolutional layers, activation functions and pooling operation, and each down-sampling layer including one or more residual blocks; and S3, performing multi-scale convolution operation, adding multi-scale 3D convolution into the shallow Unet, and performing convolution operation by using convolution kernels of different sizes. By introducing a residual path and a multi-scale 3D convolution operation, the RMCUNet can better capture hepatocellular carcinoma tumor region characteristics of different scales, especially small scales, and cope with tumor segmentation challenges of complex shapes, and for small and more complex-shaped HCC tumor samples.

Description

Technical Field

[0001] The present invention relates to the technical field of liver cancer computer tomography images, and in particular to a multi-scale hepatocellular carcinoma segmentation network method based on attention and residual paths. Background Art

[0002] Hepatocellular carcinoma (HCC) is one of the most common malignant tumors worldwide, and early diagnosis and treatment are crucial to improving patient survival. Computed tomography (CT) is a commonly used medical imaging technique for detecting and evaluating liver tumors. However, liver cancer tumors have significant differences in size, shape, and location in CT images, which poses a challenge to the neural network model to accurately segment the hepatocellular carcinoma region in abdominal CT images.

[0003] Currently, most studies use a two-stage approach, first segmenting the liver region from abdominal CT images, and then further segmenting liver cancer lesions. Although this method improves the accuracy of segmentation to a certain extent, it also increases computational complexity and time cost. In addition, the two-stage method often does not work well when dealing with situations where the tumor boundary is blurred or the tumor is highly similar to the surrounding tissue. Therefore, to solve this problem, we propose a network based on attention, 3D multi-scale convolution, and residual path, RMCUNet, for direct segmentation of hepatocellular carcinoma in 3D CT images. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a multi-scale hepatocellular carcinoma segmentation network method based on attention and residual paths, aiming to improve the accuracy and robustness of hepatocellular carcinoma lesions segmentation performance.

[0005] To achieve the above objectives, the present invention is implemented by the following technical solutions: a multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path, characterized in that it includes the following steps:

[0006] S1. Input 3D CT image data and preprocess the input 3D CT image data;

[0007] S2. Use 3D convolutional layers for feature extraction. After each downsampling layer, a maximum pooling operation is performed to reduce the spatial dimension of the feature map. Each downsampling layer contains a series of 3D convolutional layers, activation functions, and pooling operations, and contains one or more residual blocks in each downsampling layer.

[0008] S3. Multi-scale convolution operation, add multi-scale 3D convolution to the shallow Unet, use convolution kernels of different sizes to perform convolution operations, obtain multiple feature maps, splice these feature maps in the channel dimension, and fuse them through a 3D convolution layer to generate the final multi-scale feature map.

[0009] Preferably, in step S1, preprocessing includes operations such as normalization, cropping and enhancement to improve image quality.

[0010] Preferably, in step S2, the residual block introduces a short-circuit connection, that is, the input feature can directly reach the output layer.

[0011] Preferably, in step S2, the residual is calculated by the input vector and the expected result, xl represents the input vector of the lth layer, H(xl) is the expected output, and F represents the residual. The mapping result is shown as follows:

[0012] F(x l )=H(x l )-x l .

[0013] Preferably, in step S2, in each residual block, the input features may be transformed through one or more convolutional layers and then added to the original input through a short-circuit connection.

[0014] Preferably, in step S2, a 3D convolution layer is added to each residual connection, and the convolution layer is added to form a residual path to extract more liver cancer area features, and finally the extracted features are added to the features extracted by the encoder itself.

[0015] Preferably, in step S3, multi-scale 3D convolution is added to the shallow Unet, and features are extracted from different scales through the multi-scale 3D convolution, which combines convolution operations with multiple different receptive fields, thereby enhancing the model's recognition ability for complex spatial structures.

[0016] Preferably, the convolutional attention module can extract feature maps along both the channel and spatial dimensions at the same time. The output calculation formula of the channel attention module is as follows: C (F)=σ(Conv(γ(Conv(AvgPool(F))))+Conv(γ(Conv(MaxPool(F))))))

[0017] Where F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, Conv represents the convolution operation, γ represents the ReLU activation function, and σ represents the Sigmoid activation function.

[0018] Preferably, the output calculation formula of the spatial attention module is as follows:

[0019] M 5 (F)=σ(Comv(ArgPool(F);MaxPool(F)))

[0020] Where Conv represents the convolution operation, and (AvgPool(F); MaxPool(F)) means concatenating the average pooling and maximum pooling results along the channel axis.

[0021] The present invention provides a multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path. It has the following beneficial effects:

[0022] 1. By introducing residual paths and multi-scale 3D convolution operations, the present invention's RMC_UNet can better capture the regional characteristics of hepatocellular carcinoma tumors of different scales, especially small scales, and cope with the challenges of tumor segmentation with complex shapes. For smaller and more complex HCC tumor samples, RMC_UNet has higher recognition ability and more complete segmentation effect compared with other models such as Unet and Vnet, which effectively improves the problem of difficult segmentation of small-scale and complex-shaped tumors.

[0023] 2. By embedding the 3D-CBAM module, the present invention enhances the adaptive refinement capability of features in RMC_UNet not only in the channel dimension but also in the spatial dimension, which helps the model achieve better generalization performance when dealing with problems with large differences in position and shape in liver cancer data. This module, combined with the effectiveness of multi-scale 3D convolution, can extract and fuse valuable features from multiple angles, thereby improving the overall segmentation performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall model structure diagram of the present invention;

[0025] Figure 2 A schematic diagram of the deep convolutional neural network architecture of the present invention;

[0026] Figure 3 is a residual path schematic diagram of the present invention;

[0027] Figure 4 Schematic diagram of multi-scale convolution of the present invention;

[0028] Figure 5 Schematic diagram of the convolutional block attention module of the present invention;

[0029] Figure 6 It is a schematic diagram of a channel module of the present invention;

[0030] Figure 7 It is a schematic diagram of a space module of the present invention;

[0031] Figure 8 It is a schematic diagram of the segmentation result of a large-scale HCC tumor of the present invention;

[0032] Fig. 9 It is a schematic diagram of the segmentation result of the mesoscale HCC tumor of the present invention;

[0033] Fig.10 It is a schematic diagram of the segmentation result of a small-scale HCC tumor of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Example:

[0036] See also Figure 1 The embodiment of the present invention provides a multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path, comprising the following steps:

[0037] S1 inputs 3D CT image data and preprocesses the input 3D CT image data, including normalization, cropping, enhancement and other operations, to improve image quality.

[0038] S2. In the feature extraction stage, the preprocessed 3D CT image data is processed using a 3D convolution layer to extract representative features. After each downsampling layer, a maximum pooling operation is performed to reduce the spatial dimension of the feature map. Each downsampling layer contains a series of 3D convolution layers, activation functions, and pooling operations, and each downsampling layer contains one or more residual blocks. The schematic diagram of the residual block is shown in Figure 2 As shown in Figure 1, a short-circuit connection is introduced, that is, the input feature can directly reach the output layer. The residual can be calculated by the input vector and the expected result. xl represents the input vector of the lth layer, H(xl) is the expected output, and F represents the residual. The mapping result is shown in the formula:

[0039] F(x l )=H(x l )-x l

[0040] The residual block is to allow the model to learn the residual function (i.e., F(xl)) instead of directly learning the entire H(xl). When the objective function H(xl) is close to the identity mapping, the gradient vanishing or gradient exploding problem caused by the increase in network depth is avoided. In each residual block, the input features can be transformed through one or more convolutional layers and then added to the original input through a skip connection. This process allows information to be directly transmitted across multiple layers, which helps train deep networks.

[0041] like Figure 3 As shown, a 3D convolution layer is added to each residual connection, and the convolution layer is added to form a residual path to extract more liver cancer area features. Finally, the extracted features are added to the features extracted by the encoder itself.

[0042] S3. Multi-scale convolution operations, such as Figure 4 As shown in the figure, in each downsampling layer, convolution kernels of different sizes (3×3×3, 5×5×5, 7×7×7) are used for convolution operations to obtain multiple feature maps, which are concatenated in the channel dimension and fused through a 3D convolution layer to generate the final multi-scale feature map. Multi-scale 3D convolution is a convolution operation that captures features of different scales in three-dimensional space. We add multi-scale 3D convolution to the shallow Unet. Multi-scale 3D convolution combines convolution operations with multiple different receptive fields to extract features from different scales, thereby enhancing the model's ability to recognize complex spatial structures. In this way, the shallow network can be strengthened for the recognition of cancer region features of different scales to complete liver cancer segmentation. The convolution attention module can extract feature maps along both the channel and space dimensions at the same time. The attention weight is multiplied by the input feature for adaptive feature refinement to obtain the output feature, as shown in the figure. Figure 5 and Figure 6 , Figure 7 As shown in the figure, this model transforms the general 2D convolutional attention module into 3D, showing the overall structure of 3D-CBAM and describing the corresponding sub-modules.

[0043] Among them, the channel attention module (CAM) compresses the input features in the spatial dimension. First, 3D average pooling (3D-Avgpool) and 3D maximum pooling (3D-Maxpool) are used to extract the input features, and then the two features are sent to the convolution layer with shared weights for channel interaction. The output mapping is merged on the spatial axis to obtain enhanced channel features, which are activated by the sigmoid function to generate channel weights Mc and then multiplied with the original input to obtain the channel attention output features. The output calculation formula of the channel attention module is as follows:

[0044] M C (F)=σ(Conv(γ(Conv(AvgPool(F))))+Conv(γ(Conv(MaxPool(F))))))

[0045] Where F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, Conv represents the convolution operation, γ represents the ReLU activation function, and σ represents the Sigmoid activation function.

[0046] The spatial attention module (SAM) compresses the input features in the channel dimension. Two spatial features are generated through average pooling and maximum pooling operations and concatenated along the channel dimension. The output map is reduced to the same channel dimension as the original input channel by 3D convolution, and then activated by the sigmoid function to obtain the spatial weight Ms. The weight is then multiplied by the original input feature to obtain the output of the spatial attention module. The output calculation formula of the spatial attention module is as follows:

[0047] M 5 (F)=σ(Comv(ArgPool(F);MaxPool(F)))

[0048] Where Conv represents the convolution operation, and (AvgPool(F); MaxPool(F)) means concatenating the average pooling and maximum pooling results along the channel axis.

[0049] The proposed model embeds the 3D-CBAM module into the encoding part of the model to solve the problem of large position and shape differences in liver cancer data.

[0050] In order to verify the segmentation performance of the proposed method, we conducted a comprehensive and integrated evaluation and compared this method with the Unet and Vnet methods on the test set. The effectiveness of the proposed method was verified on the test set. Abdominal CT images of different shapes and scales were randomly selected from the prediction results for display. The prediction results were divided into three groups according to the scale of HCC: large scale, medium scale, and small scale. Figure 8 Showing the segmentation results of large-scale HCC tumors, Fig. 9 Showing the segmentation results of mesoscale HCC tumors, Fig.10 Shows the segmentation results of small-scale HCC tumors.

[0051] Through the segmentation results of different methods, it can be concluded that all methods can identify the specific locations of hepatocellular carcinoma areas of different scales and shapes in CT images. When segmenting HCC tumors of different scales, the segmentation results of different models are different. Most models can segment large-area HCC tumor areas in abdominal CT images well. However, for some HCC tumor samples with smaller scales and more complex shapes, the proposed network model has better recognition ability and can better adapt to changes in different scales and shapes to achieve complete segmentation of HCC tumors.

[0052] In order to quantitatively analyze the segmentation performance of our method, four image segmentation evaluation indicators were used: DSC (DiceSimilarity Coefficient), HD (Hausdorff Distance), ASD (Average Surface Distance) and Jaccard coefficient, and compared with other methods on the test set. The test results are shown in Table 1.

[0053]

[0054]

[0055] Table 1 Experimental results of the model compared with other baseline models

[0056] The HD value of the RMC_UNet proposed in this paper is 12.0760, which is the smallest compared to the selected baseline model. This means that the maximum difference between the model prediction result and the actual result is small, and the model has high accuracy and precision. At the same time, the ASD is 2.9533, which is lower than other models, reflecting that the model has a smaller error in surface distance and more accurate capture of the target boundary. The DSC value is high, which shows that the model has a certain guarantee in the accuracy of segmentation of the target area. The JCC value is the largest compared to other models, and its comprehensive performance after combining multiple modules is good.

[0057] In order to analyze the effectiveness of Res (representing residual path), CBAM, and 3DMSC in RMC_UNet, we performed the following ablation experiments on the model using the test set. The results of each module are shown in Table 1.

[0058] 1. Effectiveness of residual path: After adding residual path to Unet, all evaluation indicators are improved compared with Unet.

[0059] 2. Effectiveness of 3D CBAM: After adding 3D CBAM to Unet with residual path, the overall effect of the model is further improved.

[0060] 3. Effectiveness of 3DMSC: By observing the results of RMC_UNet, the HD and ASD coefficients decrease significantly when the DSC coefficient decreases slightly. This shows that the segmentation effect of RMC_UNet is improved after adding multi-scale convolution and 3D CBAM to Unet.

[0061] According to the experimental results, the RMC_UNet proposed by us shows a trend of gradual improvement or optimization improvement in different indicators compared with other models, demonstrating the effectiveness of our RMC_UNet. At the same time, the experimental results show that our RMC_UNet can better solve the problem of large differences in size, shape, and location of liver cancer.

[0062] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path, characterized in that: The following steps are involved: S1. Input 3D CT image data and preprocess the input 3D CT image data; S2. Use 3D convolutional layers for feature extraction. After each downsampling layer, a maximum pooling operation is performed to reduce the spatial dimension of the feature map. Each downsampling layer contains a series of 3D convolutional layers, activation functions, and pooling operations, and contains one or more residual blocks in each downsampling layer. S3. Multi-scale convolution operation, add multi-scale 3D convolution to the shallow Unet, use convolution kernels of different sizes to perform convolution operations, obtain multiple feature maps, splice these feature maps in the channel dimension, and fuse them through a 3D convolution layer to generate the final multi-scale feature map.

2. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: In step S1, preprocessing includes normalization, cropping, enhancement and other operations to improve image quality.

3. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: In step S2, the residual block introduces a short-circuit connection, that is, the input feature can directly reach the output layer.

4. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: In step S2, the residual is calculated by the input vector and the expected result, xl represents the input vector of the lth layer, H(xl) is the expected output, and F represents the residual. The mapping result is shown as follows: F(x l )=H(x l )-x l 。 5. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: In step S2, in each residual block, the input features may be transformed through one or more convolutional layers and then added to the original input through a short-circuit connection.

6. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: In step S2, a 3D convolution layer is added to each residual connection, and the convolution layer is added to form a residual path to extract more liver cancer area features, and finally the extracted features are added to the features extracted by the encoder itself.

7. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: In step S3, multi-scale 3D convolution is added to the shallow Unet. The multi-scale 3D convolution combines convolution operations with multiple different receptive fields to extract features from different scales, thereby enhancing the model's ability to recognize complex spatial structures.

8. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 1, characterized in that: The convolutional attention module can extract feature maps along both the channel and spatial dimensions at the same time. The output calculation formula of the channel attention module is as follows: M C (F)=σ(Conv(γ(Conv(AvgPool(F))))+Conv(γ(Conv(MaxPool(F))))))) Where F is the input feature map, AvgPool and MaxPool represent global average pooling and maximum pooling operations respectively, Conv represents the convolution operation, γ represents the ReLU activation function, and σ represents the Sigmoid activation function.

9. The multi-scale hepatocellular carcinoma segmentation network method based on attention and residual path according to claim 8, characterized in that: The output calculation formula of the spatial attention module is as follows: M5(F)=σ(Comv(ArgPool(F);MaxPool(F))) Where Conv represents the convolution operation, and (AvgPool(F); MaxPool(F)) means concatenating the average pooling and maximum pooling results along the channel axis.

Images