A liver tumor image segmentation method based on a CNN-Transformer hybrid architecture

Abstract

The application discloses a liver tumor image segmentation method based on a CNN-Transformer hybrid architecture, and comprises the following steps: pre-processing a liver CT image to reduce the influence of image acquisition differences on a model; a multi-stage hybrid encoder effectively combines a convolutional neural network and a Transformer network, extracts local details, takes into account the dependency relationship between elements, and obtains multi-scale features; a feature enhancement decoder performs up-sampling through a convolutional neural network, realizes multi-scale feature fusion through an improved skip connection, and finally outputs a segmentation probability graph with a size of 512*512*3; and a post-processing module converts the segmentation probability graph into a segmentation result that can be directly used for surgery planning. The application constructs a powerful segmentation model that can grasp local details and understand global context, improves calculation efficiency, and reduces the consumption of computing power resources.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a liver tumor image segmentation method based on a CNN-Transformer hybrid architecture. Background Technology

[0002] Currently, the main methods for detecting liver cancer are medical imaging examinations such as MRI, CT, and ultrasound. Treatment primarily involves surgical removal of the cancerous portion, along with chemotherapy and radiotherapy. As the largest internal organ in the human body, the liver plays a crucial role in metabolism, further highlighting the importance of precise surgical resection in liver cancer treatment. Achieving accurate resection requires not only the judgment and skill of a professional physician but also the assistance of scientific and technological methods. However, each CT scan generates hundreds to thousands of images. Relying on doctors to process each image individually would waste significant human and material resources and reduce diagnostic accuracy. Therefore, high-accuracy and high-efficiency image segmentation technology is needed to assist physicians in performing diagnostic work efficiently.

[0003] Image segmentation is a fundamental technology in computer vision, a crucial component of image understanding, and a key technique applied in medical image processing. It involves subdividing a digital image into multiple sub-regions, simplifying or altering the image's representation to make it easier to understand. In other words, image segmentation is about labeling each pixel in a digital image, giving pixels with the same label a common visual characteristic. With continuous innovation in computer and artificial intelligence technologies, machine learning and deep learning algorithms have been increasingly applied to medical image processing with excellent results. This has enabled fully automated segmentation of medical images, improving efficiency and accuracy compared to semi-automatic segmentation that combines manual operation with computer processing.

[0004] In recent years, U-Net-based convolutional neural networks have made significant progress in medical image segmentation, with their encoder-decoder structure effectively fusing multi-scale features. However, traditional U-Net relies entirely on convolutional neural networks (CNNs), and the local receptive field of its convolutional kernels limits the model's ability to model global contextual information of the image. When dealing with targets such as liver tumors, which have blurred boundaries, varied shapes, and complex spatial dependencies with surrounding tissues, incomplete segmentation and inaccurate boundaries often occur. The Transformer architecture has demonstrated powerful global relationship modeling capabilities in natural language processing and computer vision, but when directly applied to medical image segmentation, it suffers from problems such as loss of local details and high computational complexity. Summary of the Invention

[0005] The purpose of this invention is to provide a liver tumor image segmentation method based on a CNN-Transformer hybrid architecture, which constructs a powerful segmentation model that can capture both local details and understand the global context, thereby improving computational efficiency and reducing the consumption of computing resources.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A liver tumor image segmentation method based on a CNN-Transformer hybrid architecture includes the following steps: S1, preprocessing liver CT images to reduce the impact of image acquisition differences on the model; S2, the multi-stage hybrid encoder effectively combines convolutional neural networks and Transformer networks, extracting local details while taking into account the dependencies between various elements, and obtaining multi-scale features; S3, the feature enhancement decoder uses a convolutional neural network for upsampling and achieves multi-scale feature fusion through improved skip connections, and finally outputs a 512×512×3 segmentation probability map; S4, the post-processing module transforms the segmentation probability map into segmentation results that can be directly used for surgical planning.

[0007] Preferably, the specific steps of S2 are as follows: S201, the first stage uses a convolutional neural network to extract low-level features and obtain low-level detailed features of the tumor. S202, the second stage uses a convolutional neural network to extract intermediate features and obtain local intermediate features of the tumor; S203, the third stage uses a Transformer network to convert the features of the convolutional neural network into sequences for processing by the Transformer network, and models the dependencies at medium distances; S204, the fourth stage uses a Transformer network to extract global semantic dependencies and establish long-range spatial relationships between the tumor and the whole liver; S205 uses a Transformer network as the bridging layer to further refine global information and obtain multi-scale features.

[0008] Preferably, in step S201, the input is a 512×512×3 image, low-level features are extracted using a standard convolutional layer, and the receptive field is expanded by max pooling downsampling, resulting in a 256×256×64 feature map as the output.

[0009] Preferably, in step S202, the input is a feature map of 256×256×64, a residual convolutional block is used, and two convolutional blocks are repeated to extract intermediate features, and the output is a feature map of 128×128×128.

[0010] Preferably, the specific steps of S203 are as follows: S20301, the input is a 128×128×128 feature map, the feature map is divided into 16×16 feature blocks; S20302, Location Encoding, Add Spatial Location Information; S20303 employs a 4-layer multi-head self-attention mechanism to construct a medium-distance dependency relationship between the tumor and surrounding tissues, outputting a 64×64×256 feature map.

[0011] Preferably, in step S204, the input is a 64×64×256 feature map, which is used by the Transformer encoder to understand the anatomical structure of the whole liver, identify the distribution pattern of the tumor, and establish the potential association between the tumor and distant tissues. The output is a 32×32×512 feature map.

[0012] Preferably, in step S205, the input is a 32×32×512 feature map, which is processed by a Transformer encoder to extract the most abstract semantic information, establish global relationships between pixels, and provide rich contextual priors for the feature enhancement decoder, and outputs a 16×16×1024 feature map.

[0013] Preferably, the specific steps of S3 are as follows: the input is a feature map of 16×16×1024, the feature map is first upsampled in four stages, then the features extracted by the corresponding layer of the encoder are fused through skip connections, and finally the features are refined by convolutional layers to obtain a segmentation probability map of size 512×512×3.

[0014] Preferably, the specific steps of S4 are as follows: Conditional random field optimization of the boundary to make the segmentation result smoother; connected component analysis to remove small false positive areas; morphological operations to fill holes and smooth edges; and three-dimensional reconstruction to integrate the 2D slice results into a 3D model for volume calculation.

[0015] The beneficial effects of this invention are as follows: Addressing the technical problems of blurred boundaries and complex global dependencies in liver tumors, this invention proposes a U-Net-type encoder-decoder framework. It strategically combines convolutional neural networks (CNNs) and Transformer networks at different stages of the encoder: CNNs are used to extract local detail features in shallow stages, while Transformer networks are used to model global semantic dependencies in deeper stages. Furthermore, improved skip connections enable multi-scale feature fusion. This algorithm combines the precise local feature extraction capabilities of CNNs with the ability of Transformer networks to capture the dependencies between various elements in an image, thereby constructing a powerful segmentation model that can both capture local details and understand the global context, improving computational efficiency and reducing computational resource consumption. Attached Figure Description

[0016] Figure 1 This is the overall flowchart of the present invention.

[0017] Figure 2 This is a detailed flowchart of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0019] like Figures 1-2 As shown, a liver tumor image segmentation method based on a CNN-Transformer hybrid architecture includes the following steps: S1 preprocesses liver CT images to reduce the impact of image acquisition differences on the model.

[0020] The input layer receives raw medical image data, with the input specification being liver CT images, typically multi-phase abdominal CT scan sequences from hospitals, with a typical size of 512×512×3 (enhanced CT data), including arterial phase, portal venous phase, and delayed phase images.

[0021] The preprocessing module mainly includes image standardization and enhancement. The specific steps are as follows: First, multi-phase registration is performed to ensure spatial alignment. Second, the window width and window level are adjusted, and the display range is adjusted according to the CT value to highlight the liver tissue. Then, the pixel values ​​are normalized to intensity and mapped to the interval of [-1, 1] or [0, 1] to accelerate training convergence and reduce the impact of image acquisition differences on the model. Finally, random transformations that preserve anatomical structure are applied for data augmentation during training.

[0022] S2, the multi-stage hybrid encoder, effectively combines convolutional neural networks (CNN) and Transformer networks to extract local details while taking into account the dependencies between various elements, preserving the overall integrity, and ultimately obtaining multi-scale features.

[0023] The multi-stage hybrid encoder, employing an encoder-hybrid architecture, is the core innovation of this invention. Based on a U-Net-type encoder-decoder framework, this encoder addresses the technical challenges of blurred boundaries and complex global dependencies in liver tumors. It achieves feature extraction by strategically combining Convolutional Neural Networks (CNNs) and Transformer networks at different stages of the encoder. Its core design logic stems from the complementary strengths of both: CNNs excel at extracting local features (such as edges, textures, and shapes), possessing translation invariance and high computational efficiency, but their receptive field is limited when dealing with complex global dependencies between tumors and surrounding tissues in medical images (such as diffuse tumor growth and their positional relationship with blood vessels); while the Transformer network's core self-attention mechanism can capture long-range dependencies between all pixels (or blocks) in an image, establishing global contextual information, but lacks awareness of local details. Therefore, by effectively combining CNNs and Transformers on the basis of a U-Net-type encoder-decoder framework, it can extract local details while also considering the dependencies between elements, preserving the overall integrity of the image.

[0024] The multi-stage hybrid encoder consists of four feature extraction stages. The first two stages (shallow stages) use convolutional neural networks (CNNs) to extract local details and organ-level features at high resolution of the feature maps. The latter two stages use Transformer networks to capture global context and long-range dependencies at low resolution and high semantic level of the feature maps, enabling the network to understand the overall picture of the image and thus accurately address the technical challenges of liver tumor segmentation.

[0025] S201, Stage 1, uses a convolutional neural network (CNN) to extract low-level features and obtain low-level detailed features of the tumor.

[0026] The input is 512×512×3 (original image size). Standard convolutional layers are used to extract low-level features such as edges, corners, and textures, and max pooling downsampling is used to expand the receptive field. The output is a feature map of 256×256×64. Its design principle is to use convolutional neural networks (CNN) in the high-resolution stage to efficiently capture local detail features.

[0027] S202, the second stage (Stage 2) uses a convolutional neural network (CNN) to extract intermediate features and obtain local intermediate features of the tumor.

[0028] The input is a 256×256×64 feature map. The operation is to use residual convolution blocks and repeat the convolution blocks twice to extract more abstract features such as organ shape and tissue type. The output is a 128×128×128 feature map, which is of medical significance in that it can begin to identify the liver outline and major vascular structures.

[0029] S203, the third stage uses a Transformer network to convert the features of a convolutional neural network (CNN) into sequences for processing by the Transformer network, and models the medium-distance dependency between the tumor and surrounding tissues.

[0030] The third stage (Stage 3) is the transition and underlying semantic modeling layer. As a transition layer from CNN to Transformer, it converts the features of the convolutional neural network (CNN) into sequences for the Transformer network to process, and models the medium-distance dependencies between the tumor and surrounding tissues, such as the relationship between the tumor and surrounding blood vessels.

[0031] The specific steps are as follows: S20301, the input is a 128×128×128 feature map, the feature map is divided into blocks, the feature map is divided into 16×16 feature blocks.

[0032] The input is a 128×128×128 feature map. Patch partitioning divides the feature map into 16×16 feature blocks (patch).

[0033] S20302, Location Encoding, Add Spatial Location Information.

[0034] S20303 employs a 4-layer multi-head self-attention mechanism to construct a medium-distance dependency relationship between the tumor and surrounding tissues, outputting a 64×64×256 feature map.

[0035] The Transformer encoder employs a 4-layer multi-head self-attention mechanism. Its function is to construct the medium-distance dependencies between the tumor and surrounding tissues, such as the relationship between the tumor and surrounding blood vessels, and the output is a 64×64×256 feature map.

[0036] In S204, the fourth stage uses a Transformer network to extract global semantic dependencies and establish long-range spatial relationships between the tumor and the whole liver.

[0037] The fourth stage (Stage 4) is a high-level semantic modeling layer that uses a deep Transformer network to model global semantic dependencies and establish the long-range spatial relationship between the tumor and the whole liver.

[0038] The input is a 64×64×256 feature map. The operation involves using a deeper Transformer encoder (8 layers), which models global semantic information, including understanding the anatomical structure of the whole liver, identifying tumor distribution patterns (single, multiple, diffuse), and establishing potential associations between the tumor and distant tissues. The output is a 32×32×512 feature map. Its innovative advantage lies in its ability to better handle tumors with blurred boundaries and low contrast compared to pure convolutional neural networks (CNNs).

[0039] S205 uses a Transformer network as the bridging layer to further refine global information and obtain multi-scale features.

[0040] The bridging layer takes a 32×32×512 feature map as input and performs a further refinement of the Transformer encoder. Its function is to extract the most abstract semantic information, establish global relationships between pixels, and provide rich contextual priors for the feature enhancement decoder. The output is a 16×16×1024 feature map.

[0041] S3, the feature enhancement decoder uses a convolutional neural network (CNN) for upsampling and achieves multi-scale feature fusion through improved skip connections, and finally outputs a 512×512×3 segmentation probability map.

[0042] The feature enhancement decoder uses four symmetrical upsampling stages to restore resolution. Its core mechanism is that the skip-connect decoder module restores resolution through four symmetrical upsampling stages. The input is a 16×16×1024 feature map. The four stages first upsample the feature map, then fuse the features extracted by the corresponding layer of the encoder through skip connections, and finally refine the features with convolutional layers to obtain a segmentation probability map of size 512×512×3 (corresponding to the probabilities of background, liver parenchyma and tumor, respectively, with a sum of 1).

[0043] Connection 1 provides the features from Stage 4 to upsampling stage 1 (up1), fusing high-level semantic features; Connection 2 provides the features from Stage 3 to upsampling stage 2 (up2), fusing medium-distance dependent features; Connection 3 provides the features from Stage 2 to upsampling stage 3 (up3), fusing organ-level features (such as organ shape, tissue type, etc.); Connection 4 provides the features from Stage 1 to upsampling stage 4 (up4), fusing low-level detail features. Upsampling operations typically use transposed convolutions, which progressively restore spatial resolution and fuse multi-scale features. The output layer first performs final upsampling to restore the original image size, then uses a 1×1 convolution to map the number of channels to the number of categories (3 categories: background, liver, tumor), and then uses Softmax activation to generate the classification probability for each pixel. The final output is a 512×512×3 segmentation probability map (each pixel has 3 probability values, corresponding to the probabilities of background, liver parenchyma, and tumor, respectively, with a sum of 1).

[0044] S4, the post-processing module transforms the segmentation probability map into segmentation results that can be directly used for surgical planning.

[0045] Conditional Random Field (CRF) optimizes the boundary, resulting in smoother segmentation; connected component analysis removes small false positive regions; morphological operations fill holes and smooth edges; and 3D reconstruction integrates 2D slice results into a 3D model for volume calculation. These operations transform the segmentation probability map output by the feature-enhanced decoder into segmentation results that can be directly used for surgical planning.

Claims

1. A liver tumor image segmentation method based on a CNN-Transformer hybrid architecture, characterized in that, Includes the following steps: S1, preprocessing liver CT images to reduce the impact of image acquisition differences on the model; S2, the multi-stage hybrid encoder effectively combines convolutional neural networks and Transformer networks, extracting local details while taking into account the dependencies between various elements, and obtaining multi-scale features; S3, the feature enhancement decoder uses a convolutional neural network for upsampling and achieves multi-scale feature fusion through improved skip connections, and finally outputs a 512×512×3 segmentation probability map; S4, the post-processing module transforms the segmentation probability map into segmentation results that can be directly used for surgical planning.

2. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The specific steps of S2 are as follows: S201, the first stage uses a convolutional neural network to extract low-level features and obtain low-level detailed features of the tumor. S202, the second stage uses a convolutional neural network to extract intermediate features and obtain local intermediate features of the tumor; S203, the third stage uses a Transformer network to convert the features of the convolutional neural network into sequences for processing by the Transformer network, and models the dependencies at medium distances; S204, the fourth stage uses a Transformer network to extract global semantic dependencies and establish long-range spatial relationships between the tumor and the whole liver; S205 uses a Transformer network as the bridging layer to further refine global information and obtain multi-scale features.

3. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 2, characterized in that, In step S201, the input is a 512×512×3 image, low-level features are extracted using a standard convolutional layer, and the receptive field is expanded by max pooling downsampling, resulting in a 256×256×64 feature map as the output.

4. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 2, characterized in that, In step S202, the input is a feature map of 256×256×64, a residual convolution block is used, and two convolution blocks are repeated to extract intermediate features, and the output is a feature map of 128×128×128.

5. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 2, characterized in that, The specific steps of S203 are as follows: S20301, the input is a 128×128×128 feature map, the feature map is divided into 16×16 feature blocks; S20302, Location Encoding, Add Spatial Location Information; S20303 employs a 4-layer multi-head self-attention mechanism to construct a medium-distance dependency relationship between the tumor and surrounding tissues, outputting a 64×64×256 feature map.

6. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, In S204, the input is a 64×64×256 feature map, which is used by the Transformer encoder to understand the anatomical structure of the whole liver, identify the distribution pattern of the tumor, and establish the potential association between the tumor and distant tissues. The output is a 32×32×512 feature map.

7. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, In step S205, the input is a 32×32×512 feature map, which is processed by a Transformer encoder to extract the most abstract semantic information, establish global relationships between pixels, and provide rich contextual priors for the feature enhancement decoder. The output is a 16×16×1024 feature map.

8. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The specific steps of S3 are as follows: the input is a feature map of 16×16×1024, the feature map is first upsampled in four stages, then the features extracted by the corresponding layer of the encoder are fused through skip connections, and finally the features are refined by convolutional layers to obtain a segmentation probability map of size 512×512×3.

9. The liver tumor image segmentation method based on a CNN-Transformer hybrid architecture according to claim 1, characterized in that, The specific steps of S4 are as follows: Conditional random field optimization of the boundary to make the segmentation results smoother; connected component analysis to remove small false positive areas; morphological operations to fill holes and smooth edges; and three-dimensional reconstruction to integrate the 2D slice results into a 3D model for volume calculation.

Images