A pathological image segmentation method based on an improved U-Net model with an improved encoder and decoder

Abstract

This invention discloses a pathological image segmentation method based on an improved U-Net model with an enhanced encoder and decoder, comprising the following steps: preprocessing the data to improve the model's generalization ability; improving the U-Net model's encoder and decoder to establish an EEDU-Net model; training the model using training set data; obtaining the optimal model parameters, inputting test set data, and visualizing the model's prediction results. This invention modifies each layer of the encoder into a multi-branch form, increasing the encoding's ability to extract information at different scales. Simultaneously, the feature maps after skip connections are processed through channel attention and axial attention to eliminate semantic differences and ensure the effectiveness of each skip connection. Using the EEDU-Net model improves the robustness and accuracy of pathological image segmentation, demonstrating segmentation performance superior to existing models.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, and specifically to a method for segmenting pathological images based on an improved U-Net model with an encoder and decoder. Background Technology

[0002] In recent years, with the rapid development of deep learning in the image processing field, it has also made significant breakthroughs in pathological image segmentation. Pathological image segmentation can accurately separate lesion areas from normal areas in an image, helping pathologists diagnose diseases more precisely. Pathological image segmentation tasks often employ U-Net and its derivative models. U-Net's status as a cornerstone in this field is primarily due to its network model design. The U-Net model combines encoders and decoders at the same level through skip connections, helping to recover the original information after image upsampling. However, the single encoding method in U-Net cannot fully extract features, leading to varying degrees of loss of contextual information in each layer's feature map. Skip connections between layers can also result in semantic differences.

[0003] Many models have improved upon the U-Net framework, such as Res-UNet, U-Net++, and U2-Net, by modifying its encoder, skip connections, and decoder. These CNN-based models continuously improve segmentation accuracy, demonstrating the strong learning capabilities of CNNs. While CNN-based models have made significant progress in pathological image segmentation, traditional CNNs lack the ability to model long-range dependencies within images. In other words, in traditional CNNs, the receptive field is limited by the size of the convolutional kernel, causing the model to focus only on a local region of the image rather than the global context. To address this deficiency, existing methods introduce image pyramids and dilated convolutions to expand the receptive field and add the ability to model long-range dependencies to traditional CNNs. However, these operations do not completely solve the problem, and therefore still do not meet the high accuracy requirements for medical applications.

[0004] Subsequently, influenced by the great success of the Transformer in natural language processing, it was also applied to the field of computer vision, achieving satisfactory results in image classification, proving that the Transformer is equally suitable for the visual domain. Through the unique multi-head self-attention mechanism in the Transformer, the deficiency of traditional CNNs in effectively modeling long-range dependencies is effectively compensated for. Therefore, many subsequent models chose to combine U-Net with the Transformer to improve the model's segmentation ability. While improving segmentation accuracy, the introduction of the Transformer dramatically increases the number of model parameters. An increase in parameters inevitably leads to increased model training time; moreover, too many parameters may cause overfitting on a limited dataset. Summary of the Invention

[0005] The purpose of this invention is to provide a pathological image segmentation method based on an improved U-Net model with an improved encoder and decoder, which improves the robustness and accuracy of pathological image segmentation and shows a segmentation effect superior to existing models.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A pathological image segmentation method based on an improved encoder and decoder U-Net model includes the following steps:

[0008] Step 1: Perform data preprocessing operations;

[0009] Step 2: Improve the U-Net model encoder and decoder to build the EEDU-Net model;

[0010] Step 3: Train the model using the training set data to obtain the optimal model parameters;

[0011] Step 4: Input the test set data and visualize the model's prediction results.

[0012] Furthermore, the EEDU-Net model described in step two consists of an encoder, a decoder, and skip connections. The encoder extracts features from different stages of the image in a hierarchical manner; the decoder is responsible for reconstructing the encoded feature map and gradually restoring it to the original input size through upsampling; and the skip connections establish direct connections between the encoder and decoder at the same level.

[0013] Furthermore, the encoder includes a dense multi-scale extraction module (DMSEM) and a downsampling module, and the decoder includes a channel and axial hybrid attention module (CAHAM).

[0014] Furthermore, the Dense Multi-Scale Extraction Module (DMSEM) is specifically divided into four branches: Branch 1 consists of a 1×1 convolution, Branch 2 consists of max pooling and a 1×1 convolution, Branch 3 consists of a 3×3 LDM and two preceding and following 1×1 convolutions, and Branch 4 consists of a 5×5 LDM and two preceding and following 1×1 convolutions. The data is then concatenated in the channel dimension after passing through the four branches.

[0015] Furthermore, the operations at each layer of the LDM are as follows:

[0016] (1) Bacth normalization normalizes the features of the batch to a distribution with a mean of 0 and a variance of 1, reducing the variation in data distribution and accelerating the convergence of the network during training.

[0017] (2) The Rectified Linear Unit introduces non-linear features, enabling the network to learn more complex models;

[0018] (3) Depthwise Separable Convolution: First, each input channel is convolved individually using depthwise convolution, and then pointwise convolution is used to achieve a linear combination of information from different channels.

[0019] Furthermore, the feature extraction process of the Dense Multi-Scale Extraction Module (DMSEM) is as follows: the input feature map first enters the pointwise convolution of different branches, two of which will pass through LDM. The difference in the size of the convolution kernel in different LDMs represents the different learning scales. After passing through different convolutions or pooling in each branch, the feature map is finally spliced ​​in the channel dimension.

[0020] Furthermore, before concatenating along the channel dimension, a pointwise convolution operation is required. When DMSEM is in a shallow layer of the network, the proportion of branches with smaller convolution kernels is increased, while when DMSEM is in a deep layer of the network, the proportion of branches with larger convolution kernels is increased. After the feature map passes through DMSEM, the length, width, and number of channels of the feature map will not change, and the number of channels of the output feature map is equal to the sum of the number of channels output by the four branches.

[0021] Furthermore, the downsampling module operates as follows: the input data is first processed by max pooling and average pooling, then concatenated along the channel dimension, and the difference between the two is eliminated by 1×1 convolution; at the same time, the input data is processed by another convolution, and the data processed by the two different convolutions are concatenated to obtain the final downsampling result.

[0022] The specific implementation of the channel and axial hybrid attention module CAHAM is as follows:

[0023] (1) The input feature map is processed by max pooling and average pooling to extract feature maps in two different spaces. Then, a fully convolutional neural network composed of pointwise convolutions is used to generate channel attention maps. These maps are then merged by adding them element by element and multiplied element by element with the original input.

[0024] (2) After completing the above channel attention operation, the result is then input into two branches. One branch performs column attention operation first and then row attention operation, while the other branch does the opposite.

[0025] (3) Finally, the number of channels is halved by merging the results of the two branches and by point-by-point convolution.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] This invention first preprocesses the data to lay the foundation for subsequent operations and enhance generalization. An EEDU-Net model is constructed by improving the encoder and decoder of the U-Net model. The encoder employs a multi-branch approach to improve the capture of information at different scales, while the decoder combines transposed convolution and an attention module to accurately recover feature maps. Optimal model parameters are obtained using the training set, and finally, the optimal parameters are used to process the test set data and perform visualization. Using this model improves the robustness and accuracy of pathological image segmentation, demonstrating segmentation performance superior to existing models.

[0028] Furthermore, this invention enhances the encoder and decoder of U-Net. In the encoder, the multi-branch design endows the model with multi-scale learning capabilities, ensuring the model can adapt to objects of different sizes and improving its robustness and generalization ability. In the decoder, when constructing the Transformer, the feature map is divided into spatial attention and channel attention. Spatial attention is further refined into row attention and column attention. The combined use of these two approaches allows the model to maintain efficient global modeling capabilities even with a reduced number of parameters. Simultaneously, the feature map after skip connections is processed through channel attention and axial attention to eliminate semantic differences and ensure the effectiveness of each skip connection. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the EEDU-Net model of the present invention;

[0030] Figure 2 This is a schematic diagram of the Dense Multiscale Extraction (DMSEM) module structure of the encoder of the present invention;

[0031] Figure 3 This is a schematic diagram of the Lightweight Dense Module (LDM) structure of the encoder of the present invention;

[0032] Figure 4 This is a schematic diagram of the downsampling module structure of the encoder of the present invention;

[0033] Figure 5 This is a schematic diagram of the channel and axial hybrid attention module (CAHAM) structure of the decoder of the present invention;

[0034] Figure 6 To demonstrate the segmentation performance of the model of this invention on visual test images compared to other models on the MoNuSeg dataset;

[0035] Figure 7 To demonstrate the segmentation performance of the model of this invention on visual test images of other models on the GlaS dataset. Detailed Implementation

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The pathological image segmentation method based on the improved encoder and decoder U-Net model described in this embodiment includes the following steps:

[0038] Step 1: Perform data preprocessing to improve the model's generalization ability.

[0039] The image size is resized to 224×224, and a horizontal flip with a probability of 0.5 is performed. 224×224 is a widely used standard size. This size strikes a good balance between computational resources and model performance. A uniform image size facilitates batch processing of the model, improving computational efficiency. It also helps reduce performance differences on images of different sizes. Horizontal flipping is a common data augmentation technique that randomly flips images to increase data diversity, allowing the model to learn image features from different angles and orientations. Furthermore, data augmentation effectively prevents overfitting and improves generalization ability.

[0040] Step 2: Improve the U-Net model encoder and decoder to build the EEDU-Net model.

[0041] EEDU-Net model such as Figure 1 As shown, the system consists of an encoder, a decoder, and skip connections. The encoder extracts features from different stages of the image in a hierarchical manner. Specifically, the shallow encoder is responsible for extracting concrete features, while the deep encoder is responsible for extracting abstract features. The decoder is responsible for reconstructing the encoded feature map, gradually restoring it to the original input size through upsampling. Skip connections establish direct connections between the encoder and decoder at the same level, helping the encoder to better recover spatial details.

[0042] In encoders, to address the issue of insufficient feature extraction at each layer, it is necessary to enhance the network's ability to extract features at different scales across layers. To this end, a Dense Multi-Scale Extraction Module (DMSEM) was proposed based on Google Inception. This module can utilize different branches to extract feature map information at different scales, maximizing the information extracted at each layer. The DMSEM module is used in the extraction portion of each layer of the encoder to enhance the network's ability to extract features at different scales at each layer. The DMSEM structure is as follows: Figure 2 As shown, the Dense Multi-Scale Extraction (DMSEM) module is specifically divided into four branches: Branch 1 consists of a 1×1 convolution, Branch 2 consists of max pooling and a 1×1 convolution, Branch 3 consists of a 3×3 LDM followed by two 1×1 convolutions, and Branch 4 consists of a 5×5 LDM followed by two 1×1 convolutions. The data is then concatenated along the channel dimension after passing through each of the four branches.

[0043] LDM (Lightweight Dense Module) such as Figure 3 As shown, the operations at each layer of the LDM are as follows:

[0044] (1) Bacterial normalization (BN) normalizes the features of the batch to a distribution with a mean of 0 and a variance of 1, reducing the variation in data distribution and accelerating the convergence of the network during training.

[0045] (2) The Rectified Linear Unit (ReLU) introduces non-linear features, enabling the network to learn more complex models;

[0046] (3) Depthwise Separable Convolution (DSC) first performs a separate convolution operation on each input channel through depthwise convolution, and then uses pointwise convolution to achieve a linear combination of information between different channels.

[0047] The number of output channels in LDM after L1, L2, and L3 are 32, 64, and 128, respectively. Depthwise separable convolutions are used to further reduce the number of parameters, achieving a lightweight design. This design not only preserves the property of residual connections in mitigating gradient vanishing but also promotes information flow within the network. The input to each layer contains information from all previous layers, enhancing feature propagation and reducing the number of model parameters by reusing feature maps from earlier layers.

[0048] The Dense Multi-Scale Extraction Module (DMSEM) extracts features as follows: The input feature map first undergoes pointwise convolutions in different branches. This not only reduces dimensionality and computation but also allows each branch to independently learn the correlations between channels. This design fully decouples the learning of spatial correlations from channel correlations, giving the model stronger learning capabilities. Two branches pass through the LDM (Local Derivative Model), with differences in kernel size representing different learning scales. After different convolutions or pooling operations in each branch, the feature map is finally concatenated along the channel dimension. Before concatenation, another pointwise convolution operation is performed to adjust the weight of different branches in the final concatenation. When the DMSEM is at a shallow level of the network, the proportion of branches with smaller kernels should be increased; conversely, when the DMSEM is at a deep level, the proportion of branches with larger kernels should be increased. After passing through the DMSEM, the length, width, and number of channels of the feature map remain unchanged, and the number of channels in the output feature map equals the sum of the number of channels output from the four branches.

[0049] The downsampling module is a crucial component of the encoder, and its structure is as follows: Figure 4 As shown, this module is responsible for reducing the feature map size, retaining important features, and reducing computational cost. The downsampling module in the encoder works as follows: the input data is first processed by max pooling and average pooling, then concatenated along the channel dimension, and finally, a 1×1 convolution is used to eliminate the differences between the two. Simultaneously, the input data undergoes another convolution, and the data from the two different convolutions are concatenated to obtain the final downsampled result.

[0050] Using either max pooling or average pooling alone will result in varying degrees of loss of contextual information. Therefore, the downsampling module employs both pooling methods simultaneously and also includes an additional convolutional branch. This branch exhibits learning behavior, compensating for the fixed behavior of pooling operations and enhancing the feature extraction capability of downsampling. The feature maps after max pooling and average pooling are concatenated along the channel dimension, and then fused by convolution to eliminate the differences between the two. Finally, the fused feature map is element-wise added to another convolutional feature map. The downsampled feature map has its width and height reduced by half, while the number of channels is doubled.

[0051] In the decoder, to address the semantic differences arising from skip connections between the same level, a Channel and Axial Hybrid Attention Module (CAHAM) is proposed, such as... Figure 5 As shown, the decoder uses a channel and axis hybrid attention module, which is specifically implemented as follows:

[0052] (1) The input feature map is processed by max pooling and average pooling to extract feature maps in two different spaces. Then, a fully convolutional neural network composed of pointwise convolutions is used to generate channel attention maps. These maps are then merged by adding them element by element and multiplied element by element with the original input.

[0053] It should be noted that because the two tensors have different dimensions when multiplying, a broadcast mechanism will be triggered, which will expand the lower-dimensional tensor to the same size before performing the operation.

[0054] (2) After completing the above channel attention operation, the result is then input into two branches. One branch performs column attention operation first and then row attention operation, while the other branch does the opposite.

[0055] (3) Finally, the number of channels is halved by merging the results of the two branches and by point-by-point convolution.

[0056] When constructing the Transformer, the feature map is divided into spatial attention and channel attention. Spatial attention is further refined into row attention and column attention. Using only row or column attention cannot fuse global information, so both attention mechanisms are used simultaneously in each branch. This hybrid use allows the model to maintain efficient global modeling capabilities despite a reduction in the number of parameters. The different order of row and column attention in different branches enhances the model's robustness and generalization. Axial attention significantly reduces memory consumption compared to traditional attention, while also having lower computational complexity and higher computational efficiency. The feature map after CAHAM retains the same dimensions but halves the number of channels.

[0057] Step 3: Train the model using the training set data to obtain the optimal model parameters.

[0058] With the learning rate set to 0.001, the input image undergoes four feature extraction and downsampling passes through the encoder to obtain a highly abstract feature map. This map is then reconstructed layer by layer through transposed convolution. During the reconstruction process, the feature map is combined with the corresponding layer feature map from the encoder, and a channel and axis mixing attention module is used. The topmost feature map from the decoder is then convolved to obtain the final result. The final result is compared with the ground truth using a binary cross-entropy loss function to derive the loss. The parameters are then optimized using the Adam optimizer. The model parameters are continuously adjusted, and the optimal parameters are saved.

[0059] Step 4: Input test set data and visualize the model's prediction results.

[0060] First, load the trained model and switch it to evaluation mode. Next, resize the input image to 224×224 pixels to ensure consistency with the training dimensions. Then, use the model to predict the image and remove unnecessary dimensions from the prediction results. Next, convert the result to a binary image and multiply it by 255 to convert the pixel value range from 0 and 1 to an integer range of 0 to 255, making it easier to save and visualize. Finally, save the processed binary image.

[0061] Experimental section:

[0062] To comprehensively evaluate the model's performance, we compared it with other models, including U-Net, UNet++, TransUNet, UCTransNet, and Swin-Unet. During the training of EEDU-Net, the batch size was set to 4, the loss function was cross-entropy loss, the Adam optimizer was used, the learning rate was 0.001, and the epochs were 400. The test results are shown in Table 1.

[0063] Table 1 compares the results with other models on the MoNuSeg and GlaS datasets.

[0064]

[0065] Experimental results show that our proposed model ranks first in various metrics on both the MoNuSeg and GlaS datasets. In particular, the IoU metric on the MoNuSeg dataset is improved by 1.04% compared to TransUNet. Moreover, thanks to the CAHAM design, EEDU-Net has fewer parameters than other models that incorporate Transformers.

[0066] The visualization shows the segmentation performance of the test images on different models, such as... Figure 6 , Figure 7 As shown in the figure, the red box indicates the region with poor segmentation performance in the comparison model. It can be seen from the figure that the CNN-based model has overly coarse segmentation boundaries, the model with attention mechanism performs relatively better, while the model of this invention achieves smoother segmentation.

[0067] To investigate the impact of different modules on model performance, ablation experiments were conducted on the MoNuSeg and GlaS datasets. Four versions were compared: the original model, DMSEM only, CAHAM only, and both DMSEM and CAHAM. The results are shown in Table 2, which reflects the improvements of different modules to the original model.

[0068] Table 2 Ablation experimental results of DMSEM and CAHAM effectiveness

[0069]

[0070] To demonstrate the design value of multi-branch feature extraction, the original single convolution operation was replaced with DMSEM. Experimental results show that multi-branch feature extraction is an effective method to improve model performance. Compared to a single convolution operation, multi-branch features include convolution and pooling operations with different kernel sizes, which enables the model to capture information at different scales, enriching feature representation and helping the model to better locate and segment features.

[0071] To avoid inconsistencies between the information transmitted by the encoder and the semantics of the decoder after skip connections, CAHAM was used to replace the original modules. This operation significantly improved the model's metrics on both datasets. The use of attention mechanisms not only ensured the effectiveness of skip connections but also guaranteed the model's global awareness. The separation of channel attention and spatial attention allows the model to focus more on a specific dimension. Furthermore, the separate attention mechanisms can be optimized independently, giving the model greater flexibility.

Claims

1. A pathological image segmentation method based on an improved encoder and decoder U-Net model, characterized in that, Includes the following steps: Step 1: Perform data preprocessing operations; Step 2: Improve the U-Net model encoder and decoder to build the EEDU-Net model; Step 3: Train the model using the training set data to obtain the optimal model parameters; Step 4: Input the test set data and visualize the model's prediction results; The EEDU-Net model described in step two consists of an encoder, a decoder, and skip connections. The encoder extracts features from different stages of the image in a hierarchical manner. The decoder is responsible for reconstructing the encoded feature map, gradually restoring it to the original input size through upsampling; skip connections establish direct connections between the encoder and decoder at the same level; the encoder includes a Dense Multi-Scale Extraction (DMSEM) module and a downsampling module, and the decoder includes a Channel and Axial Hybrid Attention (CAHAM) module; the DMSEM module is specifically divided into four branches: branch 1 consists of a 1×1 convolution, branch 2 consists of max pooling and a 1×1 convolution, branch 3 consists of a 3×3 LDM and two preceding and following 1×1 convolutions, and branch 4 consists of a 5×5 LDM and two preceding and following 1×1 convolutions. The data is concatenated in the channel dimension after passing through the four branches; the operations of each layer of the LDM are as follows: (1) Bacth normalization normalizes the features of the batch to a distribution with a mean of 0 and a variance of 1, reducing the variation in data distribution and accelerating the convergence of the network during training. (2) The Rectified Linear Unit introduces non-linear features, enabling the network to learn more complex models; (3) Depthwise Separable Convolution first performs a separate convolution operation on each input channel through depthwise convolution, and then uses pointwise convolution to achieve a linear combination of information between different channels; The specific implementation of the channel and axial hybrid attention module CAHAM is as follows: (1) The input feature map is processed by max pooling and average pooling to extract feature maps in two different spaces. Then, a fully convolutional neural network composed of pointwise convolutions is used to generate channel attention maps. These maps are then merged by adding them element by element and multiplied element by element with the original input. (2) After completing the above channel attention operation, the result is then input into two branches. One branch performs column attention operation first and then row attention operation, while the other branch does the opposite. (3) Finally, the number of channels is halved by merging the results of the two branches and by point-by-point convolution.

2. The pathological image segmentation method based on the improved encoder and decoder U-Net model according to claim 1, characterized in that, The Dense Multi-Scale Extraction Module (DMSEM) extracts features as follows: the input feature map first enters pointwise convolution of different branches, two of which will pass through LDM. The difference in the size of the convolution kernel in different LDMs represents different learning scales. After passing through different convolutions or pooling in each branch, the feature map is finally concatenated along the channel dimension.

3. The pathological image segmentation method based on the improved encoder and decoder U-Net model according to claim 2, characterized in that, Before concatenating along the channel dimension, a pointwise convolution operation is required. When DMSEM is in a shallow layer of the network, the proportion of the branch with the smaller convolution kernel is increased, while when DMSEM is in a deep layer of the network, the proportion of the branch with the larger convolution kernel is increased. After the feature map passes through DMSEM, the length, width, and number of channels of the feature map will not change, and the number of channels of the output feature map is equal to the sum of the number of channels output by the four branches.

4. The pathological image segmentation method based on the improved encoder and decoder U-Net model according to claim 1, characterized in that, The downsampling module operates as follows: the input data is first processed by max pooling and average pooling, then concatenated along the channel dimension, and then the difference between the two is eliminated by a 1×1 convolution; at the same time, the input data is processed by another convolution, and the data processed by the two different convolutions are concatenated to obtain the final downsampling result.

Images