A medical image liver segmentation method and system combining a transformer and a U-Net

Abstract

The application particularly relates to a medical image liver segmentation method and system combining a Transformer and a U-Net. The method comprises the following steps: acquiring a medical image to be segmented; inputting the medical image to be segmented into a trained liver segmentation model to output a corresponding liver segmentation prediction result; wherein the liver segmentation model firstly extracts local feature information and global feature information of the medical image through an encoder structure and a Transformer structure of a U-Net network respectively; then fuses the local feature information and the global feature information of the medical image and assists the encoder structure to generate a liver segmentation feature image; finally performs feature positioning and image reconstruction on the liver segmentation feature image through a decoder structure of the U-Net network to generate and output the corresponding liver segmentation prediction result. The application can enhance the encoder part of the U-Net network by extracting global feature information through the Transformer, and can take into account the advantages of the Transformer and the U-Net network, so that the accuracy and comprehensiveness of the medical image liver segmentation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a medical image liver segmentation method and system combining Transformer and U-Net. BACKGROUND

[0002] The liver is the largest solid organ in the abdominal cavity of the human body, which has a very complex structure and rich blood vessels. At the same time, the liver has the characteristics of multiple types of lesions and high incidence. Liver segmentation is a key prerequisite for cancer detection and treatment. At present, in order to obtain accurate information about the volume, position and shape of the liver from abdominal medical images (such as CT images), radiologists need to manually mark each slice, which is time-consuming and prone to errors. Therefore, there is an urgent need for an automated and accurate medical image liver segmentation method in the clinic.

[0003] Existing automatic liver segmentation methods can be roughly divided into three categories: traditional methods, machine learning-based methods and deep learning-based methods. Traditional methods are mainly semi-automatic methods, which are easily affected by noise and require manual intervention for complex parameter settings. Machine learning-based methods can prove the accuracy of segmentation well. However, most machine learning-based methods require manual feature design, which can significantly affect accuracy. In contrast, deep learning-based liver segmentation methods have good accuracy. For example, the Chinese patent with publication number CN110570432A discloses a CT image liver tumor segmentation method based on deep learning, which establishes and trains a model based on the U-Net network structure, and then uses the trained model to segment the liver tumor CT image to be predicted to obtain the segmentation result.

[0004] The above-mentioned existing U-Net network-based liver segmentation method has strong rapid learning ability and excellent performance for small data sets. However, although the U-Net network model can extract local feature information of medical images, it cannot effectively extract global feature information of medical images, which leads to a narrow visual perception range of the liver segmentation model. When the liver in the medical image has problems such as edge blur, low contrast, discontinuous regions or too small shape, it cannot guarantee the accuracy and comprehensiveness of liver segmentation. Therefore, how to design a medical image liver segmentation method that can improve the accuracy and comprehensiveness of liver segmentation is a technical problem that needs to be solved. SUMMARY

[0005] In view of the above problems of the prior art, the technical problem to be solved by the present application is to provide a medical image liver segmentation method combining a Transformer and a U-Net, which can extract global feature information by the Transformer to enhance the encoder part of the U-Net network, and can take into account the advantages of the Transformer and the U-Net network, thereby improving the accuracy and comprehensiveness of medical image liver segmentation.

[0006] To solve the above technical problems, the present application adopts the following technical solutions:

[0007] A medical image liver segmentation method combining a Transformer and a U-Net, comprising:

[0008] S1: obtaining a medical image to be segmented;

[0009] S2: inputting the medical image to be segmented into a trained liver segmentation model, and outputting a corresponding liver segmentation prediction result;

[0010] The liver segmentation model is constructed based on a Transformer and a U-Net network;

[0011] The liver segmentation model first extracts local feature information and global feature information of the medical image through an encoder structure and a Transformer structure of the U-Net network, respectively; then fuses the local feature information and the global feature information of the medical image, and assists the encoder structure to generate a liver segmentation feature image; and finally performs feature positioning and image reconstruction on the liver segmentation feature image through a decoder structure of the U-Net network, to generate and output a corresponding liver segmentation prediction result;

[0012] S3: taking the liver segmentation prediction result output by the liver segmentation model as the segmentation result of the medical image to be segmented.

[0013] Preferably, the encoder structure and the decoder structure each comprise four layers of encoding Res-SE-Blocks and four layers of decoding Res-SE-Blocks, which are sequentially cascaded and used for local feature extraction; the Transformer structure comprises four layers of Swin Transformer blocks, which are sequentially cascaded and used for global feature extraction; and three layers of feature feedback enhancement layers are arranged between the encoder structure and the Transformer structure, which are sequentially cascaded and used for fusing the local feature information and the global feature information.

[0014] The inputs to the first layer Res-SE-Block encoding and the first layer Swing Transformer block are the medical images input to the liver segmentation model; the inputs to the first layer feature feedback enhancement layer are the feature map output by the first layer Swing Transformer block and the feature map output after downsampling of the first layer Res-SE-Block encoding.

[0015] The input to the second-layer encoding Res-SE-Block is the fused feature map output from the first-layer feature feedback enhancement layer; the input to the second-layer Swing Transformer block is the feature map output from the first-layer Swing Transformer block; the input to the second-layer feature feedback enhancement layer is the feature map output from the second-layer Swing Transformer block and the feature map output after downsampling from the second-layer encoding Res-SE-Block.

[0016] The input to the third layer Res-SE-Block encoding layer is the fused feature map output from the second layer feature feedback enhancement layer; the input to the third layer Swin Transformer block is the feature map output from the second layer Swin Transformer block; the input to the third layer feature feedback enhancement layer is the feature map output from the third layer Swin Transformer block and the feature map output after downsampling from the third layer Res-SE-Block encoding layer.

[0017] The input to the fourth layer Res-SE-Block is the fused feature map output from the third layer feature feedback enhancement layer; the input to the fourth layer Swing Transformer block is the feature map output from the third layer Swing Transformer block.

[0018] The inputs to the fourth layer decoding Res-SE-Block are the feature map output by the fourth layer Swing Transformer block, the feature map output by the fourth layer encoding Res-SE-Block after downsampling, and the feature map output by the fourth layer encoding Res-SE-Block.

[0019] The input to the third layer decoding Res-SE-Block is the feature map output by the fourth layer decoding Res-SE-Block after upsampling and the feature map output by the third layer encoding Res-SE-Block;

[0020] The input to the second layer decoding Res-SE-Block is the feature map output by the third layer decoding Res-SE-Block after upsampling and the feature map output by the second layer encoding Res-SE-Block;

[0021] The input to the first layer decoding Res-SE-Block is the feature map output by the second layer decoding Res-SE-Block after upsampling and the feature map output by the first layer encoding Res-SE-Block;

[0022] The output of the first layer decoding Res-SE-Block is the output of the liver segmentation model, i.e., the liver segmentation prediction result.

[0023] Preferably, an ASPP module for capturing feature map context information at multiple scales is provided between the fourth-layer encoding Res-SE-Block and the fourth-layer decoding Res-SE-Block;

[0024] The ASPP module takes as input the feature map output from the fourth layer Swin Transformer block and the feature map output after downsampling of the fourth layer encoding Res-SE-Block; the feature map output from the ASPP module and the feature map output from the fourth layer encoding Res-SE-Block serve as input to the fourth layer decoding Res-SE-Block.

[0025] Preferably, the feature feedback enhancement layer generates the fused feature map through the following steps:

[0026] S201: Obtain the feature map output by the corresponding Swin Transformer block and the feature map h output after downsampling the corresponding Res-SE-Block encoding;

[0027] S202: Upsample the feature map output by the Swin Transformer block using the nearest neighbor algorithm, and then perform convolution with Sigmoid activation to generate the corresponding feature map g.

[0028] S203: Multiply the feature map g with the feature map h output after downsampling the encoded Res-SE-Block, and add it to the feature map h to generate the corresponding intermediate attention feature h·g+h;

[0029] S204: Receives intermediate attention features h·g+h through a global average pooling layer and a fully connected layer activated by Sigmoid, and outputs the corresponding weight vector f;

[0030] S205: Multiply each channel of the intermediate attention feature h·g+h with each element of the weight vector f to generate the corresponding final attention mask (h·g+h)×f as the fused feature map output by the feature feedback enhancement layer.

[0031] Preferably, the Res-SE-Block consists of two sets of 3×3 convolutions, a set of SE blocks, and a residual connection; each set of 3×3 convolutions is followed by batch normalization and the LeaklyReLu activation function.

[0032] Preferably, the Swing Transformer block includes a LayerNorm layer, a multi-head self-attention module, a residual connection, and a two-layer MLP with GELU nonlinearity connected in sequence;

[0033] The multi-head self-attention module between two adjacent Swing Transformer blocks is alternately set as a windowed multi-head self-attention layer W-MSA and a displacement-based windowed multi-head self-attention layer SW-MSA.

[0034] Preferably, the liver segmentation model is trained through the following steps:

[0035] S211: Prepare a medical image dataset, which is divided into a training image dataset and a test image dataset, and perform image preprocessing on the training image dataset;

[0036] S212: Input the preprocessed training image data into the liver segmentation model for training until the model converges or reaches the maximum number of iterations;

[0037] S213: Test the liver segmentation model using a test image dataset.

[0038] Preferably, when training the liver segmentation model, the model parameters are optimized using the following training loss function:

[0039] L Total =αL CE +βL Dice ;

[0040]

[0041] L CE = -ylog(p) - (1-y)log(1-p);

[0042] In the formula: L Total Indicates training loss; L CE L represents the cross-entropy loss; Dice Let represent the Dice loss; α and β represent the weights of the cross-entropy loss and the Dice loss; y∈(0,1) represents the true class value of the medical image; p∈[0,1] represents the predicted probability in the medical image; N represents the total number of predicted pixels in the medical image; p i y represents the predicted probability of pixel i in a medical image; i This represents the true category of pixel i in a medical image.

[0043] Preferably, in step S211, the image preprocessing includes: using windowing techniques to open a window within the range of Hounsfield unit values ​​[-200, 200] to remove other irrelevant tissues and enhance the contrast between the liver and other tissues; then normalizing the voxel values ​​of the medical image to [-1, 1]; and finally normalizing the medical image.

[0044] This invention also discloses a medical image liver segmentation system combining Transformer and U-Net, implemented based on the medical image liver segmentation method of this invention, comprising:

[0045] The data collection module is used to collect medical image datasets containing liver images and generate training image datasets and test image datasets.

[0046] The model building module is used to build liver segmentation models using the PyTorch framework.

[0047] The model training module is used to train and test the liver segmentation model using training image datasets and test image datasets;

[0048] The liver segmentation module is used to input the medical image to be segmented into the trained liver segmentation model to output the liver segmentation prediction result;

[0049] The visualization module is used to visualize the predicted results of liver segmentation.

[0050] Compared with existing technologies, the medical image liver segmentation method and system combining Transformer and U-Net in this invention has the following advantages:

[0051] This invention extracts local and global feature information from medical images using the encoder and Transformer structures of the U-Net network, respectively. These local and global feature information are then fused to generate a liver segmentation feature image, enabling prediction of liver segmentation results. On one hand, this invention adds a Transformer structure to capture global context, building upon the U-Net network's local feature extraction via the encoder. This allows for remote interaction and spatial connections of image features, enhancing the encoder portion of the U-Net network by extracting global feature information through the Transformer. Furthermore, it improves the model's visual perception range and enhances the multi-scale, long-distance dependence of medical image segmentation. Specifically, it is applicable to medical images with blurred liver edges, low contrast, discontinuous regions, and small shapes, thereby improving the accuracy and comprehensiveness of liver segmentation. On the other hand, this invention assists the encoder structure in generating liver segmentation feature images by fusing local and global feature information. This fully utilizes the global features of the Transformer and the local features of the U-Net network, thereby inheriting the fast learning ability of the U-Net network and the excellent generalization ability of the Transformer, while taking into account the excellent performance of the U-Net network on small datasets and the ability of the Transformer to capture global context. In other words, it combines the advantages of the Transformer and U-Net networks, thereby improving the performance and decision-making ability of the liver segmentation model. Attached Figure Description

[0052] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0053] Figure 1 A block diagram of a liver segmentation method for medical images;

[0054] Figure 2 Network structure diagram of the liver segmentation model;

[0055] Figure 3 This is a network structure diagram of the ASPP module;

[0056] Figure 4 This is a network structure diagram of the feature feedback enhancement layer;

[0057] Figure 5 This is a network structure diagram of Res-SE-Block;

[0058] Figure 6 This is a network structure diagram of the Swing Transformer block;

[0059] Figure 7Images before and after image preprocessing;

[0060] Figure 8 A visual comparison of the outputs of different models on the LiTS dataset;

[0061] Figure 9 Visual comparison of different model outputs on the 3DIRCADb dataset;

[0062] Figure 10 This is a logic block diagram of a liver segmentation system for medical images. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0064] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not mean that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0065] The following detailed explanation illustrates the specific implementation methods:

[0066] Example 1:

[0067] This embodiment discloses a method for liver segmentation in medical images that combines Transformer and U-Net.

[0068] like Figure 1 As shown, a liver segmentation method for medical images combining Transformer and U-Net includes:

[0069] S1: Obtain the medical image to be segmented;

[0070] S2: Input the medical image to be segmented into the trained liver segmentation model and output the corresponding liver segmentation prediction result;

[0071] Combination Figure 2 As shown, the liver segmentation model is built based on Transformer and U-Net networks;

[0072] The liver segmentation model first extracts local and global feature information from medical images using the encoder and Transformer structures of the U-Net network, respectively. Then, it fuses the local and global feature information of the medical images and uses the encoder structure to generate a liver segmentation feature image. Finally, it uses the decoder structure of the U-Net network to perform feature localization and image reconstruction on the liver segmentation feature image to generate and output the corresponding liver segmentation prediction result.

[0073] The operating logic and training method of the encoder and decoder structures of the U-Net network in this invention are consistent with those of the existing U-Net network.

[0074] S3: Use the liver segmentation prediction results output by the liver segmentation model as the segmentation results of the medical image to be segmented.

[0075] This invention extracts local and global feature information from medical images using the encoder and Transformer structures of the U-Net network, respectively. These local and global feature information are then fused to generate a liver segmentation feature image, enabling prediction of liver segmentation results. On one hand, this invention adds a Transformer structure to capture global context, building upon the local feature extraction of the medical image by the encoder portion of the U-Net network. This allows for remote interaction and spatial connections of image features, enhancing the encoder portion of the U-Net network by extracting global feature information through the Transformer. Furthermore, it improves the model's visual perception range and enhances the multi-scale, long-distance dependence of medical image segmentation. Specifically, it is applicable to medical images with blurred liver edges, low contrast, discontinuous regions, and small shapes, thereby improving the accuracy and comprehensiveness of liver segmentation in medical images. On the other hand, this invention assists the encoder structure in generating liver segmentation feature images by fusing local and global feature information. This fully utilizes the global features of the Transformer and the local features of the U-Net network, thereby inheriting the fast learning ability of the U-Net network and the excellent generalization ability of the Transformer, while taking into account the excellent performance of the U-Net network on small datasets and the ability of the Transformer to capture global context. In other words, it combines the advantages of the Transformer and U-Net networks, thereby improving the performance and decision-making ability of the liver segmentation model.

[0076] Combination Figure 2 As shown, the encoder and decoder structures each include a four-layer cascaded encoding Res-SE-Block for local feature extraction. Figure 2 The left side of the middle Res-SE-Block) and the four-layer decoding Res-SE-Block (Figure 2 The Res-SE-Block on the right side of the middle section); The Transformer structure includes four cascaded Swing Transformer blocks for global feature extraction; Between the encoder structure and the Transformer structure, there is a three-layer feature feedback enhancement layer for fusing local and global feature information and cascading in sequence;

[0077] In this embodiment, the Res-SE-Block is the residual SE module, where the SE block is an existing mature model.

[0078] SE stands for Squeeze-and-Excitation (SE).

[0079] The inputs to the first layer Res-SE-Block encoding and the first layer Swing Transformer block are the medical images input to the liver segmentation model; the inputs to the first layer feature feedback enhancement layer are the feature map output by the first layer Swing Transformer block and the feature map output after downsampling of the first layer Res-SE-Block encoding.

[0080] The input to the second-layer encoding Res-SE-Block is the fused feature map output from the first-layer feature feedback enhancement layer; the input to the second-layer Swing Transformer block is the feature map output from the first-layer Swing Transformer block; the input to the second-layer feature feedback enhancement layer is the feature map output from the second-layer Swing Transformer block and the feature map output after downsampling from the second-layer encoding Res-SE-Block.

[0081] The input to the third layer Res-SE-Block encoding layer is the fused feature map output from the second layer feature feedback enhancement layer; the input to the third layer Swin Transformer block is the feature map output from the second layer Swin Transformer block; the input to the third layer feature feedback enhancement layer is the feature map output from the third layer Swin Transformer block and the feature map output after downsampling from the third layer Res-SE-Block encoding layer.

[0082] The input to the fourth layer Res-SE-Block is the fused feature map output from the third layer feature feedback enhancement layer; the input to the fourth layer Swing Transformer block is the feature map output from the third layer Swing Transformer block.

[0083] The inputs to the fourth layer decoding Res-SE-Block are the feature map output by the fourth layer Swing Transformer block, the feature map output by the fourth layer encoding Res-SE-Block after downsampling, and the feature map output by the fourth layer encoding Res-SE-Block.

[0084] The input to the third layer decoding Res-SE-Block is the feature map output by the fourth layer decoding Res-SE-Block after upsampling and the feature map output by the third layer encoding Res-SE-Block;

[0085] The input to the second layer decoding Res-SE-Block is the feature map output by the third layer decoding Res-SE-Block after upsampling and the feature map output by the second layer encoding Res-SE-Block;

[0086] The input to the first layer decoding Res-SE-Block is the feature map output by the second layer decoding Res-SE-Block after upsampling and the feature map output by the first layer encoding Res-SE-Block;

[0087] The output of the first layer decoding Res-SE-Block is the output of the liver segmentation model, i.e., the liver segmentation prediction result.

[0088] For example, the input to a liver segmentation model is a 224*224*1 medical image, which is fed into both the encoder structure (i.e., the CNN path) and the Transformer structure (i.e., the Transformer path). The Transformer path consists of four SwingTransformers, and the CNN path consists of four Res-SE-Blocks. The feature feedback enhancement layer uses the output of the SwingTransformer block and the downsampled output of the Res-SE-Block as input for computation, and then uses the output as the input to the next Res-SE-Block. For instance, if the output of the first SwingTransformer block is [56*56*96], and the downsampled output of the first Res-SE-Block is [112*112*64], then the output of the first feature feedback enhancement layer is equal to the output of the first Res-SE-Block, i.e., [112*112*64].

[0089] The output of the fourth layer Swing Transformer block [7*7*768] is concatenated with the downsampled output of the fourth layer encoding Res-SE-Block [14*14*512] to obtain [14*14*512], and then input into the ASPP module. The output of the ASPP module is equal to the input size [14*14*512]. Then, it is input into the upsampling layer to obtain [28*28*512] and the output of the fourth layer encoding Res-SE-Block is skipped to obtain [28*28*1024]. This is then input into the decoding Res-SE-Block of the decoder structure to obtain [28*28*512]. After repeated upsampling and convolution, the final liver segmentation prediction result is obtained as [224*224*1].

[0090] This invention designs cascaded channels in the encoder structure, enabling the extraction of local and global feature information from medical images through the encoder and Transformer structures, respectively. A feature feedback enhancement layer in each cascaded channel calculates the output of the Transformer channel (global features) and the output of the previous Res-SE-Block (local features), using the enhanced output as the input to the current Res-SE-Block. This cascaded channel design and feature feedback enhancement layer fully utilize the global features of the Transformer and the local features of the U-Net network, making it better suited for medical images with blurred liver edges, low contrast, discontinuous regions, and small shapes. This further improves the accuracy and comprehensiveness of liver segmentation in medical images.

[0091] In the specific implementation process, an ASPP module is set between the fourth-layer encoding Res-SE-Block and the fourth-layer decoding Res-SE-Block to capture feature map context information at multiple scales.

[0092] ASPP stands for Atrous Spatial Pyramid Pooling.

[0093] Combination Figure 3 As shown, the inputs of the ASPP module are the feature map output by the fourth layer Swing Transformer block and the feature map output after downsampling of the fourth layer encoding Res-SE-Block; the feature map output by the ASPP module and the feature map output by the fourth layer encoding Res-SE-Block are used as the inputs of the fourth layer decoding Res-SE-Block.

[0094] This invention captures feature map context information at multiple scales through the ASPP module, which can help to enhance the encoder part of the U-Net network by extracting global feature information through the Transformer. This makes it more suitable for medical images with blurred liver edges, low contrast, discontinuous regions, and small shapes, thereby further improving the accuracy and comprehensiveness of liver segmentation in medical images.

[0095] Combination Figure 4 As shown, the feature feedback enhancement layer generates the fused feature map through the following steps:

[0096] S201: Obtain the feature map output by the corresponding Swin Transformer block and the feature map output after downsampling the corresponding encoding Res-SE-Block;

[0097] S202: Upsample the feature map output by the Swin Transformer block using the nearest neighbor algorithm, and then perform convolution with Sigmoid activation to generate the corresponding feature map g.

[0098] S203: Multiply the feature map g with the feature map h output after downsampling the encoded Res-SE-Block, and add it to the feature map h to generate the corresponding intermediate attention feature h·g+h;

[0099] S204: Receives intermediate attention features h·g+h through a global average pooling layer and a fully connected layer activated by Sigmoid, and outputs the corresponding weight vector f;

[0100] S205: Multiply each channel of the intermediate attention feature h·g+h with each element of the weight vector f to generate the corresponding final attention mask (h·g+h)×f as the fused feature map output by the feature feedback enhancement layer.

[0101] The feature feedback enhancement layer of this invention can effectively fuse local and global features, fully utilize the global features of the Transformer and the local features of the U-Net network, thereby inheriting the fast learning ability of the U-Net network and the excellent generalization ability of the Transformer, while taking into account the excellent performance of the U-Net network on small datasets and the ability of the Transformer to capture global context. In other words, it achieves the combination of the advantages of the Transformer and U-Net networks, thereby further improving the performance and decision-making ability of the liver segmentation model.

[0102] Combination Figure 5 As shown, Res-SE-Block consists of two sets of 3×3 convolutions, a set of SE blocks, and a residual connection; each set of 3×3 convolutions is followed by batch normalization and the Leakly ReLU activation function.

[0103] The SE block performs the following operations: 1) Compresses the two-dimensional feature map (HxW) of each channel into real numbers using global average pooling; 2) Adds a non-linear transformation to the fully connected network to obtain the weights of each feature channel; 3) Applies the obtained normalized weights to the features of each channel to extract specific information.

[0104] The Res-SE-Block of this invention improves the quality of representation by explicitly simulating the interdependencies between channels of its convolutional features in the SE block, and allows the network to perform feature recalibration. It can learn to use global information to selectively emphasize informative features and suppress less useful features. That is, the Res-SE-Block module can significantly improve the accuracy of liver and tumor segmentation, and Res-SE-Block is easier to integrate into other networks compared with other attention mechanisms.

[0105] Combination Figure 6 As shown, the Swing Transformer block includes a LayerNorm layer, a multi-head self-attention module, a residual connection, and a two-layer MLP with GELU nonlinearity connected in sequence.

[0106] GELU stands for Gaussian Error Linear Units.

[0107] MLP stands for Multilayer Perceptron (MLP).

[0108] The multi-head self-attention module between two adjacent Swing Transformer blocks is alternately set as a windowed multi-head self-attention layer W-MSA and a displacement-based windowed multi-head self-attention layer SW-MSA.

[0109] The Swin-Transformer block of this invention can reflect complex spatial transformations and long-range feature dependencies, and can obtain global features through self-attention mechanism and MLP. That is, it can capture global context through the Swin-Transformer block to realize long-range interaction and spatial connection of image features, and improve the visual perception range of the model, improve the multi-scale long-distance dependency of medical image segmentation, and can be better applied to medical images with blurred liver edges, low contrast, discontinuous regions and small shapes.

[0110] In the specific implementation process, the liver segmentation model is trained through the following steps:

[0111] S211: Prepare a medical image dataset, which is divided into a training image dataset and a test image dataset, and perform image preprocessing on the training image dataset;

[0112] In this embodiment, image preprocessing includes: using windowing techniques to open windows within the range of Hounsfield unit values ​​[-200, 200] to remove other irrelevant tissues and enhance the contrast between the liver and other tissues; then normalizing the voxel values ​​of the medical image to [-1, 1]; and finally normalizing the medical image. Figure 7 The image shows a comparison before and after image preprocessing. It can be seen that the liver region is clearer after preprocessing, with clearer texture and contours, which is beneficial for better model training and thus improves the training effect and performance of the liver segmentation model.

[0113] S212: Input the preprocessed training image data into the liver segmentation model for training until the model converges or reaches the maximum number of iterations;

[0114] S213: Test the liver segmentation model using a test image dataset.

[0115] When training the liver segmentation model, the model parameters are optimized using the following training loss function:

[0116] L Total =αL CE +βL Dice ;

[0117]

[0118] L CE = -ylog(p) - (1-y)log(1-p);

[0119] In the formula: L Total Indicates training loss; L CE L represents the cross-entropy loss; Dice denoted by Dice loss; α and β represent the weights of cross-entropy loss and Dice loss, α = 0.5, β = 1; y ∈ (0, 1) represents the true class value of the medical image; p ∈ [0, 1] represents the predicted probability in the medical image; N represents the total number of predicted pixels in the medical image; p i y represents the predicted probability of pixel i in a medical image; i This represents the true category of pixel i in a medical image.

[0120] This invention trains a liver segmentation model by jointly using cross-entropy loss and Dice loss, which enables it to better adapt to the characteristics of medical images with complex organ types, sizes, and shapes, and blurred boundaries between organs and other tissues. It is also better suited for medical images with blurred liver edges, low contrast, discontinuous regions, and very small shapes, thereby improving the training effect and performance of the liver segmentation model.

[0121] In this embodiment, the liver segmentation model proposed in this invention is trained, validated, and tested on the LiTS dataset, and its generalization ability is tested on the 3D IRCADb dataset.

[0122] This experiment compares the liver segmentation model proposed in this invention (referred to as ResTransUNet) with five state-of-the-art methods to evaluate the effectiveness and robustness of ResTransUNet on the LiTS and 3D-IRCADb datasets. The comparison results of the six methods on the LiTS dataset are shown in Table 1. The results demonstrate that the proposed ResTransUNet exhibits superior performance.

[0123] In addition, visualizations of the outputs of different models are shown below. Figure 8 As shown, we can see that U-Net performs poorly in liver and tumor segmentation, while our proposed ResTransUNet performs better in both liver and tumor segmentation.

[0124] To evaluate the effectiveness and robustness of our proposed ResTransUNet, we also conducted experiments on the 3D-IRCADb dataset. The comparison and visualization results on the 3DIRCADb dataset are shown in Table 2 and [Table data would be inserted here]. Figure 9 As shown, our proposed method still outperforms other methods on the 3D-IRCADb dataset, and experimental comparisons verify the superiority of our method compared to other methods.

[0125] Table 1 Comparison of different methods on the LiTS dataset

[0126]

[0127] Table 2 Comparison of different methods on the 3D IRCADb dataset

[0128]

[0129] Example 2:

[0130] This embodiment discloses a medical image liver segmentation system that combines Transformer and U-Net, based on the medical image liver segmentation method in Embodiment 1.

[0131] like Figure 10 As shown, a medical image liver segmentation system combining Transformer and U-Net includes:

[0132] The data collection module is used to collect medical image datasets containing liver images and generate training image datasets and test image datasets.

[0133] The model building module is used to build liver segmentation models using the PyTorch framework.

[0134] The model training module is used to train and test the liver segmentation model using training image datasets and test image datasets;

[0135] The liver segmentation module is used to input the medical image to be segmented into the trained liver segmentation model to output the liver segmentation prediction result;

[0136] The visualization module is used to visualize the predicted results of liver segmentation.

[0137] This invention provides an end-to-end medical image liver segmentation system that can efficiently and accurately segment livers in medical images and visualize the liver segmentation prediction results.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A medical image liver segmentation method combining Transformer and U-Net, characterized in that, The method comprises the following steps: S1: obtaining a medical image to be segmented; S2: inputting the medical image to be segmented into a trained liver segmentation model, and outputting a corresponding liver segmentation prediction result; The liver segmentation model is constructed based on a Transformer and a U-Net network; The liver segmentation model firstly extracts local feature information and global feature information of the medical image through an encoder structure and a Transformer structure of the U-Net network respectively; then fuses the local feature information and the global feature information of the medical image, and assists the encoder structure to generate a liver segmentation feature image; finally, the liver segmentation feature image is subjected to feature positioning and image reconstruction through a decoder structure of the U-Net network, so as to generate and output the corresponding liver segmentation prediction result; The encoder structure and the decoder structure respectively comprise four layers of encoding Res-SE-Blocks and four layers of decoding Res-SE-Blocks which are sequentially cascaded and arranged for local feature extraction; the Transformer structure comprises four layers of Swin Transformer blocks which are sequentially cascaded and arranged for global feature extraction; wherein three layers of feature feedback enhancement layers which are sequentially cascaded and arranged for fusing the local feature information and the global feature information are arranged between the encoder structure and the Transformer structure; The input of the first layer of encoding Res-SE-Block and the first layer of Swin Transformer block is the medical image input into the liver segmentation model; The input of the first layer of feature feedback enhancement layer is the feature map output by the first layer of Swin Transformer block and the feature map output by the first layer of encoding Res-SE-Block after downsampling; The input of the second layer of encoding Res-SE-Block is the fusion feature map output by the first layer of feature feedback enhancement layer; the input of the second layer of Swin Transformer block is the feature map output by the first layer of Swin Transformer block; The input of the second layer of feature feedback enhancement layer is the feature map output by the second layer of Swin Transformer block and the feature map output by the second layer of encoding Res-SE-Block after downsampling; The input of the third layer of encoding Res-SE-Block is the fusion feature map output by the second layer of feature feedback enhancement layer; the input of the third layer of Swin Transformer block is the feature map output by the second layer of Swin Transformer block; The input of the third layer of feature feedback enhancement layer is the feature map output by the third layer of Swin Transformer block and the feature map output by the third layer of encoding Res-SE-Block after downsampling; The input of the fourth layer of encoding Res-SE-Block is the fusion feature map output by the third layer of feature feedback enhancement layer; the input of the fourth layer of Swin Transformer block is the feature map output by the third layer of Swin Transformer block; The input of the fourth layer decoding Res-SE-Block is the feature map output by the fourth layer Swin Transformer block, the feature map output by the fourth layer coding Res-SE-Block after downsampling, and the feature map output by the fourth layer coding Res-SE-Block; The input of the third layer decoding Res-SE-Block is the feature map output by the fourth layer decoding Res-SE-Block after upsampling and the feature map output by the third layer coding Res-SE-Block; The input of the second layer decoding Res-SE-Block is the feature map output by the third layer decoding Res-SE-Block after upsampling and the feature map output by the second layer coding Res-SE-Block; The input of the first layer decoding Res-SE-Block is the feature map output by the second layer decoding Res-SE-Block after upsampling and the feature map output by the first layer coding Res-SE-Block; The output of the first layer decoding Res-SE-Block is the output of the liver segmentation model, i.e., the liver segmentation prediction result; The feature feedback enhancement layer generates the fused feature map through the following steps: S201: Obtain the feature map output by the corresponding Swin Transformer block and the feature map h output by the corresponding coding Res-SE-Block after downsampling; S202: Upsample the feature map output by the Swin Transformer block through the nearest neighbor algorithm, and generate the corresponding feature map g through the convolution of the Sigmoid activation; S203: Multiply the feature map g and the feature map h output by the coding Res-SE-Block after downsampling, and add the feature map h to generate the corresponding intermediate attention feature h·g+h; S204: Receive the intermediate attention feature h·g+h through the global average pooling layer and the fully connected layer with Sigmoid activation, and output the corresponding weight vector f; S205: Multiply each channel of the intermediate attention feature h·g+h with each element of the weight vector f to generate the corresponding final attention mask (h·g+h)×f as the fused feature map output by the feature feedback enhancement layer; S3: Take the liver segmentation prediction result output by the liver segmentation model as the segmentation result of the medical image to be segmented.

2. The method of claim 1, wherein the method is characterized by: An ASPP module for capturing feature map context information at multiple scales is arranged between the fourth layer coding Res-SE-Block and the fourth layer decoding Res-SE-Block; The input of the ASPP module is the feature map output by the fourth layer Swin Transformer block and the feature map output by the fourth layer coding Res-SE-Block after downsampling; the feature map output by the ASPP module and the feature map output by the fourth layer coding Res-SE-Block are taken as the input of the fourth layer decoding Res-SE-Block.

3. The method of claim 1, wherein the method is characterized by: The Res-SE-Block includes two groups of 3x3 convolutions, one SE block and a residual connection; batch normalization and a Leakly ReLu activation function are used after each group of 3x3 convolutions.

4. The method of claim 1, wherein the method is characterized by: The Swin Transformer block includes a LayerNorm layer, a multi-head self-attention module, a residual connection and a two-layer MLP with GELU nonlinearity connected in sequence; The multi-head self-attention modules between adjacent two Swin Transformer blocks are alternately arranged as a window multi-head self-attention layer W-MSA and a displacement-based window multi-head self-attention layer SW-MSA.

5. The method of claim 4, wherein the method is characterized by, The liver segmentation model is trained by the following steps: S211: Prepare a medical image dataset, divide it into a training image dataset and a test image dataset, and perform image preprocessing on the training image dataset; S212: input the training image data after image preprocessing into the liver segmentation model for training until the model converges or reaches the maximum number of iterations; S213: test the liver segmentation model by using the test image dataset.

6. The method of claim 5, wherein the method is characterized by, When training the liver segmentation model, the model parameters of the liver segmentation model are optimized by the following training loss function: L Total = aL CE + bL Dice ; L CE = -y log(p) - (1 - y) log(1 - p); In the formula: L Total represents the training loss; L CE represents the cross-entropy loss; L Dice represents the dice loss; α and β represent the weights of the cross-entropy loss and the dice loss; y ∈ (0, 1) represents the class real value of the medical image; p ∈ [0, 1] represents the predicted probability in the medical image; N represents the number of all predicted pixels in the medical image; p i represents the predicted probability of pixel i in the medical image; y i represents the real class of pixel i in the medical image.

7. The method of claim 5, wherein the medical image liver segmentation method combining Transformer and U-Net is characterized by, In step S211, the image preprocessing includes: using a window technique to open a window in the Hounsfield unit value range [-200, 200] to remove other irrelevant tissues and enhance the contrast between the liver and other tissues; then normalizing the voxel value of the medical image to [-1, 1]; finally, normalizing the medical image.

8. A medical image liver segmentation system combining Transformer and U-Net, characterized in that, Based on the medical image liver segmentation method described in claim 1, comprising: a data collection module for collecting a medical image dataset containing a liver and generating a training image dataset and a test image dataset; a model building module for building a liver segmentation model through a PyTorch framework; a model training module for training and testing the liver segmentation model through the training image dataset and the test image dataset; a liver segmentation module for inputting a medical image to be segmented into the trained liver segmentation model to output a liver segmentation prediction result; a visualization module for visualizing and displaying the liver segmentation prediction result.

Images