TransUNet liver parenchyma image segmentation method based on attention mechanism

Abstract

The invention relates to a TransUNet liver parenchyma image segmentation method based on an attention mechanism, and the method comprises the steps: introducing AxialAttention and a residual connection mode in TransUNet, solving a problem of information loss in skip connection through the method, better capturing a long-range dependence relation between positions in a liver parenchyma feature map, and improving the segmentation accuracy of the liver parenchyma image. The accuracy and expression ability of the liver parenchyma segmentation model are improved.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and specifically relates to a TransUNet liver parenchyma image segmentation method based on an attention mechanism. Background Art

[0002] In the task of liver parenchyma image segmentation, the liver ultrasound image has complex edges and internal structures, and there are many organs adjacent to the liver. The existing image segmentation has the technical problem of low accuracy. Summary of the invention

[0003] In order to solve the problem of inaccurate liver image segmentation in the prior art, the technical objective of this application is to provide a TransUNet liver parenchyma image segmentation method based on an attention mechanism.

[0004] This application first provides a TransUNet liver parenchyma image segmentation method based on an attention mechanism, the method comprising the following steps:

[0005] Step 1: Preprocessing process: preprocessing the liver disease images and label image data; normalizing the preprocessed images to generate a liver disease image database for training the segmentation network and a label database for the segmentation network;

[0006] Step 2: Liver image segmentation model processing: introducing the TransUNet model with axial attention and residual connection, and learning the liver disease image database and the label database to form a deep learning model;

[0007] The deep learning model segments the sample image to obtain a segmented image;

[0008] The TransUNet model consists of three parts: encoder, decoder and skip connection. Axial Attention mechanism and residual connection are introduced in the skip connection.

[0009] Step 3: Evaluation process: Select one or more of the Dice coefficient, Jaccard index or Sensitivity to evaluate the accuracy of the segmented image.

[0010] In some embodiments, the axial attention uses two independent attention mechanisms to calculate the attention weights in the width direction and the height direction respectively.

[0011] In some embodiments, the operation process of Axial Attention is as follows:

[0012] S11: Obtaining row vectors and column vectors by applying two linear mappings to the input liver parenchyma feature map;

[0013] S12: Normalize the row vector and the column vector using the SoftMax function to obtain row attention and column attention respectively; wherein in the SoftMax function, each element of the row vector and the column vector is normalized to a non-negative number, and the sum is equal to 1;

[0014] For a given input liver feature map X, the height and width of the feature map are represented as H and W respectively, where the pixel at position (i, j) in each feature map has a vector q ij , k ij and v ij ,

[0015] The q ij represents the query vector of the feature map X at position (i, j), the k ij represents the key vector of the feature graph X at position (i, j), the v ij Represents the value vector of the feature map X at position (i, j);

[0016] For a point (i, j) on the feature map, the attention value in the width direction is as follows:

[0017]

[0018] The attention value in the height direction is as follows:

[0019]

[0020] S13: Multiply the row attention and column attention to get the final attention weight;

[0021] S14: Use the attention weights to perform weighted summation on the original liver parenchyma feature map to obtain an output liver feature map with global context information.

[0022] In some embodiments, the residual connection step is as follows: add the elements of the input x and F(x) at corresponding positions one by one to obtain a new output Y, where F(x) refers to the output feature map generated after axial attention calculation, and the expression of the residual connection is as follows: Y=F(x)+x.

[0023] In some embodiments, in step 3, the Dice coefficient is an indicator for measuring the similarity between the liver parenchyma segmentation result and the true liver parenchyma label, and its expression is as follows:

[0024]

[0025] Where A and B represent the pixel sets of the liver parenchyma segmentation result and the true liver parenchyma label, respectively. |A| represents the number of elements in A, and |B| represents the number of elements in B.

[0026] and / or;

[0027] The Jaccard index is an indicator for measuring the similarity between the liver parenchyma segmentation result and the true liver parenchyma label, called Intersection over Union (IoU), and its expression is as follows:

[0028]

[0029] and / or;

[0030] The Sensitivity index measures the model's ability to identify true positive samples, and its expression is as follows:

[0031]

[0032] Among them, TP represents true positives: the number of samples correctly predicted by the model as positive, and FN represents false negatives: the number of samples incorrectly predicted by the model as negative.

[0033] In some embodiments, the Dice coefficient, Jaccard index, and Sensitivity evaluation index reach 95% to 97%, 91% to 94%, and 94% to 98%, respectively.

[0034] In some embodiments, in step 1, the preprocessing includes any one or more of histogram equalization, image horizontal flipping, and image elastic deformation transformation;

[0035] And / or, the color depth of the pre-processed 32-bit original image is reduced and saved as a 24-bit ultrasound image.

[0036] In some implementations, the encoder performs the following processing steps:

[0037] S21: In TransUNet, the pre-trained ResNet-50 model is used as the feature extractor;

[0038] S22: Input image processing: The input liver parenchyma ultrasound image undergoes a series of convolutional layers for preliminary feature extraction; the convolutional layers capture the local texture, edge, and grayscale change information of the image through filters to generate a series of feature maps;

[0039] S23: Feature map downsampling: Use the maximum pooling layer between convolutional layers for downsampling;

[0040] S24: Features are fed into the VIT encoder: The features processed by the convolutional layer and the pooling layer are fed into the VisionTransformer / VIT encoder for further encoding;

[0041] S25: VIT encoder processing: The feature map is divided into a series of small blocks, each of which contains part of the image information; the small blocks are converted into high-dimensional vector representations through a linear embedding layer, and the position information of each small block is also encoded into a position embedding vector and added to the small block embedding vector; the self-attention mechanism helps the model capture the correlation and dependency between different regions. By calculating the relationship weight between each small block and other small blocks, the model can update the feature representation of each small block to include more global and contextual information;

[0042] S26: Feedforward neural network FFN layer: The FFN layer in the VIT encoder is used to further process the output of the self-attention layer and increase the nonlinear expression ability of the model; the FFN layer performs more complex transformations and combinations on the features to extract higher-level feature representations;

[0043] S27: VIT decoder processing: The output of the encoder is fed into the VIT decoder for decoding; the decoder contains self-attention layers and FFN layers to process features and generate segmentation maps; the self-attention mechanism in the decoder is used to focus on the relationship between different features and generate refined segmentation results;

[0044] S28: Upsampling or deconvolution operation: To generate a segmentation map of the same size as the input image, the decoder uses upsampling or deconvolution operations to gradually increase the resolution of the feature map; a low-resolution feature map can be mapped to a high-resolution segmentation map, thereby retaining more detail information and improving segmentation accuracy.

[0045] In some implementations, the decoder processes the following steps:

[0046] S31: Fusing the upsampled feature map with the feature map of the corresponding level from the encoder;

[0047] S32: The fused feature map undergoes a series of deconvolution layers and upsampling operations to finally generate a predicted segmentation image of the same size as the input image;

[0048] The feature fusion may be implemented in the following ways including but not limited to: concatenation fusion, element-by-element addition fusion or convolutional layer fusion.

[0049] In some embodiments, the liver parenchyma image is a Schistosoma japonicum liver disease image.

[0050] The following is an explanation and description of the terms used in this application:

[0051] Instruction Manual Figure 1 This is the network structure diagram of the unimproved TransUNet, Transformer structure: After the image block is embedded, the conventional Layer Norm (layer normalization) + MSA (self-attention) + MLP (multi-layer perceptron) + residual connection structure processing is performed. The complete structure of TransUNet is a U-shaped encoding and decoding structure. The encoder part is the key part of TransUNet. The encoder first performs three-layer convolution downsampling on the input image, embeds the image block on the feature map obtained by CNN, applies position encoding, and then inputs the one-dimensional vector after block embedding into the 12-layer Transformer structure. The decoder is a conventional deconvolution upsampling, combined with a high-resolution CNN feature map, to restore the feature map to the original image size and generate a pixel-level segmentation result. TransUNet introduces a hybrid encoder based on the U-Net model, combining CNN and Transformer to solve the limitations of traditional convolutional neural networks in modeling long-distance dependencies and processing large-size images.

[0052] ResNet-50 is used as a feature extractor to extract features from the input image;

[0053] Transformer Block contains multiple Transformer layers (n=12) to process the features extracted by ResNet. The Transformer layer enhances the feature representation through self-attention mechanism (MSA) and multi-layer perceptron (MLP).

[0054] Axial Attention: In Transformer Block, the Axial Attention module is used to apply attention mechanism on a specific axis, which helps the model capture local and global features of the image;

[0055] Linear Projection: Converts the hidden features extracted by ResNet into a format suitable for Transformer processing;

[0056] Conv3x3,ReLU: Use 3x3 convolution and ReLU activation function for feature processing.

[0057] Beneficial technical effects of this application:

[0058] 1. This application proposes an improved liver parenchyma image segmentation algorithm based on TransUNet, in which Axial Attention and residual connection are introduced. This method solves the problem of information loss in skip connection, and can better capture the long-range dependency between positions in the liver parenchyma feature map, thereby improving the accuracy and expression ability of the liver parenchyma segmentation model.

[0059] 2. The Dice and IoU evaluation indicators of the improved segmentation method in this application on the ultrasound dataset reached 96.5% and 93.1% respectively, showing superior segmentation performance than the original TransUNet model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is the network structure diagram of the TransUNet model that has not been improved in this application;

[0061] Figure 2 This is a flowchart of ultrasound image data preprocessing for this application;

[0062] Figure 3 This is the improved TransUNet model for this application. It introduces axial attention and residual connection methods based on the TransUNet model.

[0063] Figure 4 This is the calculation flow chart of the Axial Attention module;

[0064] Figure 5 is the residual connection structure diagram;

[0065] Figure 6 Different visualization segmentation results of U-net, original TransUNet and the improved TransUNet in this application;

[0066] Figure 7 This is a comparison chart of the effect of patch size on the segmentation performance of the model;

[0067] Figure 8 This is a comparison chart showing the impact of batch_size on the segmentation performance of the model. DETAILED DESCRIPTION

[0068] The present application is further described below by way of examples, but is not intended to limit the present application.

[0069] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0070] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0071] Unless expressly stated to the contrary, all ranges cited herein are inclusive.

[0072] The term "a" or "an" used in this application is used to describe the elements and components described in this application. This is done only for convenience and to provide a general sense of the scope of this application. This description should be understood to include one or at least one, and the singular also includes the plural, unless it is obvious that it is otherwise intended. "Multiple" means two or more.

[0073] The numbers in this application are approximate, regardless of whether the words "about" or "approximately" are used. The numerical values ​​of the numbers may vary by 1%, 2%, 5%, 7%, 8%, 10%, etc. Whenever a number with a value of N is disclosed, any number with a value of N+ / -1%, N+ / -2%, N+ / -3%, N+ / -5%, N+ / -7%, N+ / -8% or N+ / -10% will be explicitly disclosed, where "+ / -" means plus or minus, and the range between N-10% and N+10% is also disclosed.

[0074] When using ultrasound images to further diagnose patients with schistosomiasis liver disease, it is essential to segment the liver parenchyma. Ultrasound images of the liver usually present complex textures and echo patterns, and the boundaries may be blurred or overlapping, so segmentation can more clearly define the liver area. The echo intensity, texture, and morphology of the liver are important bases for the accurate diagnosis of schistosomiasis liver disease. Segmentation ensures that these features are only extracted from the liver area to avoid interference from irrelevant information. Problems such as artifacts and image quality degradation that may exist in ultrasound images also require the segmentation process to be stable and robust. Therefore, combined with the characteristics of liver ultrasound images, accurate segmentation of schistosomiasis liver ultrasound images can provide effective support for the subsequent diagnosis and treatment of the disease.

[0075] In the past few decades, many automatic liver ultrasound image segmentation methods developed by researchers can be divided into three categories: (1) based on traditional image analysis, (2) based on machine learning, and (3) based on deep learning. Based on traditional image analysis methods, most of them are semi-automatic, easily affected by noise, and the segmentation process is complicated. Although the machine learning-based methods have greatly improved the segmentation accuracy, they have high requirements for the design of identification features. However, as medical images become more and more complex, it is difficult to obtain ideal images. The deep learning-based methods can automatically learn and identify features according to the provided dataset to achieve accurate segmentation of liver ultrasound images.

[0076] TransUNet is a medical image segmentation model that combines U-Net and Transformer. It improves the performance of medical image segmentation by combining the local feature extraction capability of CNN with the global context modeling capability of Transformer. The architecture of TransUNet includes a Transformer encoder and a decoder, in which the encoder uses a self-attention mechanism to extract global context information, while the decoder uses upsampling and skip connections to achieve precise positioning. In the TransUNet model, after the image block is embedded, the conventional Layer Norm layer normalization + MSA self-attention + MLP multi-layer perceptron + residual connection structure processing is performed. The complete structure of TransUNet is a U-shaped encoding and decoding structure. The encoder part is the key part of TransUNet. The encoder first performs three-layer convolution downsampling on the input image, embeds the image block on the feature map obtained by CNN, applies position encoding, and then inputs the one-dimensional vector after the block embedding into the 12-layer Transformer structure. The decoder is a conventional deconvolution upsampling, combined with a high-resolution CNN feature map, to restore the feature map to the original image size and generate a pixel-level segmentation result. TransUNet introduces a hybrid encoder based on the U-Net model, combining CNN and Transformer to address the limitations of traditional convolutional neural networks in modeling long-distance dependencies and processing large-size images.

[0077] However, in the U-Net network, due to the limitations of the convolution operation, the model cannot effectively capture relevant contextual information when dealing with scenes with long-distance dependencies or global structural influences, which in turn affects the accuracy of the liver parenchyma segmentation model.

[0078] In order to solve these problems and optimize the z model, this application introduces Axial Attention and residual connection methods into the skip connection in the TransUNet model. In the TransUNet model, the feature map of the skip connection is reduced in resolution through downsampling in the encoder, and then restored to the original resolution through upsampling in the decoder, which may cause loss and blurring of feature information. Axial Attention can be weighted according to the importance of the input data in different dimensions, and can better capture and enhance targets with specific directional properties such as the liver edge, thereby more accurately extracting features and enhancing feature representation and information fluidity. At the same time, the use of residual connection can effectively alleviate gradient problems and information loss, enhance the expressive power of the model, and enable the liver segmentation model to better capture and understand the features in the image, thereby improving the accuracy and stability of the liver image segmentation model.

[0079] (1) Evaluation indicators:

[0080] In order to evaluate the performance and accuracy of the liver parenchyma segmentation model of this application, this application adopts multiple evaluation indicators, including Dice coefficient, Jaccard index (IoU), and Sensitivity.

[0081] Dice Coefficient

[0082] The Dice coefficient is an indicator to measure the similarity between the liver parenchyma segmentation result and the true liver parenchyma label. It calculates the overlap between the liver parenchyma segmentation result and the true liver parenchyma label. The value range is between 0 and 1. The closer to 1, the more similar the segmentation result is to the true label. Its expression is shown as follows:

[0083]

[0084] Among them, A and B represent the pixel sets of the liver parenchyma segmentation result and the true liver parenchyma label, respectively, |A| represents the number of elements in A, and |B| represents the number of elements in B.

[0085] Jaccard Index

[0086] The Jaccard index is also an indicator to measure the similarity between the liver parenchyma segmentation result and the true liver parenchyma label, also known as the intersection over Union (IoU). It calculates the ratio between the intersection and union of the liver parenchyma segmentation result and the true liver parenchyma label. The value range is also between 0 and 1. The closer to 1, the more similar the liver parenchyma segmentation result is to the true liver parenchyma label. Its expression is shown in the following formula:

[0087]

[0088] Sensitivity evaluation index

[0089] The Sensitivity index measures the model's ability to identify true positive samples. It represents the ratio of the number of samples correctly classified as positive to the number of all true positive samples. Its expression is shown in the following formula.

[0090]

[0091] Among them, TP represents true positives, which is the number of samples correctly predicted by the model as positive, and FN represents false negatives, which is the number of samples incorrectly predicted by the model as negative.

[0092] (2) Experimental environment and parameter settings:

[0093] The experimental environment and parameter settings of this application are shown in Table 1-1.

[0094] Table 1-1 Experimental environment and parameter settings

[0095]

[0096]

[0097] The input image size is 224×224, the learning rate is set to 1E-4, the batch size is set to 16, and the patch size is set to 16×16 according to the settings of TransUNet and ViT. The number of skip connections is set to 3 according to the number of skip connections in TransUNet, and the SGD optimizer is used (learning rate is 0.01, momentum is 0.9, weight decay is 1E-4), and a combination of cross entropy loss (CE) and Dice loss (Diceloss) is used, as shown in the following formula. CE_Loss is used to evaluate the performance of classification tasks, and Dice_Loss is used to measure the performance of segmentation tasks.

[0098] Loss=0.5×(CE_Loss+Dice_Loss)

[0099] (3) Preprocessing: Liver parenchyma segmentation dataset:

[0100] The liver parenchyma segmentation dataset used in this application is a preprocessed dataset of 520 images, of which 417 images are selected as the training set, and 103 images are retained as the test set.

[0101] The preprocessing process is as follows: The data set only contains 130 ultrasound images of schistosomiasis liver disease. However, in practical applications, for deep learning models, the number of such data sets is relatively limited, which may lead to model overfitting and poor generalization ability. To solve this problem, a series of image preprocessing and data amplification techniques such as histogram equalization, horizontal flipping, and elastic degeneration are used to expand the data set to 520 images. At the same time, in order to meet specific storage or processing requirements, the color depth of the 32-bit original image after the preprocessing operation is reduced and saved as a 24-bit ultrasound image, thereby optimizing storage space and processing efficiency while retaining key color information. First, the schistosomiasis liver disease image and label image data are preprocessed, and then the preprocessed images are normalized, see the attached figure of the specification, to generate schistosomiasis liver disease image data for training the segmentation network, and label data for the segmentation network, see the attached figure of the specification. Figure 2 .

[0102] (4) Construction of liver parenchyma segmentation model based on attention mechanism and TransUNet:

[0103] The improved TransUNet model of this application introduces Axial Attention and residual connection methods based on the existing TransUNet model, see the attached figure of the specification Figure 3 The model consists of three parts: encoder, decoder and skip connection.

[0104] 4.1 Encoder:

[0105] In TransUNet, a pre-trained ResNet-50 model is used as a feature extractor (back-bone) to extract the liver parenchyma features of the input image. The input liver parenchyma ultrasound image first passes through a series of convolutional layers for preliminary feature extraction. These convolutional layers capture information such as local texture, edges, and grayscale changes in the image through filters to generate a series of feature maps. In order to reduce the amount of computation and memory consumption while retaining important feature information, the maximum pooling layer is used between the convolutional layers for downsampling operations. This can reduce the resolution of the feature map while increasing the receptive field, allowing the model to capture more global information.

[0106] After being processed by the convolutional layer and the pooling layer, the extracted features are sent to the VIT (Vision Transformer) encoder for further encoding. In the VIT encoder, the feature map is divided into a series of small patches, each of which contains part of the image information. These patches are converted into high-dimensional vector representations through a linear embedding layer, and in order to retain the spatial position information, the position information of each patch is also encoded into a position embedding vector and added to the patch embedding vector. Then in the ultrasound image of the liver parenchyma, the self-attention mechanism can help the model capture the correlation and dependency between different regions. By calculating the relationship weight between each patch and all other patches, the model can update the feature representation of each patch to contain more global and contextual information.

[0107] In addition to the self-attention layer, the feedforward neural network (FFN) layer in the VIT encoder is used to further process the output of the self-attention layer and increase the nonlinear expression capability of the model. Through the FFN layer, the model can perform more complex transformations and combinations on features to extract higher-level feature representations.

[0108] The output of the encoder is fed into the VIT decoder for decoding. Similar to the encoder, the decoder also contains self-attention layers and FFN layers to process features and generate segmentation maps. However, in the decoder, the self-attention mechanism is used to focus on the relationship between different features and generate refined segmentation results.

[0109] In order to generate a segmentation map of the same size as the input image, the decoder needs to gradually increase the resolution of the feature map by upsampling or deconvolution operations. These operations can map low-resolution feature maps to high-resolution segmentation maps, thereby retaining more detail information and improving segmentation accuracy.

[0110] 4.2 Decoder

[0111] In order to improve segmentation accuracy and retain more detailed information, the decoder will fuse the upsampled feature map with the feature map of the corresponding level from the encoder. This fusion operation allows the model to use both low-level and high-level feature information during the decoding process, thereby obtaining more accurate and detailed segmentation results. In liver parenchymal ultrasound images, this feature fusion helps the model better capture information such as the boundaries and internal structures of the liver parenchyma.

[0112] Feature fusion can be achieved in different ways, such as concatenation, element-by-element addition, or fusion using convolutional layers. The fused feature map then goes through a series of deconvolution layers and upsampling operations to ultimately generate a predicted segmentation image of the same size as the input image. In this process, the model will gradually recover the detailed structure and boundary information of the liver parenchyma.

[0113] 4.3Axial Attentional Mechanism

[0114] Introduce axial attention and residual connection in skip connection.

[0115] First, the pre-trained ResNet-50 encoder is used to extract liver parenchyma features from the input image to generate corresponding feature maps. These feature maps are then used as input to the Axial Attention module, which then decomposes the feature maps into two axes: rows and columns. Self-attention calculations are performed on each axis to obtain global contextual information in the axial direction, capturing the global information of the input liver parenchyma feature image.

[0116] Then, the liver parenchyma feature map output by axial attention is added to the liver parenchyma feature map of the encoder layer through residual connection, which retains the information of the original liver parenchyma feature map. This avoids information loss and helps the network learn local detail features. The introduction of residual connection further enhances the ability of information transmission and helps the decoder to better restore the edge details of the liver parenchyma. In this way, the network can better utilize the feature map of the encoder layer to help the decoder generate accurate segmentation results.

[0117] Axial Attention is an attention mechanism based on the Transformer model for processing two-dimensional images or sequence data. The traditional attention mechanism treats the entire input feature map as a sequence and calculates the attention weight of each position at all positions. However, for two-dimensional images, this processing method may ignore the long-range dependencies between positions in the image. Therefore, Axial Attention uses two independent attention mechanisms to calculate the attention weights in the width direction and the height direction respectively. Among them, Axial Attention can be weighted according to the importance of the input data in different dimensions, and can better capture and strengthen targets with specific directional properties such as the liver edge, so as to more accurately extract features, enhance feature representation and information fluidity. At the same time, the use of residual connections can effectively alleviate gradient problems and information loss, enhance the expressiveness of the model, enable the liver segmentation model to better capture and understand the features in the image, and thereby improve the accuracy and stability of the liver image segmentation model.

[0118] It is specifically carried out in the following way: first, two linear mappings are applied to the input liver parenchyma feature map to obtain row vectors and column vectors. Next, the row vectors and column vectors are normalized using the SoftMax function to obtain row attention and column attention. In the SoftMax function, each element of the row vector and column vector is normalized to a non-negative number, and their sum is equal to 1. Therefore, row attention and column attention represent the relative correlation between each position and all other positions. For a given input liver feature map X, the height and width of the feature map are represented as H and W, respectively, where each pixel at position (i, j) in the feature map has a vector q ij , k ij and v ij , where q ij represents the query vector at position (i, j) of feature map X, where k ij represents the key vector of feature graph X at position (i, j), where v ij Represents the value vector at position (i, j) of feature map X. The attention value in the width direction for a point (i, j) on the feature map is shown as follows.

[0119]

[0120] Similarly, the attention value in the height direction is as follows:

[0121]

[0122] Finally, the row attention and column attention are multiplied to obtain the final attention weight. The original liver parenchyma feature map is weighted summed using the attention weight to obtain the output liver feature map with global context information. The calculation process of the AxialAttention module is illustrated in the figure Figure 4 As shown in the figure, Axial Attention can effectively model the global and local relationship in the liver parenchyma feature image by performing independent attention calculations on different axes of the liver parenchyma feature image, thereby improving the accuracy of the liver parenchyma segmentation results.

[0123] 4.4 Residual Connection

[0124] Residual connection refers to adding the output of the previous layer directly to the input of the next layer. It constitutes a cross-layer connection method. The main purpose of this cross-layer connection is to solve the problem of gradient disappearance and network degradation, and to accelerate the training and convergence of the network. Figure 5 Specifically, the input x and F(x) are added one by one at the corresponding position to obtain a new output Y. F(x) refers to the output feature map generated after the Axial Attention calculation. The expression of the residual connection is shown in the following formula.

[0125] Y=F(x)+x.

[0126] In order to better explain the present application, the following is a detailed description in conjunction with specific embodiments. The experimental methods used in the following examples are all conventional methods unless otherwise specified.

[0127] Example 1

[0128] A method for image segmentation using the above method is provided.

[0129] Visualized result analysis shows that the attached figure of the specification Figure 6 The visualization results of liver parenchyma segmentation are compared between three randomly selected test set input ultrasound images and their corresponding original liver parenchyma labels.

[0130] In the drawings of the specification Figure 6 In (a), U-net, the original TransUNet, and the improved TransUNet using the algorithm of the present application all demonstrate the ability to effectively segment the liver parenchyma. Figure 6 In the comparison between (b) and (c), it can be clearly seen that the improved TransUNet using the algorithm of the present application is more accurate in identifying the edge of the liver parenchyma. In contrast, U-net and the original TransUNet cannot capture and depict the contour of the liver parenchyma as accurately as the model of the present application when processing edge details.

[0131] Judging from the visualization results, the TransUNet model improved by the algorithm of this application shows better performance in the liver image segmentation task, and can more accurately identify the liver edge and provide more refined segmentation results.

[0132] In order to explore the impact of different factors on model performance, this application will analyze the impact of patch size, batch_size, input image resolution, and model structure design on model performance.

[0133] Verification Example 1: Design comparison and analysis of model structure results

[0134] Table 2 shows the design comparison of the model structure, comparing only introducing axial attention and combining axial attention with residual connection. The experimental results show that the model combining axial attention with residual connection has better segmentation effect.

[0135] Table 2 Design comparison of model structures

[0136]

[0137] Verification Example 2: Analysis of the impact of patch size and batch_size on the segmentation performance of the model

[0138] Table 3 and Figure 7 The effect of patch size on the segmentation performance of the model is shown in Table 3. As can be seen from Table 3, when the patch size is 8, its Dice coefficient and IoU coefficient are 96.7% and 93.4% respectively, which are higher than those when the patch size is 16 and 24. Figure 7 The data in Table 3 show that in the liver image segmentation task, a smaller Patch size may be more conducive to maintaining the accuracy of segmentation. However, this application sets the Patch size to 16×16 according to the settings of TransUNet and ViT.

[0139] Table 3 Comparison of the impact of patch size on the segmentation performance of the model

[0140]

[0141] Analysis of the impact of batch_size on the segmentation performance of the model

[0142] Table 4 uses the Dice coefficient, IoU and SE three evaluation indicators to evaluate the impact of different batch_sizes on the model segmentation performance. Table 4 and Figure Figure 8 It shows that when batch_size is 16, the Dice coefficient and IoU reach the highest values ​​of 96.5% and 93.1%, and SE is also close to the best. Therefore, based on comprehensive consideration of model performance and training efficiency, this application selects batch_size of 16 for model training to ensure the stability of model training and improve the segmentation accuracy of the model.

[0143] Table 4 Comparison of the impact of batch_size on the segmentation performance of the model

[0144]

[0145] Verification Example 3: The impact of input image size on the segmentation performance of the model

[0146] Table 5 shows the effect of input image size on the segmentation performance of the model. The experimental results show that the segmentation accuracy is higher when the original data image size is 512×512, while the image size of 224×224 is obtained by reducing the original image, which may lose some feature information and lead to unsatisfactory segmentation results. However, the size of the input image has high requirements on the experimental equipment during training. Due to the limitation of the experimental equipment, this application uses a 224×224 input image.

[0147] Table 5 Comparison of the impact of input image size on the segmentation performance of the model

[0148]

[0149]

[0150] Verification Example 4: Comparative Experimental Results Analysis of Different Liver Segmentation Models

[0151] Table 6 shows the comparison results between the improved TransUNet model of this application and different models on the schistosomiasis liver disease dataset. The performance of liver segmentation is evaluated by three evaluation indicators: Dice, IoU, and SE. The Dice, IoU, and SE evaluation indicators of the improved TransUNet model of this application reached 96.5%, 93.1%, and 97.0%, respectively. The Dice, IoU, and SE evaluation indicators of the original TransUNet model were improved by 1.0 percentage points, 1.8 percentage points, and 0.9 percentage points, respectively. The results show that the segmentation performance of the improved TransUNet model of this application is better than that of the original TransUNet model.

[0152] Table 6 Comparative experimental results of different liver segmentation models

[0153]

[0154] Experimental results show that the Dice and IoU evaluation indicators of the improved model in this application on the ultrasound dataset reached 96.5% and 93.1% respectively, showing superior segmentation performance than the original TransUNet model.

[0155] In summary, this application proposes an improved liver parenchyma image segmentation algorithm based on TransUNet, in which Axial Attention and residual connection are introduced. This method solves the problem of information loss in skipconnection and can better capture the long-range dependency between positions in the liver parenchyma feature map, thereby improving the accuracy and expression ability of the liver parenchyma segmentation model.

[0156] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A TransUNet liver parenchyma image segmentation method based on attention mechanism, characterized by: The method comprises the following steps: Step 1: Preprocessing process: preprocessing the liver disease images and label image data; normalizing the preprocessed images to generate a liver disease image database for training the segmentation network and a label database for the segmentation network; Step 2: Liver image segmentation model processing: introducing the TransUNet model with AxialAttention and residual connection, and learning the liver disease image database and the label database to form a deep learning model; The deep learning model segments the sample image to obtain a segmented image; The TransUNet model consists of three parts: encoder, decoder and skip connection. The skip connection introduces AxialAttention mechanism and residual connection. Step 3: Evaluation process: Select one or more of the Dice coefficient, Jaccard index or Sensitivity to evaluate the accuracy of the segmented image.

2. The segmentation method according to claim 1, characterized in that: The AxialAttention uses two independent attention mechanisms to calculate the attention weights in the width direction and the height direction respectively.

3. The segmentation method according to any one of claims 1 to 2, characterized in that: The operation process of AxialAttention is as follows: S11: Obtaining row vectors and column vectors by applying two linear mappings to the input liver parenchyma feature map; S12: Normalize the row vector and the column vector using the SoftMax function to obtain row attention and column attention respectively; wherein in the SoftMax function, each element of the row vector and the column vector is normalized to a non-negative number, and the sum is equal to 1; For a given input liver feature map X, the height and width of the feature map are represented as H and w respectively, where the pixel at position (i, j) in each feature map has a vector q ij , k ij and v ij , The q ij represents the query vector of the feature map X at position (i, j), the k ij represents the key vector of the feature graph X at position (i, j), the v ij Represents the value vector of the feature map X at position (i, j); For a point (i, j) on the feature map, the attention value in the width direction is as follows: The attention value in the height direction is as follows: S13: Multiply the row attention and column attention to get the final attention weight; S14: Use the attention weights to perform weighted summation on the original liver parenchyma feature map to obtain an output liver feature map with global context information.

4. The segmentation method according to any one of claims 1 to 2, characterized in that: The steps of the residual connection are as follows: add the elements of the corresponding positions of the input x and F(x) one by one to obtain a new output Y, where F(x) refers to the output feature map generated after AxialAttention calculation. The expression of the residual connection is shown as follows: Y=F(x)+x.

5. The segmentation method according to any one of claims 1 to 2, characterized in that: In step 3, the Dice coefficient is an indicator for measuring the similarity between the liver parenchyma segmentation result and the true liver parenchyma label, and its expression is as follows: Where A and B represent the pixel sets of the liver parenchyma segmentation result and the true liver parenchyma label, respectively. |A| represents the number of elements in A, and |B| represents the number of elements in B. and / or; The Jaccard index is an indicator for measuring the similarity between the liver parenchyma segmentation result and the true liver parenchyma label, called Intersection over Union (IoU), and its expression is as follows: and / or; The Sensitivity index measures the model's ability to identify true positive samples, and its expression is as follows: Among them, TP represents true positives: the number of samples correctly predicted by the model as positive, and FN represents false negatives: the number of samples incorrectly predicted by the model as negative.

6. The segmentation method according to claim 5, characterized in that: Among them, the Dice coefficient, Jaccard index and Sensitivity evaluation indicators reached 95%-97%, 91%-94%, and 94%-98% respectively.

7. The segmentation method according to any one of claims 1 to 2, characterized in that: In the step 1, the preprocessing includes any one or more of histogram equalization, image horizontal flipping and image elastic deformation transformation; And / or, the color depth of the pre-processed 32-bit original image is reduced and saved as a 24-bit ultrasound image.

8. The segmentation method according to any one of claims 1 to 2, characterized in that: The processing steps of the encoder are as follows: S21: In TransUNet, the pre-trained ResNet-50 model is used as the feature extractor; S22: Input image processing: The input liver parenchyma ultrasound image undergoes a series of convolutional layers for preliminary feature extraction; the convolutional layers capture the local texture, edge, and grayscale change information of the image through filters to generate a series of feature maps; S23: Feature map downsampling: Use the maximum pooling layer between convolutional layers for downsampling; S24: Features are fed into the VIT encoder: The features processed by the convolutional layer and the pooling layer are fed into the VisionTransformer / VIT encoder for further encoding; S25: VIT encoder processing: The feature map is divided into a series of small blocks, each of which contains part of the image information; the small blocks are converted into high-dimensional vector representations through a linear embedding layer, and the position information of each small block is also encoded into a position embedding vector and added to the small block embedding vector; the self-attention mechanism helps the model capture the correlation and dependency between different regions. By calculating the relationship weight between each small block and other small blocks, the model can update the feature representation of each small block to include more global and contextual information; S26: Feedforward neural network FFN layer: The FFN layer in the VIT encoder is used to further process the output of the self-attention layer and increase the nonlinear expression ability of the model; the FFN layer performs more complex transformations and combinations on the features to extract higher-level feature representations; S27: VIT decoder processing: The output of the encoder is fed into the VIT decoder for decoding; the decoder contains self-attention layers and FFN layers to process features and generate segmentation maps; the self-attention mechanism in the decoder is used to focus on the relationship between different features and generate refined segmentation results; S28: Upsampling or deconvolution operation: To generate a segmentation map of the same size as the input image, the decoder uses upsampling or deconvolution operations to gradually increase the resolution of the feature map; a low-resolution feature map can be mapped to a high-resolution segmentation map, thereby retaining more detail information and improving segmentation accuracy.

9. The segmentation method according to any one of claims 1 to 2, characterized in that: The processing steps of the decoder are as follows: S31: Fusing the upsampled feature map with the feature map of the corresponding level from the encoder; S32: The fused feature map undergoes a series of deconvolution layers and upsampling operations to finally generate a predicted segmentation image of the same size as the input image; The feature fusion may be implemented in the following ways including but not limited to: concatenation fusion, element-by-element addition fusion or convolutional layer fusion.

10. The segmentation method according to any one of claims 1 to 2, characterized in that: The liver parenchyma image is an image of Schistosoma japonicum liver disease.

Images