Medical image segmentation method, device and medium based on a two-scale encoder network

Abstract

The application discloses a medical image segmentation method and device based on a double-scale encoder network and a medium, the method uses a trained double-scale encoder network DSE-Net to segment the target of an image for a medical image to be segmented; the DSE-Net is composed of a coarse segmentation part and a fine segmentation part; the coarse segmentation part uses a double-scale encoder to encode the medical image layer by layer, and uses a decoder to up-sample the feature map output by the encoding layer by layer until the input image size is restored, and outputs a predicted coarse segmentation map; wherein the encoder includes two scale encoding blocks; the fine segmentation part uses a refining module based on a Transformer to refine the coarse segmentation map to obtain a refined medical image segmentation map. The application uses a double-scale encoder network, can capture different field of view information, obtain more abundant context information, complement each other for the lost information, and realize more accurate and complete segmentation of the medical image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and relates to a medical image segmentation method, device and medium based on a double-scale encoder network. BACKGROUND

[0002] Medical image segmentation is an important and challenging research topic, and image segmentation provides an important scientific basis for medical image quantitative analysis. Researching a precise medical image segmentation network framework not only improves the diagnosis efficiency, but also improves the accuracy of diagnosis. U-Net has achieved great success in medical image segmentation tasks, and many models based on U-type network improvement are constantly improving the performance of medical image segmentation. In the U-type network, the encoder provides the context information captured by the image to the decoder to enhance the information lost by the decoder, and the decoder recovers to the resolution size of the input image through layer-by-layer upsampling. However, due to the limitations of the convolutional layer in the U-type segmentation framework, long-distance dependencies cannot be captured. In recent years, a novel structure Transformer has been widely used in computer vision. As an encoder, Transformer can capture long-range dependencies in images, and visual Transformer (ViT) has shown excellent performance in image segmentation. Meanwhile, in the encoding stage of the segmentation framework, single-scale convolutional layers ignore some information of rough or fine texture. SUMMARY

[0003] The problem solved in this paper is to overcome the shortcomings of the prior art, and to propose a medical image segmentation method, device and medium based on a double-scale encoder network, which can realize more accurate and complete medical image segmentation performance.

[0004] To achieve the above technical purpose, the present application adopts the following technical scheme:

[0005] A medical image segmentation method based on a double-scale encoder network, for a medical image to be segmented, uses a trained double-scale encoder network DSE-Net to segment the target of the image;

[0006] Wherein the DSE-Net is composed of a coarse segmentation part and a fine segmentation part;

[0007] The coarse segmentation part uses a double-scale encoder to encode the original input medical image layer by layer, and uses a decoder to upsample the encoded output feature map layer by layer until the input image size is restored, and outputs a predicted coarse segmentation map; wherein the double-scale encoder includes two scale encoding blocks;

[0008] The fine segmentation part uses a Transformer-based refinement module to refine the predicted coarse segmentation map to obtain a refined medical image segmentation map.

[0009] Further, the coarse segmentation part adopts a U-shaped network structure composed of multi-layer encoders and decoders; an initial convolution block is included before the first layer encoder to perform initial convolution on the original input medical image; between each layer encoder after the initial convolution block, a pooling operation is used to down-sample the input feature map step by step; and a bilinear interpolation is used between each layer decoder to up-sample step by step.

[0010] Further, in each layer encoder, each scale of the encoding block is composed of 4 basic residual blocks and 2 layers of 4-head attention Transformer, and the output is obtained by calculating the residual of the output of the basic residual block and the output of the Transformer.

[0011] Further, in the bridging layer between the last layer encoder and the decoder, the double-scale feature map obtained by the last layer encoder is fused through concatenation, convolution, BN layer and ReLU layer operation, and then input into the bridging layer built by several convolution blocks, and the output feature map of the bridging layer is concatenated with the output feature map of the last layer of the jump connection, which is used as the input feature map of the last layer decoder.

[0012] Further, in the jump connection between the same layer encoder and decoder, a channel attention fusion module is set to fuse the double-scale feature map output by the encoder; the channel attention fusion module: on the one hand, the double-scale feature map Fs i and Fl i are concatenated, and then conv, BN and ReLU layers are operated to obtain a feature map Ff i ; on the other hand, the two-scale feature maps Fs i and Fl i are fused into the channel attention module to capture the dependency between channels to obtain a feature map Finally, the feature maps Ff i and are calculated by residual to obtain the output feature map of the channel attention fusion module Where subscript i represents the level of the layer where the encoder and the decoder are located.

[0013] Further, the two-scale feature maps Fs i and Fl i are fused into the channel attention module to capture the dependency between channels, specifically:

[0014] The small-scale feature map Fs i and the large-scale feature map Fl i are reshaped into a series of flat two-dimensional patches, and then the tokens of the two-scale feature maps are spliced as key and value, that is, and tokens of small-scale and large-scale feature maps, respectively, and contact represents and concatenation operation, by concatenating vectors and ; wherein patch is a sequence block, tokens is a query, key and value are keys and values, respectively;

[0015] The corresponding weight matrix is integrated into the Transformer for calculating attention value, and the output O is calculated i :

[0016]

[0017]

[0018]

[0019] O i = MCA i + MLP(Q i + MCA i )

[0020] In the formula, and represent the i-th layer query matrix of small-scale and large-scale, K i and V i represent the i-th layer key and value matrix, and represent the i-th layer query weight matrix of small-scale feature map, query weight matrix of large-scale feature map, weight matrix of key and weight matrix of value, CA i represents the i-th layer channel fusion attention value, MCA i represents the average of N attention heads of the i-th layer, MLP represents multi-layer perceptron, d is the number of two-dimensional sequence blocks, M i is a similarity matrix, represents the i-th layer quries of 2 scale feature maps;

[0021] The output O is reshaped to form and and the i-th layer decoder feature map D i ∈R Ci×Hi×Wi is taken as the input of channel cross attention, O i and D i perform global average pooling to obtain and The output feature map is generated by multiplying the channel attention mask σ(·) and the feature map O i

[0022]

[0023] wherein σ(·) represents the relevance of each channel, and α1 and α2 are the weights of the two linear layers respectively.

[0024] Further, the Transformer-based refining module first performs an initial convolution on the predicted coarse segmentation map S1 to change the channel of S1 to obtain S2; then performs a concatenation, a conv and a ReLU operation on the double-scale feature map output by the first layer encoder to obtain a feature map F1∈R 64×H×W The output S2 of the initial convolution and F1 are concatenated, and a convolution layer, a normalization layer and a ReLU layer are used to obtain a feature map output C; the output feature map C is input into a Transformer with 4 layers of multi-head attention for learning, and the mask output by the Transformer is used for residual calculation with the coarse segmentation map S1 to obtain a final refined segmentation map.

[0025] Further, the loss function for training the double-scale encoder network DSE-Net is set as the sum of all side loss outputs:

[0026]

[0027] wherein Loss is the total loss, l k is the loss of the kth side output, and l k (k = 1, …, M) represent the loss of the total M-1 decoder outputs and the loss of the sub-segmentation part output respectively, wherein represents the dice loss and the binary cross-entropy loss of the kth side output.

[0028] An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to enable the processor to implement the medical image segmentation method based on the double-scale encoder network.

[0029] A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the medical image segmentation method based on the double-scale encoder network.

[0030] Advantages

[0031] ​The application uses a trained DSE-Net to segment medical images, wherein the DSE-Net is built based on a pytorch deep learning framework and is composed of coarse segmentation and fine segmentation, the former generates a coarse segmentation map, and the latter generates a fine segmentation map, a skeleton network adopts a U-shaped structure, each layer of encoder uses two scales of encoding blocks and each encoding block uses a basic residual block and a Transformer to extract features. Secondly, in order to better enhance the features lost by the decoder upsampling, a channel attention fusion mechanism is used to deeply fuse the double-scale features of the encoder. Thirdly, the loss function is constructed by using the output loss sum of each layer of decoder and the fine segmentation part to train the DSE-Net. The application uses convolutional layers of different scales to capture different field of view information, and the multi-scale convolutional kernels jointly encode to obtain more rich context information, which not only can complement each other for the lost information, but also is more conducive to improving the medical image segmentation performance, and can produce more accurate and complete segmentation regions for medical images. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Structure diagram of double-scale encoder network DSE-Net.

[0033] Figure 2 Channel attention fusion structure diagram for fusing double-scale feature maps.

[0034] Figure 3 Segmentation comparison diagram of DSE-Net and other advanced models on MoNuSeg dataset.

[0035] Figure 4 Segmentation comparison diagram of DSE-Net and other advanced models on GlaS dataset.

[0036] Figure 5 Segmentation comparison diagram of DSE-Net and other advanced models on KvasirCapsule-SEG dataset. DETAILED DESCRIPTION

[0037] The embodiments of the application are described in detail below. The embodiments are based on the technical solutions of the application and give detailed implementation modes and specific operation processes, and further explain and describe the technical solutions of the application.

[0038] The embodiment provides a medical image segmentation method based on a double-scale encoder network. For a medical image to be segmented, a trained double-scale encoder network DSE-Net is used for target segmentation of the image. The DSE-Net is composed of coarse segmentation and fine segmentation parts, and its network structure and training method are described in detail as follows.

[0039] 1. Selecting a dataset

[0040] First, the public medical image datasets MoNuSeg, GlaS and KvasirCapsule-SEG are selected, and these datasets are divided into training set and test set. At the same time, in order to prevent the model from overfitting, data augmentation is needed, including random enlargement, rotation and flip operations.

[0041] 2, build a double-scale encoder network DSE-Net model (as shown in Figure 1 )

[0042] The double-scale encoder network DSE-Net is composed of coarse segmentation and fine segmentation. The coarse segmentation part adopts a U-shaped network structure composed of multiple layers of double-scale encoders and decoders. Each layer of double-scale encoder includes two scale encoding blocks.

[0043] (1) Encoder stage

[0044] Let the input medical image of the double-scale encoder network DSE-Net be an image x∈R C×H×W .

[0045] First, build a convolution block that performs initial convolution on the input image x∈R C×H×W . Specifically, use a 3x3 convolution layer, a BN layer and a ReLU layer to perform initial convolution on the input image.

[0046] Then build a 4-layer encoder. In each encoding block of each layer of the encoder, use a 4-layer basic residual block and a 2-layer 4-head Transformer to extract features from the input feature map. The output is obtained by calculating the residual of the output of the basic residual block and the output of the Transformer.

[0047] The convolution scales of the two encoding blocks in each layer are 3x3 and 5x5 respectively, to obtain fine and coarse texture feature maps for information complementation. The output sizes of the 4-layer encoding blocks with small scale (3x3) are: HxW, H / 2xW / 2, H / 4xW / 4, H / 8xW / 8. The output sizes of the 4-layer encoding blocks with large scale (5x5) are the same as those of the small scale.

[0048] Between every two layers of the encoder, use the pooling operation to downsample the input feature map step by step.

[0049] (2) Skip connection

[0050] The channel attention fusion module is arranged in the skip connection between the same layer encoder and the decoder to perform channel fusion on the double scale feature maps output by the encoder, effectively utilize the double scale feature maps from the same layer encoder to enhance the information lost by the decoder upsampling. Specifically, a 4-layer multi-head attention Transformer is built.

[0051] Assuming the small scale feature and the large scale feature i=(1,2,3,4), i represents the i-th layer.

[0052] Fs i and Fl i are first concatenated, and then conv, BN and ReLU layers are operated to obtain for later use in calculating the residual output of channel attention.

[0053] Fs i and Fl i , i=(1,2,3,4) are reshaped into a series of flat two-dimensional patches (sequence blocks) with sizes p, p / 2, p / 4, p / 8 respectively. The tokens (queries) and i=(1,2,3,4) of the two scale feature maps are spliced as keys and values, i.e.

[0054]

[0055] wherein d is the number of patches. By generating a similarity matrix M i , and weighting with V i , the attention value is obtained.

[0056]

[0057] wherein represents the queries of the two scale feature maps of the i-th layer, represents the tokens of the double scale feature maps of the i-th layer.

[0058]

[0059] O i =MCA i +MLP(Q i +MCA i )

[0060] wherein N is the number of attention heads. The output is obtained by applying a simple MLP and a residual operator.

[0061] The output Reshaping formation And will And the i-th layer decoder feature map As the input of channel cross attention, O i And D i Perform global average pooling to get And The output feature map is generated using the following formula, that is, the channel attention mask σ(·) is multiplied by the feature map O i The output of the channel attention module is obtained

[0062]

[0063] Where σ(·) represents the correlation of each channel, and α1, α2 are the corresponding weights.

[0064] Finally, Ff i With Residual calculation is obtained Channel attention fusion module output feature

[0065]

[0066] (3) Bridge layer

[0067] In the bridge layer, 3 ordinary convolution blocks are built. First, the double-scale feature map obtained by the 4th layer encoder is fused through cascading, convolution, BN layer and ReLU layer operation, to obtain a feature map with a size of (512×H / 8×W / 8), and then the obtained feature map is input into the bridge layer built by several convolution blocks. Then the output feature map after the bridge layer is cascaded with the feature map of the 4th layer jump connection, which is used for the 4th layer decoder to recover the resolution size after upsampling.

[0068] (4) Decoder stage, first, the output feature map O i ∈R C×H×W And the feature map of the same layer decoder are cascaded to get R 2C×H×W , and then conv, BN, ReLU operations are used to recover the channel dimension to get the feature map of R C×H×W .

[0069] Bilinear interpolation is used for upsampling between every two decoder layers until the resolution size of the input image is restored.

[0070] (5) Transformer-based refinement module

[0071] As Figure 1The illustrated coarse segmentation part output feature S1 ∈ R 1×H×W S2 ∈ R is obtained through the conv layer, BN layer and ReLU layer 64×H×W Secondly, the first layer of double-scale features is passed through the cascade, conv layer, BN layer and ReLU layer to obtain the feature map F1 ∈ R 64 ×H×W S2 ∈ R 64×H×W C ∈ R is obtained by cascading and using convolution, normalization and ReLU 64×H×W C is sent to the 4-layer 4-attention-head Transformer refinement module for learning, and the output mask is calculated with the predicted mask to obtain the final mask.

[0072] 3. Constructing a loss function, the loss function is set as the sum of all side loss outputs, as shown in FIG. DSE-Net structure Figure 1 .

[0073]

[0074] wherein, l k is the loss of the kth side output, and M represents the total number of outputs. As shown in FIG. DSE-Net structure Figure 1 The segmentation framework is supervised in depth by 5 outputs (i.e. M = 5), including 4 outputs from the decoder prediction and one output from the refinement module.

[0075] 4. Training the DSE-Net model

[0076] The data set obtained in the foregoing is input into the model for training according to the batch size. The experiment uses a pytorch deep learning framework. The initial learning rate is set to le-3, and the epoch is set to 2000. In order to prevent model overfitting, an early stopping training strategy is adopted. Dice and IoU are used as evaluation indicators for the three data sets. The batch size of the MoNuSeg and KvasirCapsule-SEG data sets is set to 6, and the batch size of the GlaS data set is set to 4. The best Dice model is saved on the validation set.

[0077] 5. Test verification

[0078] The best saved model on MoNuSeg, GlaS and KvasirCapsule-SEG datasets is generalized to the test set for testing. The Dice and IoU obtained on the MoNuSeg dataset are: 81.171% and 68.410%. The Dice and IoU obtained on the GlaS dataset are: 91.334% and 84.853%. The Dice and IpU obtained on the KvasirCapsule-SEG dataset are: 96.421% and 93.132. Figure 3 The visualization comparison chart of DSE-Net and other models on the MoNuSeg dataset. The red box is the area where the DSE-Net model of the application performs better than other methods. Figure 4 The visualization comparison chart of DSE-Net and other models on the GlaS dataset. The red box is the area where other models perform poorly compared with the DSE-Net model of the application. It can be seen that the segmentation result of the DSE-Net model of the application is closer to the real segmentation graph. Figure 5 The visualization comparison chart of DSE-Net and other models on the KvasirCapsule-SEG dataset. The boundary segmented by the DSE-Net model of the application is closer to the actual segmentation graph. It can be seen that the DSE-Net model of the application has better generalization ability and generation ability.

[0079] In summary, the performance of the DSE-Net model for medical image segmentation has more superior competitiveness. The experimental results on the MoNuSeg, GlaS and KvasirCapsule-SEG datasets show that the DSE-Net can generate more accurate and complete segmentation regions for images.

Claims

1. A medical image segmentation method based on a two-scale encoder network, characterized in that, For a medical image to be segmented, a trained double-scale encoder network DSE-Net is used to segment the target of the image; The DSE-Net is composed of a coarse segmentation part and a fine segmentation part; The coarse segmentation part uses a double-scale encoder to encode the original input medical image layer by layer, and uses a decoder to up-sample the encoded output feature map layer by layer until the input image size is restored, and outputs a predicted coarse segmentation map; wherein the double-scale encoder includes two scales of encoding blocks; The fine segmentation part uses a Transformer-based refinement module to refine the predicted coarse segmentation map to obtain a refined medical image segmentation map; The coarse segmentation part adopts a U-shaped network structure composed of multiple layers of encoders and decoders; an initial convolution block is included before the first layer of encoder, which is used to perform initial convolution on the original input medical image; between each layer of encoder after the initial convolution block, the input feature map is down-sampled step by step using a pooling operation; and bilinear interpolation is used between each layer of decoder to up-sample step by step; Each scale of encoding block in each layer of encoder is composed of 4 basic residual blocks and 2 layers of 4-head attention Transformer, and the output is obtained by calculating the residual of the output of the basic residual block and the output of the Transformer; In the bridging layer between the last layer of encoder and the decoder, the double-scale feature map obtained by the last layer of encoder is fused through concatenation, convolution, BN layer and ReLU layer operations, and then input into the bridging layer built by several convolution blocks, and the output feature map of the bridging layer is concatenated with the output feature map of the last layer of jump connection for the input feature map of the last layer of decoder; In the skip connection between the same layer encoder and decoder, a channel attention fusion module is arranged to perform channel fusion on the double-scale feature maps output by the encoder; the channel attention fusion module: on the one hand, the double-scale feature maps output by the encoder and are cascaded, and then a conv layer, a BN layer and a ReLU layer are operated to obtain a feature map ; on the other hand, the feature maps of the two scales and are fused into a channel attention module to capture the dependency between channels, and a feature map is obtained; finally, the feature maps and are subjected to residual calculation to obtain the output feature map of the channel attention fusion module ; wherein the subscript represents the level of the layer where the encoder and the decoder are located.

2. The medical image segmentation method according to claim 1, characterized in that, Feature maps of two scales and This is integrated into the channel attention module to capture dependencies between channels, specifically: small-scale feature maps and large-scale feature maps are reshaped into a series of flat two-dimensional patches, and then the tokens of the two-scale feature maps are spliced as key and value, that is , and represent the tokens of small-scale and large-scale feature maps respectively, and contact represents and splicing operation, by splicing vectors and ; wherein patch is a sequence block, token is a query, key and value are keys and values respectively; The corresponding weight matrix is integrated into the Transformer for calculating attention values, and the output is calculated : ， ， ， ； ； ； ； In the formula, and Let represent the query matrix at the i-th level, both at small and large scales. and Let these represent the key and value matrices of the i-th layer, respectively. , , and Let represent the query weight matrix of the small-scale feature map of the i-th layer, the query weight matrix of the large-scale feature map, the weight matrix of the key, and the weight matrix of the value, respectively. This represents the attention value for channel fusion in the i-th layer. This represents averaging the N attention heads in the i-th layer, where MLP stands for Multilayer Perceptron, and d is the number of two-dimensional sequence blocks. This is a similarity matrix. Indicates the first Queries of feature maps at two scales of the layer; output reshaping formation and and the i-th layer decoder feature map as the input of channel cross attention, we have and perform global average pooling to obtain and the output feature map is generated using the following formula, i.e., channel attention mask is multiplied by the feature map to obtain the output of the channel attention module : ； wherein, represents the correlation of each channel, , are the weights of the two linear layers, respectively.

3. The medical image segmentation method according to claim 1, characterized in that, The Transformer-based refinement module first refines the predicted coarse segmentation image. Performing initial convolution changes its channels to obtain Then, the dual-scale feature maps output by the first encoder are concatenated, subjected to conv and ReLU operations to obtain the final feature map. Then convert the output of the initial convolution... and The layers are cascaded, and feature maps are obtained using convolutional layers, normalization layers, and ReLU layers. The output feature map C is then fed into a Transformer with four layers of multi-head attention for learning, and the mask output by the Transformer is compared with the coarse segmentation map. The final refined segmentation map is obtained by performing residual calculations.

4. The medical image segmentation method of claim 1, characterized by, The loss function for training the double-scale encoder network DSE-Net is set as the sum of all side loss outputs: ； wherein, is the total loss, is the loss of the kth edge output, respectively represent the loss of the total of decoder outputs and the loss of the sub- partition outputs, wherein denotes the dice loss and the binary cross-entropy loss of the kth edge output. 5.An electronic device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-4.

Images