Medical image segmentation method based on channel-mixed codec network

Abstract

The present invention particularly relates to a medical image segmentation method based on a channel-mixed codec network, comprising the following steps: obtaining a medical image to be segmented; importing the medical image to be segmented into a trained network model for identification to obtain a segmented medical image; the basic structure of the trained network model is a symmetrical U-Net network structure, and the feature map of the decoder part in the U-Net network structure is obtained by processing and fusing the feature map of the same size in the encoder through a self-attention module and a maximum upsampling index matrix. In the medical image segmentation method proposed in the present invention, most of the segmentation algorithms refer to the skip connection of U-Net to fuse information at different scales, thereby guiding the purpose of information transmission; different from the traditional algorithm, the present method proposes a channel-mixed self-attention module to replace the skip-connect structure and the maximum index upsampling to replace the transposed convolution to achieve feature upsampling of the decoding network.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image recognition, and in particular to a medical image segmentation method based on a channel-mixed encoding and decoding network. Background Art

[0002] At present, computer vision technology is applied to multiple scenarios, including image classification, target detection, 3D reconstruction, and semantic segmentation. With the rapid development of Internet communications, the competitiveness of intelligent products requires technological breakthroughs in more advanced semantic scene understanding. Therefore, semantic segmentation, as a core issue of computer vision, can help more and more products automatically and efficiently understand the relevant knowledge or semantics in images or videos, thereby achieving intelligent goals, reducing manual interaction operations, and improving customer comfort. Currently, these products have been widely used in autonomous driving, human-computer interaction, computational photography, image search engines, augmented reality and other fields.

[0003] The semantic segmentation problem in computer vision is essentially a process from rough reasoning to refined reasoning. At the beginning, it can be traced back to the classification problem, that is, roughly predicting the object category in the input sample, and then the positioning and detection of the target object, which not only predicts the category of the object, but also provides additional information about the spatial position of each category, such as the center point or the border of the object area. On this basis, semantic segmentation can be understood as a fine-grained prediction in the detection field. The test image is input into the segmentation network so that the predicted heat map size is consistent with the input image. The number of channels is equal to the number of categories, which represents the probability of each spatial position belonging to each category, that is, it can be classified pixel by pixel.

[0004] The fully convolutional network (FCN) has become the cornerstone of deep learning technology applied to semantic segmentation problems. It can accept input images of any size, and upsample and decode the feature map of the last convolution of the encoding network through several deconvolution layers to restore it to the same size as the input image, so that a prediction can be generated for each pixel while retaining the spatial information in the original input image. Subsequently, based on the FCN network, a variety of semantic segmentation models were derived, such as the symmetric network U-net with jump connections between the encoder and decoder, the DeepLab series of networks that introduce dilated convolutions and use conditional random fields (CRF) for post-processing optimization, and ParseNet that combines contextual information for feature fusion.

[0005] Medical image segmentation is a complex and critical step in the field of medical image processing and analysis. Its purpose is to segment parts of medical images with certain special meanings and extract relevant features to provide a reliable basis for clinical diagnosis and pathological research, and to assist doctors in making more accurate diagnoses. However, it is still a difficult task to automatically segment targets from medical images using some common algorithms, because medical images have high complexity and lack simple linear features, and there are partial volume effects, grayscale inhomogeneity, artifacts, different grayscale values ​​between soft tissues and other phenomena, resulting in low segmentation accuracy. Summary of the invention

[0006] The purpose of the present invention is to provide a medical image segmentation method based on a channel-mixed encoding and decoding network, which can better achieve the segmentation of complex images.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is: a medical image segmentation method based on a channel-mixed codec network, comprising the following steps: obtaining a medical image to be segmented; importing the medical image to be segmented into a trained network model for recognition to obtain a segmented medical image; the basic structure of the trained network model is a symmetric U-Net network structure, and the feature map of the decoder part in the U-Net network structure is obtained by processing and fusing the feature map of the same size in the encoder through a self-attention module and a maximum upsampling index matrix.

[0008] Compared with the prior art, the present invention has the following technical effects: the medical image segmentation method proposed in the present invention, most of the segmentation algorithms borrow the skip connection of U-Net to fuse information at different scales, so as to guide the purpose of information transmission; different from the traditional algorithm, the present method proposes a channel-mixed self-attention module to replace the skip-connect structure and the maximum index upsampling to replace the transposed convolution to realize the feature upsampling of the decoding network; in addition, a combined loss strategy of segmented training is proposed to achieve a trade-off between training speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a structural block diagram of the network model in the present invention;

[0010] Figure 2 It is a specific structural diagram of the network model in the present invention;

[0011] Figure 3 It is a specific structural diagram of the self-attention module in the present invention;

[0012] Figure 4 This is a diagram of the maximum upsampling index matrix connection. DETAILED DESCRIPTION

[0013] Combine the following Figures 1 to 4 , the present invention is further described in detail.

[0014] See also Figure 1 , a medical image segmentation method based on a channel-mixed encoding and decoding network, comprising the following steps: obtaining a medical image to be segmented; importing the medical image to be segmented into a trained network model for identification to obtain a segmented medical image; the basic structure of the trained network model is a symmetrical U-Net network structure, and the feature map of the decoder part in the U-Net network structure is obtained by processing and fusing the feature map of the same size in the encoder through a self-attention module and a maximum upsampling index matrix. In the medical image segmentation method proposed in the present invention, most segmentation algorithms refer to the skip connection of U-Net to fuse information at different scales, thereby guiding the purpose of information transmission; different from the traditional algorithm, this method proposes a channel-mixed self-attention module to replace the skip-connect structure and the maximum index upsampling to replace the transposed convolution to realize the feature upsampling of the decoding network. The self-attention module can encode the global feature information of a large area and add it to the local feature information, so that the local features of the feature map carry the information dependent on the global spatial features, thereby enhancing the representation ability of the module. The maximum upsampling index matrix is ​​used to save the pooling point position information in the symmetrical encoding stage. In the decoding stage, for the upsampling operation, the index matrix of the symmetrical encoding record will be used to determine which area of ​​the previous 2x2 each pooled 1x1 feature point comes from. Through the processing of the self-attention module and the maximum upsampling index matrix, it is ensured that more information can be integrated when restoring the image semantics of the feature map, thereby improving the final segmentation accuracy.

[0015] There are many ways to set up the U-Net network structure based on the self-attention module and the maximum upsampling index matrix. In the present invention, it is preferred to adopt Figure 2 In the scheme shown, since the network structure in the present invention is relatively complex and it is not convenient to directly describe it from the structure, the network structure is presented here by describing the specific steps of image processing. Specifically, the U-Net network structure processes the input image into a segmented medical image in the following steps: S100, convolution operation is performed on the input image to obtain a feature map a1, where the number of channels * height * width of the input image is recorded as C*H*W; S200, convolution operation is performed on the feature map a1; i Perform convolution and pooling operations to obtain feature map a i+1 , where i∈{1,2,…,n-1}, that is: convolution and pooling operations are performed on feature map a1 to obtain feature map a2, convolution and pooling operations are performed on feature map a2 to obtain feature map a3, and so on, finally obtaining feature map a n ; S300, using the self-attention module to the feature map ai Processing, using the maximum upsampling index matrix to feature map b i+1 After processing, the processing results of the two are fused to obtain the feature map b i ; S400, perform convolution operation on feature map b1 to obtain feature map c; S500, perform softmax operation on feature map c to obtain the segmented medical image; In the above steps, the input image has the same width and height as feature map c, and the number of channels of feature map c is K, where K is the number of categories, and the eigenvalues ​​in each category represent the probability that each pixel position belongs to that category. In the step S500, the number of channels K of feature map c is the number of categories after the medical image is segmented; the segmented medical image is a heat map, and each K category in the heat map is displayed in a different color. Feature map a i and b i The size is exactly the same, and the size is exactly the same, that is, the number of channels, height, and width of the two are equal. Through these steps, the input image can be easily processed into a classified heat map.

[0016] Here is a special explanation as follows: In a general U-Net network model, there are an encoder and a decoder. The encoder extracts high-level semantic features from the feature map, and the decoder restores the image semantics from the feature map. The encoder and the decoder are connected through the bottom-level feature map. This bottom-level feature map can be divided into the encoder or the decoder, and of course it can also exist independently as the bottom-level feature map. In the present invention, for the convenience of description, the feature map a n , feature map b n And the lowest level feature map refers to the same thing, namely Figure 2 The characteristic graph of C4*H3*W3 in .

[0017] In traditional algorithm models, upsampling is generally achieved by transposed convolution and the feature map in the encoder is directly copied to the decoder. The present invention is aimed at improving this. There are many solutions that can implement the self-attention module and the maximum upsampling index matrix processing. In the present invention, preferably, in step S200, the feature map a i Generate the maximum upsampling index matrix d when performing the pooling operation i , the matrix is Figure 4 As shown in , it expresses the pooling point position information in the encoding stage. Step S300 includes the following steps: S310, feature map a i After being processed by the self-attention module, the feature map b is obtained i,1 ; S320, feature map b i+1 Perform convolution operation to obtain feature map ; S330, characteristic map According to the maximum upsampling index matrix dj Perform maximum pooling upsampling to obtain feature map b i,2 ; S340, feature map b i,1 And feature map b i,2 Perform channel fusion to obtain feature map b i ; In the above steps, feature map a i 、b i,1 、b i,2 、b i The sizes are exactly the same; channel fusion means: from feature map b i,1 Select some channels and select them from the feature map b i,2 Select another part of the channels to jointly form the feature map b i In the present invention, preferably, in step S340, respectively from the feature map b i,1 And feature map b i,2 Select 1 / 2 of the number of channels to form the feature map b i Through the above steps to restore the image and speech, the final classification heat map is more accurate.

[0018] See also Figure 3 Further, the step S310 includes the following steps: S311, for the feature map a i Convolution operation is performed on some channels of the feature map f and feature map g respectively; S312, feature map a i Perform convolution operation to obtain feature map h; S313, process feature maps f and g into two-dimensional matrices of number of channels × (height * width) according to their number of channels, height, and width; S314, transpose the two-dimensional matrix corresponding to feature map f and multiply it with the two-dimensional matrix corresponding to feature map g to obtain the attention layer matrix; S315, multiply feature map h by the attention layer matrix to obtain feature map b i,1 Preferably, in step S311, the “part” in the partial channels is 1 / 4 or 1 / 8, that is, the number of channels of the feature map f and the feature map g is the same as that of the feature map a. i The ratio of the number of channels is 1 / 4 or 1 / 8. In this embodiment, the ratio of 1 / 8 is used. Figure 3 The above steps are elaborated in detail.

[0019] Assume that the feature map to be processed is a iThe size of is C×H×W, and the "part" in some channels is equal to 1 / 8. After the convolution operation, the size of the feature maps f and g is (C / 8)×H×W. At this time, the two feature maps are converted into a two-dimensional matrix, that is, a matrix of size (C / 8)×(H×W), that is, the matrix has C / 8 columns and each row of the matrix has H×W elements. Then transpose the matrix corresponding to the feature map f to obtain a matrix of (H×W)×(C / 8). Multiply this matrix with the matrix corresponding to the feature map g to obtain a two-dimensional matrix of size (H×W)×(H×W), which is the attention layer matrix; finally, multiply the feature map h by the attention layer matrix to obtain the same matrix as the feature map a. i The same size feature map b i,1 Through the above steps, the global feature information of a large area can be encoded and added to the local feature information, so that the local features of the feature map carry information that is dependent on the global spatial features.

[0020] For medical image segmentation, common loss functions are cross entropy loss or dice loss. In this invention, a weighted combination of the two is used, and the weight is proportional to the time coefficient. Specifically, the loss function of the U-Net network during training is as follows:

[0021]

[0022] In the formula, represents the predicted probability that the i-th sample belongs to the j-th class, It represents the label probability that the i-th sample belongs to the j-th category, N represents the number of samples, and K represents the number of categories. represents the smoothing factor; is the time coefficient, which is used to change the weighting of the two losses. Dice loss has a good optimization effect on the problem of sample category imbalance, but when the values ​​of p and q are very small, the calculated gradient value may be very large, which will lead to unstable training. Therefore, the time coefficient is introduced in this case. , early stage of training, is close to 1, and the loss function loss is As the training progresses, Gradually decrease, 1- Gradually increases, making The weight is reduced, The weight increases, and later in training, is close to 0, and the loss function loss is For the main.

[0023] Time coefficient There are various ways to calculate , and it is only necessary to ensure that its value is less than and close to 1 in the early stage of training and greater than and close to 0 in the later stage of training. In the present invention, preferably, the calculation formula of the time coefficient is as follows: , t represents the ratio of the current number of iterations to the number of loss switching times T, and T are determined hyperparameters, the The value range is an integer between 5 and 10. The two loss weights determined are more reasonable.

[0024] After building the network model according to the previous steps, it needs to be trained. When training, you can follow the steps below:

[0025] 1. Prepare a public dataset and determine the input size of the dataset;

[0026] 2. Build a training network, determine the network parameters and hyperparameters during training. The hyperparameters mentioned above and T, and also includes some parameters of the conventional U-Net network during training, such as the initial learning rate, the total epoch, etc. Generally, after a certain number of iterations, the learning rate will decrease. In this case, the time coefficient Likewise, as iterations proceed, its value gradually decreases;

[0027] 3. Input the sample set and perform batch sampling training;

[0028] 4. Finally, use the validation set to verify and test the model. After the network model is saved after the test, it can recognize the images to be detected.

Claims

1. A medical image segmentation method based on a channel-mixed codec network, characterized in that: The steps include: Acquire a medical image to be segmented; Importing the medical image to be segmented into the trained network model for recognition to obtain the segmented medical image, specifically includes: S100, performing a convolution operation on the medical image to be segmented to obtain a feature map a1; S200, the characteristic graph a i Perform convolution and pooling operations to obtain feature map a i+1 , where i∈{1,2,…,n-1}, a n That is the lowest level feature map b n , wherein the feature map a i Generate the maximum upsampling index matrix d when performing the pooling operation i ; S300, using the self-attention module to i Processing is performed using the maximum upsampling index matrix d i For feature map b i+1 After processing, the processing results of the two are fused to obtain the feature map b i ; S400, performing a convolution operation on the feature map b1 to obtain a feature map c; S500, performing a softmax operation on the feature map c to obtain a segmented medical image; In the above steps, the medical image to be segmented has the same width and height as the feature map c. i and b i The dimensions are exactly the same; The basic structure of the trained network model is a symmetrical U-Net network structure, and the feature map b of the decoder part in the U-Net network structure i The feature map a with the same size as the encoder i Through the self-attention module and the feature map b having the same size i+1 By the maximum upsampling index matrix d i After processing and fusion, it is obtained, including: S310, the feature map a i After being processed by the self-attention module, the feature map b is obtained. i,1 ; S320, the characteristic graph b i+1 Perform convolution operation to obtain feature map b′ i+1 ; S330, the characteristic graph b′ i+1 According to the maximum upsampling index matrix d i Perform maximum pooling upsampling to obtain feature map b i,2 ; S340, the feature map b i,1 and the characteristic graph b i,2 Perform channel fusion to obtain the feature map b i ; In the above steps, the feature map a i 、b i,1 、b i,2 、b i The sizes are exactly the same; channel fusion means: from the feature map b i,1 Select some channels, from the feature map b i,2 Select another part of the channels to jointly form the feature map b i ; Wherein, the step S310 includes the following steps: S311, the characteristic graph a i Convolution operations are performed on some channels to obtain feature maps f and g respectively; S312, the characteristic graph a i Perform convolution operation to obtain feature map h; S313, processing the feature graphs f and g into two-dimensional matrices of number of channels × (height*width) according to the number of channels, height, and width of the feature graphs f and g; S314, transposing the two-dimensional matrix corresponding to the feature map f and multiplying it by the two-dimensional matrix corresponding to the feature map g to obtain an attention layer matrix; S315: Multiply the feature map h by the attention layer matrix to obtain the feature map b i,1 .

2. The medical image segmentation method based on a channel-mixed codec network as claimed in claim 1, characterized in that: In the step S340, respectively, from the feature map b i,1 and the characteristic graph b i,2 Select 1 / 2 of the number of channels to form the feature map b i .

3. The medical image segmentation method based on a channel-mixed codec network as claimed in claim 1, characterized in that: In step S500, the number of channels K of the feature map c is the number of categories after the medical image is segmented; the segmented medical image is a heat map, and each K category in the heat map is displayed in a different color.

4. The medical image segmentation method based on a channel-mixed codec network as claimed in claim 1, characterized in that: In step S311, the "part" in the partial channel is 1 / 4 or 1 / 8.

5. The medical image segmentation method based on a channel-mixed codec network according to any one of claims 1 to 4, characterized in that: The loss function loss(P,Q,t) during U-Net network training is as follows: In the formula, p(x ij ) represents the predicted probability that the i-th sample belongs to the j-th class, q(x ij ) represents the label probability that the i-th sample belongs to the j-th class, N represents the number of samples, K represents the number of categories, δ represents the smoothing factor, and ω(t) is the time coefficient, which is used to change the weighting of the two losses.

6. The medical image segmentation method based on a channel-mixed codec network as claimed in claim 5, characterized in that: The calculation formula of the time coefficient is as follows: ω(t)=exp(-λ*t 2 ), t represents the ratio of the current iteration number to the loss switching number T, and λ and T are determined hyperparameters.

7. The medical image segmentation method based on a channel-mixed codec network as claimed in claim 6, characterized in that: The value range of λ is an integer between 5 and 10.

Images