Medical image segmentation method, device and storage medium fusing multi-scale features

Abstract

The application relates to a medical image segmentation method and device fusing multi-scale features and a storage medium, the method uses a pre-trained medical image segmentation model for segmentation, wherein the medical image segmentation model comprises: an encoder module comprising two groups of different encoding blocks based on a transformer and a convolutional neural network, used for extracting multi-scale features from an image to be segmented; a feature fusion module connected with each encoding block respectively, used for encoding and fusing the multi-scale features based on a cross-attention mechanism; and a decoder module connected with the encoder module and the feature fusion module respectively, comprising a plurality of decoding blocks, used for outputting a final segmentation image based on the fused features. The application takes the transformer as a backbone network, simultaneously extracts low-scale features by means of the CNN, and then fuses the features, so that the model has stronger feature extraction and feature fusion capabilities, and more accurate segmentation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, in particular to a medical image segmentation method and device fusing multi-scale features and a storage medium. BACKGROUND

[0002] Medical auxiliary systems are gradually becoming a reality, which can provide help for doctors to diagnose and treat through processing and analysis of medical images. In the process of medical image processing, image segmentation is a particularly key step, which is the basis for subsequent auxiliary diagnosis and treatment. At the same time, medical images themselves have high complexity, so it is a difficult task to automatically segment the target from the medical images. The existing segmentation methods mainly use single-dimensional feature extraction methods, which cannot well utilize multi-scale feature information.

[0003] Chinese patent application publication No. CN116563204A discloses a medical image segmentation method fusing multi-scale residual attention, the steps of which are: dividing the training set, the validation set and the test set after preprocessing the collected data set; data enhancement, in the training process, using a probability of 0.5 to randomly vertically flip, randomly horizontally flip and randomly rotate the data; constructing a multi-scale residual attention network, in the encoder part, the network reduces the channel number of the double convolution intermediate part, and moderately reduces the network parameters; designing a multi-scale residual attention module in the decoder to capture the detail features of the medical image under different scales, improve the utilization rate of the image feature map, alleviate the gradient disappearance problem and degradation problem caused by the network being too deep, and further improve the stability of the network; testing the model on the test set to obtain the segmentation result of the model.

[0004] The above-mentioned application overcomes the problem of blurred boundary between the target region and the background of the medical image, but the above-mentioned application still has the problem of insufficient multi-scale feature extraction. SUMMARY

[0005] The present application is to overcome the defects of the prior art and provide a medical image segmentation method, device and storage medium fusing multi-scale features, which extracts and fuses the multi-scale features of the image to improve the learning ability of the model.

[0006] The object of the present application can be achieved by the following technical solutions:

[0007] In one aspect of the present application, a medical image segmentation method fusing multi-scale features is provided, which uses a pre-trained medical image segmentation model to segment the acquired medical image, wherein the medical image segmentation model comprises:

[0008] The encoder module comprises two groups of different encoding blocks based on a transformer and a convolutional neural network, and is used for extracting multi-scale features from an image to be segmented;

[0009] The feature fusion module is connected with each encoding block respectively, and is used for encoding and fusing the multi-scale features based on a cross-attention mechanism.

[0010] The decoder module is connected with the encoder module and the feature fusion module respectively, and comprises a plurality of decoding blocks, and is used for outputting a final segmentation map based on the fused features.

[0011] As a preferred technical solution, the two groups of different encoding blocks comprise a plurality of multi-layer convolutional neural network-based encoding blocks and a plurality of multi-layer transformer-based encoding blocks, the transformer-based encoding blocks comprise a plurality of CSWinTransformer blocks, and the convolutional neural network-based encoding blocks comprise a plurality of convolutional blocks.

[0012] As a preferred technical solution, the feature fusion block comprises:

[0013] The feature encoding submodule is used for mapping the multi-scale features output by the encoder module to the same region and splicing in the channel dimension.

[0014] The channel attention calculation submodule is used for cross-attention calculation on the spliced output.

[0015] The fusion submodule is used for fusion processing on the cross-attention calculation result.

[0016] As a preferred technical solution, the channel attention calculation submodule comprises a first calculation submodule and a second calculation submodule, wherein, as the query, as the key and value as the input of the first calculation submodule, as the query, as the key and value as the input of the second calculation submodule, wherein, , is a feature vector of two channels in the feature encoding submodule after being mapped to the same region, is a feature vector of two channels in the feature encoding submodule after being spliced in the channel dimension.

[0017] As a preferred technical solution, the feature fusion module comprises a plurality of decoding blocks, and each decoding block comprises a plurality of CSWinTransformer blocks.

[0018] As a preferred technical solution, the encoder module is connected with the decoder module through an intermediate layer.

[0019] As a preferred technical solution, the intermediate layer comprises a CSWin Transforemer block.

[0020] As a preferred technical solution, the medical image segmentation model is trained based on a loss function to obtain the pre-trained medical image segmentation model, wherein the loss function comprises a Dice function and a cross-entropy function.

[0021] In another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory having stored therein one or more programs, the one or more programs comprising instructions for performing the medical image segmentation method of fusing multi-scale features.

[0022] In another aspect of the present application, a computer-readable storage medium is provided, comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for performing the medical image segmentation method of fusing multi-scale features.

[0023] Compared with the prior art, the present application has the following advantages:

[0024] (1) Two networks are used for feature extraction to improve model training capability: Unlike some existing methods that use a single network for feature extraction, the present application uses a medical image segmentation model based on transformer and convolutional neural network, uses transformer as the backbone network, and also uses CNN to extract low-scale features, then fuses the features, so that the model has stronger feature extraction and feature fusion capability, and realizes more accurate segmentation.

[0025] (2) Can quickly establish global relationship: the traditional transformer structure lacks induction of global relationship to some extent, the present application uses a CSWin Transformer block to construct the model, and through the calculation of the cross-window attention mechanism, the global relationship can be established through two layers of calculation.

[0026] (3) Can effectively establish the relationship between different scale features: the two channel attention calculation modules of the present application calculate the features extracted by CNN and Transformer respectively, and use cross-channel attention to better establish the relationship between different scale features. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a schematic diagram of a medical image segmentation model for fusing multi-scale features.

[0028] Figure 2 Fig. 1 is a schematic diagram of a feature fusion module (CMT) in a medical image segmentation model fusing multi-scale features.

[0029] Figure 3 Fig. 1 is a schematic diagram of a feature fusion module (CMT) in a medical image segmentation model fusing multi-scale features. Figure 2 Fig. 1 is a schematic diagram of a feature fusion module (CMT) in a medical image segmentation model fusing multi-scale features.

[0030] Figure 4 Fig. 1 is a schematic diagram of a feature fusion module (CMT) in a medical image segmentation model fusing multi-scale features. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0032] Embodiment 1

[0033] In view of the problems of the prior art, the present embodiment provides a medical image segmentation method fusing multi-scale features. The method uses a pre-trained medical image segmentation model to segment the obtained medical image, and comprises the following steps:

[0034] S1, obtaining a medical image to be segmented and labeling a target region;

[0035] S2, preprocessing the original data;

[0036] S3, dividing the image data set into a training set and a test set according to a preset ratio;

[0037] S4, generating a medical image segmentation model, inputting the training set of the data set into the medical image segmentation model to train the medical image segmentation model;

[0038] S5, inputting the test set of the data set into the trained image segmentation model to obtain a segmented image.

[0039] Specifically, the structure of the medical image segmentation model is as follows: Figure 1 The model is designed based on transformer and CNN, and the structure is as follows:

[0040] ① Constructing an encoding module, which mainly includes two parts, and the structure is as follows:

[0041] Based on the convolutional neural network encoding block, three layers of encoding blocks are used in the encoding structure of the network, and each layer of encoding block includes two convolutional layers, a ReLU activation layer and a maximum pooling layer (MaxPooling).

[0042] Based on the transformer encoding block, three layers of encoding blocks are used in the encoding structure of the network, and each layer of encoding block is composed of a CSWin Transformer block, and the number of CSWin Transformer blocks is 1, 2, 21 respectively.

[0043] ② Construct a feature fusion block (CMT), see Figure 2 The feature fusion module mainly includes three parts of feature encoding module, attention calculation module and fusion module, and the structure is as follows:

[0044] Feature encoding module. Given the output of each stage of CNN and transformer and , ) respectively. First, the encoder features of each scale need to be mapped to the same region, and the output of the encoder of each stage is respectively reshaped to , and , and .

[0045] Where Contact is the channel concatenation operation.

[0046] Two channel attention calculation modules. Take as query, as key and value as the input of one channel attention module, and as query, as key and value as the input of the other channel attention module for attention calculation:

[0047]

[0048]

[0049] Where and respectively represent the weight parameter matrix of different inputs of two attention modules, represents the length of the input sequence, represents the number of channels of different stages. and are used to perform cross-attention calculation (CA) respectively:

[0050]

[0051]

[0052] wherein denotes a normalization operation, denotes a softmax function. In the case of N heads, the multi-head cross attention is calculated as:

[0053]

[0054]

[0055] Fusion module. The output of the and correspond to the input feature fusion module (MFB) respectively. The structure of the feature fusion module is as described in Figure 3 . First, the and are converted into and , and then the feature fusion operation is calculated to obtain the output :

[0056]

[0057]

[0058] wherein , and finally down-sampling and RELU are performed to obtain the output , and finally the is connected with the up-sampling feature of the i-th level decoder.

[0059] ③Intermediate layer. Mainly contains a CSWin Transformer module.

[0060] ④Decoder block. In the decoding structure of the network, corresponding to the encoder block, four layers of encoding blocks are used, including three layers of decoding blocks corresponding to the encoder, and each layer of decoding block is composed of a CSWin Transformer block, wherein the number of CSWin Transformer blocks is 1, 2, 21 respectively; one layer of expansion block, including a linear layer and a convolution layer.

[0061] Referring to Figure 4 is a schematic diagram of the CSWin Transformer in this embodiment, including a first LN layer, a visual Transformer backbone network with a cross-shaped window, a second LN layer and a multi-layer perceptron connected in sequence.

[0062] The specific structure parameters of the convolutional neural network coding block in the medical image segmentation model in the embodiment are as shown in Table 1.

[0063] Table 1 Structure parameters of convolutional neural network coding block

[0064]

[0065] In the training process of the model, the loss function is defined as a new loss function composed of the Dice function and the cross-entropy function The expression is as follows:

[0066]

[0067]

[0068]

[0069] wherein y is a standard segmentation, represents a segmentation map output by the decoder block, is a multi-class cross-entropy loss function, is a Dice loss function.

[0070] The following will be described by taking an actual application as an example, including the following steps:

[0071] Step 1, obtaining a medical image to be segmented, labeling a target region, and forming a data set;

[0072] For example, taking prostate organ segmentation, the organ segmentation data set used in the experiment includes 150 patients, and each sample contains 4 labels, and the labeled organs are prostate, intestine and vesicle.

[0073] Step 2, preprocessing the original data, in the data preprocessing process, in order to ensure that the images have the same size and resolution, the spatial resolution of the original slice image and the labeled image is uniformly sampled to 224x224;

[0074] Step 3, dividing the image data set into a training set and a test set according to a preset proportion, the training set is 120 cases, and the test set is 30 cases.

[0075] Step 4, generating a medical image segmentation model, inputting the training set of the data set into the medical image segmentation model, and training the medical image segmentation model;

[0076] The network uses the SGD algorithm to optimize the loss function, the batch size is set to 24, the epoch is 250, and the learning rate is set to 0.05.

[0077] Step 5, input the test set of the data set into the trained image segmentation model to obtain a segmented image.

[0078] The segmentation result is evaluated by the Dice Similarity Coefficient (DSC) and the average Hausdorff distance (HD).

[0079] After multiple rounds of experiments, as shown in Table 2, on the prostate organ segmentation data set, the DSC reached 84.41%, and the HD reached 3.11.

[0080] Table 2 Experimental results

[0081]

[0082] The application fuses the features of transformer and CNN, improves the traditional U-shaped structure, uses transformer as the backbone network, extracts low-scale features with the help of CNN, fuses the features, makes the model have stronger feature extraction and feature fusion capability, and realizes more accurate segmentation.

[0083] Embodiment 2

[0084] The embodiment provides an electronic device, including one or more processors and a memory, the memory has one or more programs stored therein, and the one or more programs include instructions for executing the medical image segmentation method fusing multi-scale features as described in embodiment 1.

[0085] Embodiment 3

[0086] The embodiment provides a computer-readable storage medium, including one or more programs for an electronic device to execute, and the one or more programs include instructions for executing the medical image segmentation method fusing multi-scale features as described in embodiment 1.

[0087] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the application, and these modifications or replacements should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A medical image segmentation method fusing multi-scale features, characterized in that, The pre-trained medical image segmentation model is used to segment the obtained medical image, wherein the medical image segmentation model comprises: An encoder module comprising two groups of different encoding blocks based on a transformer and a convolutional neural network, for extracting multi-scale features from the image to be segmented; A feature fusion module connected to each encoding block, for encoding and fusing multi-scale features based on a cross-attention mechanism; A decoder module connected to the encoder module and the feature fusion module, comprising a plurality of decoding blocks, for outputting a final segmentation map based on the fused features, The feature fusion block comprises: A feature encoding submodule for mapping the multi-scale features output by the encoder module to the same region and concatenating them in the channel dimension; A channel attention calculation submodule for cross-attention calculation on the concatenated output; A fusion submodule for fusion processing of the cross-attention calculation result, For the feature encoding sub-module, given the CNN and transformer outputs of each stage respectively and , first map the encoder features of each scale to the same region, and the outputs of the encoder of each stage are respectively reshaped to and , and then respectively concatenated to and , where Contact is a channel concatenation operation,​ The channel attention calculation submodule includes a first calculation submodule and a second calculation submodule, wherein, as the input of the first calculation submodule, as the query, as the key and value, as the query, as the key and value, as the input of the second calculation submodule: wherein and respectively represent the weight parameter matrix of different inputs of two attention modules, wherein, , is the feature vector of two channels after mapping to the same area in the feature encoding submodule, is the feature vector of two channels after splicing in the channel dimension in the feature encoding submodule, and to respectively perform cross-attention computation: wherein denotes a normalization operation, denotes a softmax function, in the case of N heads, computed as: For the fusion sub-module, the following is obtained and is converted to and and then a feature fusion operation is performed to obtain the output : wherein , down-sampling and RELU are performed to obtain the output , and finally is concatenated with the up-sampled features of the decoder of the i-th stage.

2. The method of claim 1, wherein, The two groups of different encoding blocks comprise a plurality of convolutional neural network-based encoding blocks and a plurality of transformer-based encoding blocks, the transformer-based encoding blocks comprise a plurality of CSWin Transformer blocks, and the convolutional neural network-based encoding blocks comprise a plurality of convolutional blocks. 3.The method of claim 1, wherein, The decoder module comprises a plurality of decoding blocks, each of which comprises a plurality of CSWin Transformer blocks.

4. The method of claim 1, wherein, The encoder module is connected to the decoder module through an intermediate layer.

5. The method of claim 4, wherein, The intermediate layer comprises a CSWin Transforemer block.

6. The method of claim 1, wherein, The medical image segmentation model is trained based on a loss function to obtain the pre-trained medical image segmentation model, wherein the loss function comprises a Dice function and a cross-entropy function.

7. An electronic device, comprising: Comprise: One or more processors and a memory, the memory storing one or more programs, the one or more programs comprising instructions for performing the medical image segmentation method of fusing multi-scale features according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, Comprise one or more programs for one or more processors of an electronic device to execute, the one or more programs comprising instructions for performing the medical image segmentation method of fusing multi-scale features according to any one of claims 1-6.

Images