Medical image segmentation method and device based on multi-scale and global context information

Abstract

This invention relates to a medical image segmentation method and apparatus based on multi-scale and global contextual information. The method includes: acquiring a medical image to be processed; extracting image features from the medical image to obtain a feature map; learning multi-scale global contextual information from the feature map; guiding the fusion of the multi-scale global contextual information with decoder branch features; and combining the multi-scale global contextual information fused with the decoder branch features according to a preset correspondence between scale and weight to obtain the target medical image. This invention addresses, to some extent, the problems of small receptive field, shallow model depth, and significant information loss during encoding and decoding in related fully convolutional network models. Furthermore, this invention also addresses, to some extent, the problem of inaccurate combination results in related fully convolutional network models due to the use of average pooling strategies to combine multi-scale results during multi-scale inference.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method and device based on multi-scale and global context information. Background Art

[0002] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing targets of interest. Computer-Aided Diagnostics (CAD) technology based on machine learning has been widely used in medical image analysis in recent years. Machine learning-based algorithms can train model parameters through various features of medical images, and use the trained model to predict the extracted features, which can well solve the classification, regression, clustering and association problems in medical images. At the same time, deep learning technology in machine learning can automatically obtain high-dimensional features directly from data, which greatly overcomes the defects of traditional segmentation algorithms that require strong prior constraints and manual intervention, improves the robustness and effectiveness of the algorithm, and automatically adjusts model parameters through forward propagation and reverse adjustment algorithms, so that the performance of the model in related tasks is optimized.

[0003] Medical image segmentation methods based on deep learning are mainly based on convolutional neural networks. At present, convolutional neural network models used for medical image segmentation include full convolutional network models based on full-image prediction. The full convolutional network model is an end-to-end model that removes the fully connected layer of the last layer of the traditional convolutional network and uses segmentation masks as labels for model training, which greatly improves the segmentation performance and speed of the model. However, the full convolutional network model based on full-image prediction in the related art has the problems of small perception field of view, shallow model depth and large information loss in the encoding process. Summary of the invention

[0004] In view of this, a medical image segmentation method and device based on multi-scale and global context information are provided to solve the problems of small receptive field of view, shallow model depth and large information loss in the encoding process of the fully convolutional network model in the related art.

[0005] The present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a medical image segmentation method based on multi-scale and global context information, comprising:

[0007] Acquiring a medical image to be processed;

[0008] Extracting image features from the medical image to be processed to obtain a feature map;

[0009] Learning multi-scale global context information of the feature map;

[0010] Guiding the fusion of the multi-scale global context information and decoder branch features;

[0011] The multi-scale global context information fused with the decoder branch features is combined according to the correspondence between the preset scales and the weights to obtain a target medical image.

[0012] Preferably, before extracting the image features in the medical image to be processed and obtaining the feature map, the medical image segmentation method based on multi-scale and global context information of the present invention further includes:

[0013] Performing bias correction on the medical image to be processed;

[0014] Determining a target area in the bias-corrected medical image to be processed;

[0015] Cropping the medical image to be processed according to the target area;

[0016] The cropped medical image to be processed is subjected to intensity standardization.

[0017] Preferably, before obtaining the medical image to be processed, the medical image segmentation method based on multi-scale and global context information of the present invention further includes:

[0018] A preset medical image segmentation model is trained based on the original medical image; the preset medical image segmentation model is used to implement a medical image segmentation method based on multi-scale and global context information.

[0019] Preferably, the training of a preset medical image segmentation model based on the original medical image includes:

[0020] Acquiring the original medical image;

[0021] Dividing the original medical images into a training set and a validation set;

[0022] Preprocessing the original medical image;

[0023] Perform data enhancement processing on the preprocessed original medical images;

[0024] Performing model training based on the training set training;

[0025] Evaluate whether the trained model meets the preset standards;

[0026] When the trained model does not meet the preset standard, the steps of: performing data enhancement processing on the preprocessed original medical image;

[0027] When the trained model reaches the preset standard, saving the model reaching the preset standard;

[0028] Verifying the model that meets the preset standard based on the verification set;

[0029] The model that passes verification is defined as the preset medical image segmentation model.

[0030] Preferably, the data enhancement processing method includes: random up and down flipping, random left and right flipping, random cropping, random translation, random scaling and random rotation.

[0031] In a second aspect, the present invention provides a medical image segmentation device based on multi-scale and global context information, comprising: an encoder module, an M module, a G module, a decoder module and a scale attention module;

[0032] The encoder module is used to obtain a medical image to be processed, extract image features in the medical image to be processed, obtain a feature map, and send the feature map to the M module;

[0033] The M module is used to learn multi-scale global context information of the feature map, and sends the multi-scale global context information to the decoder module and the G module;

[0034] The decoder module is used to decode the multi-scale global context information to obtain decoder branch features;

[0035] The G module is used to guide the fusion of the multi-scale global context information and the decoder branch features;

[0036] The decoder module is also used to send the multi-scale global context information fused with the decoder branch features to the scale attention module;

[0037] The scale attention module is used to combine the multi-scale global context information fused with the decoder branch features according to the corresponding relationship between the preset scale and the weight to obtain the target medical image.

[0038] Preferably, the encoder module includes four layers of Dense Block and an attention mechanism;

[0039] The decoder module includes four layers of Dense Block.

[0040] Preferably, each layer of the Dense Block includes four convolutional layers;

[0041] Each of the convolutional layers includes 64 output channels; the filter size of each of the convolutional layers is 3*3; the output of each of the convolutional layers is processed by regularization and ReLU activation function.

[0042] Preferably, the output of each layer of the Dense Block is downsampled by a pooling layer with a kernel size of 2×2 and a stride of 2.

[0043] The present invention adopts the above technical scheme, a medical image segmentation method based on multi-scale and global context information, including: obtaining a medical image to be processed; extracting image features in the medical image to be processed to obtain a feature map; learning the multi-scale global context information of the feature map; guiding the fusion of the multi-scale global context information with the decoder branch feature; combining the multi-scale global context information fused with the decoder branch feature according to the correspondence between the preset scale and the weight to obtain the target medical image. Based on this, the present invention solves the problems of small receptive field, shallow model depth and large information loss in the encoding and decoding process of the full convolutional network model of the related art to a certain extent by learning the multi-scale global context information of the feature map and guiding the fusion of the multi-scale global context information with the decoder branch feature. In addition, the present invention solves the problem of inaccurate combination results of the full convolutional network model of the related art due to the method of combining multi-scale results using the average pooling strategy in multi-scale reasoning by combining the multi-scale global context information fused with the decoder branch feature according to the correspondence between the preset scale and the weight. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flowchart of a medical image segmentation method based on multi-scale and global context information provided by an embodiment of the present invention.

[0046] Figure 2 It is a schematic diagram of the training process of a preset medical image segmentation model provided in an embodiment of the present invention.

[0047] Figure 3 A network framework diagram of a medical image segmentation device based on multi-scale and global context information provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0049] Figure 1 FIG. 1 is a flow chart of a medical image segmentation method based on multi-scale and global context information provided by an embodiment of the present invention. Figure 1 As shown, the medical image segmentation method based on multi-scale and global context information of an embodiment of the present invention includes:

[0050] S101, obtaining a medical image to be processed;

[0051] S102, extracting image features from the medical image to be processed to obtain a feature map;

[0052] S103, learning multi-scale global context information of the feature graph;

[0053] S104, guiding the fusion of the multi-scale global context information and decoder branch features;

[0054] S105 , combining the multi-scale global context information fused with the decoder branch features according to the correspondence between the preset scales and the weights to obtain a target medical image.

[0055] Specifically, the decoder decodes the multi-scale global context information to obtain decoder branch features. The correspondence between the preset scales and weights is specifically manifested in that different scales correspond to different weights.

[0056] This embodiment adopts the above technical solution, a medical image segmentation method based on multi-scale and global context information, including: obtaining a medical image to be processed; extracting image features in the medical image to be processed to obtain a feature map; learning the multi-scale global context information of the feature map; guiding the fusion of the multi-scale global context information with the decoder branch feature; combining the multi-scale global context information fused with the decoder branch feature according to the correspondence between the preset scale and the weight to obtain the target medical image. Based on this, this embodiment solves the problems of small receptive field, shallow model depth and large information loss in the encoding and decoding process of the full convolutional network model of the related technology to a certain extent by learning the multi-scale global context information of the feature map and guiding the fusion of the multi-scale global context information with the decoder branch feature. In addition, this embodiment solves the problem of inaccurate combination results of the full convolutional network model of the related technology due to the method of combining multi-scale results using the average pooling strategy in multi-scale reasoning by combining the multi-scale global context information fused with the decoder branch feature according to the correspondence between the preset scale and the weight.

[0057] Preferably, before extracting the image features in the medical image to be processed and obtaining the feature map, the medical image segmentation method based on multi-scale and global context information of this embodiment further includes:

[0058] Performing bias correction on the medical image to be processed;

[0059] Determining a target area in the bias-corrected medical image to be processed;

[0060] Cropping the medical image to be processed according to the target area;

[0061] The cropped medical image to be processed is subjected to intensity standardization.

[0062] In detail, the bias correction is to determine the correction threshold using the Otsu Thresholding method, and the image contrast is adjusted according to the threshold during fusion. Cropping is to grayscale the image and then filter the image to change the image contrast so that the difference between the target area and the background is obvious, so as to determine the approximate position of the target and cut off the background outside the target. Intensity normalization is to use the z-score method to standardize each modality image, that is, to standardize the data of each modality to zero mean and unit standard deviation. In this way, the noise in the medical image to be processed can be removed to a certain extent, and the problem of uneven intensity in the medical image to be processed can be alleviated, which is convenient for the neural network to learn.

[0063] Preferably, before obtaining the medical image to be processed, the medical image segmentation method based on multi-scale and global context information of this embodiment further includes:

[0064] A preset medical image segmentation model is trained based on the original medical image; the preset medical image segmentation model is used to implement a medical image segmentation method based on multi-scale and global context information.

[0065] Figure 2 FIG. 1 is a schematic diagram of a training process of a preset medical image segmentation model provided by an embodiment of the present invention. Figure 2 As shown, the present embodiment trains a preset medical image segmentation model based on the original medical image, including:

[0066] S201: Acquire the original medical image.

[0067] S202: Divide the original medical images into a training set and a validation set.

[0068] S203: Preprocess the original medical image.

[0069] Specifically, first, the original medical image is bias-corrected; then, a target area is determined in the original medical image, an image cropping frame is obtained, and the bias-corrected image is cropped using the cropping frame to remove unnecessary background information; finally, the cropped image is intensity-normalized to facilitate learning by the neural network.

[0070] S204: Perform data enhancement processing on the pre-processed original medical image.

[0071] In detail, in order to improve the generalization ability of the model and alleviate the overfitting problem generated during the model training process, the preprocessed original medical images are subjected to data enhancement processing before the model training, thereby increasing the amount of data and enriching the diversity of the data. In the specific data enhancement process, the data enhancement processing methods for the preprocessed original medical images include random up and down flipping, random left and right flipping, random cropping, random translation, random scaling and random rotation.

[0072] The specific process of random translation is to randomly translate the image by 0.3 times the size of the image on the x-axis and y-axis. The specific process of random scaling is to randomly scale the image by scaling factors of 0.6 and 0.3 on the x-axis and y-axis. The specific process of random rotation is to randomly rotate the image using a rotation angle between -100° and 100°.

[0073] S205: Perform model training based on the training set training.

[0074] In detail, during the training process, the model will automatically extract features from the training set data, and add the extracted features to the designed model structure for forward propagation. At the same time, the loss value is calculated using the target area mask as the label, so as to reversely adjust the model parameters in multiple iterations to achieve the purpose of optimizing the model performance.

[0075] S206, evaluating whether the trained model meets the preset standard; when the trained model does not meet the preset standard, executing step S204, and when the trained model meets the preset standard, executing step S207.

[0076] Specifically, the model is evaluated by evaluation indicators such as Dice coefficient and IoU (Intersection over Union).

[0077] S207: Save the model that meets the preset standard.

[0078] S208: Verify the model that meets the preset standard based on the verification set.

[0079] S209: Define the verified model as the preset medical image segmentation model.

[0080] Based on a general inventive concept, the present invention also provides a medical image segmentation device based on multi-scale and global context information. Figure 3 A network framework diagram of a medical image segmentation device based on multi-scale and global context information provided by an embodiment of the present invention. Figure 3 As shown, the medical image segmentation device based on multi-scale and global context information of this embodiment includes: an encoder module 31, an M module 32, a G module 33, a decoder module 34 and a scale attention module 35.

[0081] Among them, the encoder module 31 is used to obtain the medical image to be processed, and to extract the image features in the medical image to be processed, to obtain a feature map, and to send the feature map to the M module; the M module 32 is used to learn the multi-scale global context information of the feature map, and to send the multi-scale global context information to the decoder module and the G module; the decoder module 34 is used to decode the multi-scale global context information to obtain the decoder branch features; the G module 33 is used to guide the fusion of the multi-scale global context information with the decoder branch features; the decoder module 34 is also used to send the multi-scale global context information fused with the decoder branch features to the scale attention module; the scale attention module 35 is used to combine the multi-scale global context information fused with the decoder branch features according to the correspondence between the preset scale and the weight to obtain the target medical image.

[0082] Specifically, the M module 32 is a Mutil-Scale Fusion (MSF) module, the G module 33 is a Global Context Information Guidance (GCIG) module, and the scale attention module 35 is a Scale-Attention module.

[0083] The G module 33 is specifically used to process the feature maps from different coding layers through a two-dimensional convolution with a kernel of 1 to unify the number of channels, and upsample the feature maps with the unified number of channels to the same scale through a two-dimensional convolution with a kernel of 3, and then cat (stack) all the feature maps, and pass them through three parallel separable convolutions with different dilation rates to learn the global context information.

[0084] The M module 32 is specifically used to firstly process the feature map from the encoding branch through three parallel dilated convolutions with different expansion rates to obtain features of different scales. Then, the feature maps of different scales are fused in pairs under the guidance of attention to obtain the multi-scale context information we need.

[0085] Preferably, the encoder module includes four layers of Dense Block and an attention mechanism; the decoder module includes four layers of Dense Block. The attention mechanism is integrated into the encoding layer, which can effectively highlight the characteristics of the target area and suppress noise.

[0086] Preferably, each layer of the Dense Block includes four convolutional layers; each convolutional layer includes 64 output channels; the filter size of each convolutional layer is 3*3; the output of each convolutional layer is processed by regularization and ReLU activation function. The output of each layer of the Dense Block is downsampled by a pooling layer with a kernel size of 2×2 and a stride of 2.

[0087] Specifically, each Dense Block has four convolutional layers. The outputs of all previous convolutional layers in the Dense Block are added to the subsequent convolutional layers. This dense connection can promote gradient flow and allow the network to learn better feature representations. Each convolutional layer in the Dense Block has 64 output channels and a filter size of 3×3. The output of each convolutional layer is processed by regularization and ReLU activation function. The output of the Dense Block is downsampled by a maximum pooling layer with a kernel size of 2×2 and a stride of 2, and the downsampled output is used as the input of the next Dense Block.

[0088] It should be noted that this embodiment and the above embodiments are based on a general inventive concept, have the same or corresponding execution processes and beneficial effects, and will not be described in detail here.

[0089] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0090] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0091] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0092] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0093] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0094] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0095] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0096] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0097] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A medical image segmentation method based on multi-scale and global context information, characterized in that: include: Acquire a medical image to be processed through an encoder module, extract image features in the medical image to be processed, obtain a feature map, and send the feature map to the M module and the G module; Through the M module, the feature map from the encoding branch first undergoes three parallel dilation convolutions with different dilation rates to obtain features of different scales. Then, the feature maps of different scales are fused in pairs under the guidance of attention to obtain multi-scale context information, which is then sent to the decoder module. The feature maps from different coding layers are processed by the G module through a two-dimensional convolution with a kernel of 1 to unify the number of channels, and the feature maps with the unified number of channels are upsampled to the same scale through a two-dimensional convolution with a kernel of 3. Then all the feature maps are stacked and passed through three parallel separable convolutions with different dilation rates to learn the global context information. Decoding the multi-scale context information through a decoder module to obtain a decoder branch feature, wherein the decoder branch feature identifies the multi-scale context information; Under the guidance of the G module, the decoder branch features are fused with the global context information to obtain multi-scale global context information and send it to the scale attention module; The scale attention module combines the multi-scale global context information according to the correspondence between the preset scale and the weight to obtain a target medical image.

2. The medical image segmentation method based on multi-scale and global context information according to claim 1, characterized in that: Before extracting the image features from the medical image to be processed and obtaining the feature map, the method further includes: Performing bias correction on the medical image to be processed; Determining a target area in the bias-corrected medical image to be processed; Cropping the medical image to be processed according to the target area; The cropped medical image to be processed is subjected to intensity standardization.

3. The medical image segmentation method based on multi-scale and global context information according to claim 1, characterized in that: Before obtaining the medical image to be processed, the method further includes: A preset medical image segmentation model is trained based on the original medical image; the preset medical image segmentation model is used to implement a medical image segmentation method based on multi-scale and global context information.

4. The medical image segmentation method based on multi-scale and global context information according to claim 3, characterized in that: The method of training a preset medical image segmentation model based on the original medical image includes: Acquiring the original medical image; Dividing the original medical images into a training set and a validation set; Preprocessing the original medical image; Perform data enhancement processing on the preprocessed original medical images; Performing model training based on the training set training; Evaluate whether the trained model meets the preset standards; When the trained model does not meet the preset standard, the steps of: performing data enhancement processing on the preprocessed original medical image; When the trained model reaches the preset standard, saving the model reaching the preset standard; Verifying the model that meets the preset standard based on the verification set; The model that passes verification is defined as the preset medical image segmentation model.

5. The medical image segmentation method based on multi-scale and global context information according to claim 4, characterized in that: The data enhancement processing method includes: random up and down flipping, random left and right flipping, random cropping, random translation, random scaling and random rotation.

6. A medical image segmentation device based on multi-scale and global context information, characterized in that: include: Encoder module, M module, G module, decoder module and scale attention module; The encoder module includes four layers of Dense Block and an attention mechanism, and the encoder module is used to obtain a medical image to be processed, extract image features in the medical image to be processed, obtain a feature map, and send the feature map to the M module and the G module; The M module is used to firstly process the feature map from the encoding branch through three parallel dilated convolutions with different dilation rates to obtain features of different scales. Then, the feature maps of different scales are fused in pairs under the guidance of attention to obtain multi-scale context information, and the multi-scale context information is sent to the decoder module. The G module is used to process the feature maps from different coding layers through a two-dimensional convolution with a kernel of 1 to unify the number of channels, and upsample the feature maps with the unified number of channels to the same scale through a two-dimensional convolution with a kernel of 3, and then stack all the feature maps and pass them through three parallel separable convolutions with different dilation rates to learn global context information; The decoder module includes four layers of Dense Block, and the decoder module is used to decode the multi-scale context information to obtain decoder branch features, and the decoder branch features identify the multi-scale context information; Under the guidance of the G module, the decoder branch features are fused with the global context information to obtain multi-scale global context information and send it to the scale attention module; The scale attention module is used to combine the multi-scale global context information according to the correspondence between the preset scale and the weight to obtain a target medical image.

7. The medical image segmentation device based on multi-scale and global context information according to claim 6, characterized in that: Each layer of the Dense Block includes four convolutional layers; Each of the convolutional layers includes 64 output channels; the filter size of each of the convolutional layers is 3*3; the output of each of the convolutional layers is processed by regularization and ReLU activation function.

8. The medical image segmentation device based on multi-scale and global context information according to claim 6, characterized in that: The output of each layer of DenseBlock is downsampled by a pooling layer with a kernel size of 2×2 and a stride of 2.

Images