Pulmonary nodule classification segmentation model fusing uncertainty region based on deep learning

Abstract

The invention discloses a pulmonary nodule classification segmentation model fusing an uncertainty region based on deep learning. Comprising a multi-confidence segmentation block and an uncertain feature fusion block. The multi-confidence segmentation module can segment an original CT image with different confidence levels based on an uncertain sensing segmentation network, and extracts related information from a high-confidence region and a low-confidence region of a pulmonary nodule. And generating an intersection mask including high-confidence information and a union mask including the high-confidence information and low-confidence information based on the extracted related information. The uncertain feature fusion block can generate corresponding CT images through intersection masks and union masks on the basis of the original CT images, segmentation features are extracted from the generated CT images and fused with basic features extracted from the original CT images, and a final classification result is obtained. Therefore, the feature information of the uncertainty region with low confidence and the feature information of the de-determination region with high confidence are fused, and the classification performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions. Background Art

[0002] In recent years, early detection and classification of lung nodules has become an important research direction in the field of medical imaging. With the popularization of CT (computed tomography) technology, the detection rate of lung nodules has increased significantly. However, traditional imaging analysis mainly relies on the experience of radiologists and is easily affected by subjective factors, resulting in inconsistent diagnostic results. In order to improve the accuracy and efficiency of lung nodule classification, researchers have gradually applied deep learning technology to medical image analysis, especially in the automatic detection and classification of lung nodules.

[0003] At present, a variety of deep learning-based technical solutions have been proposed for the classification of lung nodules. These solutions mainly include convolutional neural networks (CNN), deep residual networks (ResNet), and transfer learning methods. Through training on a large amount of labeled data, these models can automatically extract the features of lung nodules and perform good classification. Some studies also combine image segmentation technology to improve the recognition accuracy of nodule areas. In addition, some commercial products and open source tools (such as Google's DeepMind, IBM Watson, etc.) have also achieved certain application results in lung nodule detection and classification.

[0004] Although existing deep learning technologies have made some progress in the classification of lung nodules, there are still some significant problems. When constructing the current data set, different doctors may have different annotations for the surrounding areas of the same lung nodule, that is, there are uncertainty areas. Due to the low confidence of the uncertainty area, it is often ignored. However, these uncertainty areas often appear in the edge areas of the lung nodules, which usually contain the ground glass part and burr part of the lung nodules. And some studies have shown that the more obvious these features are, the more malignant the lung nodules are. Therefore, if these features can be emphasized by obtaining information from the uncertainty area, more accurate information can be provided to the model, thereby improving the classification performance of the model. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention proposes a lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions, which can improve the classification performance of the model. The specific technical solution is as follows:

[0006] A lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions is provided. In a first achievable method, the model includes:

[0007] a multi-confidence segmentation block configured to segment the original CT image at different confidence levels and generate corresponding intersection masks and union masks, wherein the intersection mask includes high-confidence information and the union mask includes high-confidence information and low-confidence information;

[0008] The uncertain feature fusion block is configured to generate a corresponding CT image based on the original CT image through an intersection mask and a union mask, and to extract a segmentation feature map and a basic feature map from the generated CT image and the original CT image and fuse them to obtain a final classification result.

[0009] In combination with the first implementable manner, in a second implementable manner, the multi-confidence segmentation block includes:

[0010] A feature extraction unit is configured to extract high confidence information, low confidence information and fusion information of the original CT image at different confidence levels and fuse them to obtain a corresponding fusion feature map;

[0011] The mask extraction unit is configured to extract the image mask of the fused feature map through two mask branches respectively, and generate the intersection mask and the union mask respectively.

[0012] In combination with the second possible implementation manner, in a third possible implementation manner, the feature extraction unit includes:

[0013] A feature extraction block is configured to extract features from the original CT image to obtain a corresponding feature map;

[0014] an uncertainty perception block configured to extract high confidence information, low confidence information and fused information from the feature map at feature layers with different confidence levels, respectively;

[0015] The intersection constraint block is configured to further extract features from the high confidence information, the low confidence information and the fused information respectively, and fuse the extracted features with the feature map to generate a fused feature map.

[0016] In combination with the third possible implementation, in a fourth possible implementation, the feature extraction block includes an encoder-decoder consisting of multiple downsampling layers and multiple upsampling layers, and corresponding upsampling and downsampling layers are jump-connected.

[0017] In combination with the third implementable manner, in a fifth implementable manner, the intersection constraint block uses a Gabor filter or an OTSU threshold algorithm to perform feature extraction.

[0018] In combination with the second implementable manner, in a sixth implementable manner, the mask branch includes two convolution-regularization blocks and a convolution block, and the convolution-regularization block includes a convolution layer, a normalization layer, and a ReLU activation layer.

[0019] In combination with the first implementable manner, in a seventh implementable manner, the uncertain feature fusion block includes:

[0020] A deep convolutional network configured to extract a basic feature map from the original CT image;

[0021] A spatial attention module, configured to extract segmentation feature maps from the two generated CT images respectively;

[0022] The feature map fusion module is configured to fuse the basic feature map and the segmentation feature map to obtain a final classification result.

[0023] In combination with the seventh possible implementation, the eighth possible implementation further includes a convolutional layer group, which is configured to process the segmentation feature map extracted by the spatial attention module to reduce the height and width of the segmentation feature map.

[0024] In combination with the eighth possible implementation, in a ninth possible implementation, the convolutional layer group includes multiple convolutional layers, and each convolutional layer is configured with a channel-spatial attention block.

[0025] Beneficial effects: The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions of the present invention can segment lung nodules at different confidence levels through multi-confidence segmentation blocks, thereby extracting effective feature information from high-confidence regions and low-confidence regions of lung nodules to guide classification. Basic features can be extracted from the original CT image through the uncertain feature fusion block, and the basic features are fused with the feature information from the multi-confidence segmentation block, thereby fusing the feature information of the uncertain region to improve the classification performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific implementation of the present invention, the following will briefly introduce the drawings required for use in the specific implementation. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0027] Figure 1 A structural diagram of a lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions provided in one embodiment of the present invention;

[0028] Figure 2 A structural diagram of a feature extraction unit provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0030] like Figure 1 The structural diagram of the lung nodule classification and segmentation model based on deep learning and fusion uncertainty region is shown, and the segmentation model includes:

[0031] a multi-confidence segmentation block configured to segment the original CT image at different confidence levels and generate corresponding intersection masks and union masks, wherein the intersection mask includes high-confidence information and the union mask includes high-confidence information and low-confidence information;

[0032] The uncertain feature fusion block is configured to generate a corresponding CT image based on the original CT image through an intersection mask and a union mask, and to extract a segmentation feature map and a basic feature map from the generated CT image and the original CT image and fuse them to obtain a final classification result.

[0033] Specifically, the segmentation model includes a multi-confidence segmentation block and an uncertain feature fusion block. Among them, the multi-confidence segmentation module can segment the original CT image at different confidence levels based on the uncertainty perception segmentation network, and extract relevant information from the high confidence area and low confidence area of ​​the lung nodule. And based on the extracted relevant information, an intersection mask including high confidence information and a union mask containing high confidence information and low confidence information are generated. The uncertain feature fusion block can generate corresponding CT images based on the original CT image through the intersection mask and the union mask respectively to enhance the structure of the lung nodules and provide texture information for subsequent classification. Then the segmentation features are extracted from the enhanced CT image and fused with the basic features extracted from the original CT image to obtain the final classification result. In this way, the feature information of the low-confidence uncertainty area and the high-confidence undetermined area can be fused, thereby improving the classification accuracy of the model.

[0034] In this embodiment, optionally, the multi-confidence segmentation block includes:

[0035] A feature extraction unit is configured to extract high confidence information, low confidence information and fusion information of the original CT image at different confidence levels and fuse them to obtain a corresponding fusion feature map;

[0036] The mask extraction unit is configured to extract the image mask of the fused feature map through two mask branches respectively, and generate the intersection mask and the union mask respectively.

[0037] Specifically, the multi-confidence segmentation block uses UGMCS-Net as the backbone network, in which the feature extraction unit can extract high-confidence information, low-confidence information and fusion information of the original CT image at different confidence levels. And the high-confidence information, low-confidence information and fusion information are fused with the feature map extracted from the original CT image to generate a fusion feature map. The mask extraction unit includes two mask branches, which can respectively extract corresponding image masks from the fusion feature map to generate the intersection mask and union mask, thereby achieving accurate segmentation of lung nodules.

[0038] In this embodiment, optionally, the feature extraction unit includes:

[0039] A feature extraction block is configured to extract features from the original CT image to obtain a corresponding feature map;

[0040] an uncertainty perception block configured to extract high confidence information, low confidence information and fused information from the feature map at feature layers with different confidence levels, respectively;

[0041] The intersection constraint block is configured to further extract features from the high confidence information, the low confidence information and the fused information respectively, and fuse the extracted features with the feature map to generate a fused feature map.

[0042] Specifically, if Figure 2 As shown, Feature Extractor Module represents a feature extraction block, Uncertainty-Aware Module represents an uncertainty perception block, and Intersection-Union Constraining Module represents an intersection-union constraint block. The feature extraction unit includes a feature extraction block, an uncertainty perception block, and an intersection-union constraint block. The feature extraction block can extract features through the original CT image input by the encoder-decoder to obtain a corresponding feature map. The uncertainty perception block includes three feature layers with different confidence levels, through which high-confidence information, low-confidence information, and fusion information can be extracted from the feature map respectively. The intersection-union constraint block can use three filters to further process the extracted high-confidence information, low-confidence information, and fusion information respectively. The three filters can be set according to the corresponding learning objectives to extract the features most relevant to the learning objectives from the high-confidence information, low-confidence information, and fusion information respectively. In this way, a balance can be achieved between the three types of information, thereby generating a more reasonable and accurate segmentation prediction, thereby improving the performance and robustness of the segmentation task. The features obtained after processing by the intersection constraint block can be fused through the connection layer, and the feature extraction block is connected through the jump connection to merge the feature map captured by the feature extraction block to generate the fused feature map R Final .

[0043] In this embodiment, optionally, the feature extraction block includes an encoder-decoder composed of multiple downsampling layers and multiple upsampling layers, and the corresponding upsampling and downsampling layers are jump-connected.

[0044] Specifically, the feature extraction block can adopt a general encoder-decoder composed of four downsampling layers and upsampling layers. In order not to lose the previous semantic information, the corresponding downsampling layers and upsampling layers can be connected by skipping.

[0045] In this embodiment, optionally, the intersection constraint block uses a Gabor filter or an OTSU threshold algorithm to perform feature extraction.

[0046] Specifically, the intersection constraint block can use a Gabor filter or a filter using the OTSU threshold algorithm. Among them, the Gabor filter is sensitive to the edges of the image and can provide good direction selection and scale selection characteristics, so that it can capture local structural features in multiple directions in the local area of ​​the image. Therefore, the Gabor filter can extract relevant features from low-confidence information and fused information. The OTSU threshold algorithm is sensitive to density features and can more accurately identify high-density tissues. Therefore, a filter based on the OTSU threshold algorithm can be used to extract relevant features from high-confidence information. After the relevant features of high-confidence information, low-confidence information, and fused information are extracted through the filter, the following calculation formula can be used to fuse the features of the three types of information:

[0047]

[0048] Among them, R Uni ′、R LC ′、R HC ′ are the relevant features extracted by the intersection constraint blocks, is the fusion information R extracted from the uncertain perception block Uni Related features R Uni The cosine similarity between ′, is low confidence information R LC Related features R LC The cosine similarity between ′, is high confidence information R HC Related features R HC The cosine similarity between ′.

[0049] The fused feature R AugThe feature map R extracted by the feature extraction block is spliced ​​and merged to obtain more effective feature information. This splicing method of feature map and segmentation feature map can retain more information from the original CT image, which helps to improve the accuracy and robustness of the segmentation model.

[0050] In this embodiment, optionally, the mask branch includes two convolution-regularization blocks and a convolution block, and the convolution-regularization block includes a convolution layer, a normalization layer and a ReLU activation layer.

[0051] Specifically, the mask branch includes two convolution-regularization CR blocks and a convolution block Conv. Among them, the convolution-regularization block CR consists of a convolution layer, a normalization layer and a ReLU activation layer. The features of the fused feature map can be extracted through the convolution layer. The normalization layer can be used to reduce the internal covariate shift of the features extracted by the convolution layer, accelerate training and improve the stability of the model. The ReLU activation layer can introduce nonlinear characteristics, allowing the model to learn and represent complex functional relationships.

[0052] In this embodiment, optionally, the uncertain feature fusion block includes:

[0053] A deep convolutional network configured to extract a basic feature map from the original CT image;

[0054] A spatial attention module, configured to extract segmentation feature maps from the two generated CT images respectively;

[0055] The feature map fusion module is configured to fuse the basic feature map and the segmentation feature map to obtain a final classification result.

[0056] Specifically, after obtaining the intersection mask and the union mask, the uncertain feature fusion block can multiply the intersection mask and the union mask with the original CT image respectively to obtain the corresponding CT image. This can enhance the structural features of pulmonary nodules in the CT image and provide texture information for subsequent processing. The uncertain feature fusion block includes a deep convolutional network, two spatial attention modules, and a feature map fusion module. Among them, the deep convolutional network can extract a 2×2×2048 basic feature map from the original CT image. The two spatial attention modules SA can respectively extract segmentation feature maps from the CT images generated based on the intersection mask and the union mask. The feature map fusion module can fuse the segmentation feature maps extracted by the two spatial attention modules with the basic feature maps extracted by the deep convolutional network through the convolutional layer to obtain the final classification result.

[0057] In this embodiment, optionally, a convolutional layer group is also included, which is configured to process the segmentation feature map extracted by the spatial attention module to reduce the height and width of the segmentation feature map.

[0058] Specifically, the uncertain feature fusion block also includes two groups of convolutional layer groups, which can respectively perform convolution processing on the segmentation feature maps output by the two spatial attention modules to reduce the height and width of the feature maps while increasing their depth.

[0059] In this embodiment, optionally, the convolutional layer group includes multiple convolutional layers, and each convolutional layer is configured with a channel-spatial attention block.

[0060] Specifically, the convolutional layer group includes multiple convolutional layers consisting of convolution-regularization blocks CR and channel-spatial attention blocks CB. Channel and spatial information can be highlighted by the channel-spatial attention blocks CB. The reduction rate of the channel-spatial attention block is set to 16 and the kernel size is 7. The number of channels in each CBAM module matches the number of channels in the previous convolutional layer.

[0061] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions, characterized in that: include: a multi-confidence segmentation block configured to segment the original CT image at different confidence levels and generate corresponding intersection masks and union masks, wherein the intersection mask includes high-confidence information and the union mask includes high-confidence information and low-confidence information; The uncertain feature fusion block is configured to generate a corresponding CT image based on the original CT image through an intersection mask and a union mask, and to extract a segmentation feature map and a basic feature map from the generated CT image and the original CT image and fuse them to obtain a final classification result.

2. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 1, characterized in that: The multi-confidence segmentation block includes: A feature extraction unit is configured to extract high confidence information, low confidence information and fusion information of the original CT image at different confidence levels and fuse them to obtain a corresponding fusion feature map; The mask extraction unit is configured to extract the image mask of the fused feature map through two mask branches respectively, and generate the intersection mask and the union mask respectively.

3. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 2, characterized in that: The feature extraction unit comprises: A feature extraction block is configured to extract features from the original CT image to obtain a corresponding feature map; an uncertainty perception block configured to extract high confidence information, low confidence information and fused information from the feature map at feature layers with different confidence levels, respectively; The intersection constraint block is configured to further extract features from the high confidence information, the low confidence information and the fused information respectively, and fuse the extracted features with the feature map to generate a fused feature map.

4. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 3 is characterized in that: The feature extraction block includes an encoder-decoder composed of multiple downsampling layers and multiple upsampling layers, and the corresponding upsampling and downsampling layers are skip-connected.

5. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 3, characterized in that: The intersection constraint block uses a Gabor filter or an OTSU threshold algorithm to extract features.

6. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 2, characterized in that: The mask branch includes two convolution-regularization blocks and a convolution block, and the convolution-regularization block includes a convolution layer, a normalization layer and a ReLU activation layer.

7. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 1, characterized in that: The uncertain feature fusion block includes: A deep convolutional network configured to extract a basic feature map from the original CT image; A spatial attention module, configured to extract segmentation feature maps from the two generated CT images respectively; The feature map fusion module is configured to fuse the basic feature map and the segmentation feature map to obtain a final classification result.

8. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 7, characterized in that: It also includes a convolutional layer group, which is configured to process the segmentation feature map extracted by the spatial attention module to reduce the height and width of the segmentation feature map.

9. The lung nodule classification and segmentation model based on deep learning and fusion uncertainty regions according to claim 8, characterized in that: The convolutional layer group includes multiple convolutional layers, and each convolutional layer is configured with a channel-spatial attention block.

Images