A lung nodule segmentation method and device based on target consistent multi-resolution fusion

Abstract

The present application relates to a kind of lung nodule segmentation method and device based on target consistent multi-resolution fusion, which comprises the following steps: S1: collecting lung CT image data and using the object-consistent multi-resolution input strategy (OCMIS), high-resolution and low-resolution data are generated;S2: by small nodule fine-grained enhanced encoder (SNFGEE) and multi-granularity enhanced encoder (MGEE), high and low resolution data are respectively extracted features, and high and low resolution features of small nodule are obtained;S3: by muti-resolution lossless feature fusion module (MLFF), high and low resolution features are fused, and fusion features are generated;S4: the fusion features are input into the decoder, and segmentation results are generated.The present application can realize the accurate segmentation of complex morphology lung nodule.The present application is a kind of lung nodule segmentation method and device based on target consistent multi-resolution fusion, which can be widely applied in the field of medical image segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation, in particular to a lung nodule segmentation method and device based on object-consistent multi-resolution fusion. BACKGROUND

[0002] Accurate segmentation of lung nodules is of great significance for early diagnosis and treatment of lung cancer. With the widespread application of computed tomography (CT) technology, lung CT images have become an indispensable tool in clinical diagnosis. However, due to the complex shape, varying size and low contrast with surrounding tissues of lung nodules, automated and accurate segmentation remains a challenging task.

[0003] Existing lung nodule segmentation methods mainly include traditional image processing-based algorithms and deep learning-based algorithms. Traditional methods rely on manual feature extraction and rules, making it difficult to cope with the diversity and complexity of nodule shapes. Deep learning-based methods, especially convolutional neural networks (U-Net), have made significant progress in feature extraction and segmentation accuracy. However, current deep learning methods have limitations in feature fusion at different resolutions, especially in the identification and segmentation of small nodules, which are easily affected by resolution limitations, resulting in unsatisfactory segmentation results.

[0004] In addition, existing methods often lack object consistency strategies in the multi-resolution feature fusion process, resulting in insufficient information transfer between high-resolution and low-resolution features, affecting the final segmentation results. Therefore, how to effectively fuse features at multiple resolutions and improve the segmentation accuracy of complex morphological lung nodules remains a technical challenge to be solved.

[0005] In summary, existing technologies have limitations in multi-resolution feature fusion and object consistency for lung nodules, and there is an urgent need to develop new segmentation methods to improve the accuracy and reliability of segmentation. SUMMARY

[0006] To solve the above technical problems, the present application provides a lung nodule segmentation method and device based on object-consistent multi-resolution fusion, which can provide users with a more efficient and accurate lung nodule segmentation scheme.

[0007] Specifically, the method comprises the following steps:

[0008] S1: Collecting lung CT image data and using an object-consistent multi-resolution input strategy (OCMIS) to generate high-resolution and low-resolution data;

[0009] S2: feature extraction is performed on high and low resolution data respectively by a small nodule fine-grained enhanced encoder (SNFGEE) and a multi-granularity enhanced encoder (MGEE) to obtain high and low resolution features of the small nodule;

[0010] S3: high and low resolution features are fused by a muti-resolution lossless feature fusion (MLFF) module to generate fused features;

[0011] S4: the fused features are input into a decoder to generate a segmentation result.

[0012] Preferably, S1 comprises the following steps:

[0013] S1.1: lung CT image data is collected and the center coordinates and size of each nodule are obtained;

[0014] S1.2: image data and nodule labels of different resolutions are cut and generated according to the center coordinates and size of each nodule;

[0015] S1.3: image data of different resolutions are taken as input images and nodule labels are taken as input labels.

[0016] Preferably, S2 comprises the following steps:

[0017] S2.1: a small nodule fine-grained enhanced encoder (SNFGEE) and a multi-granularity enhanced encoder (MGEE) are constructed;

[0018] S2.2: high resolution features f h are obtained by performing feature extraction calculation on high resolution input according to the SNFGEE module;

[0019] S2.3: low resolution features f l are obtained by performing feature extraction calculation on low resolution input according to the MGEE module;

[0020] Preferably, S2.1 comprises the following steps:

[0021] The global information of high-resolution data is extracted through the combined CT module of convolution and Transformer. The multi-granularity features are enhanced according to the multi-differential information through the Fine-Grained Enhanced Convolution (FGEC) module to obtain the SNFGEE encoder:

[0022] f1=Conv 1×1 (Conv 3×3 (Transformer(Conv 1×1 (Conv 3×3 (x))) ×2 ))

[0023]

[0024] In the above formula, x represents the input data, Transformer() represents the Transformer module, the subscript ×2 represents that the operation is performed twice, f1 and f2 represent the intermediate features, and f CT represents the output of the CT module, k i (i=1,2,3) represent the convolution kernels of standard convolution, center difference, and angle difference, respectively. n×n () represents the convolution kernel of size n, f h Represents features of high-resolution data, Represents the splicing operation, represents the matrix addition operation, Represents a matrix multiplication operation;

[0025] Large kernel convolution is used to extract features from low-resolution data, retaining fine-grained information and improving feature learning efficiency. Efficient Edge Feature Preserving Convolution (EEFPC) is used to enhance the expression of deep features by utilizing spatial-to-depth recombinant features, resulting in the MGEE encoder:

[0026] MLP(.)=Conv 5×5 (Relu(Conv 5×5 (.)))

[0027] scale(.)=wAvgPool(.)+b

[0028] EEFPC(x)=SE(NConv(SPD(x,scale(x))))

[0029] f l =EEFPC(MLP(x)) ×4

[0030] In the above formula, x represents input data, MLP(.) represents a multi-layer perceptron, Relu() represents a Relu activation function, scale(.) represents an adaptive scaling factor, w and b represent learnable weights and bias parameters, AvgPool(.) represents an average pooling function, SE(.) represents a Squeeze-and-Excitation module, NConv(.) represents a Non-stridedConvolution, SPD(.) represents a Space-to-depth convolution, and f l represents a feature of low-resolution data, and the subscript x4 represents that the operation is performed 4 times.

[0031] Preferably, S3 comprises the following steps:

[0032] S3.1: constructing a multi-resolution lossless feature fusion module (MLFF);

[0033] S3.2: inputting high-resolution features f h and low-resolution features f l to the MLFF to obtain fused features;

[0034] Preferably, S3.1 comprises the following steps:

[0035] By extracting rich feature information of nodules in a low-resolution image, the nodule features of a high-resolution image are strengthened, so that important detail information is not lost in the fusion process, accurate feature fusion is achieved, and the MLFF structure is obtained:

[0036]

[0037] In the above formula, sigma represents a sigmoid activation function, f1 represents features after multi-resolution channel information fusion, f2 represents features after multi-resolution spatial information fusion, and f out represents a multi-resolution fused feature.

[0038] Preferably, S4 comprises the following steps:

[0039] S4.1: inputting the fused features into a decoder, restoring the resolution through layer-by-layer upsampling to obtain a lung nodule segmentation result;

[0040] S4.2: obtaining a segmentation model of the lung nodule segmentation method based on target consistent multi-resolution fusion.

[0041] The second technical solution adopted by the present application is: a lung nodule segmentation device based on target consistent multi-resolution fusion, comprising:

[0042] Preprocessing module: used for collecting data, preprocessing data according to OCMIS;

[0043] SNFGEE encoder: used for extracting features of high-resolution data;

[0044] MGEE encoder: used for extracting features of low-resolution data;

[0045] MLFF module: used for fusing high and low resolution features to obtain fused features.

[0046] The method has the advantages that: the method adopts the target-consistent multi-resolution input strategy (OCMIS) to first collect lung CT image data and generate high-resolution and low-resolution data. Then, the small nodule fine-grained enhancement encoder (SNFGEE) and the multi-granularity enhancement encoder (MGEE) are used to extract features of the high-resolution and low-resolution data, respectively, to obtain high-resolution and low-resolution features of the small nodule. Then, the multi-resolution lossless feature fusion module (MLFF) is used to effectively fuse the high-resolution and low-resolution features to generate fused features. Finally, the fused features are input into the decoder to generate the segmentation result of the lung nodule, so as to realize accurate segmentation of complex morphological lung nodules. At the same time, the method can effectively cope with the challenges of existing neural network models in lung nodule segmentation. On the one hand, the target-consistent multi-resolution input strategy (OCMIS) is used to effectively promote the model to capture the subtle features of the lung nodule at different scales and enhance the recognition ability of the model to complex morphological nodules. On the other hand, the small nodule fine-grained enhancement encoder (SNFGEE) and the multi-granularity enhancement encoder (MGEE) can effectively improve the extraction ability of the model to small nodules and multi-scale features, and ensure that the features at different resolutions are fully utilized. At the same time, the multi-resolution lossless feature fusion module (MLFF) improves the utilization rate and integration efficiency of the model to different resolution features, and finally improves the effect and reliability of lung nodule segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings needed to be used in the prior art and embodiments, and the following drawings are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a flowchart of a lung nodule segmentation method based on target-consistent multi-resolution fusion of the present application;

[0049] Figure 2 is a model architecture diagram of a lung nodule segmentation method based on target-consistent multi-resolution fusion of the present application;

[0050] Figure 3 is a SNFGEE module structure schematic diagram of a lung nodule segmentation method based on target consistent multi-resolution fusion of the present application;

[0051] Figure 4 is a MGEE module structure schematic diagram of a lung nodule segmentation method based on target consistent multi-resolution fusion of the present application;

[0052] Figure 5 is a MLFF module structure schematic diagram of a lung nodule segmentation method based on target consistent multi-resolution fusion of the present application; DETAILED DESCRIPTION

[0053] In order to make the purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0054] The embodiments of the present application provide a lung nodule segmentation method and device based on target consistent multi-resolution fusion, which is used for efficient and accurate segmentation of nodules in lung CT images at the algorithm level, thereby improving the accuracy and efficiency of diagnosis.

[0055] In a typical embodiment of the present application, the LUNA16 dataset is taken as an example, and the method is as follows: Figure 1 The method comprises the following steps:

[0056] S1: Collecting lung CT image data and using an object-consistent multi-resolution input strategy (OCMIS) to generate high-resolution and low-resolution data;

[0057] S2: Extracting features from the high-resolution and low-resolution data by a small nodule fine-grained enhanced encoder (SNFGEE) and a multi-granularity enhanced encoder (MGEE) respectively to obtain high-resolution and low-resolution features of small nodules;

[0058] S3: fusing the high-resolution and low-resolution features through a multi-resolution lossless feature fusion module (MLFF) to generate fused features;

[0059] S4: inputting the fused features into a decoder to generate a segmentation result.

[0060] Further, S1 includes the following steps:

[0061] S1.1: collecting lung CT image data and obtaining the center coordinates and size of each nodule;

[0062] Further, the step of collecting lung CT image data and obtaining the center coordinates and size of each nodule specifically includes:

[0063] Download the lung CT images contained in the LUNA16 dataset, and ensure that the center coordinates and size information of each nodule are accurately recorded.

[0064] S1.2: according to the center coordinates and size of each nodule, cutting and generating image data and nodule labels of different resolutions;

[0065] Further, the step of cutting and generating data and nodule labels of different resolutions according to the center coordinates and size of each nodule specifically includes:

[0066] Taking the center coordinates of each nodule as the center, cut the local area containing the nodule;

[0067] According to the size of the nodule, adjust the scale of the cutting area to generate high-resolution (128x128 pixels) and low-resolution (64x64 pixels) image blocks;

[0068] Corresponding to the generated high-resolution and low-resolution image blocks, create the corresponding nodule label masks to mark the nodule region.

[0069] S1.3: input the image data of different resolutions as images and the nodule labels as labels.

[0070] Further, the step of inputting the image data of different resolutions as images and the nodule labels as labels specifically includes:

[0071] Use the Numpy library to convert the high-resolution and low-resolution image blocks and their corresponding label masks into NPY format files;

[0072] The high-resolution image blocks are saved as images_high.npy, the low-resolution image blocks are saved as images_low.npy, and the corresponding label masks are saved as labels_high.npy and labels_low.npy.

[0073] Further, with reference to Figure 3 、 4 , S2 includes the following steps:

[0074] S2.1: Construct a small nodule fine-grained enhanced encoder (SNFGEE) and a multi-granularity enhanced encoder (MGEE);

[0075] S2.2: Perform feature extraction calculation on the high-resolution input according to the SNFGEE module to obtain high-resolution features f h ;

[0076] S2.3: Perform feature extraction calculation on the low-resolution input according to the MGEE module to obtain low-resolution features f l .

[0077] Further, S2.1 includes the following steps:

[0078] Global information of the high-resolution data is extracted through a combination CT module of convolution and Transformer. Multi-granularity features are enhanced according to multi-difference information through a fine-grained enhanced convolution (FGEC) module to obtain the SNFGEE encoder:

[0079] f1=Conv 1×1 (Conv 3×3 (Transformer(Conv 1×1 (Conv 3×3 (x))) ×2 ))

[0080]

[0081]

[0082] In the above formula, x represents input data, Transformer() represents a Transformer module, subscript ×2 represents that the operation is performed twice, f1 and f2 represent intermediate features, f CT represents the output of the CT module, k i(i = 1, 2, 3) represent the convolution kernel of standard convolution, center difference, and angle difference, respectively, Conv n×n () represents the convolution kernel with size n, f h represents the feature of high-resolution data, represents the concatenation operation, represents the matrix addition operation, represents the matrix multiplication operation;

[0083] The feature of low-resolution data is extracted by large kernel convolution, fine-grained information is maintained, and the efficiency of feature learning is improved. Efficient edge feature preserving convolution (EEFPC) uses the reorganized feature from space to depth to enhance the expression of depth feature, and obtains the MGEE encoder:

[0084] MLP(.) = Conv 5×5 (Relu(Conv 5×5 (.)))

[0085] scale(.) = wAvgPool(.) + b

[0086] EEFPC(x) = SE(NConv(SPD(x, scale(x))))

[0087] f l = EEFPC(MLP(x)) ×4

[0088] In the above formula, x represents the input data, MLP(.) represents the multi-layer perceptron, Relu() represents the Relu activation function, scale(.) represents the adaptive scaling factor, w and b represent the learnable weight and bias parameters, AvgPool(.) represents the average pooling function, SE() represents the Squeeze-and-Excitation module, NConv() represents the Non-stridedConvolution, SPD() represents the Space-to-depth convolution, f l represents the feature of low-resolution data, and the subscript x4 represents that the operation is performed 4 times.

[0089] Further, with reference to Figure 5 , S3 includes the following steps:

[0090] S3.1: Constructing a multi-resolution lossless feature fusion module (Muti-Resolution Lossless Feature Fusion, MLFF);

[0091] S3.2: Concatenating high and low resolution features fh l Input into the MLFF to obtain the fusion feature.

[0092] Further, S3.1 comprises the following steps:

[0093] By extracting the rich feature information of the nodule in the low-resolution image, the nodule feature of the high-resolution image is strengthened, so as to ensure that important detail information will not be lost in the fusion process, realize accurate feature fusion, and obtain the MLFF structure:

[0094]

[0095]

[0096] In the above formula, sigma represents a sigmoid activation function, f1 represents a feature after multi-resolution channel information fusion, f2 represents a feature after multi-resolution spatial information fusion, and f represents a multi-resolution fusion feature. out

[0097] Further, S4 comprises the following steps:

[0098] S4.1: input the fusion feature into the decoder, restore the resolution through layer-by-layer upsampling, and obtain the lung nodule segmentation result;

[0099] S4.2: obtain a segmentation model of the lung nodule segmentation method based on target consistent multi-resolution fusion;

[0100] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make other equivalent modifications or replacements without departing from the spirit of the application. These equivalent modifications or replacements are included in the scope defined by the claims.​​

Claims

1. A pulmonary nodule segmentation method based on target consistent multi-resolution fusion, characterized in that: The method comprises the following steps: S1: Collect lung CT image data and use a multi-resolution input strategy with consistent goals to generate high-resolution and low-resolution data; S2: The high-resolution and low-resolution features of the nodules are extracted through the fine-grained enhancement encoder and the multi-granularity enhancement encoder respectively. Among them, S2 includes the following steps: S2.1: Construct a small nodule fine-grained enhancement encoder and a multi-granularity enhancement encoder; S2.2: Perform feature extraction calculation on the high-resolution input according to the SNFGEE module to obtain the high-resolution feature f h ; S2.3: Perform feature extraction calculation on the low-resolution input according to the MGEE module to obtain the low-resolution feature f l ; S2.1 includes the following steps: The global information of high-resolution data is extracted through the combined CT module of convolution and Transformer; the multi-granularity features are enhanced according to the multi-difference information through the fine-grained enhancement convolution module to obtain the SNFGEE encoder: In the above formula, x represents the input data, Transformer() represents the Transformer module, the subscript ×2 represents that the operation is performed twice, f1 and f2 represent the intermediate features, and f CT represents the output of the CT module, k i (i=1,2,3) represent the convolution kernels of standard convolution, center difference, and angle difference, respectively. n×n () represents the convolution kernel of size n, f h Represents features of high-resolution data, Represents the splicing operation, represents the matrix addition operation, Represents a matrix multiplication operation; The features of low-resolution data are extracted through large kernel convolution, which maintains fine-grained information and improves feature learning efficiency. The MGEE encoder is obtained by utilizing the spatial-to-depth reorganization features through efficient edge feature maintenance convolution to enhance the expression of deep features: MLP(.)=Conv 5×5 (Release(Conv 5×5 (.))) scale(.)=wAvgPool(.)+b EEFPC(x)=SE(NConv(SPD(x,scale(x)))) fl l =EEFPC(MLP(x)) ×4 In the above formula, x represents input data, MLP(.) represents multi-layer perceptron, Relu() represents Relu activation function, scale(.) represents adaptive scaling factor, w and b represent learnable weights and bias parameters, AvgPool(.) represents average pooling function, SE() represents Squeeze-and-Excitation module, NConv() represents Non-strided Convolution, SPD() represents Space-to-depth convolution, f l Represents the features of low-resolution data, and the subscript ×4 means that the operation is performed 4 times; S3: Fuse high-resolution and low-resolution features through the multi-resolution lossless feature fusion module to generate fused features; Among them, S3 includes the following steps: S3.1: Construct a multi-resolution lossless feature fusion module; S3.2: High and low resolution features f h 、f l Input into MLFF to obtain fusion features; Among them, S3.1 includes the following steps: By extracting the rich feature information of nodules in low-resolution images, the nodule features of high-resolution images are enhanced, ensuring that important detail information is not lost during the fusion process, achieving accurate feature fusion, and obtaining the MLFF structure: In the above formula, σ represents the sigmoid activation function, f1 represents the feature after multi-resolution channel information fusion, f2 represents the feature after multi-resolution spatial information fusion, and f out Represents multi-resolution fusion features; S4: Input the fused features into the decoder to generate the segmentation results.

2. The method and device for pulmonary nodule segmentation based on target consistent multi-resolution fusion according to claim 1, characterized in that: S1 includes the following steps: S1.1: Collect lung CT image data and obtain the center coordinates and size of each nodule; S1.2: Intercept and generate image data and nodule annotations of different resolutions based on the center coordinates and size of each nodule; S1.3: Use image data of different resolutions as input images and nodule annotations as input labels.

3. The method and device for pulmonary nodule segmentation based on target consistent multi-resolution fusion according to claim 1, characterized in that: S4 includes the following steps: S4.1: The fused features are input into the decoder, and the resolution is restored by upsampling layer by layer to obtain the lung nodule segmentation result; S4.2: Obtain the segmentation model of the lung nodule segmentation method based on target consistent multi-resolution fusion.

4. A pulmonary nodule segmentation device based on target consistent multi-resolution fusion, characterized in that: include: Preprocessing module: used to collect data and preprocess the data according to OCMIS strategy to obtain high-resolution and low-resolution data; SNFGEE encoder: used to extract high-resolution features of small nodules; the extraction method consists of a CT module combined with convolution and Transformer, and a fine-grained enhanced convolution module. The data x obtains global information through the CT module, and then obtains high-resolution features f with multi-granularity feature enhancement through fine-grained enhanced convolution. h ; MGEE encoder: used to extract low-resolution features of small nodules; the extraction method consists of large kernel convolution and efficient edge feature maintenance convolution. The data x first obtains low-resolution features through large kernel convolution, and then obtains low-resolution features f with deep feature enhancement through efficient edge feature maintenance convolution. l ; MLFF module: used to fuse high-resolution and low-resolution features to obtain fused features; the fusion method is to use the low-resolution feature f l Enhance high-resolution features f h , get the multi-resolution fusion feature f that retains detail information out .

Images