Pulmonary nodule segmentation method and device based on target consistent multi-resolution fusion

Abstract

The invention relates to a pulmonary nodule segmentation method and device based on target consistent multi-resolution fusion, and the method comprises the steps: S1, collecting lung CT (Computed Tomography) image data, and generating high-resolution data and low-resolution data by adopting a target consistent multi-resolution input strategy (OCMIS); s2, respectively carrying out feature extraction on the high-resolution data and the low-resolution data through a small nodule fine granularity enhancement encoder (SNFGEE) and a multi-granularity enhancement encoder (MGEE), so as to obtain the high-resolution features and the low-resolution features of the small nodules, and carrying out the feature extraction on the high-resolution data and the low-resolution data of the small nodules through the multi-granularity enhancement encoder (SNFGEE) and the multi-granularity enhancement encoder (MGEE), so as to obtain the high-resolution features and the low-resolution features of the small nodules; s3, through a multi-resolution lossless feature fusion module (Mti-Resolution Lossless Feature Fusion, MLFF), fusion is carried out on the high-resolution feature and the low-resolution feature, and a fusion feature is generated; and S4, inputting the fusion features into a decoder to generate a segmentation result. According to the invention, accurate segmentation of pulmonary nodules in complex forms can be realized. The pulmonary nodule segmentation method and device based on target consistent multi-resolution fusion can be widely applied to the field of medical image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a method and device for segmenting pulmonary nodules based on target consistent multi-resolution fusion. Background Art

[0002] Accurate segmentation of lung nodules is of great significance for the diagnosis and treatment of early lung cancer. With the widespread application of computed tomography (CT) technology, lung CT images have become an indispensable tool in clinical diagnosis. However, due to the complex morphology, different sizes and low contrast between lung nodules and surrounding tissues, automated and accurate segmentation remains a challenging task.

[0003] Existing pulmonary nodule segmentation methods mainly include algorithms based on traditional image processing and algorithms based on deep learning. Traditional methods rely on manual feature extraction and rules, which are difficult to cope with the diversity and complexity of nodule morphology. Deep learning-based methods, especially convolutional neural networks (U-Net), have made significant progress in feature extraction and segmentation accuracy. However, current deep learning methods are insufficient in processing feature fusion at different resolutions, especially in the identification and segmentation of small nodules, which are easily limited by resolution, resulting in unsatisfactory segmentation results.

[0004] In addition, existing methods often lack target consistency strategies in the process of multi-resolution feature fusion, resulting in insufficient information transfer between high-resolution and low-resolution features, affecting the final segmentation results. Therefore, how to effectively fuse features at multiple resolutions and improve the segmentation accuracy of complex morphological lung nodules remains a technical problem that needs to be solved urgently.

[0005] In summary, the existing technology has shortcomings in multi-resolution feature fusion and target consistency of lung nodules, and it is urgent to develop new segmentation methods to improve the accuracy and reliability of segmentation. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a lung nodule segmentation method and device based on target consistent multi-resolution fusion, which can provide users with a more efficient and accurate lung nodule segmentation solution.

[0007] Specifically, the method comprises the following steps:

[0008] S1: Collect lung CT image data and use the Object-Consistent Muti-Resolution Input Strategy (OCMIS) to generate high-resolution and low-resolution data;

[0009] S2: The high-resolution and low-resolution features of the data are extracted by Small Nodule Fine-Grained Enhanced Encoder (SNFGEE) and Multi-Granularity Enhanced Encoder (MGEE) respectively to obtain the high-resolution and low-resolution features of the small nodules.

[0010] S3: The high-resolution and low-resolution features are fused through the multi-resolution lossless feature fusion module (MLFF) to generate fused features;

[0011] S4: Input the fused features into the decoder to generate the segmentation result.

[0012] Preferably, S1 comprises the following steps:

[0013] S1.1: Collect lung CT image data and obtain the center coordinates and size of each nodule;

[0014] S1.2: Intercept and generate image data and nodule annotations of different resolutions according to the center coordinates and size of each nodule;

[0015] S1.3: Use image data of different resolutions as input images and nodule annotations as input labels.

[0016] Preferably, S2 comprises the following steps:

[0017] S2.1: Construct Small Nodule Fine-Grained Enhanced Encoder (SNFGEE) and Multi-Granularity Enhanced Encoder (MGEE);

[0018] S2.2: Perform feature extraction calculation on the high-resolution input according to the SNFGEE module to obtain the high-resolution feature f h ;

[0019] S2.3: Perform feature extraction calculation on the low-resolution input according to the MGEE module to obtain the low-resolution feature f l ;

[0020] Preferably, S2.1 comprises the following steps:

[0021] The global information of high-resolution data is extracted through the combined CT module of convolution and Transformer. The multi-granularity features are enhanced according to the multi-difference information through the Fine-Grained Enhanced Convolution (FGEC) module to obtain the SNFGEE encoder:

[0022] f 1 =Conv 1×1 (Conv 3×3 (Transformer(Conv 1×1 (Conv 3×3 (x))) ×2 ))

[0023]

[0024] In the above formula, x represents the input data, Transformer() represents the Transformer module, the subscript ×2 represents that the operation is performed twice, and f 1 、f 2 represents the intermediate feature, f CT represents the output of the CT module, k i (i=1,2,3) represent the convolution kernels of standard convolution, center difference, and angle difference, respectively. n×n () represents the convolution kernel of size n, f h Represents the characteristics of high-resolution data, Represents a splicing operation, represents the matrix addition operation, Represents a matrix multiplication operation;

[0025] The features of low-resolution data are extracted through large kernel convolution, which maintains fine-grained information and improves the efficiency of feature learning. The MGEE encoder is obtained by using the spatial-to-depth reorganization features through the Efficient Edge Feature Preserving Convolution (EEFPC):

[0026] MLP(.)=Conv 5×5 (Relu(Conv 5×5 (.)))

[0027] scale(.)=wAvgPool(.)+b

[0028] EEFPC(x)=SE(NConv(SPD(x,scale(x))))

[0029] f l=EEFPC(MLP(x)) ×4

[0030] In the above formula, x represents input data, MLP(.) represents multi-layer perceptron, Relu() represents Relu activation function, scale(.) represents adaptive scaling factor, w and b represent learnable weights and bias parameters, AvgPool(.) represents average pooling function, SE() represents Squeeze-and-Excitation module, NConv() represents Non-stridedConvolution, SPD() represents Space-to-depth convolution, and f l Represents the features of low-resolution data, and the subscript ×4 means that the operation is performed 4 times.

[0031] Preferably, S3 comprises the following steps:

[0032] S3.1: Construct a multi-resolution lossless feature fusion module (MLFF);

[0033] S3.2: High and low resolution features f h 、f l Input into MLFF to obtain fusion features;

[0034] Preferably, S3.1 comprises the following steps:

[0035] By extracting the rich feature information of nodules in low-resolution images, the nodule features of high-resolution images are enhanced, thereby ensuring that important detail information is not lost during the fusion process, achieving accurate feature fusion, and obtaining the MLFF structure:

[0036]

[0037] In the above formula, σ represents the sigmoid activation function, f 1 represents the features after fusion of multi-resolution channel information, f 2 represents the feature after fusion of multi-resolution spatial information, f out Represents multi-resolution fusion features.

[0038] Preferably, S4 comprises the following steps:

[0039] S4.1: Input the fused features into the decoder, restore the resolution by upsampling layer by layer, and obtain the lung nodule segmentation result;

[0040] S4.2: Obtain the segmentation model of the lung nodule segmentation method based on target consistent multi-resolution fusion.

[0041] The second technical solution adopted by the present invention is: a pulmonary nodule segmentation device based on target consistent multi-resolution fusion, comprising:

[0042] Preprocessing module: used to collect data and preprocess the data according to OCMIS;

[0043] SNFGEE encoder: used to extract features of high-resolution data;

[0044] MGEE encoder: used to extract features of low-resolution data;

[0045] MLFF module: used to fuse high- and low-resolution features to obtain fused features.

[0046] The beneficial effect of the method of the present invention is as follows: the present invention first collects lung CT image data and generates high-resolution and low-resolution data by adopting a target-consistent multi-resolution input strategy (OCMIS). Then, the small nodule fine-grained enhancement encoder (SNFGEE) and the multi-granularity enhancement encoder (MGEE) are used to extract features from the high-resolution and low-resolution data, respectively, to obtain high-resolution and low-resolution features of the small nodules. Then, the high-resolution and low-resolution features are effectively fused through the multi-resolution lossless feature fusion module (MLFF) to generate fused features. Finally, the fused features are input into the decoder to generate the segmentation results of the lung nodules, thereby achieving accurate segmentation of complex morphological lung nodules. At the same time, the present invention can effectively address the common challenges of existing neural network models in pulmonary nodule segmentation. On the one hand, the use of the target consistent multi-resolution input strategy (OCMIS) can effectively promote the model to capture the subtle features of pulmonary nodules at different scales, and enhance the model's recognition ability for complex morphological nodules; on the other hand, the use of the small nodule fine-grained enhancement encoder (SNFGEE) and the multi-granularity enhancement encoder (MGEE) can effectively improve the model's ability to extract small nodules and multi-scale features, ensuring that features at different resolutions are fully utilized; at the same time, through the multi-resolution lossless feature fusion module (MLFF), the utilization and integration efficiency of the model for features of different resolutions are improved, ultimately improving the effect and reliability of pulmonary nodule segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the prior art and the drawings required for use in the embodiments. The following drawings are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a flowchart of a lung nodule segmentation method based on target consistent multi-resolution fusion of the present invention;

[0049] Figure 2 It is a schematic diagram of the model architecture of a lung nodule segmentation method based on target consistent multi-resolution fusion of the present invention;

[0050] Figure 3 It is a schematic diagram of the SNFGEE module structure of a pulmonary nodule segmentation method based on target consistent multi-resolution fusion of the present invention;

[0051] Figure 4 It is a schematic diagram of the MGEE module structure of a lung nodule segmentation method based on target consistent multi-resolution fusion of the present invention;

[0052] Figure 5 It is a schematic diagram of the MLFF module structure of a pulmonary nodule segmentation method based on target consistent multi-resolution fusion of the present invention; Specific implementation plan

[0053] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all other embodiments obtained by ordinary technicians in this field without creative work based on the embodiments of the present invention are within the scope of protection of the present invention.

[0054] The embodiments of the present invention provide a method and device for segmenting lung nodules based on target consistent multi-resolution fusion, which are used to efficiently and accurately segment nodules in lung CT images at the algorithm level, thereby improving the accuracy and efficiency of diagnosis.

[0055] A typical embodiment of the present invention, taking the LUNA16 dataset as an example, refer to Figure 1 , the method comprises the following steps:

[0056] S1: Collect lung CT image data and use the Object-Consistent Muti-Resolution Input Strategy (OCMIS) to generate high-resolution and low-resolution data;

[0057] S2: The high-resolution and low-resolution features of the data are extracted by Small Nodule Fine-Grained Enhanced Encoder (SNFGEE) and Multi-Granularity Enhanced Encoder (MGEE) respectively to obtain the high-resolution and low-resolution features of the small nodules.

[0058] S3: The high-resolution and low-resolution features are fused through the multi-resolution lossless feature fusion module (MLFF) to generate fused features;

[0059] S4: Input the fused features into the decoder to generate the segmentation result.

[0060] Further, S1 comprises the following steps:

[0061] S1.1: Collect lung CT image data and obtain the center coordinates and size of each nodule;

[0062] Furthermore, the step of collecting lung CT image data and obtaining the center coordinates and size of each nodule specifically includes:

[0063] Download the lung CT images contained in the LUNA16 dataset and ensure that the center coordinates and size information of each nodule are accurately recorded.

[0064] S1.2: Intercept and generate image data and nodule annotations of different resolutions according to the center coordinates and size of each nodule;

[0065] Furthermore, the step of intercepting and generating data of different resolutions and nodule annotation according to the center coordinates and size of each nodule specifically includes:

[0066] Taking the central coordinates of each nodule as the center, intercept the local area containing the nodule;

[0067] The scale of the cropped area is adjusted according to the size of the nodule to generate high-resolution (128×128 pixels) and low-resolution (64×64 pixels) image blocks;

[0068] Corresponding to the generated high-resolution and low-resolution image blocks, corresponding nodule annotation masks are created to mark the nodule area.

[0069] S1.3: Use image data of different resolutions as input images and nodule annotations as input labels.

[0070] Furthermore, the step of using the image data of different resolutions as input images and the nodule annotations as input labels specifically includes:

[0071] Use the Numpy library to convert high-resolution and low-resolution image patches and their corresponding annotation masks into NPY format files;

[0072] The high-resolution image patches are saved as images_high.npy, and the low-resolution image patches are saved as images_low.npy; the corresponding annotation masks are saved as labels_high.npy and labels_low.npy.

[0073] Further, refer to Figure 3 , 4 , S2 includes the following steps:

[0074] S2.1: Construct Small Nodule Fine-Grained Enhanced Encoder (SNFGEE) and Multi-Granularity Enhanced Encoder (MGEE);

[0075] S2.2: Perform feature extraction calculation on the high-resolution input according to the SNFGEE module to obtain the high-resolution feature f h ;

[0076] S2.3: Perform feature extraction calculation on the low-resolution input according to the MGEE module to obtain the low-resolution feature f l .

[0077] Further, S2.1 includes the following steps:

[0078] The global information of high-resolution data is extracted through the combined CT module of convolution and Transformer. The multi-granularity features are enhanced according to the multi-difference information through the Fine-Grained Enhanced Convolution (FGEC) module to obtain the SNFGEE encoder:

[0079] f 1 =Conv 1×1 (Conv 3×3 (Transformer(Conv 1×1 (Conv 3×3 (x))) ×2 ))

[0080]

[0081]

[0082] In the above formula, x represents the input data, Transformer() represents the Transformer module, the subscript ×2 represents that the operation is performed twice, and f 1 、f 2 represents the intermediate feature, f CTrepresents the output of the CT module, k i (i=1,2,3) represent the convolution kernels of standard convolution, center difference, and angle difference, respectively. n×n () represents the convolution kernel of size n, f h Represents the characteristics of high-resolution data, Represents a splicing operation, represents the matrix addition operation, Represents a matrix multiplication operation;

[0083] The features of low-resolution data are extracted through large kernel convolution, which maintains fine-grained information and improves the efficiency of feature learning. The MGEE encoder is obtained by using the spatial-to-depth reorganization features through the Efficient Edge Feature Preserving Convolution (EEFPC):

[0084] MLP(.)=Conv 5×5 (Relu(Conv 5×5 (.)))

[0085] scale(.)=wAvgPool(.)+b

[0086] EEFPC(x)=SE(NConv(SPD(x,scale(x))))

[0087] f l =EEFPC(MLP(x)) ×4

[0088] In the above formula, x represents input data, MLP(.) represents multi-layer perceptron, Relu() represents Relu activation function, scale(.) represents adaptive scaling factor, w and b represent learnable weights and bias parameters, AvgPool(.) represents average pooling function, SE() represents Squeeze-and-Excitation module, NConv() represents Non-stridedConvolution, SPD() represents Space-to-depth convolution, and f l Represents the features of low-resolution data, and the subscript ×4 means that the operation is performed 4 times.

[0089] Further, refer to Figure 5 , S3 includes the following steps:

[0090] S3.1: Construct a multi-resolution lossless feature fusion module (MLFF);

[0091] S3.2: High and low resolution features f h 、f l Input into MLFF to obtain fusion features.

[0092] Further, S3.1 includes the following steps:

[0093] By extracting the rich feature information of nodules in low-resolution images, the nodule features of high-resolution images are enhanced, thereby ensuring that important detail information is not lost during the fusion process, achieving accurate feature fusion, and obtaining the MLFF structure:

[0094]

[0095]

[0096] In the above formula, σ represents the sigmoid activation function, f 1 represents the features after fusion of multi-resolution channel information, f 2 represents the feature after fusion of multi-resolution spatial information, f out Represents multi-resolution fusion features.

[0097] Further, S4 comprises the following steps:

[0098] S4.1: Input the fused features into the decoder, restore the resolution by upsampling layer by layer, and obtain the lung nodule segmentation result;

[0099] S4.2: Obtain a segmentation model for the lung nodule segmentation method based on target consistent multi-resolution fusion;

[0100] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make other equivalent modifications or substitutions without violating the spirit of the invention, and these equivalent modifications or substitutions are included in the scope defined by the application claims.

Claims

1. A pulmonary nodule segmentation method and device based on target consistent multi-resolution fusion, characterized in that: The method comprises the following steps: S1: Collect lung CT image data and use the Object-Consistent Muti-Resolution Input Strategy (OCMIS) to generate high-resolution and low-resolution data; S2: The high-resolution and low-resolution features of the data are extracted by using the Small Nodule Fine-Grained Enhanced Encoder (SNFGEE) and Multi-Granularity Enhanced Encoder (MGEE) to obtain the high-resolution and low-resolution features of the nodules. S3: The high-resolution and low-resolution features are fused through the multi-resolution lossless feature fusion module (MLFF) to generate fused features; S4: Input the fused features into the decoder to generate the segmentation result.

2. The method and device for pulmonary nodule segmentation based on target consistent multi-resolution fusion according to claim 1, characterized in that: S1 includes the following steps: S1.1: Collect lung CT image data and obtain the center coordinates and size of each nodule; S1.2: Intercept and generate image data and nodule annotations of different resolutions according to the center coordinates and size of each nodule; S1.3: Use image data of different resolutions as input images and nodule annotations as input labels.

3. The method and device for pulmonary nodule segmentation based on target consistent multi-resolution fusion according to claim 1, characterized in that: S2 includes the following steps: S2.1: Construct Small Nodule Fine-Grained Enhanced Encoder (SNFGEE) and Multi-Granularity Enhanced Encoder (MGEE); S2.2: Perform feature extraction calculation on the high-resolution input according to the SNFGEE module to obtain the high-resolution feature f h ; S2.3: Perform feature extraction calculation on the low-resolution input according to the MGEE module to obtain the low-resolution feature f l ; Among them, S2.1 includes the following steps: The global information of high-resolution data is extracted through the combined CT module of convolution and Transformer. The multi-granularity features are enhanced according to the multi-difference information through the Fine-Grained Enhanced Convolution (FGEC) module to obtain the SNFGEE encoder: f1=Conv 1×1 (Conv 3×3 (Transformer(Conv 1×1 (Conv 3×3 (x))) ×2 )) In the above formula, x represents the input data, Transformer() represents the Transformer module, the subscript ×2 represents that the operation is performed twice, f1 and f2 represent intermediate features, and f CT represents the output of the CT module, k i (i=1,2,3) represent the convolution kernels of standard convolution, center difference, and angle difference, respectively. n×n () represents the convolution kernel of size n, f h Represents the characteristics of high-resolution data, Represents a splicing operation, represents the matrix addition operation, Represents a matrix multiplication operation; The features of low-resolution data are extracted through large kernel convolution, which maintains fine-grained information and improves the efficiency of feature learning. The MGEE encoder is obtained by using the spatial-to-depth reorganization features through the Efficient Edge Feature Preserving Convolution (EEFPC): MLP(.)=Conv 5×5 (Release(Conv 5×5 (.))) scale(.)=wAvgPool(.)+b EEFPC(x)=SE(NConv(SPD(x,scale(x)))) f l =EEFPC(MLP(x)) ×4 In the above formula, x represents input data, MLP(.) represents multi-layer perceptron, Relu() represents Relu activation function, scale(.) represents adaptive scaling factor, w and b represent learnable weights and bias parameters, Avgpool(.) represents average pooling function, SE() represents Squeeze-and-Excitation module, NConv() represents Non-strided Convolution, SPD() represents Space-to-depth convolution, d l Represents the features of low-resolution data, and the subscript ×4 means that the operation is performed 4 times.

4. The method and device for pulmonary nodule segmentation based on target consistent multi-resolution fusion according to claim 1, characterized in that: S3 includes the following steps: S3.1: Construct a multi-resolution lossless feature fusion module (MLFF); S3.2: High and low resolution features f h 、f l Input into MLFF to obtain fusion features; Among them, S3.1 includes the following steps: By extracting the rich feature information of nodules in low-resolution images, the nodule features of high-resolution images are enhanced, thereby ensuring that important detail information is not lost during the fusion process, achieving accurate feature fusion, and obtaining the MLFF structure: In the above formula, σ represents the sigmoid activation function, f1 represents the feature after multi-resolution channel information fusion, f2 represents the feature after multi-resolution spatial information fusion, and f out Represents multi-resolution fusion features.

5. The method and device for pulmonary nodule segmentation based on target consistent multi-resolution fusion according to claim 1, characterized in that: S4 includes the following steps: S4.1: Input the fused features into the decoder, restore the resolution by upsampling layer by layer, and obtain the lung nodule segmentation result; S4.2: Obtain the segmentation model of the lung nodule segmentation method based on target consistent multi-resolution fusion.

6. A pulmonary nodule segmentation device based on target consistent multi-resolution fusion, characterized in that: include: Preprocessing module: used to collect data and preprocess the data according to OCMIS; SNFGEE encoder: used to extract features of high-resolution data; MGEE encoder: used to extract features of low-resolution data; MLFF module: used to fuse high- and low-resolution features to obtain fused features.

Images