Multi-scale information fusion complementary lightweight pulmonary nodule segmentation model and implementation method thereof

Abstract

The invention discloses a multi-scale information fusion complementary lightweight pulmonary nodule segmentation model and an implementation method thereof. The segmentation model comprises a feature mining coding module, a feature enhancement module, a feature mapping decoding module and a feature mining coding module. The feature mining coding module is used for deeply mining deep and shallow layer information of a pulmonary nodule region from a CT image by using convolution residual blocks with different scales, and introducing an EMA attention mechanism to enrich semantic information of a feature map; an E-MLCA attention mechanism is designed in the feature enhancement module and promotes complementary fusion of semantic information from two dimensions of global and local and channel and space; an RMF module is used for enhancing multi-scale features, and the problem of detail information missing between channels is solved. And the feature mapping decoding module realizes smoothness of the marginal area of the lesion area and truly reflects the shape of the nodule area through recombination and fusion of features of four decoding layers and sub-channel reinforcement of the segmentation effect picture. According to the method, the excellent segmentation effect is achieved while the small parameter quantity is kept, the segmented feature map is smooth, the subjective effect is good, and the method has the important clinical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and relates to a multi-scale information fusion and complementary lightweight lung nodule segmentation model and an implementation method thereof. Background Art

[0002] Cancer is a difficult problem that modern medicine cannot overcome. The mortality rate of lung cancer ranks first among all cancers, which has a great impact on the health and life of patients. Relevant data show that the survival rate of advanced lung cancer is only 15%, while the survival rate of early lung cancer can reach 40%-70%. Early lung cancer usually exists in the form of benign nodules. If early screening is performed through computed tomography (CT), the pulmonary nodule lesion area is discovered in time and corresponding diagnosis and treatment measures are taken, the patient's 5-year survival rate can be increased to 57%. However, due to the uneven distribution and irregular shape of pulmonary nodules, it has a great impact on the subjective judgment of doctors. In recent years, with the widespread application of artificial intelligence technology in the field of medical image processing, the study of accurate and efficient pulmonary nodule segmentation algorithms is of great significance for the early screening of lung cancer and the timely diagnosis and treatment of patients.

[0003] Traditional pulmonary nodule segmentation is mainly based on shallow features such as grayscale value and texture. Although this type of segmentation method has certain effects, it requires manually designed feature extraction operators to extract features, has a strong reliance on manual work, is easily affected by subjective factors, and is also easily affected by underlying features such as lighting and posture; and cannot provide robust and accurate segmentation results. There is still room for improvement in performance and generalization ability, and it cannot meet the needs of current clinical diagnosis.

[0004] With the improvement of deep learning theory and the continuous progress of computer hardware and software, domestic and foreign scholars have carried out extensive research in the field of lung nodule segmentation based on convolutional neural networks. Ronneberger et al. first used a symmetrical U-shaped network structure to process the detail information in medical images, and achieved good results in segmenting small target areas, but its parameters were large and overfitting problems were prone to occur during training. For details, see "Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention (MICCAI), 2015, 234-241". Oktay et al. added an attention mechanism to the end of the jump connection of U-Net to form a gating signal, filter irrelevant feature information, and better process the detail information in medical images. For details, see "Attention U-Net: Learning where to look for the pancreas. arXiv preprint arXiv: 1804.03999, 2018". Zhou et al. densely replaced the cropping and concatenation operations in the U-Net jump connection with convolution operations to obtain better feature information and compensate for the information loss caused by sampling. For details, see "Zhou Z, Siddiquee MM, Tajbakhsh N, et al. Unet++: Anested u-net architecture for medical imagesegmentation. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018, 3-11". Jha et al. stacked two U-Net architectures together to capture more information. For details, see "Jha D, Riegler MA, Johansen D, et al. DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation. 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS), 2020, 558-564".Xu et al. used a split-attention U-net to effectively utilize low-level and high-level semantic information, solving the problem that U-Net does not allow sufficient feature information to be extracted from different depths. For details, see "Xu Q, Ma Z, He N, et al. DCSAU-Net: A deeper and more compact split-attention U-Net for medical imagesegmentation. Computers in Biology and Medicine, 2023, 154". Zhang et al. introduced residual connections and dense connections to improve the model's feature extraction capabilities and obtain more information about lung nodule areas. For details, see "Zhang X, Kong S, Han Y, et al. Lung Nodule CT Image Segmentation Model Based on Multiscale Dense Residual Neural Network. Mathematics, 2023, 11(6)".

[0005] Although the above methods fully mine useful information of the lesion area from CT images, the model parameters are too large to meet the needs of clinical real-time diagnosis. In order to reduce the number of model parameters and facilitate hardware deployment, Zainab proposed a hybrid neural network that integrates the MobileNetV2 and UNet architectures. The pre-trained MobileNetV2 is used as the encoder of the UNet model to extract features to reduce the amount of calculation. For details, see "Zainab R, Bangul K, Saad A, et al. Lung Tumor Image Segmentation from Computer Tomography Images Using MobileNetV2 and Transfer Learning. Bioengineering-Basel, 2023, 10 (8)". Wang introduced the attention mechanism and integrated multi-scale information to reduce the number of model parameters while ensuring the segmentation effect. For details, see "Wang C, Shao S, Yin J, et al. MSA-UNet: A Multiscale Lightweight U-Net Lung CT Image Segmentation Algorithm Under Attention Mechanism. International Journal on Artificial Intelligence Tools, 2024, 33 (02)". Zhou achieved performance improvement by combining residual enhanced connections with adaptive attention mechanisms to capture key features in the image and reduce model parameters. For details, see "Zhou H, HuB, Yi N, et al. Balancing High-performance and Lightweight: HL-UNet for 3DCardiac Medical Image Segmentation. Academic Radiology, 2024". Cao reduced the complexity of the model by fusing only the important information at each stage and ensured that the decoder produced better results. For details, see "Cao D, Zhang Rand Zhang Y. MFLUnet: multi-scale fusion lightweight Unet for medical imagesegmentation. Biomedical Optics Express, 2024, 15(10): 5574-5591".Pang uses a lightweight GhostV2 bottleneck block and a convolutional attention block that captures key features to reduce redundant information and reduce the number of model parameters. For details, see "Pang B, Chen L, Tao Q, et al. GA-UNet: A Lightweight Ghost and Attention U-Net for Medical Image Segmentation. Journal of Imaging Informatics in Medicine, 2024, 37(4): 1874-1888". Although the above method reduces the number of model parameters, it does not establish a good balance between model performance and accuracy. The reduction in the number of parameters causes the loss of some key information, and the subjective effect of the feature map after segmentation needs to be improved, which cannot meet the clinical diagnosis needs of medical image processing.

[0006] The inventors have found that the existing deep learning-based lung nodule segmentation method has poor segmentation effect, and has the following main defects: (1) The existing segmentation algorithm has poor segmentation effect on small target areas of nodules and cannot meet the needs of actual clinical diagnosis; (2) The existing model has a large number of parameters and poor universality, which is not convenient for hardware deployment. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a lightweight lung nodule segmentation model with multi-scale information fusion and complementarity. The model establishes a relative balance between high precision and light weight, and is particularly accurate in segmenting small nodule areas. The segmented images are smoother, which solves the problems existing in the prior art.

[0008] The second objective of the present invention is to provide a lightweight lung nodule segmentation method with multi-scale information fusion and complementarity.

[0009] A third object of the present invention is to provide an electronic device.

[0010] A fourth object of the present invention is to provide a computer storage medium.

[0011] A multi-scale information fusion and complementary lightweight pulmonary nodule segmentation model, characterized by comprising a feature mining encoding module, a feature enhancement module and a feature mapping decoding module.

[0012] The feature mining encoding module combines ordinary convolution, hole convolution, and deep convolution to build a six-layer multi-convolution residual fusion feature mining encoding module, which deeply mines the fuzzy edge and texture information of the lesion area and fuses shallow features to solve the gradient disappearance and gradient explosion problems during network training, and obtains feature maps with rich semantic information in the lesion area;

[0013] The feature enhancement module includes the feature deep fusion submodule FDFM and the multi-scale fusion module RMF. The FDFM module implements the enhanced fusion of feature maps X2 to X5 by designing the E-MLCA attention mechanism. The RMF module enhances multi-scale features to solve the problem of missing detail information between channels, and sends the enhanced feature map to the feature map decoding module. The RMF module is responsible for deep mining of the feature map X6.

[0014] Multi-dimensional attention mechanism E-MLCA, in order to enhance the high-level semantic features of low-frequency signals in the lung nodule area, the feature enhancement module uses the attention mechanism E-MLCA that is enhanced in two dimensions, global and local, channel and space. Local features extract local spatial information through feature reorganization depth, and global features are used in both spatial and channel directions to achieve significant information mining through weight allocation and fusion with features;

[0015] The feature map decoding module includes four layers of decoding convolutional layers and an enhanced output module EOM. The decoding convolutional layers are responsible for decoding the output of the feature enhancement module. The EOM module is responsible for enhancing the semantic features of the pulmonary nodule lesion area, enriching the semantic information of the pulmonary nodule area in the feature map, and smoothing the output feature map to obtain the output of the feature map decoding module. The output of the feature map decoding module is fused with the initial features of the network, and the final pulmonary nodule segmentation image is obtained through convolution with a convolution kernel size of 3×3.

[0016] The enhanced output module EOM reorganizes and fuses the four-scale features output by the decoding convolutional layer, enriches the semantic information of the lung nodule area in the feature map, separates the features of the fused feature map along the channel direction, concatenates the global feature map and fuses it with the local features, and uses two layers of convolution with a kernel size of 3×3 for feature smoothing.

[0017] Furthermore, the feature mining encoding module is composed of six layers of sub-modules, namely, a layer of initial feature residual extraction block IFREB, two layers of global depth-by-depth convolution blocks GDCB, a layer of feature mapping enhancement block FMEB and two layers of ordinary convolution blocks.

[0018] The first layer is used to obtain shallow features of the original lung CT image through a 3×3 convolution;

[0019] The second layer is used to perform deep mining on the shallow features extracted from the first layer through a residual connection block IFREB consisting of a 3×3 convolution and a 3×3 dilated convolution.

[0020] The third and fourth layers are used to enhance the high-level semantic features of the low-frequency signals of the lesions through the residual connection blocks GDCB ​​composed of a 1×1 convolution and a 3×3 depth convolution on the outputs of the second and third layers respectively.

[0021] The fifth layer is used to deeply enhance the lesion features of the output of the fourth layer through a residual connection block consisting of a 1×1 convolution and a 3×3 deep convolution and an EMA attention mechanism, which is composed of FMEB.

[0022] The sixth layer is used to smooth the features of the output of the fifth layer through a 3×3 convolution to obtain the output of the feature mining encoding module.

[0023] Furthermore, the feature enhancement module fuses and enhances the features extracted from adjacent layers through three FDFM submodules, and uses one RMF module to enhance the features extracted from the sixth layer.

[0024] Furthermore, the FDFM submodule first fuses the features of two adjacent layers, and strengthens the fused feature map in the spatial and channel, global and local dimensions through the E-MLCA attention mechanism. The feature map strengthened by E-MLCA is weighted and features are fused with the two original inputs to obtain the output of the FDFM module.

[0025] Furthermore, the RMF submodule uses maximum pooling with pooling sizes of 0, 5, 9, and 13 for the feature maps output by the sixth layer of the feature mining encoding module to retain the input features to the maximum extent without introducing parameters, and uses global depth-wise convolution blocks with expansion rates of 1, 3, and 5 for the pooled and original feature maps to enhance the features. After the feature cascade of each branch is completed, the feature maps of four scales are superimposed to obtain the feature maps with enhanced multi-scale information.

[0026] Furthermore, the feature map decoding module includes four layers of decoding convolutional layers and an enhanced output module EOM.

[0027] Furthermore, the four-layer decoding convolution layer uses deconvolution to expand the feature map to obtain the feature map of the same size as the previous level of the feature enhancement module. After the cascade is completed, the features are passed through the decoding convolution layer to obtain a four-dimensional feature map that fuses information of different dimensions.

[0028] Furthermore, the EOM module reorganizes and fuses the four-scale features, separates the features of the fused feature map along the channel direction, concatenates the global feature map and fuses it with the local features, and uses two layers of convolution with a kernel size of 3×3 to smooth the features.

[0029] The output of the feature map decoding module is fused with the initial features of the network, and the lesion area is optimized through a convolution with a convolution kernel size of 3×3 to generate the final lung nodule segmentation image.

[0030] A lightweight pulmonary nodule segmentation method with multi-scale information fusion and complementation is carried out in the following steps:

[0031] S1. The original lung nodule CT image is input into the feature mining coding module of the multi-scale information fusion complementary lightweight lung nodule segmentation model. The feature mining coding module extracts features according to the following formula:

[0032]

[0033] X n =[f ReLU (BN(C 1×1 (X n-1 ))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X n-1 )))))))];

[0034] X4' (1) =Group H (Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4)))))))]));

[0035] X4' (2) =Group W (Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4)))))))]));

[0036] X4' (3) =Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))]);

[0037] X4”=Group(X4')×f f H (C 1×1 ([Avg(X4' (1) ),Avg(X4' (2) )]))×f f W (C 1×1([Avg(X4' (1) ),Avg(X4' (2) )]));

[0038] X M1 =f Softmax (Avg(GN(X4”)))·C 3×3 (X4' (3) );

[0039] X M2 =GN(X4”)·f Softmax (Avg(C 3×3 (X4' (3) )));

[0040] X6=C 3×3 (Group(X4')×f Sigmoid (X M1 +X M2 ));

[0041] Among them, C 3×3 represents a normal convolution with a convolution kernel size of 3×3, C 1×1 represents a normal convolution with a convolution kernel size of 1×1, represents a dilated convolution with a kernel size of 3×3 and a dilation rate of 2, BN represents batch normalization, and f ReLU represents the ReLU activation function. n-1 and X n (n=3,4) represent the input and output of the third and fourth layers of the feature mining encoding module, respectively. DP3×3 represents a depthwise convolution with a kernel size of 3×3, [] represents cascade fusion, X4' (k) (k=1,2,3) represents the feature map of the kth branch after grouping. Group represents the grouping operation along the channel direction. H and Group W Respectively represent the slice grouping along the height and width directions, Avg represents average pooling, and f f H and f f W They represent the vector decomposition along the height and width directions respectively, X4" represents the output feature map of the fusion of the weights in the X and Y directions, X M1 and X M2 Represents the output of matrix dot multiplication in two dimensions, f Softmax represents the Softmax activation function, GN represents GroupNorm normalization, · represents the dot product of the matrix, and X5 represents the output of the feature enhancement block;

[0042] S2, the outputs X1 to X6 of each convolution block of the feature mining encoding module are sent to the feature enhancement module. The FDFM submodule and RMF submodule in the feature enhancement module perform deep-level enhancement of features according to the following formula:

[0043] X C =[Adap(C 3×3 (X k-1 )),Adap(f up (C 3×3 (X k )))];

[0044] X C '=Re(C 1×1 (Re(LAP(X C ))))+GAP(LAP(X C ))·f Sigmoid (C 1×1 (GAP(LAP(X C ))));

[0045] X C ” = f UNAP (X C ')×X C +X C ;

[0046]

[0047] Y k-2a =λ×X C ”×(C 3×3 (X k-1 ))+(1-λ)×X C ”(f up (C 3×3 (X k )));

[0048]

[0049] Among them, X k-1 and X k (k=3,4,5) represents the input features of the feature enhancement module, Adap represents the adaptive pooling operation, LAP and GAP represent the local pooling and global pooling operations respectively, Re represents the reorganization of the features, and f Sigmoid and f UNAP Represent the Sigmoid activation function and the anti-pooling process, respectively, up represents the upsampling process, T i (i∈[0,g]) represents the i-th subchannel in channel T, λ represents the weight coefficient of each branch at the output of the feature enhancement module, Y k-2a(a=0,1,2 when k=3,4,5) represents the output of FDFM module, X C and X C ” represent the input and output features of the E-MLCA attention mechanism, X C ' represents the features after the fusion of the two branches in the E-MLCA attention mechanism, They represent global depth-wise convolution with kernel size of 3×3 and dilation rates of 1, 5, and 9, respectively. t (t=5,9,13) represent the maximum pooling operations with sizes of 5, 9, and 13 respectively, Y4 h (h=1, 2, 3) represent the outputs of the branches with pooling sizes of 5, 9, and 13, respectively, and Y4 represents the output of the RMF module.

[0050] S3. Send the output of the feature enhancement module to the feature map decoding module. The feature map decoding module processes the four input feature maps according to the following formula:

[0051] D1=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y3,C tr (Y4)]))))));

[0052] D2=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y2,C tr (D1)]))))));

[0053] D3=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y1,C tr (D2)]))))));

[0054] D4=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([X2,C tr (D3)]))))));

[0055] D add =Re(D1)+Re(D2)+Re(D3)+Re(D4);

[0056] D add '=f sp-G (D add )+fBilinear (Pat(f sp-L (D add )));

[0057] Y'=f PReLU (BN(C 3×3 (f PReLU (BN(C 3×3 (D add '))))));

[0058] Among them, D b (b∈[1,4]) represents the output of the b-th decoding convolutional layer, C tr represents the deconvolution operation, D add and D add 'represent the output of the features after four-scale reorganization and fusion and the feature map separation and enhancement, respectively, sp-G and f sp-L Represent the global features and local features after separation, respectively, Bilinear represents the bilinear interpolation process, f PReLU represents the PReLU activation function, and Y' represents the feature map output by the enhanced output module.

[0059] S4, the output of the feature map decoding module is fused with the initial features of the network, and the final lung nodule segmentation image Y is obtained by convolution with a convolution kernel size of 3×3. The mathematical expression of the above process is:

[0060] Y=C 3×3 (Y'+f Bilinear (Y',X1))

[0061] Among them, Y represents the output of the network model.

[0062] An electronic device adopts the above method to achieve lung nodule region segmentation.

[0063] A computer storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by a processor to implement the above-mentioned image segmentation method.

[0064] The beneficial effects of the present invention are:

[0065] The present invention makes full use of the multi-scale features extracted by different convolution kernels and the correlation between feature maps in multi-dimensional directions. In the feature mining encoding module, convolution residual blocks of different scales are used to dig deep and shallow information of the lung nodule area from the CT image, and the EMA attention mechanism is introduced to enrich the semantic information of the feature map. Then the feature enhancement module is used to mine and utilize the correlation between the features output by each submodule of the feature mining encoding part. In the feature enhancement module, the E-MLCA attention mechanism is designed, which promotes the complementary fusion of semantic information from the two dimensions of global and local, channel and space; the RMF module is used to enhance multi-scale features and solve the problem of missing detail information between channels. The feature mapping decoding module realizes the smoothing of the edge area of ​​the lesion area through the reorganization and fusion of the features of the four decoding layers and the channel-by-channel enhancement of the segmentation effect map, and truly reflects the shape of the nodule area. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a structural diagram of a segmentation model according to an embodiment of the present invention.

[0068] Figure 2 It is a structural diagram of a feature mining encoding module in a segmentation model according to an embodiment of the present invention.

[0069] Figure 3 It is a structural diagram of the FDFM module in the segmentation model of an embodiment of the present invention.

[0070] Figure 4 It is a structural diagram of the RMF module in the segmentation model of an embodiment of the present invention.

[0071] Figure 5 It is a structural diagram of the EOM module in the segmentation model of an embodiment of the present invention.

[0072] Figure 6 It is a subjective comparison diagram of the segmentation effect of the embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] Embodiment 1,

[0075] A lightweight pulmonary nodule segmentation model with multi-scale information fusion and complementation, whose structure is as follows Figure 1 As shown, it includes a feature mining encoding module, a feature enhancement module, and a feature mapping decoding module connected in sequence;

[0076] Among them, the feature mining encoding module aims to address the problem that the captured images are blurred due to changes in external environmental factors and the influence of different substances in the patient's body during the capture of lung CT images, and the characteristics of lung nodules change greatly at different stages, making it difficult to accurately identify significant features. It combines ordinary convolution, hole convolution, and deep convolution to construct a six-layer multi-convolution residual fusion structure to deeply mine the fuzzy edges and texture information of the lesion area and fuse shallow features to solve the problems of gradient disappearance and gradient explosion in the network training process, and obtain feature maps with rich semantic information in the lesion area; at the same time, the EMA attention mechanism is introduced to enrich the semantic information of the feature map.

[0077] like Figure 2 As shown in the figure, the feature mining encoding module is mainly composed of a layer of initial feature residual extraction block IFREB responsible for deep mining of shallow features, two layers of global depth-by-depth convolution blocks GDCB ​​responsible for deep mining of high-level semantic features of low-frequency signals in the lesion area, a layer of feature mapping enhancement block FMEB responsible for deep-level enhancement of lesion features, and two layers of ordinary convolution blocks responsible for preliminary extraction of lesion features and feature smoothing.

[0078] The feature mining encoding module sends the shallow feature X1 to Figure 2 The residual structure shown in (a) performs deep mining on X1 by connecting ordinary convolution with a convolution kernel size of 3×3 and a dilated convolution in series, fuses shallow features to enhance detail information, and achieves deep extraction of the lung nodule lesion area to obtain feature map X2 without increasing the number of parameters. In order to highlight the high-level semantic information of the low-frequency signal in the lesion area, the third and fourth layers use Figure 2 The GDCB ​​shown in (b) performs deep mining of feature maps. By introducing deep convolution, the input features X2 and X3 are divided into 32 and 64 groups, respectively. The convolution with a convolution kernel size of 3×3 is used to enhance the features of each group, highlighting the detail information of each sub-channel. The initial features and the sub-features obtained from each group are fused in a cascade manner to obtain feature maps X3 and X4, thereby achieving deep extraction of texture and contour information in the lesion area.

[0079] After four layers of different convolution blocks, the shallow and deep features of the nodule area are fully mined. In order to strengthen the deep features, the fifth layer is constructed by integrating GDCB ​​and EMA attention mechanisms. Figure 2 (c) shows the feature enhancement block FMEB. FMEB first uses global depth-wise convolution to enhance the features of X4 to obtain feature map X4' and divides it into 128 sub-features in the channel dimension direction. Each sub-feature extracts the attention weight of the grouped feature map through three parallel routes, that is, global average pooling is used to encode the channel features in the two spatial directions of X and Y, and convolution with a convolution kernel size of 3×3 is used in the third route to capture multi-scale features. In order to make full use of the information in the two spatial dimensions, the information extracted in the X and Y directions is fused and re-aggregated to achieve cross-channel feature interaction. In order to make full use of the shallow features extracted in the third route, the fused features are multiplied with the input in another route to form a new matrix. After the two matrices are fused, the sigmoid function and the feature are reorganized to form a new feature map X5 that highlights the texture and contour information of the lesion area, and the features are smoothed by the sixth layer of ordinary convolution blocks with a convolution kernel size of 3×3 to form the output X6 of the feature mining encoding module.

[0080] The feature enhancement module consists of two parts: the feature deep fusion submodule FDFM that enhances the fusion of feature maps X2 to X5 and the RMF module that performs deep mining on feature map X6. In the FDFM module, the multi-dimensional attention mechanism E-MLCA is used to deeply enhance the extracted feature maps from the two dimensions of global and local, channel and space. The local features extract local spatial information through the reorganization depth of the features, and the global features are used in the two directions of space and channel to achieve significant information mining through weight distribution and fusion with features; the RMF module retains the input features to the maximum extent without introducing parameters by using global depth-by-depth convolutions of different sizes.

[0081] like Figure 3 As shown in Figure 1, the FDFM module consists of two parts: the attention mechanism E-MLCA that integrates global features and local features, spatial and channel multi-dimensional features, and dual-path feature fusion. The module first integrates the low-level features X generated by the previous encoding layer. n-1 and the high-level features X generated by the current encoding layer n Forming feature map X C However, the fused feature map contains a lot of redundant information that affects the segmentation effect. Therefore, the attention mechanism E-MLCA, which integrates global features and local features, spatial and channel multi-dimensional features, is used to enhance the segmentation effect. CAfter global average pooling, two branches are connected in parallel to enhance the features: one branch performs adaptive feature search within the channel range M after global pooling, and multiplies it with the original feature map to obtain the final output; the other branch forms a feature map X by reorganizing the features and fusing the enhanced global features. C '. X C 'After unpooling and combined with X C Fusion forms the output X of E-MLCA C FDFM will be X C "The weights are assigned and features are fused with the original input features to obtain a feature map that removes redundant information.

[0082] like Figure 4 As shown in the figure, in order to enhance the multi-scale features, the RMF module is used to enhance the features of the generated X6 feature map. The maximum pooling with pooling sizes of 0, 5, 9, and 13 is used to retain the input features to the maximum extent without introducing parameters. The global depth-wise convolution blocks with expansion rates of 1, 3, and 5 are used to enhance the features of the pooled and original feature maps. After the feature cascade of each branch is completed, the feature maps of the four scales are superimposed to obtain the feature map with enhanced multi-scale information.

[0083] In order to better express the features, the feature map decoding module fuses and enhances the decoded features, uses the EOM module to enhance the semantic features of the pulmonary nodule lesion area, enriches the semantic information of the pulmonary nodule area in the feature map, and smoothes the output feature map.

[0084] like Figure 5 As shown in the figure, the feature map decoding module uses four layers of decoding convolutional layers as the main architecture, uses deconvolution to expand the feature map, and obtains the feature map of the same size as the previous level of the feature enhancement module. After the cascade is completed, the features are passed through the decoding convolution layer to obtain a four-dimensional feature map that fuses information of different dimensions. Although the context features of different scales are fused through the four layers of decoding convolutional layers, the relative independence of the features of each scale has a great impact on the segmentation effect. In order to deeply explore the internal connection of the four-scale features, the feature map decoding module is finally used as shown in the figure. Figure 5 The enhanced output module EOM shown in the figure enhances the semantic features of the pulmonary nodule lesion area. EOM reorganizes and fuses the four-scale features, enriches the semantic information of the pulmonary nodule area in the feature map, separates the features of the fused feature map along the channel direction, and fuses the global feature map with the local features after splicing, and uses two layers of convolution with a kernel size of 3×3 to smooth the features.

[0085] like Figure 1 As shown in FIG. 1 , the output of the feature map decoding module is fused with the initial features of the network, and the lesion area is optimized by a convolution with a convolution kernel size of 3×3 to generate the final lung nodule segmentation image Y.

[0086] The design of network models based on deep learning is based on the application background. For example, image super-resolution reconstruction focuses on the nonlinear mapping relationship between low-resolution images and high-resolution images. By extracting each feature information in the image, the original weak contour features are enhanced to improve details and textures; while image segmentation focuses on the analysis and understanding of image content. In model design, it is more inclined to focus on the identification of the target and separate the target information from the whole. In order to fully extract the useful information of the lesion area and minimize the high-frequency details, the embodiment of the present invention proposes a lightweight lung nodule segmentation model and segmentation method with multi-scale information fusion and complementarity. The feature mining encoding module that combines ordinary convolution, deep convolution, and hole convolution deeply mines the significant information of the lesion area from the CT image through a structure of six layers of different convolution residual connections, and introduces the EMA attention mechanism to enrich the semantic information of the feature map. The feature enhancement module makes full use of the relationship between features and fuses features of adjacent layers. The FDFM module that integrates the E-MLCA attention mechanism strengthens detail information from the global and local, spatial and channel dimensions, and deeply mines the intermediate layer features of the feature mining encoding module to highlight significant information. The RMF module uses the maximum pooling with pooling sizes of 0, 5, 9, and 13 to retain the input features to the maximum extent without introducing parameters. The global depth-by-depth convolution blocks with expansion rates of 1, 3, and 5 are used to strengthen the deep features of the pooled and original feature maps, enhance multi-scale features, and refine detail information. The feature map decoding module uses the EOM module to reorganize and fuse the decoded convolution layer information of different dimensions to enrich the semantic information of the lung nodule area in the feature map. The fused feature map is separated along the channel direction. The global feature map is fused with the local features after splicing, and the feature is smoothed using two layers of convolution with a kernel size of 3×3. The smoothed features are fused with the initial features of the network to obtain the final segmentation result map.

[0087] Embodiment 2,

[0088] A segmentation method of a lightweight pulmonary nodule segmentation model with multi-scale information fusion and complementation is performed in the following steps:

[0089] S1. The original lung nodule CT image is input into the feature mining coding module of the multi-scale information fusion complementary lightweight lung nodule segmentation model. The feature mining coding module extracts features according to the following formula:

[0090]

[0091] X n =[f ReLU (BN(C 1×1 (X n-1 ))),f ReLU(BN(C DP3×3 (f ReLU (BN(C 1×1 (X n-1 ))))))];

[0092] X4' (1) =Group H (Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))]));

[0093] X4' (2) =Group W (Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))]));

[0094] X4' (3) =Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))]);

[0095] X4”=Group(X4')×f f H (C 1×1 ([Avg(X4' (1) ),Avg(X4' (2) )]))×f f W (C 1×1 ([Avg(X4' (1) ),Avg(X4' (2) )]));

[0096] X M1 =f Softmax (Avg(GN(X4”)))·C 3×3 (X4' (3) );

[0097] X M2 =GN(X4”)·f Softmax(Avg(C 3×3 (X4' (3) )));

[0098] X6=C 3×3 (Group(X4')×f Sigmoid (X M1 +X M2 ));

[0099] Among them, C 3×3 represents a normal convolution with a convolution kernel size of 3×3, C 1×1 represents a normal convolution with a convolution kernel size of 1×1, represents a dilated convolution with a kernel size of 3×3 and a dilation rate of 2, BN represents batch normalization, and f ReLU represents the ReLU activation function. n-1 and X n (n=3,4) represent the input and output of the third and fourth layers of the feature mining encoding module, respectively. DP3×3 represents a depthwise convolution with a kernel size of 3×3, [] represents cascade fusion, X4' (k) (k=1,2,3) represents the feature map of the kth branch after grouping. Group represents the grouping operation along the channel direction. H and Group W Respectively represent the slice grouping along the height and width directions, Avg represents average pooling, and f f H and f f W They represent the vector decomposition along the height and width directions respectively, X4" represents the output feature map of the fusion of the weights in the X and Y directions, X M1 and X M2 Represents the output of matrix dot multiplication in two dimensions, f Softmax represents the Softmax activation function, GN represents GroupNorm normalization, · represents the dot product of the matrix, and X5 represents the output of the feature enhancement block;

[0100] S2, the outputs X1 to X6 of each convolution block of the feature mining encoding module are sent to the feature enhancement module. The FDFM submodule and RMF submodule in the feature enhancement module perform deep-level enhancement of features according to the following formula:

[0101] X C =[Adap(C 3×3 (X k-1 )),Adap(f up (C 3×3 (X k )))];

[0102] XC '=Re(C 1×1 (Re(LAP(X C ))))+GAP(LAP(X C ))·f Sigmoid (C 1×1 (GAP(LAP(X C ))));

[0103] X C ” = f UNAP (X C ')×X C +X C ;

[0104]

[0105] Y k-2a =λ×X C ”×(C 3×3 (X k-1 ))+(1-λ)×X C ”(f up (C 3×3 (X k )));

[0106]

[0107] Among them, X k-1 and X k (k=3,4,5) represents the input features of the feature enhancement module, Adap represents the adaptive pooling operation, LAP and GAP represent the local pooling and global pooling operations respectively, Re represents the reorganization of the features, and f Sigmoid and f UNAP Represent the Sigmoid activation function and the anti-pooling process, respectively, up represents the upsampling process, T i (i∈[0,g]) represents the i-th subchannel in channel T, λ represents the weight coefficient of each branch at the output of the feature enhancement module, Y k-2a (a=0,1,2 when k=3,4,5) represents the output of FDFM module, X C and X C ” represent the input and output features of the E-MLCA attention mechanism, X C ' represents the features after the two branches are fused in the E-MLCA attention mechanism, They represent global depth-wise convolutions with kernel size of 3×3 and dilation rates of 1, 5, and 9, respectively, and they represent maximum pooling operations with sizes of 5, 9, and 13, respectively. h (h=1,2,3) represent the pooling size Max t(t=5,9,13) are the outputs of branches 5, 9, and 13, and Y4 represents the output of the RMF module.

[0108] S3. Send the output of the feature enhancement module to the feature map decoding module. The feature map decoding module processes the four input feature maps according to the following formula:

[0109] D1=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y3,C tr (Y4)]))))));

[0110] D2=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y2,C tr (D1)]))))));

[0111] D3=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y1,C tr (D2)]))))));

[0112] D4=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([X2,C tr (D3)]))))));

[0113] D add =Re(D1)+Re(D2)+Re(D3)+Re(D4);

[0114] D add '=f sp-G (D add )+f Bilinear (Pat(f sp-L (D add )));

[0115] Y'=f PReLU (BN(C 3×3 (f PReLU (BN(C 3×3 (D add '))))));

[0116] Among them, D b (b∈[1,4]) represents the output of the b-th decoding convolutional layer, Ctr represents the deconvolution operation, D add and D add 'represent the output of the features after four-scale reorganization and fusion and the feature map separation and enhancement, respectively, sp-G and f sp-L Represent the global features and local features after separation, respectively, Bilinear represents the bilinear interpolation process, f PReLU represents the PReLU activation function, and Y' represents the feature map output by the enhanced output module.

[0117] S4, the output of the feature map decoding module is fused with the initial features of the network, and the final lung nodule segmentation image Y is obtained by convolution with a convolution kernel size of 3×3. The mathematical expression of the above process is:

[0118] Y=C 3×3 (Y'+f Bilinear (Y',X1))

[0119] Among them, Y represents the output of the network model.

[0120] During the model training process, in order to avoid introducing unnecessary training techniques and speed up the convergence speed, the binary cross entropy loss function and the Dice similarity coefficient loss function are used to form a composite loss function to quantify the difference between the predicted results and the actual results of the model. The calculation formula is:

[0121] f Loss =αf BCE +βf Dice ;

[0122]

[0123] Among them, f BCE and f Dice They represent the binary cross entropy loss function and the Dice similarity coefficient loss function, respectively. Loss Represents the total loss function of model training, N represents the number of samples, y j represents the true label of the jth sample, p j Represents the probability that the jth sample is a positive class, R q and T q The predicted result and the true label are the values ​​at the qth pixel respectively, and α and β represent hyperparameters respectively.

[0124] In order to verify the effectiveness of the multi-scale information fusion and complementary lightweight pulmonary nodule segmentation method of the embodiment of the present invention, the LIDC-IDIR dataset and the LUNA16 dataset are used to synthesize a new dataset for model training and verification. Since the original formats of the two datasets are 3D medical image formats, they cannot be directly used as input for the 2D medical image segmentation algorithm. At the same time, in order to improve the stability and convergence speed of the model so that different images have similar data distributions, the pixel values ​​of the image are first scaled to between 0 and 1 by normalization, and then the image data matrix and the corresponding mask matrix are created according to image and cmask; finally, by locating the center point, the image slices in the three-dimensional space direction are resampled and sliced ​​to a size of 64×64 to form a pulmonary nodule CT image and a corresponding label image. In order to improve the generalization performance of the model, the samples of the completed data set were expanded. Considering various situations that may occur during the CT image shooting process, data expansion methods such as slight angle transformation, translation, and introduction of random noise were used to expand the samples. The expanded data were randomly divided into training set, validation set, and test set in a ratio of 7:1:2. After the division, the training set, validation set, and test set contained 9253, 1322, and 2643 samples and corresponding label data, respectively.

[0125] Select different lung nodule segmentation images and use Ronneberger's algorithm (Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention (MICCAI), 2015, 234-241.); Oktay's algorithm (Oktay O, Schlemper J, Folgoc LL, et al. Attention U-Net: Learning where to look for the pancreas. arXiv preprint arXiv: 1804.03999, 2018.); Zhou's algorithm (Zhou Z, Siddiquee MM, Tajbakhsh N, et al. Unet++: A nested u-net architecture for medical image segmentation. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018, 3-11.); Jha's algorithm (Jha D, Riegler MA, Johansen D, et al.DoubleU-Net: A Deep Convolutional NeuralNetwork for Medical Image Segmentation.2020IEEE 33rd International Symposiumon Computer-Based Medical Systems(CBMS),2020,558-564.); Valanarasu's algorithm (Valanarasu Jand Patel V.UNeXt:MLP-based Rapid Medical Image SegmentationNetwork[J].arXiv Preprint, 2022, arXiv:2203.04967.); Ruan’s algorithm (Ruan J, Gao J, XieM, et al.Learning Multi-axis Representation in Frequency Domain for Medical Image Segmentation[J]. arXiv Preprint, 2023, arXiv:2312.17030.); Xu's algorithm (Xu Q, Ma Z, He N, et al. DCSAU-Net: A deeper and more compact split-attention U-Net for medical image segmentation. Computers in Biology and Medicine, 2023, 154.); Bougourzi's algorithm (Bougourzi F, Distante C, Dornaika F, et al. PDAtt-Unet: Pyramid Dual-Decoder Attention Unet for Covid-19 infection segmentation from CT-scans[J]. Medical Image Analysis, 2023, 86.); Zhang's algorithm (Zhang J, Chen D, Ma D, et al. CdcSegNet: Automatic COVID-19 Infection Segmentation From CT Images. IEEE Transactions on Instrumentation and Measurement, 2023, 72:1-13.); Yang's algorithm (Yang Z and Farsiu S. Directional Connectivity-based Segmentation of Medical Images. 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, 11525-11535.); Rezkellah's algorithm (Rezkellah N, Pierre-Yves B, Aurélien J, et al. Shape-aware synthesis of pathological lung CT scans using CycleGAN for enhanced semi-supervised lung segmentation.arXiv Preprint, 2024, arXiv: 2405.08556.); Fu's algorithm (Fu B, Peng Y, He J, et al. HmsU-Net: A hybrid multi-scale U-net based on a CNN and transformer for medical image segmentation. Computers in Biology and Medicine, 2024, 170.); and the experimental results of the present invention are compared and analyzed in both subjective and objective aspects. .

[0126] like Figure 6 As shown in the figure, it is an experimental effect diagram of the multi-scale information fusion complementary lightweight lung nodule segmentation method provided by the embodiment of the present invention and other algorithm segmentation. Figure 6 (A) is the original CT image of the nodule area. Figure 6 (B) is the label of the lesion area in the original image. Figure 6 (C) is the segmentation result diagram of Ronneberger's UNet method. Figure 6 (D) is the segmentation result diagram of Oktay’s AttentionUNet method. Figure 6 (E) is the segmentation result of Zhou’s UNet++ method. Figure 6 (F) is the segmentation result of Jha’s Doubleunet method. Figure 6 (G) is the segmentation result of Valanarasu’s UNext method. Figure 6 (H) is the segmentation result diagram of Ruan’s MEWUNet method. Figure 6 (I) is the segmentation result diagram of Xu’s DCSAU_Net method. Figure 6 (J) is the segmentation result of Bougourzi's PDAtUnet method. Figure 6 (K) is the segmentation result diagram of Zhang’s cdcSegNet method. Figure 6 (L) is the segmentation result diagram of Yang’s DconnNet method. Figure 6 (M) is the segmentation result image of Rezkellah’s CycleGANUNet method. Figure 6 (N) is the segmentation result diagram of Fu’s HmsUnet method. Figure 6(O) is the segmentation result diagram of the embodiment of the present invention. It can be seen from the subjective comparison that the basic model UNet cannot accurately locate the position and contour information of the lesion area, and the edge area after segmentation is greatly different from the real boundary. Although AttentionUNet and UNet++ introduce the attention mechanism and use the nested structure, they cannot accurately locate the boundary of the lesion area when processing images with no obvious difference in foreground information and background information contrast, resulting in the lack of some key areas in the segmented image, affecting the actual clinical application. Although the Doubleunet stacked with two layers of U-Net improves a certain segmentation effect, the features of the edge and texture areas still have certain defects. Although UNext and DCSAU_Net have small parameters, they cannot accurately locate the edge area during the segmentation process, and the segmented nodule area has the problem of being too large or too small. Although DconnNet has higher objective indicators, the subjective effect has a certain gap with the real image. HmsUnet based on Transformer does not pay attention to the interaction between different features, and the quality of the segmented image needs to be improved. The method of the embodiment of the present invention uses the FDFM module to deeply explore the relationship between features at each scale, strengthens the significance information by separating global features and local features, and highlights the detail information of the lesion area; the EOM module strengthens the edge texture information by reorganizing and splicing features, and has a better effect on lung nodule areas where contrast changes are not obvious.

[0127] In order to avoid the deviation caused by qualitative analysis, this example uses IoU, Dice coefficient (Dice), Accuracy (ACC), Sensitivity (SE), Precision (PR), and Specificity (SP) to comprehensively and objectively evaluate the segmentation effect of the model. At the same time, Params and GFLOPS are introduced to compare the performance of the model, as shown in Table 1.

[0128] Table 1 Objective index evaluation table

[0129]

[0130] According to the longitudinal analysis of the data in Table 1, the method of the present invention achieves the best results in IoU, Dice and ACC, and obtains suboptimal indicators in SE, PR and SP. Although the number of parameters and floating-point operations of the present invention are improved compared with the extremely lightweight algorithms UNext and DCSAU_Net, the number of parameters has decreased by 82.38% and 38.54% respectively, and the IoU values ​​have increased by 1.32% and 1.86% respectively compared with the current mainstream algorithms CycleGANUNet and HmsUnet. Compared with the basic algorithm UNet for medical image segmentation, the IoU, Dice and SE of the algorithm of the present invention have increased by 1.16%, 0.73% and 1.54% respectively; compared with the four algorithms of cdcSegNet, DconnNet, CycleGANUNet and HmsUnet, the segmentation accuracy of the present invention has increased by 0.16%, 0.03%, 0.03% and 0.05% respectively. Compared with the latest HmsUnet algorithm, the IoU, Dice, ACC, SE, and PR values ​​of the algorithm of the present invention have increased by 1.86%, 1.10%, 0.05%, 0.17%, and 2.40%, respectively. Compared with the comparison model, the significant improvement in the segmentation effect of the present invention is due to the fact that its feature mining encoding module fully mines the feature information of small target areas from CT images, and E-MLCA fuses information of similar scales and strengthens it in multiple dimensions, highlighting the significant information and improving its feature expression ability. Compared with AttentionUNet and UNet, the SE, PR, and SP of the present invention have room for improvement. The three indicators all reflect the false detection and missed detection from the side. SP represents the prediction ability of negative samples. Since the pictures are basically all nodule areas during the slicing process of the data set, there is a slight difference in this value, which does not affect the overall prediction effect of the model. Since the UNet model is only an end-to-end encoding and decoding structure, the information of the subtle target area is retained, so the probability of false detection is small; AttentionUNet introduces an attention mechanism at the end of UNet, directly merging the same-scale features of the encoding part with the features of the previous decoding layer, and effective information is retained. The E-MLCA attention mechanism used in the present invention will fuse information of different scales. The upsampling operation in the fusion process may cause the loss of some tiny features, thus causing a slight missed detection phenomenon. However, the gap between the above three indicators does not affect the expression of the overall segmentation. After comprehensive analysis of all the data, the segmentation effect of the present invention is relatively close to the real label area, which better solves the problem of large parameters and poor segmentation effect in the existing lung nodule segmentation algorithm. The moderate parameter amount facilitates the deployment of the algorithm on the hardware side, and the excellent segmentation effect has important clinical application value.

[0131] If the CT image segmentation method described in the embodiment of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the image segmentation method described in the embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., various media that can store program codes.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A lightweight pulmonary nodule segmentation model with multi-scale information fusion and complementation, characterized in that: It includes feature mining encoding module, feature enhancement module and feature mapping decoding module. The feature mining encoding module combines ordinary convolution, hole convolution, and deep convolution to construct a six-layer multi-convolution residual fusion feature mining encoding module, deeply mines the fuzzy edge and texture information of the lesion area and fuses shallow features to solve the gradient disappearance and gradient explosion problems in the network training process, and obtains a feature map with rich semantic information in the lesion area; The feature mining encoding module is composed of six layers of submodules, namely, a layer of initial feature residual extraction block IFREB, two layers of global depth-by-depth convolution blocks GDCB, a layer of feature map enhancement block FMEB and two layers of ordinary convolution blocks; The first layer is used to obtain shallow features of the original lung CT image through a 3×3 convolution; The second layer is used to perform deep mining on the shallow features extracted from the first layer through a residual connection block IFREB consisting of a 3×3 convolution and a 3×3 dilated convolution. The third and fourth layers are used to enhance the high-level semantic features of the low-frequency signals of the lesions through the residual connection blocks GDCB ​​composed of a 1×1 convolution and a 3×3 depth convolution on the outputs of the second and third layers respectively. The fifth layer is used to deeply enhance the lesion features of the output of the fourth layer through a residual connection block consisting of a 1×1 convolution and a 3×3 deep convolution and an EMA attention mechanism, which is composed of FMEB; The sixth layer is used to smooth the features of the output of the fifth layer through a 3×3 convolution to obtain the output of the feature mining encoding module; The feature enhancement module includes a feature deep fusion submodule FDFM and a multi-scale fusion module RMF; the FDFM module implements enhanced fusion of feature maps X2 to X5 by designing an E-MLCA attention mechanism; the RMF module enhances multi-scale features to solve the problem of missing detail information between channels, and sends the enhanced feature map to the feature mapping decoding module. The RMF module is responsible for deep mining of the feature map X6; The multi-dimensional attention mechanism E-MLCA is used to enhance the high-level semantic features of low-frequency signals in the lung nodule area. In the feature enhancement module, the attention mechanism E-MLCA is used to enhance the global and local, channel and space dimensions in parallel. The local features extract local spatial information through the reorganization depth of the features, and the global features are used to mine significant information in the space and channel directions through weight allocation and fusion with the features. The feature map decoding module includes four layers of decoding convolutional layers and an enhanced output module EOM. The decoding convolutional layers are responsible for decoding the output of the feature enhancement module. The EOM module is responsible for enhancing the semantic features of the pulmonary nodule lesion area, enriching the semantic information of the pulmonary nodule area in the feature map, and smoothing the output feature map to obtain the output of the feature map decoding module. The output of the feature map decoding module is fused with the initial features of the network, and the final pulmonary nodule segmentation image is obtained by convolution with a convolution kernel size of 3×3. The enhanced output module EOM reorganizes and fuses the four-scale features output by the decoding convolution layer, enriches the semantic information of the lung nodule area in the feature map, separates the features of the fused feature map along the channel direction, fuses the global feature map with the local features after splicing, and uses two layers of convolution with a convolution kernel size of 3×3 for feature smoothing.

2. According to claim 1, a lightweight pulmonary nodule segmentation model with multi-scale information fusion and complementation, characterized in that: The multi-dimensional attention mechanism E-MLCA uses two branches in parallel to enhance features after global average pooling: one branch performs adaptive feature search within the channel range after global pooling, and multiplies it with the original feature map to obtain the final output of the first branch; the other branch reorganizes and fuses the enhanced global features, and obtains the output of the second branch after de-pooling and fusion with the original input; the output of the E-MLCA attention mechanism is obtained by allocating weights to the features of the two branches and adding and fusing them.

3. According to claim 1, a lightweight pulmonary nodule segmentation model with multi-scale information fusion and complementation, characterized in that: The RMF module uses maximum pooling with pooling sizes of 0, 5, 9, and 13 for the feature maps output by the sixth layer of the feature mining encoding module to retain the input features to the maximum extent without introducing parameters, and uses global depth-wise convolution blocks with expansion rates of 1, 3, and 5 for feature enhancement of the pooled and original feature maps. After the feature cascade of each branch is completed, the feature maps of four scales are superimposed to obtain a feature map with enhanced multi-scale information.

4. A lightweight pulmonary nodule segmentation method with multi-scale information fusion and complementation, characterized in that: Follow these steps: S1. Input the original lung nodule CT image into the feature mining encoding module of the multi-scale information fusion complementary lightweight lung nodule segmentation model. The feature mining encoding module extracts features according to the following formula: X n =[f ReLU (BN(C 1×1 (X n-1 ))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X n-1 ))))))); X4' (1) =Group H (Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))])); X4' (2) =Group W (Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))])); X4' (3) =Group([f ReLU (BN(C 1×1 (X4))),f ReLU (BN(C DP3×3 (f ReLU (BN(C 1×1 (X4))))))]); X4”=Group(X4')×f f H (C 1×1 ([Avg(X4' (1) ),Avg(X4' (2) )]))×f f W (C 1×1 ([Avg(X4' (1) ),Avg(X4' (2) )])); X M1 =f Softmax (Avg(GN(X4”)))·C 3×3 (X4' (3) ); X M2 =GN(X4”)·f Softmax (Avg(C 3×3 (X4' (3) ))); X6=C 3×3 (Group(X4')×f Sigmoid (X M1 +X M2 )); Among them, C 3×3 represents a normal convolution with a convolution kernel size of 3×3, C 1×1 represents a normal convolution with a convolution kernel size of 1×1, represents a dilated convolution with a kernel size of 3×3 and a dilation rate of 2, BN represents batch normalization, and f ReLU represents the ReLU activation function. n-1 and X n (n=3,4) represent the input and output of the third and fourth layers of the feature mining encoding module, respectively. DP3×3 represents a depthwise convolution with a kernel size of 3×3, [] represents cascade fusion, X4' (k) (k=1,2,3) represents the feature map of the kth branch after grouping. Group represents the grouping operation along the channel direction. H and Group W Respectively represent the slice grouping along the height and width directions, Avg represents average pooling, and f f H and f f W They represent the vector decomposition along the height and width directions respectively, X4" represents the output feature map of the fusion of the weights in the X and Y directions, X M1 and X M2 Represents the output of matrix dot multiplication in two dimensions, f Softmax represents the Softmax activation function, GN represents GroupNorm normalization, · represents the dot product of the matrix, and X5 represents the output of the feature enhancement block; S2, the outputs X1 to X6 of each convolution block of the feature mining encoding module are sent to the feature enhancement module, and the FDFM submodule and RMF submodule in the feature enhancement module respectively enhance the input feature map according to the following formulas: X C =[Adap(C 3×3 (X k-1 )),Adap(f up (C 3×3 (X k )))]; X C '=Re(C 1×1 (Re(LAP(X C ))))+GAP(LAP(X C ))·f Sigmoid (C 1×1 (GAP(LAP(X C )))); X C ”=f UNAP (X C ')×X C +X C ; Y k-2a =λ×X C ”×(C 3×3 (X k-1 ))+(1-λ)×X C ”(f up (C 3×3 (X k ))); Among them, X k-1 and X k (k=3,4,5) represents the input features of the feature enhancement module, Adap represents the adaptive pooling operation, LAP and GAP represent the local pooling and global pooling operations respectively, Re represents the reorganization of the features, and f Sigmoid and f UNAP Represent the Sigmoid activation function and the anti-pooling process, respectively, up represents the upsampling process, T i (i∈[0,g]) represents the i-th subchannel in channel T, λ represents the weight coefficient of each branch at the output of the feature enhancement module, Y k-2a (a=0,1,2 when k=3,4,5) represents the output of FDFM module, X C and X C ” represent the input and output features of the E-MLCA attention mechanism, X C ' represents the features after the fusion of the two branches in the E-MLCA attention mechanism, They represent global depth-wise convolution with kernel size of 3×3 and dilation rates of 1, 5, and 9, respectively. t (t=5,9,13) represents the maximum pooling operation with size 5, 9, and 13 respectively, Y4h(h=1,2,3) represents the output of the branch with pooling size 5, 9, and 13 respectively, and Y4 represents the output of the RMF module. S3, the output of the feature enhancement module is sent to the feature mapping decoding module, and the feature mapping decoding module decodes and smoothes the features of the four input feature maps according to the following formula: D1=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y3,C tr (Y4)])))))); D2=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([Y2,C tr (D1)])))))); <h2 style=";text-align:left;direction:ltr">D3=f<h2 style=";text-align:left;direction:ltr"> ReLU <h2 style=";text-align:left;direction:ltr"> (BN(C<h2 style=";text-align:left;direction:ltr"> 3×3 <h2 style=";text-align:left;direction:ltr"> (f<h2 style=";text-align:left;direction:ltr"> ReLU <h2 style=";text-align:left;direction:ltr"> (BN(C<h2 style=";text-align:left;direction:ltr"> 3×3 <h2 style=";text-align:left;direction:ltr"> ([Y1,C<h2 style=";text-align:left;direction:ltr"> tr <h2 style=";text-align:left;direction:ltr"> (D2)])))))); D4=f ReLU (BN(C 3×3 (f ReLU (BN(C 3×3 ([X2,C tr (D3)])))))); D add =Re(D1)+Re(D2)+Re(D3)+Re(D4); D add '=f sp-G (D add )+f Bilinear (Pat(f sp-L (D add ))); Y'=f PReLU (BN(C 3×3 (f PReLU (BN(C 3×3 (D add ')))))); Among them, D b (b∈[1,4]) represents the output of the b-th decoding convolutional layer, C tr represents the deconvolution operation, D add and D add 'represent the output of the features after four-scale reorganization and fusion and the feature map separation and enhancement, respectively, sp-G and f sp-L Represent the global features and local features after separation, respectively, Bilinear represents the bilinear interpolation process, f PReLU represents the PReLU activation function, and Y' represents the feature map output by the enhanced output module. S4. The output of the feature map decoding module is fused with the initial features of the network, and the final lung nodule segmentation image is obtained by convolution with a convolution kernel size of 3×3: Y=C 3×3 (Y'+f Bilinear (Y',X1)); Among them, Y represents the output of the network model.

5. An electronic device, characterized in that: The method as claimed in claim 4 is used to implement lung nodule image segmentation.

6. A computer storage medium, characterized in that: The storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by the processor to implement the lung nodule image segmentation method as claimed in claim 4.

Images