Pulmonary nodule detection method, system and equipment based on Mama and global multi-scale feature fusion

Abstract

The invention discloses a pulmonary nodule detection method, system and device based on Mama and global multi-scale feature fusion, and the method comprises the steps: carrying out the improvement of a Center Net algorithm, constructing a pulmonary nodule detection model, replacing an original feature extractor in the Center Net with a feature extractor based on Mama, and carrying out the construction of a pulmonary nodule detection model; adding a multi-scale feature aggregation module for fusing features from different scales; detecting the to-be-detected lung CT image by using the pulmonary nodule detection model, wherein the method comprises the following steps: performing feature coding on the to-be-detected lung CT image; performing feature decoding on the encoded multi-scale feature map; inputting the multi-scale feature maps of different decoding stages into a multi-scale feature aggregation module, and aggregating features with different functions; bounding box regression prediction is carried out, the pulmonary nodules are positioned through central point prediction, offset prediction and size prediction, and a final detection result is obtained through combination; compared with a traditional method, the method can improve the automatic and accurate detection capability of the pulmonary nodules in the CT image.

Description

Technical Field

[0001] The present invention relates to a lung nodule detection method, system and device, and in particular to a lung nodule detection method, system and device based on Mamba and global multi-scale feature fusion, belonging to the technical field of medical image processing. Background Art

[0002] Lung cancer is a disease that seriously threatens human health. Its early detection and diagnosis are crucial to reducing mortality. Currently, the medical field relies on high-precision automated algorithms to detect lung nodules in CT images. However, due to the randomness of lung nodules in size and location, traditional detection methods still face challenges in achieving comprehensive and accurate identification.

[0003] Existing technologies usually rely on anchor point mechanisms for detection, which is inefficient when dealing with irregular nodules. At the same time, the existing global feature extraction capabilities are limited, which leads to missed detection or false detection in complex image backgrounds. In addition, multi-scale feature fusion modules are usually manually designed, with limited generalization capabilities, and are difficult to adapt to changes in different types of image data.

[0004] In the prior art, the original CenterNet algorithm has the problem of low accuracy in lung nodule detection, which easily leads to false detection and missed detection, and it is difficult to achieve the expected effect. Since lung nodules are small in size, have different shapes, have blurred edges, and are similar to background information such as blood vessels and trachea, it is difficult for existing algorithms to accurately extract nodule feature areas, resulting in inaccurate positioning. During the multiple sampling process of the network, feature image pixels are lost, resulting in feature information loss, affecting the accuracy of the detection results. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method, system and device for lung nodule detection based on Mamba and global multi-scale feature fusion, which can improve detection speed and accuracy.

[0006] Technical solution: The present invention discloses a method for detecting lung nodules based on Mamba and global multi-scale feature fusion, comprising:

[0007] The CenterNet algorithm is improved to build a lung nodule detection model, wherein the improvements include replacing the original feature extractor in CenterNet with a feature extractor based on Mamba, adding a multi-scale feature aggregation module for fusing features from different scales, and combining bounding box regression and loss function to train the model and output detection results;

[0008] A lung nodule detection model is used to detect a lung CT image to be detected, including: performing feature encoding on the lung CT image to be detected to obtain an encoded multi-scale feature map; performing feature decoding on the encoded multi-scale feature map to obtain a decoded multi-scale feature map; inputting the multi-scale feature maps at different decoding stages into a multi-scale feature aggregation module, aggregating features with different functions, and obtaining a feature map after multi-scale feature aggregation; performing bounding box regression prediction, and locating lung nodules through center point prediction, offset prediction, and size prediction; and combining the center point prediction result, the offset prediction result, and the size prediction result to obtain the final detection result.

[0009] Furthermore, the structure of the Mamba-based feature extractor is as follows:

[0010] Linearize the 3D features into 2D features through the linearization layer:

[0011] x'=LN(Flatten(x0))

[0012] Among them, x0 is the input feature, x'∈R C，H,W represents the linearized feature, R represents the set of feature images, C is the depth of the feature image, H is the height of the feature image, W is the width of the feature image, LN() is layer normalization, and Flatten() represents the operation of linearizing three-dimensional features into two-dimensional;

[0013] Then, the linearized features are globally modeled through the SSM layer and expressed in linear space:

[0014] h′ t =A·h t +B·x′ t

[0015] y t =C·h′ t +x′ t

[0016] where x′ t represents the tth point in the linearized feature, h′ t Represents the characteristics of the hidden state, y t represents the t-th point of the output feature, A, B and C represent learnable parameters, which are initialized randomly; t The combination is performed to obtain y0, and after being processed by the SSM model, the feature space of the linear feature is changed to:

[0017] y'=Unflatten(y0)

[0018] Where Unflatten() represents the inverse operation of Flatten(), y'∈RC，H，W ;

[0019] A small network consisting of two convolutional layers is used to add further transformation to the global features:

[0020] y″=Conv(Conv(y'))+y'

[0021] Where y″ represents the transformed global feature, and Conv() represents the convolution operation;

[0022] A small CNN network is used to extract fine-grained features from global features:

[0023] y″=σ(BN(Conv(y″))+x)

[0024] Where y″′ represents the final output feature, BN() represents the normalization operation, and σ() represents the ReLU activation function.

[0025] Furthermore, the feature encoding of the lung CT image to be detected is performed to obtain an encoded multi-scale feature map:

[0026] A convolutional layer is used to preliminarily extract the features of the lung CT image to be tested;

[0027] Four feature extraction blocks are used to perform global feature representation on the initially extracted features to generate encoded multi-scale feature maps; a maximum pooling layer is used before each feature extraction block to halve the scale of the features and double the number of channels.

[0028] Furthermore, the structure of the feature extraction block includes:

[0029] The convolution layer with a convolution kernel size of 1 is used to expand the channel of the input features;

[0030] BN layer, used to eliminate the influence of outlier samples;

[0031] Mamba block, a feature extractor based on Mamba, is used for linear spatial representation;

[0032] A small CNN network is used to process global features and retain the ability to perceive fine-grained features.

[0033] Furthermore, the step of performing feature decoding on the encoded multi-scale feature map to obtain a decoded multi-scale feature map includes:

[0034] Upsample the encoded multi-scale feature map to increase the width and height of the feature and restore the size of the image;

[0035] The upsampled feature map is input into the Mamba-based feature extractor for feature extraction and generates a multi-scale feature map. Three layers of Mamba feature extractors are stacked continuously to restore the feature size of the multi-scale feature map to half the size of the input image, and the decoded multi-scale feature map is obtained.

[0036] Furthermore, the multi-scale feature maps of different decoding stages are input into a multi-scale feature aggregation module, and features with different functions are aggregated to obtain a feature map after multi-scale feature aggregation, including:

[0037] The feature maps of the three stages are obtained from the decoded multi-scale feature map, which are defined as: and

[0038] Features of different scales are upsampled, and then the number of channels is transformed through a convolution layer with a convolution kernel of 1 to ensure that features of different scales have the same shape:

[0039] x′1=Conv(Up(x1))

[0040] x′2=Conv(Up(x2))

[0041] Among them, Up() means upsampling the features using bilinear interpolation, and Conv() means a convolution layer with a convolution kernel size of 1; the feature size after processing is: and

[0042] Add an extra empty dimension to the three features that are preprocessed to the same size, and concatenate and fuse along the empty dimension to obtain Then, a 3D convolution layer with a convolution kernel of 3×3×3 is used to process the concatenated and fused features, and then a 3D Bn layer is used to eliminate the negative impact of outlier data. Finally, the ReLU activation function is used to obtain the final result:

[0043] x′ f =σ(Bn 3D (Conv 3D (x f )))

[0044] Among them, Bn 3D () represents a three-dimensional batch normalization layer, Conv 3D () represents a 3D convolutional layer;

[0045] The output of the feature fusion unit is processed using a maximum pooling layer with a window of 3×1×1:

[0046] x′ f ′=Maxpool3D (x′ f )

[0047] Among them, Maxpool 3D () represents the maximum pooling layer in three dimensions, is the feature after maximum pooling.

[0048] Furthermore, the bounding box regression prediction is performed to locate the lung nodules through center point prediction, offset prediction and size prediction, including:

[0049] Use the center point prediction loss function to get the center point of the nodule:

[0050]

[0051] Among them, N represents the number of key points in the image, α, β are hyperparameters, and Y xyc represents the center point heat map predicted by the network, c represents the number of categories finally detected, and x and y represent the center point position of the nodule;

[0052] To perform offset prediction, the offset prediction loss function is defined as follows:

[0053]

[0054] Among them, p represents the predicted node, The output of the offset prediction branch. Indicates the error value that occurs when downsampling the label data by 2 times;

[0055] The size of the located nodules is predicted to predict the size of the lung nodules, and the size prediction loss function is defined as:

[0056]

[0057] where s pk =[w pk ,h pk ] represents the size of the true label, [w pk ,h pk ] represent the width and height of the current nodule, s pk ′ represents the nodule size predicted by the nodule detection network.

[0058] Furthermore, the center point prediction result, the offset prediction result and the size prediction result are combined to obtain the final detection result, including:

[0059] L total =a·L k +b·L o +d·L wh

[0060] Among them, L k is the prediction result of the center point, L o is the offset prediction result, L wh is the size prediction result, a, b and d are the hyperparameters of network training.

[0061] Based on the same inventive concept, the present invention also provides a pulmonary nodule detection system based on Mamba and global multi-scale feature fusion, comprising:

[0062] A construction module is used to improve the CenterNet algorithm and build a lung nodule detection model, wherein the improvement includes replacing the original feature extractor in CenterNet with a feature extractor based on Mamba, adding a multi-scale feature aggregation module for fusing features from different scales, and combining bounding box regression and loss function to train the model and output detection results;

[0063] The detection module is used to detect the lung CT image to be detected using the lung nodule detection model, including: feature encoding the lung CT image to be detected to obtain an encoded multi-scale feature map; feature decoding the encoded multi-scale feature map to obtain a decoded multi-scale feature map; inputting the multi-scale feature maps at different decoding stages into the multi-scale feature aggregation module, aggregating features with different functions, and obtaining a feature map after multi-scale feature aggregation; performing bounding box regression prediction, and locating lung nodules through center point prediction, offset prediction, and size prediction; combining the center point prediction result, the offset prediction result, and the size prediction result to obtain the final detection result.

[0064] Based on the same inventive concept, the present invention also provides a computing device, comprising: one or more processors, one or more memories, and one or more programs, wherein the programs are stored in the memories and configured to be executed by the processors, and when the programs are loaded into the processors, the steps of the lung nodule detection method based on Mamba and global multi-scale feature fusion are implemented according to any one of the above items.

[0065] Beneficial effects: Compared with the prior art, the present invention achieves high-precision positioning and identification of lung nodules by introducing an anchor-free frame detection framework, a Mamba-based feature extractor and a global multi-scale feature fusion module, reduces false detection and missed detection rates, and improves the accuracy of lung nodule detection; the present invention utilizes a feature encoder to extract multi-scale features of CT images, and combines a feature decoder and a multi-scale fusion module to fully mix the features, thereby more accurately extracting nodule feature areas; the Mamba-based feature extractor of the present invention can achieve comprehensive perception of global information and local detail information with linear complexity, and reduce feature information loss; the global multi-scale feature fusion module of the present invention aggregates and maps multi-scale features to a high-dimensional feature space, and uses a 3D neural network to achieve its adaptive fusion, thereby enhancing the robustness of the model and having stronger generalization ability; the algorithm proposed in the present invention can achieve higher detection efficiency while ensuring high accuracy, meet practical application needs, and is suitable for the field of medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0067] Figure 2 is an overall network structure diagram of an embodiment of the present invention;

[0068] Figure 3 This is a structural diagram of a feature extractor based on Mamba according to an embodiment of the present invention;

[0069] Figure 4 Schematic diagram of a multi-scale feature aggregation module according to an embodiment of the present invention;

[0070] Figure 5 4 is a comparison chart of the detection effects of various models in the embodiments of the present invention. DETAILED DESCRIPTION

[0071] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0072] As attached Figure 1 As shown, the lung nodule detection method based on Mamba and global multi-scale feature fusion in this embodiment includes:

[0073] Step 1: Improve the CenterNet algorithm and build a pulmonary nodule detection model. The improvement includes replacing the original feature extractor in CenterNet with a Mamba-based feature extractor, adding a multi-scale feature aggregation module for fusing features from different scales, and combining bounding box regression and loss function to train the model and output detection results.

[0074] Step 2: Use the lung nodule detection model to detect the lung CT image to be detected, including:

[0075] Step 2.1: feature encoding is performed on the lung CT image to be detected to obtain an encoded multi-scale feature map;

[0076] Step 2.2: Decode the encoded multi-scale feature map to obtain a decoded multi-scale feature map;

[0077] Step 2.3: Input the multi-scale feature maps of different decoding stages into the multi-scale feature aggregation module, aggregate the features with different functions, and obtain the feature maps after multi-scale feature aggregation;

[0078] Step 2.4: Perform bounding box regression prediction to locate lung nodules through center point prediction, offset prediction, and size prediction;

[0079] Step 2.5: Combine the center point prediction results, offset prediction results and size prediction results to obtain the final detection result.

[0080] The lung nodule detection model described in this embodiment is as follows Figure 2 As shown, specifically:

[0081] Step 2.1: Feature encoding of the input lung CT image.

[0082] Preliminary feature extraction: A convolutional layer is used to perform preliminary feature extraction on the input lung CT image to extract local texture information in the image.

[0083] Mamba feature extractor: Four feature extraction blocks based on Mamba feature extractor are used to perform global feature representation on the initially extracted features and generate multi-scale feature maps. A maximum pooling layer is used before each feature extraction block to halve the scale of the feature, thereby obtaining feature maps of different scales. After processing by the maximum pooling layer and the Mamba feature extractor, the length and width of the feature are reduced to 0.5 of the previous layer, and the number of feature channels is increased to twice that of the previous layer. Among them, the number of feature channels after convolution processing is 16.

[0084] Internal structure of feature extraction block:

[0085] ① The convolution layer (convolution kernel size is 1) performs channel expansion on the input features.

[0086] ②BN layer eliminates the influence of outlier samples.

[0087] ③ Mamba block, a feature extractor based on Mamba, performs linear space expression, including linearization, SSM model processing and further transformation of CNN network.

[0088] ④The small CNN network processes the global features and retains the ability to perceive fine-grained features.

[0089] Based on the Mamba feature extractor structure Figure 3 As shown: Linearization layer: linearizes 3D features into 2D features. SSM layer: globally models the linearized features and expresses them in linear space. Small CNN network: extracts fine-grained features from global features. The specific structure is as follows:

[0090] Linearize the 3D features into 2D features through the linearization layer:

[0091] x'=LN(Flatten(x0))

[0092] Among them, x0 is the input feature, x'∈R C，H,W represents the linearized feature, R represents the set of feature images, C is the depth of the feature image, H is the height of the feature image, W is the width of the feature image, LN() is layer normalization, and Flatten() represents the operation of linearizing three-dimensional features into two-dimensional;

[0093] Then, the linearized features are globally modeled through the SSM layer and expressed in linear space:

[0094] h′ t =A·h t +B·x′ t

[0095] y t =C·h′ t +x′ t

[0096] where x′ t represents the tth point in the linearized feature, h′ t Represents the characteristics of the hidden state, y t represents the t-th point of the output feature, A, B and C represent learnable parameters, which are initialized randomly; t By combining them, we can get y0. Through the above formula, we can globally model an input feature sequence, and the computational complexity remains at o(n). However, the feature space involved in the pulmonary nodule detection network in this paper is R C,H,W Therefore, after being processed by the SSM model, the feature space of the linear feature is changed to:

[0097] y'=Unflatten(y0)

[0098] Where Unflatten() represents the inverse operation of Flatten(), y'∈R C，H,W ;

[0099] After global perception, this paper uses a small network consisting of two layers of convolution to add further transformation to the global features:

[0100] y″=Conv(Conv(y'))+y'

[0101] Where y″ represents the transformed global feature, and Conv() represents the convolution operation;

[0102] After extracting the global features of the input lung CT image, in order to ensure that the network can have fine-grained feature perception capabilities, this paper uses a small CNN network to further process the global features:

[0103] y″′=σ(BN(Conv(y″))+x)

[0104] Where y″′ represents the final output feature, BN() represents the normalization operation, and σ() represents the ReLU activation function.

[0105] Steps:

[0106] Through preliminary feature extraction, local texture information in the image is obtained.

[0107] The Mamba feature extractor is used to extract global feature information from the image and generate multi-scale feature maps to obtain semantic information and texture information at different scales.

[0108] Through the maximum pooling layer, the size of the feature map is reduced to obtain feature maps of different scales.

[0109] Step 2.2: Perform feature decoding on the encoded multi-scale feature map to obtain a decoded multi-scale feature map.

[0110] Upsampling: Upsample the multi-scale feature map output by the feature encoder to increase the width and height of the feature to restore the image size and ensure the accuracy of bounding box regression. The upsampling operation can use bilinear interpolation or other suitable interpolation methods.

[0111] Mamba feature extractor: The upsampled feature map is input into the Mamba-based feature extractor for further feature extraction. Three layers of Mamba feature extractors are stacked continuously to restore the feature size to half the size of the input image. Features at different decoding stages have different scales. Small-scale features contain more semantic information of lung nodules, while large-scale features reflect more texture detail information.

[0112] Steps:

[0113] Through the upsampling operation, the width and height of the feature are increased and the size of the image is restored.

[0114] The upsampled feature map is further extracted through the Mamba feature extractor to generate a multi-scale feature map.

[0115] Step 2.3: Input the multi-scale feature maps of different decoding stages into the multi-scale feature aggregation module, aggregate the features with different functions, and obtain the feature maps after multi-scale feature aggregation.

[0116] Multi-scale feature aggregation module, such as Figure 4 As shown: the feature maps of different decoding stages are input into the multi-scale feature aggregation module to aggregate features with different functions.

[0117] The multi-scale feature aggregation module can adopt the following structure: feature maps of different scales are spliced ​​and processed using convolutional layers. The attention mechanism is used to perform weighted fusion of features of different scales. The fused feature map contains richer semantic information and texture information, providing more accurate feature representation for subsequent lung nodule detection.

[0118] Steps:

[0119] Through the multi-scale feature aggregation module, feature maps of different scales are aggregated to obtain a fused feature map, which contains richer semantic information and texture information. The details are as follows:

[0120] The feature maps of the three stages are obtained from the decoded multi-scale feature map, which are defined as: and

[0121] Features of different scales are upsampled, and then the number of channels is transformed through a convolution layer with a convolution kernel of 1 to ensure that features of different scales have the same shape:

[0122] x′1=Conv(Up(x1))

[0123] x′2=Conv(Up(x2))

[0124] Among them, Up() means upsampling the features using bilinear interpolation, and Conv() means a convolution layer with a convolution kernel size of 1; the feature size after processing is: and

[0125] Add an extra empty dimension to the three features that are preprocessed to the same size, and concatenate and fuse along the empty dimension to obtain Then, a 3D convolution layer with a convolution kernel of 3×3×3 is used to process the concatenated and fused features, and then a 3D Bn layer is used to eliminate the negative impact of outlier data. Finally, the ReLU activation function is used to obtain the final result:

[0126] x′ f =σ(Bn 3D (Conv 3D (x f )))

[0127] Among them, Bn 3D () represents a three-dimensional batch normalization layer, Conv 3D () represents a 3D convolutional layer;

[0128] After that, in order to remove the extra dimensions and ensure that the fused features can be input into the detection head for bounding box regression, the output of the feature fusion unit is processed using a maximum pooling layer with a window of 3×1×1:

[0129] x′ f ′=Maxpool 3D (x′ f )

[0130] Among them, Maxpool 3D () represents the maximum pooling layer in three dimensions, The feature after maximum pooling, next, the extra third dimension is removed, and the two-dimensional fusion feature is input into the detection head to obtain the final detection result.

[0131] Step 2.4: Perform bounding box regression prediction to locate lung nodules through center point prediction, offset prediction, and size prediction.

[0132] Detection head: The feature map after multi-scale feature aggregation is input into three identical detection heads. Each detection head performs the following functions: Center point prediction: predicts the location of the center point of the lung nodule. Offset prediction: ensures the accuracy of the center point location predicted by the nodule detection network on the low-resolution feature map. Size prediction: predicts the size of the lung nodule.

[0133] Loss function: The following loss functions are used for training: Center point prediction loss function: used to calculate the difference between the predicted center point location and the true center point location. Size prediction loss function: used to calculate the difference between the predicted nodule size and the true nodule size.

[0134] Use the center point prediction loss function to get the center point of the nodule:

[0135]

[0136] Among them, N represents the number of key points in the image, α, β are hyperparameters, and Y xyc represents the center point heat map predicted by the network, c represents the number of categories finally detected, and x and y represent the center point position of the nodule;

[0137] In this way, the detection network can achieve preliminary positioning of the nodule. Considering that the positioning accuracy will be lost when locating the center point of the downsampled feature map, it is necessary to further predict the offset term of the center point obtained by the above formula, and define the offset prediction loss function as follows:

[0138]

[0139] Among them, p represents the predicted node, The output of the offset prediction branch. Represents the error value that occurs when downsampling the label data by 2 times; in this way, the accuracy of the center point position predicted by the nodule detection network on the low-resolution feature map is guaranteed.

[0140] After completing the prediction of the center point, it is necessary to predict the size of the located nodules to predict the size of the lung nodules and define the size prediction loss function:

[0141]

[0142] where s pk =[w pk ,h pk ] represents the size of the true label, [w pk ,h pk ] represent the width and height of the current nodule, s pk ′ represents the nodule size predicted by the nodule detection network;

[0143] Step 2.5: Combine the center point prediction results, offset prediction results and size prediction results to obtain the final detection result.

[0144] The center point prediction results, offset prediction results and size prediction results are combined to obtain the final detection result.

[0145] L total =a·L k +b·L o +d·L wh

[0146] Among them, L k is the prediction result of the center point, L o is the offset prediction result, L whis the size prediction result. a, b and d are the hyperparameters of network training, which are set to 1, 0.1 and 1 respectively.

[0147] This example selects common indicators in target detection as evaluation indicators of the model quality, including F1, mAP, precision (P), and recall (R).

[0148] This paper selects common indicators in target detection as evaluation indicators of the quality of the model, including F1, mAP, precision (P), and recall (R). Among them:

[0149]

[0150] Among them, TP represents the number of positive samples correctly detected by the model, and FP represents the number of positive samples incorrectly detected by the model.

[0151]

[0152] Where FN represents the number of positive targets that the model failed to detect.

[0153]

[0154] For the calculation of mAP, this paper adopts the All-point interpolation method, the specific formula is:

[0155]

[0156] Where N is the number of points on the PR curve, P(R n+1 ) is the maximum accuracy value at point n+1. On this basis, mAP is obtained, which is defined as follows:

[0157]

[0158] Among them, AP i represents the average precision of the i-th category.

[0159] The code of this embodiment is implemented by Pytorch 2.0. This paper selects the LUNA16 dataset and uses the preprocessing method to process the 3D raw data to obtain 3538 2D images, of which 3085 images are used for training, 100 images are used for verification during training, and 353 are used for testing. During the training process, the initial learning rate is set to 0.001, the training rounds are set to 200, of which the feature encoder and feature decoder are frozen in the first 50 rounds, the batch size is set to 64, and the Adam optimizer is used to update the network parameters. All experiments in this paper were completed on Ubuntu20.04 using an Nvidia 3090.

[0160] Experiments were conducted on the LUNA16 dataset, and the results showed that:

[0161] The mAP of the method of the present invention reaches 97.22%, and the F1 value reaches 0.99, which is better than the comparison method.

[0162] The method of the present invention can process 40 CT images per second, meeting the real-time requirements. Figure 5 As shown:

[0163] Table 1 Comparison of indicators of various models on Luna16

[0164]

[0165] Table 2. Comparison of ablation experiment indicators on Luna16

[0166]

[0167] Table 3 Comparison of operating efficiency analysis of different methods

[0168]

[0169] Description of the English abbreviations and codes used in this embodiment:

[0170] Mamba: A global feature extraction method based on linear complexity.

[0171] SSM: Self-Supervised Module, used for linear space expression.

[0172] BN: Batch Normalization, batch normalization.

[0173] ReLU: Rectified Linear Unit, activation function.

[0174] CNN: Convolutional Neural Network, convolutional neural network.

[0175] CenterNet: Object detection algorithm based on key point detection.

[0176] FPN: Feature Pyramid Network, feature pyramid network.

[0177] SSD: Single Shot MultiBox Detector, single shot multi-box detector.

[0178] YOLO: You Only Look Once, target detection algorithm.

[0179] EfficientDet: Object detection algorithm based on the EfficientNet backbone network.

[0180] FCos: Fully Convolutional One-Stage Object Detection, full convolutional one-stage target detection algorithm.

[0181] Retinanet: Object detection algorithm based on Focal Loss.

[0182] mAP: Mean Average Precision, average precision mean.

[0183] F1: F1 value, a comprehensive indicator, measures precision and recall.

[0184] P:Precision, accuracy.

[0185] R:Recall, recall rate.

[0186] FPS: Frames Per Second, frames per second.

[0187] GFLOPs: Gigaflops, one billion floating-point operations per second.

[0188] Parms:Parameters, parameter quantity.

[0189] Based on the same inventive concept, this embodiment also provides a pulmonary nodule detection system based on Mamba and global multi-scale feature fusion, including:

[0190] A construction module is used to improve the CenterNet algorithm and build a lung nodule detection model, wherein the improvement includes replacing the original feature extractor in CenterNet with a feature extractor based on Mamba, adding a multi-scale feature aggregation module for fusing features from different scales, and combining bounding box regression and loss function to train the model and output detection results;

[0191] The detection module is used to detect the lung CT image to be detected using the lung nodule detection model, including: feature encoding the lung CT image to be detected to obtain an encoded multi-scale feature map; feature decoding the encoded multi-scale feature map to obtain a decoded multi-scale feature map; inputting the multi-scale feature maps at different decoding stages into the multi-scale feature aggregation module, aggregating features with different functions, and obtaining a feature map after multi-scale feature aggregation; performing bounding box regression prediction, and locating lung nodules through center point prediction, offset prediction, and size prediction; combining the center point prediction result, the offset prediction result, and the size prediction result to obtain the final detection result.

[0192] Based on the same inventive concept, this embodiment also provides a computing device, including: one or more processors, one or more memories, and one or more programs, wherein the programs are stored in the memories and configured to be executed by the processors, and when the programs are loaded into the processors, the steps of the lung nodule detection method based on Mamba and global multi-scale feature fusion are implemented according to any of the above items.

Claims

1. A lung nodule detection method based on Mamba and global multi-scale feature fusion, characterized in that: include: The CenterNet algorithm is improved to build a lung nodule detection model, wherein the improvements include replacing the original feature extractor in CenterNet with a feature extractor based on Mamba, adding a multi-scale feature aggregation module for fusing features from different scales, and combining bounding box regression and loss function to train the model and output detection results; A lung nodule detection model is used to detect a lung CT image to be detected, including: performing feature encoding on the lung CT image to be detected to obtain an encoded multi-scale feature map; performing feature decoding on the encoded multi-scale feature map to obtain a decoded multi-scale feature map; inputting the multi-scale feature maps at different decoding stages into a multi-scale feature aggregation module, aggregating features with different functions, and obtaining a feature map after multi-scale feature aggregation; performing bounding box regression prediction, and locating lung nodules through center point prediction, offset prediction, and size prediction; and combining the center point prediction result, the offset prediction result, and the size prediction result to obtain the final detection result.

2. The method for detecting pulmonary nodules based on Mamba and global multi-scale feature fusion according to claim 1, characterized in that: The structure of the Mamba-based feature extractor is as follows: Linearize the 3D features into 2D features through the linearization layer: x′=LN(Flatten(x0)) Among them, x0 is the input feature, x′∈R C,H,W represents the linearized feature, R represents the set of feature images, C is the depth of the feature image, H is the height of the feature image, W is the width of the feature image, LN() is layer normalization, and Flatten() represents the operation of linearizing three-dimensional features into two-dimensional; Then, the linearized features are globally modeled through the SSM layer and expressed in linear space: h t ′=A·h t +B·x t ′ y t =C·h t ′+x t ′ where x t ′ represents the tth point in the linearized feature, h t ′ represents the characteristics of the hidden state, y t represents the t-th point of the output feature, A, B and C represent learnable parameters, which are initialized randomly; t The combination is performed to obtain y0, and after being processed by the SSM model, the feature space of the linear feature is changed to: y′=Unflatten(y0) Where Unflatten() represents the inverse operation of Flatten(), y′∈R C,H,W ; A small network consisting of two convolutional layers is used to add further transformation to the global features: y″=Conv(Conv(y′))+y′ Where y″ represents the transformed global feature, and Conv() represents the convolution operation; A small CNN network is used to extract fine-grained features from global features: y″′=σ(BN(Conv(y″))+x) Where y″′ represents the final output feature, BN() represents the normalization operation, and σ() represents the ReLU activation function.

3. The lung nodule detection method based on Mamba and global multi-scale feature fusion according to claim 1, characterized in that: The feature encoding of the lung CT image to be detected is performed to obtain an encoded multi-scale feature map: A convolutional layer is used to preliminarily extract the features of the lung CT image to be tested; Four feature extraction blocks are used to perform global feature representation on the initially extracted features to generate encoded multi-scale feature maps; a maximum pooling layer is used before each feature extraction block to halve the scale of the features and double the number of channels.

4. The method for detecting pulmonary nodules based on Mamba and global multi-scale feature fusion according to claim 3, characterized in that: The structure of the feature extraction block includes: The convolution layer with a convolution kernel size of 1 is used to expand the channel of the input features; BN layer, used to eliminate the influence of outlier samples; Mamba block, a feature extractor based on Mamba, is used for linear spatial representation; A small CNN network is used to process global features and retain the ability to perceive fine-grained features.

5. The method for detecting pulmonary nodules based on Mamba and global multi-scale feature fusion according to claim 1, characterized in that: The step of decoding the encoded multi-scale feature map to obtain a decoded multi-scale feature map includes: Upsample the encoded multi-scale feature map to increase the width and height of the feature and restore the size of the image; The upsampled feature map is input into the Mamba-based feature extractor for feature extraction and generates a multi-scale feature map. Three layers of Mamba feature extractors are stacked continuously to restore the feature size of the multi-scale feature map to half the size of the input image, and the decoded multi-scale feature map is obtained.

6. The method for detecting pulmonary nodules based on Mamba and global multi-scale feature fusion according to claim 1, characterized in that: The multi-scale feature maps at different decoding stages are input into the multi-scale feature aggregation module, and features with different functions are aggregated to obtain feature maps after multi-scale feature aggregation, including: The feature maps of the three stages are obtained from the decoded multi-scale feature map, which are defined as: and Features of different scales are upsampled, and then the number of channels is transformed through a convolution layer with a convolution kernel of 1 to ensure that features of different scales have the same shape: x′1=Conv(Up(x1)) x′2=Conv(Up(x2)) Among them, Up() means upsampling the features using bilinear interpolation, and Conv() means a convolution layer with a convolution kernel size of 1; the feature size after processing is: and Add an extra empty dimension to the three features that are preprocessed to the same size, and concatenate and fuse along the empty dimension to obtain Then, a 3D convolution layer with a convolution kernel of 3×3×3 is used to process the concatenated and fused features, and then a 3D Bn layer is used to eliminate the negative impact of outlier data. Finally, the ReLU activation function is used to obtain the final result: x′ f =σ(Bn 3D (Conv 3D (x f ))) Among them, Bn 3D () represents a three-dimensional batch normalization layer, Conv 3D () represents a 3D convolutional layer; The output of the feature fusion unit is processed using a maximum pooling layer with a window of 3×1×1: x′ f ′=Maxpool 3D (x′ f ) Among them, Maxpool 3D () represents the maximum pooling layer in three dimensions, is the feature after maximum pooling.

7. The method for detecting pulmonary nodules based on Mamba and global multi-scale feature fusion according to claim 1, characterized in that: The bounding box regression prediction is performed to locate the lung nodules through center point prediction, offset prediction and size prediction, including: Use the center point prediction loss function to get the center point of the nodule: Among them, N represents the number of key points in the image, α, β are hyperparameters, and Y xyc represents the center point heat map predicted by the network, c represents the number of categories finally detected, and x and y represent the center point position of the nodule; To perform offset prediction, the offset prediction loss function is defined as follows: Among them, p represents the predicted node, The output of the offset prediction branch. Indicates the error value that occurs when downsampling the label data by 2 times; The size of the located nodules is predicted to predict the size of the lung nodules, and the size prediction loss function is defined as: where s pk =[w pk ,h pk ] represents the size of the true label, [w pk ,h pk ] represent the width and height of the current nodule, s pk ′ represents the nodule size predicted by the nodule detection network.

8. The method for detecting pulmonary nodules based on Mamba and global multi-scale feature fusion according to claim 1, characterized in that: The center point prediction result, the offset prediction result and the size prediction result are combined to obtain the final detection result, including: L total =a·L k +b·L o +d·L wh Among them, L k is the prediction result of the center point, L o is the offset prediction result, L wh is the size prediction result, a, b and d are the hyperparameters of network training.

9. A pulmonary nodule detection system based on Mamba and global multi-scale feature fusion, characterized in that: include: A construction module is used to improve the CenterNet algorithm and build a lung nodule detection model, wherein the improvement includes replacing the original feature extractor in CenterNet with a feature extractor based on Mamba, adding a multi-scale feature aggregation module for fusing features from different scales, and combining bounding box regression and loss function to train the model and output detection results; The detection module is used to detect the lung CT image to be detected using the lung nodule detection model, including: feature encoding the lung CT image to be detected to obtain an encoded multi-scale feature map; feature decoding the encoded multi-scale feature map to obtain a decoded multi-scale feature map; inputting the multi-scale feature maps at different decoding stages into the multi-scale feature aggregation module, aggregating features with different functions, and obtaining a feature map after multi-scale feature aggregation; performing bounding box regression prediction, and locating lung nodules through center point prediction, offset prediction, and size prediction; combining the center point prediction result, the offset prediction result, and the size prediction result to obtain the final detection result.

10. A computing device, characterized in that: include: One or more processors, one or more memories, and one or more programs, wherein the programs are stored in the memories and configured to be executed by the processors, and when the programs are loaded into the processors, the steps of the lung nodule detection method based on Mamba and global multi-scale feature fusion according to any one of claims 1 to 8 are implemented.

Images