A pulmonary nodule detection method and system based on three-dimensional attention mechanism

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a lung nodule detection method and system based on a three-dimensional attention mechanism; the method comprises: a lung nodule detection model based on a three-dimensional attention mechanism comprises a candidate lung nodule detection branch and a false positive deletion branch; the candidate lung nodule detection branch is a 3D U-Net model with a PSV-AM module added to both an encoding path and a decoding path; the false positive deletion branch consists of 5 continuous convolutions, 5 PSV-AM modules and a fully connected classification module; a CT image is acquired, the CT image is sheared, and the sheared CT image is input into a trained lung nodule detection model based on a three-dimensional attention mechanism to obtain a target lung nodule block; the model of the present invention has good training effect, high detection accuracy and good application prospect.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a lung nodule detection method and system based on a three-dimensional attention mechanism. Background Art

[0002] The main goals of the lung nodule detection task are twofold: one is to detect nodules, that is, to detect nodules; the other is to remove false positive nodules to obtain accurate classification of real nodules.

[0003] The goal of lung nodule detection is to accurately detect nodules of varying scales and morphologies. Previous research has demonstrated significant success with target detection models. For example, two RPNs (region proposal networks) and a deconvolution strategy were used to construct a network. A model trained on 3D slices was used to integrate 3D lung information. A fine-tuned 2D CNN (two-dimensional convolutional neural network) and a fully convolutional Faster RCNN (fast region convolutional neural network) were used to detect candidate nodules and eliminate false-positive nodules.

[0004] Another approach to nodule detection is to leverage the idea of ​​an end-to-end model for lesion region segmentation. The combination of U-Net and residual networks overcomes the potential ineffectiveness of U-Net due to insufficient depth. Due to the limited utilization of inherent 3D data by 2D and 2.5D networks, the network is not very effective in distinguishing nodules from surrounding blood vessels, resulting in many misjudgments of nodules. Taking into account the three-dimensional features of CT images, a multi-region construction network employs different receptive fields and multi-scale feature fusion strategies, thereby improving the network's sensitivity to detecting nodules of varying scales. At the same time, in order to better integrate high-resolution and strong semantic features, a multi-scale attention module is added to adaptively learn the relationship between different feature maps.

[0005] Misidentification and missed detection are another common problem in lung nodule detection. One study used a pyramid network-based FPN model to detect candidate lung nodules and proposed a conditionally constrained three-dimensional non-maximum suppression (3D-NMS) algorithm to eliminate redundant candidate nodules while reducing false positives through an attention mechanism.

[0006] In recent years, the exploration of attention mechanisms to guide networks to focus on regions of interest has received widespread attention. In lung nodule detection, attention mechanisms can be used to adapt to varying nodule sizes. A representative study incorporated convolutional block attention (CBMA) into a multi-stream CNN architecture for false positive reduction, achieving promising results.

[0007] However, in medical image processing, the way images are presented differs from natural images, making it difficult to distinguish the boundaries between target regions and surrounding tissue, leading to high misjudgment rates. Furthermore, most research on lung nodule detection and classification focuses on 2D CT slices, paying less attention to the connections between slices in real 3D structures, and suffers from poor interpretability. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention proposes a lung nodule detection method and system based on a three-dimensional attention mechanism. The method comprises: acquiring a CT image, cropping the CT image, and inputting the cropped CT image into a trained lung nodule detection model based on a three-dimensional attention mechanism to obtain a target lung nodule block;

[0009] The pulmonary nodule detection model based on the three-dimensional attention mechanism includes a candidate pulmonary nodule detection branch and a false positive removal branch. The candidate pulmonary nodule detection branch is a 3D U-Net model with PSV-AM modules added to both the encoding and decoding paths. The false positive removal branch consists of five consecutive convolutions, an upsampling module, four PSV-AM modules, and a fully connected classification module.

[0010] Among them, the PSV-AM module extracts texture features, spatial features and channel features of CT images, and integrates the extracted texture features, spatial features and channel features to obtain complete image features.

[0011] Preferably, the process of training a lung nodule detection model based on a three-dimensional attention mechanism includes: obtaining a CT image dataset with nodule labels; preprocessing the CT images in the dataset, inputting the preprocessed CT images into a candidate lung nodule detection branch for training, and obtaining candidate lung nodule blocks; migrating the parameters of the candidate lung nodule detection branch into a false positive deletion branch, and inputting the preprocessed CT images into a false positive deletion branch after the complete migration parameters for training, thereby achieving the deletion of false positive nodules in the candidate lung nodule blocks and obtaining target lung nodule blocks.

[0012] Furthermore, the process of preprocessing CT images includes: cropping all CT images to 512×512 pixels; regularizing all CT image slices according to the thickness of the CT image slices to unify the thickness of the CT image slices; cutting all CT images into multiple 96×96 pixel image blocks, with the image block containing the nodule marker coordinates as the center, and intercepting a total of 16 CT image blocks before and after the coordinates and adjacent to them as detection labels; performing horizontal, vertical, and flipping operations on the detection labels with nodules to expand the detection labels to 40 times the original size.

[0013] Preferably, the process of extracting features from CT images by the PSV-AM module includes:

[0014] A convolution layer with a convolution kernel of 1 and a maximum pooling layer are used to process the CT image to obtain texture features;

[0015] After performing the maximum pooling and average pooling operations on the CT image in sequence, a convolution with a convolution kernel of 7×7 is used to learn the spatial structure of the image along the channel direction with a longitudinal length of 1 to obtain the intermediate features. The intermediate features are linearized with the sigmoid function to obtain the spatial features.

[0016] The maximum pooling operation and the average pooling operation are performed on the CT image respectively. The maximum pooling result and the average pooling result are added together and then linearized with the sigmoid function to obtain the channel features.

[0017] Furthermore, the expression for extracting texture features is:

[0018] featur_plane(X)=relu[down_sample(X)+conv(X)]

[0019] Among them, featur_plane(X) represents the plane feature of patch x, down_sample(X) represents the downsampling of patch X, conv(X) represents the convolution of patch X, and relu represents the activation function.

[0020] Furthermore, the expression for extracting channel features is:

[0021]

[0022] Among them, featur_chan(X) represents the channel feature of patch X, σ represents the sigmoid nonlinear function, and W represents the pooling weight. represents the average pooling of patch X, Represents the maximum pooling of patch X.

[0023] Preferably, the formula for integrating texture features, spatial features and channel features is:

[0024] featur(X)=featur_chan(X)(featur_plane(X)+featur_spatial(X))

[0025] Among them, featur(X) represents the complete feature image of patch X, featur_chan(X) represents the channel feature of patchx, featur_plane(X) represents the plane feature of patch X, and featur_spatial(X) represents the spatial feature of patch X.

[0026] Preferably, each feature learning module of the false positive deletion branch includes three convolutional layers, each feature learning module is connected to a PSV-AM module, and the last PSV-AM module is connected to a fully connected classification module; wherein the full classification module includes a fully connected layer with 256 neurons and a convolutional layer with 512 neurons.

[0027] A pulmonary nodule detection system based on a three-dimensional attention mechanism, comprising: an image preprocessing module, a candidate pulmonary nodule detection module, a false positive removal module, and a target pulmonary nodule display module;

[0028] The image preprocessing module is used to perform shearing processing on the CT image;

[0029] The candidate pulmonary nodule detection module is used to perform candidate pulmonary nodule detection on the cropped CT image to obtain candidate pulmonary nodules;

[0030] The false positive deletion module is used to perform false positive deletion on candidate lung nodules to obtain target lung nodule blocks;

[0031] The target lung nodule display module displays the detection result of the target lung nodule block.

[0032] The beneficial effects of the present invention are as follows: the lung nodule detection method based on the three-dimensional attention mechanism of the present invention takes into account the three-dimensional CT image, designs a three-dimensional attention mechanism module for extracting the three-dimensional features of the CT image, and improves the 3D U-Net model according to the three-dimensional attention mechanism to obtain the candidate lung nodule detection results, and designs a false positive deletion branch network to achieve false positive deletion of candidate lung nodules, thereby improving the detection accuracy of the target lung nodules. Compared with the existing technology, the present invention pays more attention to using prior knowledge as a basis to guide the network model to learn the image of the lung nodule and extract features, and learns from the three dimensions of the cross-section, longitudinal plane, and sagittal plane of the image slice to fully wrap the nodule, thereby reducing the incidence of false positive detection. In clinical work, computer-aided diagnosis can significantly improve the work efficiency of radiologists, realize batch processing of images, and provide clinicians with auxiliary judgment basis. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the lung nodule detection method based on the three-dimensional attention mechanism in the present invention;

[0034] Figure 2 Schematic diagram of the structure of the lung nodule detection model based on the three-dimensional attention mechanism in the present invention;

[0035] Figure 3 This is a schematic diagram of the false positive deletion branch network structure in the present invention;

[0036] Figure 4 This is a visualization image of the lung nodule detection results of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] The present invention proposes a lung nodule detection method and system based on a three-dimensional attention mechanism. Figure 1 、 Figure 2 As shown, the method includes the following steps: acquiring a CT image, cropping the CT image, and inputting the cropped CT image into a trained lung nodule detection model based on a three-dimensional attention mechanism to obtain a target lung nodule block;

[0039] The pulmonary nodule detection model based on the 3D attention mechanism includes a candidate pulmonary nodule detection branch and a false positive removal branch. The candidate pulmonary nodule detection branch is a 3D U-Net model with the PSV-AM module (3D attention mechanism module) added to both the encoding and decoding paths. The false positive removal branch consists of five consecutive convolutions, an upsampling module, four PSV-AM modules, and a fully connected classification module.

[0040] Among them, the PSV-AM module extracts texture features, spatial features and channel features of CT images, and integrates the extracted texture features, spatial features and channel features to obtain complete image features.

[0041] The process of training the 3D attention-based lung nodule detection model includes the following:

[0042] A CT image dataset with nodule markings is obtained, where the nodule markings are the regions identified as nodules and the coordinates of the center positions.

[0043] The CT images in the dataset are preprocessed: First, due to the different sources of CT images in the dataset, from the perspective of network input and running efficiency, all CT images are cropped to 512×512 pixels. At the same time, since the thickness of CT image slices may be inconsistent, all CT image slices are regularized according to the thickness of CT image slices to unify the thickness of CT image slices; then, in order to enable the network to extract nodules more accurately, all CT images are cut into multiple image blocks of 96×96 pixels; according to actual experience, the largest nodule occupies the position of the front and back 16 slices. Due to the differences in the three-dimensional structure and morphological size of nodules, the image block containing the nodule mark coordinates is taken as the center, and a total of 16 CT image blocks adjacent to the coordinates are intercepted as detection labels. Then, the size of the input set for the final detection network is (96, 96, 16)*N; this method uses the area identified as the nodule as the label sequence of the training sample, rather than the entire image, which can more clearly extract the target features and reduce redundant learning.

[0044] In a given label queue, there is a significant difference in the number of positive and negative samples, with images with nodules far fewer than those without. Therefore, we perform horizontal, vertical, and flip operations on the detection labels with nodules to expand the detection labels by 40 times the original number, resulting in a positive-to-negative sample ratio of approximately 1:10.

[0045] In order to focus on learning the features of nodule regions, this paper designs a cube attention module named PSV-AM, such as Figure 3 As shown in Figure 2, based on the morphology of nodules on tomographic slices, high-dimensional features of images dominate image classification in deep learning. The cube module is a three-dimensional structure that can learn the structure of nodules surrounded by cross-sectional, coronal, and sagittal planes.

[0046] The PSV-AM module is used to extract texture features, spatial features, and channel features from CT images, and integrate the extracted texture features, spatial features, and channel features to obtain complete image features. The module's feature extraction process for CT images includes:

[0047] The morphology of the nodules is dominant in a slice. A convolution layer with a convolution kernel of 1 and a maximum pooling layer are used to process the CT image (single slice), so that the texture features composed of the changes in the pixel grayscale values ​​in the slice are highlighted again. Each convolution operation is accompanied by batch normalization and activation function mapping; the expression for extracting texture features is:

[0048] featur_plane(X)=relu[down_sample(X)+conv(X)]

[0049] Among them, featur_plane(X) represents the plane feature of patch X, down_sample(X) represents downsampling of patch X, conv(X) represents convolution of patch X, and relu represents the activation function.

[0050] For three-dimensional slice information, it is easy to ignore the existence of the slice's spatial structure. Most existing technologies directly compress it into a unified channel for feature learning. The present invention also learns the third dimension, namely the thickness dimension. Learning spatial features includes: performing maximum pooling and average pooling operations on the CT image (single slice) in sequence, using a convolution kernel of 7×7 to learn the image's spatial structure along the channel direction with a longitudinal length of 1 to obtain intermediate features, and performing sigmoid function linearization on the intermediate features to obtain spatial features; the expression for extracting spatial features is:

[0051] featur_spatial(X)=σ{conv (7×7×7) [AvgPool(X);MaxPool(X)]⊙X}

[0052] Among them, featur_spatial(X) represents the spatial features of patch X, σ represents the sigmoid nonlinear function, conv (7×7×7) Represents a convolution operation with a kernel of 7×7×7, AvgPool(X) represents average pooling of patch X, and MaxPool(X) represents maximum pooling of patch X.

[0053] Channel-wise attention learning is primarily performed through two operations (a cascade of maximum pooling and average pooling). Unlike spatial attention learning, channel stacking increases the number of parameters by dozens of times. Therefore, through multi-layer perceptron learning, the sigmoid nonlinear function can effectively reduce the number of descriptive parameters. Learning channel features involves performing maximum pooling and average pooling operations on CT images (stacked slices), adding the maximum and average pooling results, and then linearizing them with the sigmoid function to obtain channel features. The expression for extracting channel features is:

[0054]

[0055] Among them, featur_chan(X) represents the channel feature of patch X, σ represents the sigmoid nonlinear function; W represents the pooling weight, represents the average pooling of patch X, Represents the maximum pooling of patch X.

[0056] The extracted texture features, spatial features and channel features are integrated to obtain the complete image features. The integration formula is:

[0057] featur(X)=featur_chan(X)(featur_plane(X)+featur_spatial(X))

[0058] The preprocessed CT image is fed into the candidate pulmonary nodule detection branch for training, generating candidate pulmonary nodule blocks. The candidate pulmonary nodule detection branch consists of a 3D U-Net model with PSV-AM modules added to both the encoding and decoding paths. The data flow of the CT image in the 3D U-Net model is as follows: the input image first enters the contraction path of the U-Net network, which consists of four contraction modules. These modules repeatedly use 3×3×3 padded convolutional layers, each followed by a Relu-like function. After the convolution, the feature map is downsampled using four 2×2×2 max pooling layers with a stride of 2. Each contraction block is followed by a PSV-AM module for knowledge-based feature extraction. The expansion path consists of four expansion modules, each of which performs feature map upsampling. The first three expansion modules use 2×2×2 convolutional layers, each followed by a Relu-like function, and are connected to the response feature map in the contraction path. Similarly, each expansion module is followed by a PSV-AM module, and the last expansion module is a 1×1×1 convolutional layer, which maps each 32-component feature vector to the required number of classes to obtain the inspection results of the candidate lung nodes.

[0059] After the candidate nodule detection branch is trained, its parameters are transferred to the false positive removal branch, which consists of five sequential feature learning modules, five PSV-AM modules, and a fully connected classification module. The preprocessed CT image is then input into the false positive removal branch, after the parameters have been fully transferred, for training. This eliminates false positive nodules from the candidate nodule blocks and yields the target nodule blocks. The data flow of the CT image in the false positive removal branch is as follows: the parameters of the convolutional blocks in the false positive removal branch are derived from the candidate nodule detection branch, thus migrating the training modules. The cropped input image then passes through five feature learning modules consisting of three convolutional layers. Each feature learning module is followed by a PSV-AM module, which extracts slice features under knowledge guidance. Finally, the classification results are output after a fully connected layer with 256 neurons and a fully connected layer with 512 neurons, respectively.

[0060] Before output, the high-dimensional coefficient tensor of the image is flattened and scored by two fully connected layers in the fully connected module, and the final nodule discrimination result is obtained according to the probability value classification, that is, whether the image belongs to a lung nodule block and the probability of belonging to a lung nodule block; the role of the attention module in the false positive removal network is to enhance the network's guidance on the real lung nodule features during the classification process, guide the network to learn classification, and pay more attention to the comparison between misclassified nodules and real nodules.

[0061] The present invention detects lung nodules in CT images by identifying them. The obtained lung nodule detection results can be used as a basis for judgment, assisting doctors to better diagnose the benign or malignant nature and degree of infiltration of lung nodules.

[0062] The present invention also provides a pulmonary nodule detection system based on a three-dimensional attention mechanism, comprising: an image preprocessing module, a candidate pulmonary nodule detection module, a false positive deletion module, and a target pulmonary nodule display module;

[0063] The image preprocessing module is used to perform shearing processing on the CT image;

[0064] The candidate pulmonary nodule detection module is used to perform candidate pulmonary nodule detection on the cropped CT image to obtain candidate pulmonary nodules;

[0065] The false positive deletion module is used to perform false positive deletion on candidate lung nodules to obtain target lung nodule blocks;

[0066] The target lung nodule display module displays the detection result of the target lung nodule block.

[0067] To evaluate this invention, we used the LUNA16 public lung nodule dataset as the experimental object. The dataset was divided into a training set and a test set with a ratio of 80% and 20%, respectively. The relevant learning parameters were set as follows: learning rate = 0.0001, batch_size = 16, dropout = 0.5, momentum = 0.9, and epochs = 100 * input_num. The detection accuracy during training varied with the number of training iterations. During training, the original and predicted images were saved in multiples of 10 training cycles. The resulting visualization is shown in the figure below. Figure 4 As shown in the figure, src is the original nodule image, mask is the label image, and predict is the detected image.

[0068] The false-positive nodule subtraction network architecture is labeled RPSV-Net, and the relevant parameter settings are the same as those for the detection branch. The training and test sets, containing both true and false nodules, serve as input to the subtraction network, and the prediction results are saved. The results of the positive nodule subtraction network are shown in Table 1, and the overall network performance is shown in Table 2. The subtraction network achieved a sensitivity of 97.6% and a detection accuracy of 96%. The area under the receiver operating characteristic (ROC) curve was 0.9998.

[0069] Table 1 Nodule classification prediction results

[0070]

[0071] Table 2 RPSC-Net structural performance measurement indicators

[0072]

[0073] Experiments have shown that the classification accuracy of the present invention reaches 98%, and the average false positive detection rate of the scan is 89.3%. Therefore, the present invention can effectively detect lung nodules with high detection accuracy and can be compared with the standards of experienced radiologists. The various image features of lung nodules obtained by the detection method of the present invention can be used as an effective basis for judgment, assisting doctors to better diagnose the benign and malignant nature and degree of infiltration of lung nodules, and has good application prospects.

[0074] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A lung nodule detection method based on a three-dimensional attention mechanism, characterized in that: include: Acquire a CT image, crop the CT image, and input the cropped CT image into a trained lung nodule detection model based on a three-dimensional attention mechanism to obtain a target lung nodule block; The lung nodule detection model based on the three-dimensional attention mechanism includes a candidate lung nodule detection branch and a false positive removal branch; The candidate lung nodule detection branch is a 3D U-Net model with PSV-AM modules added to both the encoding and decoding paths; the false positive reduction branch consists of 5 consecutive feature learning modules, 5 PSV-AM modules and a fully connected classification module; Among them, the PSV-AM module extracts texture features, spatial features and channel features from CT images, and integrates the extracted texture features, spatial features and channel features to obtain complete image features; The process of feature extraction of CT images by PSV-AM module includes: A convolution layer with a convolution kernel of 1 and a maximum pooling layer are used to process the CT image to obtain texture features; the expression for extracting texture features is: featur_plane(X)=relu[down_sample(X)+conv(X)] Among them, feature_plane(X) represents the plane feature of patch X, down_sample(X) represents downsampling of patch X, conv(X) represents convolution of patch X, and relu represents the activation function; After performing the maximum pooling operation and the average pooling operation on the CT image in sequence, the convolution kernel of 7×7 is used to learn the spatial structure of the image along the channel direction with a longitudinal length of 1 to obtain the intermediate features. The intermediate features are linearized by the sigmoid function to obtain the spatial features. The expression for extracting the spatial features is: featur_spatioal(X)=σ{conv (7×7×7) [AvgPool(X);MaxPool(X)]⊙X} Among them, featur_spatial(X) represents the spatial features of patch X, σ represents the sigmoid nonlinear function, and conv (7×7×7) represents a convolution operation with a kernel of 7×7×7, AvgPool(X) represents average pooling of patchX, and MaxPool(X) represents maximum pooling of patch X; The maximum pooling operation and the average pooling operation are performed on the CT image respectively. The maximum pooling result and the average pooling result are added and then linearized with the sigmoid function to obtain the channel feature. The expression for extracting the channel feature is: Among them, featur_Chan(X) represents the channel feature of patchX and W represents the pooling weight. represents the average pooling of patchX, Indicates the maximum pooling of patchX; The formula for integrating texture features, spatial features and channel features is: featur(X)=featur_chan(X)(featur_plane(X)+featur_spatial(X)) Among them, featur(X) represents the complete feature image of patch X.

2. According to claim 1, a lung nodule detection method based on a three-dimensional attention mechanism is characterized in that: The process of training the lung nodule detection model based on the three-dimensional attention mechanism includes: obtaining a CT image dataset with nodule labels; preprocessing the CT images in the dataset, inputting the preprocessed CT images into the candidate lung nodule detection branch for training, and obtaining candidate lung nodule blocks; migrating the parameters of the candidate lung nodule detection branch to the false positive deletion branch, and inputting the preprocessed CT images into the false positive deletion branch after the complete migration parameters for training, so as to achieve the deletion of false positive nodules in the candidate lung nodule blocks and obtain the target lung nodule blocks.

3. The method for detecting lung nodules based on a three-dimensional attention mechanism according to claim 2, characterized in that: The process of preprocessing CT images includes: cropping all CT images to 512×512 pixels; regularizing all CT image slices according to the thickness of CT image slices to unify the thickness of CT image slices; cutting all CT images into multiple 96×96 pixel image blocks, taking the image block containing the nodule marking coordinates as the center, and intercepting a total of 16 CT image blocks adjacent to the coordinates before and after the coordinates as detection labels; performing horizontal, vertical, and flipping operations on the detection labels with nodules to expand the detection labels to 40 times the original size.

4. The method for detecting lung nodules based on a three-dimensional attention mechanism according to claim 1, characterized in that: Each feature learning module of the false positive removal branch includes three convolutional layers. Each feature learning module is connected to a PSV-AM module, and the last PSV-AM module is connected to a fully connected classification module; among them, the full classification module includes a fully connected layer with 256 neurons and a convolutional layer with 512 neurons.

5. A pulmonary nodule detection system based on a three-dimensional attention mechanism, the system being used to execute a pulmonary nodule detection method based on a three-dimensional attention mechanism as described in any one of claims 1 to 4, characterized in that: include: Image preprocessing module, candidate lung nodule detection module, false positive deletion module and target lung nodule display module; The image preprocessing module is used to perform shearing processing on the CT image; The candidate pulmonary nodule detection module is used to perform candidate pulmonary nodule detection on the cut CT image to obtain candidate pulmonary nodules; The false positive deletion module is used to perform false positive deletion on candidate lung nodules to obtain target lung nodule blocks; The target lung nodule display module displays the detection result of the target lung nodule block.

Images