Deep learning model system for benign and malignant diagnosis of solid pulmonary nodules

Abstract

The invention belongs to the technical field of benign and malignant diagnosis model systems for solid pulmonary nodules, and particularly relates to a benign and malignant diagnosis deep learning model system for solid pulmonary nodules. Comprising an overall architecture system of the model, a data set introduction system, a data set preprocessing system, a lung CT image segmentation system, a lung nodule detection system, a lung nodule category system, an experiment parameter setting system, an evaluation index system and an experiment setting and result system. A lower half branch of the network is mainly designed, the lower half branch network extracts multi-scale feature information of a lung CT image through an Inception module, and convolutional attention modules such as ECA and CBAM are introduced to enhance the extraction effect of the lower half branch network on position and shape features of the lung CT image.

Description

Technical Field

[0001] The present invention belongs to the technical field of benign and malignant solid pulmonary nodules diagnosis model system, and specifically relates to a benign and malignant solid pulmonary nodule diagnosis deep learning model system. Background Art

[0002] The benign resection rate of lung nodules is generally required to be around 10%, and the actual clinical benign resection rate is above 30%. The main benign resection lesions appear as solid lesions on lung CT, that is, the high benign resection rate of solid lung nodules is a clinical problem that is difficult to avoid. The core of the above problem is that the prognosis of solid lung cancer is much worse than that of subsolid lung cancer (ground glass), and doctors must avoid missed diagnosis and overcome unnecessary surgical impulses. The lack of accurate judgment tools and methods for the limited diagnosis of lung cancer in the clinic makes diagnostic surgery the most reliable way. The core of solving the problem is to find effective methods and tools for the limited diagnosis of lung cancer, and provide effective reference for clinical preoperative decision-making. CT is still a clinically available, sufficiently low-cost, and efficient tool for limited diagnosis of lung cancer, but its ultimate performance in the diagnosis of benign and malignant solid lung nodules is not clear, and whether it can provide sufficient decision support for surgical decisions is even more unclear.

[0003] Existing algorithms for diagnosing benign and malignant lung nodules are mainly divided into two categories: one is machine learning algorithms (ML) and the other is deep learning algorithms (DL). Machine learning algorithms are traditional, two-stage algorithms that require manual selection of features that are closely related to the task of diagnosing benign and malignant lung nodules. Then, appropriate machine learning algorithms are used to classify these features, thereby achieving the effect of diagnosing benign and malignant lung nodules. Warkentin et al. [1] used cross-validation and grid search methods to evaluate the ability of three machine learning algorithms, eXtreme Gradient Boosted Trees, random forest, and LASSO, to classify benign and malignant lung nodules. The results showed that the least absolute shrinkage and selection operator (LASSO) had the best classification effect and could accurately predict the benign and malignant nature of lung nodules. In response to the problem of poor performance of a single machine learning algorithm, Muzammil et al. proposed an ensemble learning algorithm based on the average prediction score and the maximum image. It can integrate the classification decisions of the two algorithms, SVM (support vector machine) and AdaBoostM2, and can more effectively predict the benign and malignant nature of lung nodules.

[0004] However, the quality of manually extracted features largely determines the performance of machine learning classification algorithms, and experts need to spend a lot of time manually extracting these key features. For example, in the diagnosis of benign and malignant lung nodules, the morphological and texture features, size, and original data of each patient (age, gender, etc.) of lung nodules are important manual features, and obtaining these features requires a lot of manual costs. In addition, the classifiers used by machine learning algorithms are often relatively shallow, making it difficult to mine information from complex high-dimensional data, and are prone to underfitting and overfitting problems.

[0005] With the continuous research of researchers on artificial intelligence technology, deep learning algorithms have achieved significant success in computer vision, speech recognition, natural language processing and other fields. From the great success of convolutional neural network AlexNet on the ImageNet dataset to the current large language model technology based on the Transformer architecture, deep neural network technology is becoming more and more mature and its application is becoming more and more extensive. Its powerful feature extraction ability makes it widely used in many fields. End-to-end deep learning algorithms do not require manual feature extraction and can learn complex feature information. Therefore, a large number of studies based on deep learning methods have been conducted to solve the problem of diagnosis of benign and malignant pulmonary nodules. Yin et al. proposed a multimodal feature fusion network to classify benign and malignant pulmonary nodules. MobileNetV3 was used as the backbone network, and a dual-branch network was designed to effectively extract features from two modal data, the original CT image of the pulmonary nodule and the extracted nodule image. While extracting the global information of the original CT image, the local feature information of the pulmonary nodules was also extracted. Song et al.[7] based on the multi-task learning method, by mining the potential common features between the tasks of lung nodule detection and lung nodule segmentation, enabled each task to complete training and learning efficiently, thus improving the effects of lung nodule detection and segmentation. Cao et al. proposed the MSM-ViT network based on CNN and MobileViT network. First, dilated convolution and ordinary convolution were used to extract multi-scale local information of lung nodules to make up for the deficiency of local information extraction of the later ViT model. Then, the MovileViT module was used to extract features of the global information of lung nodules. This model can effectively extract feature information of each scale and improve the accuracy of benign and malignant classification of lung nodules. Zhang et al. used the deep learning model of ResNet combined with convolutional attention module (CBAM) to classify benign and malignant lung nodules based on graphics, morphological features, and clinical information, and compared it with the traditional machine learning algorithm NSDTCT-SVM. The results showed that the deep learning model combined with additional information is better than the traditional machine learning algorithm. Summary of the invention

[0006] The purpose of the present invention is to provide a deep learning model system for diagnosing benign and malignant solid pulmonary nodules. The system extracts clinical experience and feature information from lung CT images through two independent branches, and effectively integrates the two to achieve a more accurate prediction of benign and malignant pulmonary nodules. This combination not only fully utilizes the advantages of deep learning algorithms in feature extraction and classification, but also incorporates the clinical experience and professional knowledge of doctors, providing a more comprehensive and reliable solution for the diagnosis of pulmonary nodules.

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a deep learning model system for diagnosing benign and malignant solid pulmonary nodules, including an overall architecture system of the model, a data set introduction system, a data set preprocessing system, a lung CT image segmentation system, a lung nodule detection system, a lung nodule classification system, an experimental parameter setting system, an evaluation index system, and an experimental setting and result system. The overall architecture system of the model consists of an upper-branch network module and a lower-branch network module. The upper-branch network module uses a deep neural network composed of a residual network Resnet and a self-attention mechanism (Self-Attention) to extract features from a category matrix. The lower-branch network module uses a multi-scale deep neural network to extract features from a picture composed of ten pictures, and extracts the location and shape information of the nodules. Finally, the nodule size features are normalized and input into the penultimate layer, and are spliced ​​with the category features and the extracted nodule image features, and are input into the final fully connected layer with an activation function to obtain the predicted probability of malignancy of the pulmonary nodules.

[0008] The lower half branch network module includes an Inception module, an ECA module, and a CBAM module, and the Inception module is the core module of GoogLeNet. The Inception module extracts multi-scale features from the input feature map by using convolution kernels of different sizes, and then splices these feature maps with multi-scale information in the channel dimension. Subsequently, further convolution operations can be performed to fuse multi-scale feature information. The Inception module uses a 1×1 convolution kernel to first reduce the dimension of the feature map, and then uses multiple small convolution kernels to replace the large convolution kernel, which greatly reduces the parameters and calculation amount of the network and improves the efficiency of model training. In addition, residual connections are used in the lower branch of Inception, which can make the model converge faster and avoid the problem of gradient disappearance to a certain extent.

[0009] The ECA module is improved based on the defects of SENet, and the ECA module replaces the two-layer fully connected operations in the SE module with a convolution operation. Compared with the fully connected operation, the convolution operation can share parameters, reduce the number of model parameters, and provide appropriate channel interaction.

[0010] The CBAM module is an advanced attention mechanism, and the CBAM module consists of two core parts: channel attention and spatial attention, which aims to refine the importance of features from the channel dimension and spatial dimension respectively, realize the adaptive weighting of features, and thus improve the recognition accuracy and generalization ability of the network. The channel attention generates two channel feature vectors by performing global average pooling and global maximum pooling on the feature map. These two pooling operations can capture the statistical information between channels and reflect the overall distribution and salient parts of the features. Subsequently, these two feature vectors pass through a series of fully connected layers and a Sigmoid function to generate a normalized weight vector to indicate the importance of each channel, and in the back propagation process, these weights guide the network to pay more attention to those channels that are more discriminative for the task, effectively suppressing irrelevant or noise information. The spatial attention module focuses on the spatial dimension, and extracts the correlation of spatial features by applying two one-dimensional convolutions (along the width and height directions, respectively) to each channel, combined with the ReLU activation function, and then, a two-dimensional weight score is generated by the Sigmoid function, which represents the contribution of each position in the feature map to the final decision.

[0011] Preferably, the data set introduction system uses a lung CT data set provided by Changhai Hospital containing 1,279 cases, of which 388 are benign cases and 899 are malignant cases. The data set contains real-time, continuous Dicom data that can reflect the specific conditions of the patient's lungs, and the doctor selects multiple continuous lung CT images with nodules from the patient's Dicom data and puts them into a folder, and uses the folder as the key diagnostic data of the case. Each folder contains an original lung CT image and a labeled lung CT image annotated by the doctor, and the diagnostic information of the case is given by experienced doctors through comprehensive evaluation of the lung CT images in the corresponding folder. Based on clinical experience, the doctor divides the lung nodules in the lung CT images into 13 categories, including clear and smooth, halo sign, clear and rough, lobed, spinous process, burr, pleural retraction, bronchial truncation, vascular bundle, cavitation, inflation, multi-nodule aggregation, and straight, and all lung CT images of each case are labeled, and the nodule area is marked with a red circle.

[0012] Preferably, the data set preprocessing system uses the TransUnet network to preprocess the data set, and then uses the processed lung CT images to train the YOLOv5 model to detect whether the CT images contain nodules, and detects all real-time Dicom data of a patient according to the trained detection model, and detects 10 consecutive Dicom data with lung nodules as a data set for subsequent real-time classification of benign and malignant lung nodules, and records the detection results of these 10 images in a txt file, and the information in the txt file can be used to calculate the position of the lung nodules in the CT image, and this part of the nodule area is cropped on the original Dicom data, and the cropped image is classified using the pre-trained lung nodule thirteen classification model EfficientNet, and the output category probability distribution is used as the category feature vector.

[0013] Preferably, the lung CT image segmentation system summarizes the lung CT images of all cases in the data set, and in order to avoid interference of some text information in the lung nodule image with the detection and classification of nodules, the trained lung segmentation model TransUnet is used to segment the lung CT image to obtain a contour image of the lung CT image, where the pixel values ​​in the contour image are 0 or 1, and the pixels in the contour and its interior are 1, and the remaining pixel values ​​are 0. The original image and the Mask image are multiplied pixel by pixel to obtain a preliminarily processed image, which is used for subsequent detection and classification tasks.

[0014] Preferably, the lung nodule detection system uses the YOLOv5 algorithm. The YOLOv5 is a classic single-stage target detection algorithm with high detection accuracy and fast inference speed. It is now widely used in various target detection tasks. The YOLOv5 algorithm trains and detects nodules in lung CT images, and detects the original Dicom data based on the trained YOLOv5 model, and automatically selects the Dicom data with nodules. The lung nodule target detection is divided into a training stage and an application stage.

[0015] Preferably, the lung nodule classification system is trained to automatically identify thirteen specific categories of lung nodules in medical images. The categories are precisely divided based on clinical experience to ensure the comprehensiveness and accuracy of the model classification. Subsequently, this classification model performs expert-level classification on the target detected nodule areas. The specific process of the thirteen classifications is divided into a training phase and an application phase.

[0016] Preferably, the experimental parameter setting system divides the entire data set in a ratio of 8:2, with 80% of the case data as a training set and 20% of the case data as a test set. In the upper and lower branch network modules, 10 pictures of each patient are spliced ​​in the channel dimension as picture features and input into the lower branch network. The category features of these 10 pictures are spliced ​​to obtain a category matrix and input into the upper branch network. The size features of the lung nodules are added before the final classifier. The sizes of various features are shown in Table 1. This experiment was completed on a server with two 1080 graphics cards, 8 CPU cores and 128G of running memory. The parameter configuration of the model during training is shown in Table 2:

[0017] Table 1 Dimensions of input features

[0018]

[0019] Table 2 Training parameter settings

[0020]

[0021] Preferably, the evaluation index system uses accuracy and F1 score as evaluation indicators to evaluate the classification effect of the designed classification model on real-time benign and malignant pulmonary nodules. The higher the ACC and F1 index values, the better the classification effect. The calculation formula is as follows:

[0022]

[0023]

[0024]

[0025]

[0026] Where: TP represents that both the predicted value and the true value are positive, TN represents that both the predicted value and the true value are negative, and these two categories are predicted correctly, FN represents that the predicted value is negative and the true value is positive, and FP represents that the predicted value is positive and the true value is negative.

[0027] Preferably, the experimental setting and results system set up a set of control experiments to verify the necessity of reintegrating clinical experience in the task of benign and malignant lung nodule classification. Experiment one used the lower branch in the two-branch network for the experiment, and experiment two used the entire two-branch network for the experiment. Both experiments integrated the size characteristics of the nodules in the fully connected layer.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes a deep learning model system for diagnosing benign and malignant solid pulmonary nodules, and is mainly designed for the lower branch of the network. The lower branch network extracts multi-scale feature information of lung CT images through the Inception module, and introduces convolutional attention modules such as ECA and CBAM to enhance the extraction effect of the lower branch network on the position and shape features of lung CT images. The upper branch network effectively extracts the category information of lung nodules through the Resnet and Self-Attention modules. Experimental results show that in the real-time classification task of benign and malignant lung nodules, the deep learning-based method based on the integration of clinical experience proposed in this paper can meet the requirements of clinical diagnosis in terms of evaluation indicators such as F1 and accuracy, and can be used as an auxiliary means to help doctors improve the accuracy of diagnosing benign and malignant lung nodules. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the overall architecture diagram of the dual-branch network integrating clinical experience of the present invention.

[0030] Figure 2 This is a schematic diagram of the network structure of the Inception module of the present invention.

[0031] Figure 3 is a schematic diagram of the network structure of the ECA attention module of the present invention,

[0032] Figure 4 is a schematic diagram of the network structure of the CBAM attention module of the present invention,

[0033] Figure 5 is a schematic diagram of the number of pictures contained in the case folder of the present invention,

[0034] Figure 6 is the overall flow chart of data processing of the present invention,

[0035] Figure 7 This is a schematic diagram of a picture obtained by merging the original picture and the segmented picture of the present invention,

[0036] Figure 8 Schematic diagram of different types of lung nodules obtained by cutting out the present invention.

[0037] Fig. 9 A schematic diagram of the number of different nodule categories of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Please see attached Figure 1-9 , several embodiments provided by the present invention:

[0040] The deep learning model system for diagnosis of benign and malignant solid pulmonary nodules includes the overall architecture system of the model, the data set introduction system, the data set preprocessing system, the lung CT image segmentation system, the lung nodule detection system, the lung nodule classification system, the experimental parameter setting system, the evaluation index system, the experimental setting and result system. The overall architecture system of the model consists of an upper-branch network module and a lower-branch network module. The upper-branch network module uses a deep neural network composed of a residual network Resnet and a self-attention mechanism (Self-Attention) to extract features from the category matrix. The lower-branch network module uses a multi-scale deep neural network to extract features from a picture composed of ten pictures, and extracts the location and shape information of the nodules. Finally, the nodule size features are normalized and input into the penultimate layer, and then spliced ​​with the category features and the extracted nodule image features, and input into the final fully connected layer with an activation function to obtain the predicted probability of malignancy of the lung nodules.

[0041] The lower half branch network modules include the Inception module, the ECA module, and the CBAM module. The Inception module is the core module of GoogLeNet. The Inception module uses convolution kernels of different sizes to extract multi-scale features from the input feature map, and then splices these feature maps with multi-scale information in the channel dimension. The multi-scale feature information can be integrated through further convolution operations. The Inception module uses a 1×1 convolution kernel to reduce the dimension of the feature map first, and then uses multiple small convolution kernels to replace the large convolution kernel, which greatly reduces the network parameters and calculation amount, and improves the efficiency of model training. In addition, the residual connection is used in the lower branch of Inception, which can make the model converge faster and avoid the problem of gradient disappearance to a certain extent.

[0042] The ECA module is improved based on the defects of SENet. The ECA module replaces the two-layer fully connected operations in the SE module with a convolution operation. Compared with the fully connected operation, the convolution operation can share parameters, reduce the number of model parameters, and provide appropriate channel interaction.

[0043] The CBAM module is an advanced attention mechanism, and the CBAM module consists of two core parts: channel attention and spatial attention, which aims to refine the importance of features from the channel dimension and spatial dimension respectively, realize the adaptive weighting of features, and thus improve the recognition accuracy and generalization ability of the network. Channel attention generates two channel feature vectors by performing global average pooling and global maximum pooling on the feature map. These two pooling operations can capture the statistical information between channels and reflect the overall distribution and salient parts of the features. Subsequently, these two feature vectors pass through a series of fully connected layers and a Sigmoid function to generate a normalized weight vector to indicate the importance of each channel. In the back-propagation process, these weights guide the network to pay more attention to those channels that are more discriminative for the task, and effectively suppress irrelevant or noisy information. The spatial attention module focuses on the spatial dimension, and extracts the correlation of spatial features by applying two one-dimensional convolutions to each channel (along the width and height directions, respectively), combined with the ReLU activation function. After that, a two-dimensional weight score is generated by the Sigmoid function, which represents the contribution of each position in the feature map to the final decision. The structure of the entire CBAM is as follows: Figure 4 As shown in Figure 2, through the channel attention and spatial attention mechanisms, the deep network model can more effectively extract important and key features from lung CT images.

[0044] Furthermore, the data set introduction system uses a lung CT data set provided by Changhai Hospital containing 1,279 cases, of which 388 are benign cases and 899 are malignant cases. The data set contains real-time, continuous Dicom data, which can reflect the specific conditions of the patient's lungs. The doctor selects multiple continuous lung CT images with nodules from the patient's Dicom data and puts them into a folder. The folder is used as the key diagnostic data of the case. Each folder contains an original lung CT image and a labeled lung CT image annotated by the doctor. The diagnostic information of the case is given by experienced doctors who comprehensively judge the lung CT images in the corresponding folder. Based on clinical experience, the doctor divides the lung nodules in the lung CT images into 13 categories, including clear and smooth, halo sign, clear and rough, lobed, spinous process, burr, pleural retraction, bronchial truncation, vascular bundle, cavitation, inflation, multi-nodule aggregation, and straight. All lung CT images of each case are labeled, and the nodule area is marked with a red circle. The basic structure of the data set is as follows Figure 5 shown.

[0045] Furthermore, the data set preprocessing system uses the TransUnet network to preprocess the data set, and then uses the processed lung CT images to train the YOLOv5 model to detect whether the CT images contain nodules, and uses the trained detection model to detect all real-time Dicom data of a patient, and detects 10 consecutive Dicom data with lung nodules as the subsequent real-time classification of benign and malignant lung nodules. The detection results of these 10 images are recorded in a txt file, and the information in the txt file can be used to calculate the position of the lung nodules in the CT image, and this part of the nodule area is cropped on the original Dicom data. The cropped image is classified using the pre-trained lung nodule thirteen classification model EfficientNet, and the output category probability distribution is used as the category feature vector.

[0046] Furthermore, the lung CT image segmentation system summarizes the lung CT images of all cases in the data set, and in order to avoid the interference of some text information in the lung nodule image on the detection and classification of nodules, the trained lung segmentation model TransUnet is used to segment the lung CT images to obtain the contour image of the lung CT image. The pixel values ​​in the contour image are 0 or 1, and the pixels in the contour and its interior are 1, and the remaining pixel values ​​are 0. The original image and the Mask image are multiplied pixel by pixel to obtain the initially processed image, which is used for subsequent detection and classification tasks.

[0047] Furthermore, the lung nodule detection system uses the YOLOv5 algorithm. YOLOv5 is a classic single-stage target detection algorithm with high detection accuracy and fast inference speed. It is now widely used in various target detection tasks. The YOLOv5 algorithm trains and detects nodules in lung CT images, and detects the original Dicom data based on the trained YOLOv5 model, automatically selecting Dicom data with nodules. Lung nodule target detection is divided into training stage and application stage:

[0048] Training phase: In this detection task, the detection goal is to determine whether a lung CT image contains nodules. Use the segmented lung nodule images as the training and test data of the YOLOv5 model. According to the corresponding annotated lung CT images in the original data set, use LabelImg software to label the original lung CT images. The label here is only one type of nodule. Since the contour of the segmented image is relatively accurate and most of the lung nodules are located within the contour, the label obtained here can be directly used as the label of the segmented lung CT image. Based on this batch of data, the YOLOv5 model is trained.

[0049] Application stage: The real-time Dicom data of each patient is preprocessed through the TransUNet network, that is, the segmented image and the original image are multiplied element by element to obtain the image type that can be used for YOLOv5 reasoning. The trained YOLOv5 model is used to reason about each image. Since the lung CT images of patients with lung nodules are continuous, we will use the data of more than ten consecutive images detected as the data set for subsequent real-time benign and malignant classification.

[0050] Furthermore, the lung nodule classification system can automatically identify thirteen specific categories of lung nodules in medical images after training. The categories are accurately divided based on clinical experience to ensure the comprehensiveness and accuracy of the model classification. In the subsequent classification model, the target detected nodule area is classified at the expert level. The specific process of the thirteen classifications is divided into the training stage and the application stage:

[0051] Training phase: Based on the label information obtained above, cropping is performed on the original lung CT image. Since each CT image corresponds to this expert-level nodule category label, this category label is used as the category label of the cropped nodule area. The cropped image has been labeled as follows: Figure 8 The data set consisting of the cropped images and the corresponding category labels is used as the data set for training the pulmonary nodule 13 classification model, and the number of images in each category is preliminarily counted, as shown in Fig. 9 As shown in the figure, the number of nodules in the three categories of bronchial truncation, inflation, and straightness is relatively small. In order to enable the classification model to accurately learn the characteristics of each type of nodule, we perform data enhancement on these minority categories. The specific method is to flip the image horizontally and vertically. The classification model selects EfficientNet, and the classification network is trained using the enhanced data set.

[0052] Application stage: The data set obtained from the detection task is processed in a certain way. The original image is cropped according to the detected area. The cropped image is inferred through the trained EfficientNet to obtain a 13-dimensional category probability distribution vector. The above operation is performed on all CT images of a patient. Finally, these category probability feature vectors are spliced ​​to obtain a category probability matrix for subsequent benign and malignant classification tasks.

[0053] Furthermore, the experimental parameter setting system divides the entire data set in a ratio of 8:2, with 80% of the case data as the training set and 20% of the case data as the test set. In the upper and lower branch network modules, 10 pictures of each patient are spliced ​​in the channel dimension as picture features and input into the lower branch network. The category features of these 10 pictures are spliced ​​to obtain a category matrix and input into the upper branch network. The size features of the lung nodules are added before the final classifier. The sizes of various features are shown in Table 1. This experiment was completed on a server with two 1080 graphics cards, 8 CPU cores and 128G of running memory. The parameter configuration of the model during training is shown in Table 2:

[0054] Table 1 Dimensions of input features

[0055]

[0056] Table 2 Training parameter settings

[0057]

[0058]

[0059] Furthermore, the evaluation index system uses accuracy and F1 score as evaluation indicators to evaluate the classification effect of the designed classification model on the real-time classification of benign and malignant pulmonary nodules. The higher the ACC and F1 index values, the better the classification effect. The calculation formula is as follows:

[0060]

[0061]

[0062]

[0063]

[0064] Where: TP represents that both the predicted value and the true value are positive, TN represents that both the predicted value and the true value are negative, and these two categories are predicted correctly, FN represents that the predicted value is negative and the true value is positive, and FP represents that the predicted value is positive and the true value is negative.

[0065] Furthermore, in order to verify the necessity of integrating clinical experience in the task of classifying benign and malignant lung nodules, a set of control experiments was set up. Experiment one used the lower branch in the two-branch network for the experiment, and experiment two used the entire two-branch network for the experiment. Both experiments integrated the size characteristics of the nodules in the fully connected layer. For the task of classifying benign and malignant lung nodules, the two-branch network that integrated clinical experience had better effect, with the F1 score and accuracy increased by 5.8% and 6.8% respectively, which was significantly higher than the lower branch network without integrating clinical experience.

[0066] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A deep learning model system for diagnosis of benign and malignant solid pulmonary nodules, characterized by: The system includes an overall architecture system of the model, a data set introduction system, a data set preprocessing system, a lung CT image segmentation system, a lung nodule detection system, a lung nodule classification system, an experimental parameter setting system, an evaluation index system, and an experimental setting and result system. The overall architecture system of the model consists of an upper-branch network module and a lower-branch network module. The upper-branch network module uses a deep neural network composed of a residual network Resnet and a self-attention mechanism to extract features from a category matrix. The lower-branch network module uses a multi-scale deep neural network to extract features from a picture composed of ten pictures, and extracts the location and shape information of nodules. Finally, the nodule size features are normalized and input into the penultimate layer, and are spliced ​​with the category features and the extracted nodule image features, and are input into the final fully connected layer with an activation function to obtain the predicted probability of malignancy of the lung nodules. The lower half branch network module includes an Inception module, an ECA module, and a CBAM module, and the Inception module is the core module of GoogLeNet. The Inception module extracts multi-scale features from the input feature map by using convolution kernels of different sizes, and then splices these feature maps with multi-scale information in the channel dimension. Subsequently, further convolution operations can be performed to fuse multi-scale feature information. The Inception module uses a 1×1 convolution kernel to first reduce the dimension of the feature map, and then uses multiple small convolution kernels to replace the large convolution kernel, which greatly reduces the parameters and calculation amount of the network and improves the efficiency of model training. In addition, residual connections are used in the lower branch of Inception, which can make the model converge faster and avoid the problem of gradient disappearance to a certain extent. The ECA module is improved based on the defects of SENet, and the ECA module replaces the two-layer fully connected operations in the SE module with a convolution operation. Compared with the fully connected operation, the convolution operation can share parameters, reduce the number of model parameters, and provide appropriate channel interaction. The CBAM module is an advanced attention mechanism, and the CBAM module consists of two core parts: Composition: Channel attention and spatial attention are designed to refine the importance of features from the channel dimension and spatial dimension respectively, realize adaptive weighting of features, and thus improve the recognition accuracy and generalization ability of the network. The channel attention generates two channel feature vectors by performing global average pooling and global maximum pooling on the feature map. These two pooling operations can capture the statistical information between channels and reflect the overall distribution and salient parts of the features. Subsequently, these two feature vectors pass through a series of fully connected layers and a Sigmoid function to generate a normalized weight vector to indicate the importance of each channel. In the back propagation process, these weights guide the network to pay more attention to channels that are more discriminative for the task and effectively suppress irrelevant or noise information. The spatial attention module focuses on the spatial dimension, extracts the correlation of spatial features by applying two one-dimensional convolutions to each channel (along the width and height directions respectively), and combines the ReLU activation function. After that, a two-dimensional weight score is generated by the Sigmoid function, which represents the contribution of each position in the feature map to the final decision.

2. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The data set introduction system uses a lung CT data set provided by Changhai Hospital containing 1,279 cases, of which 388 are benign cases and 899 are malignant cases. The data set contains real-time, continuous Dicom data, which can reflect the specific conditions of the patient's lungs. The doctor selects multiple continuous lung CT images with nodules from the patient's Dicom data and puts them into a folder. The folder is used as the key diagnostic data of the case. Each folder contains an original lung CT image and a labeled lung CT image annotated by the doctor. The diagnostic information of the case is given by experienced doctors through comprehensive evaluation of the lung CT images in the corresponding folder. Based on clinical experience, the doctor divides the lung nodules in the lung CT images into 13 categories, including clear and smooth, halo sign, clear and rough, lobed, spinous process, burr, pleural retraction, bronchial truncation, vascular bundle, cavitation, inflation, multi-nodule aggregation, and straight. All lung CT images of each case are labeled with categories, and the nodule area is marked with a red circle.

3. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The data set preprocessing system uses the TransUnet network to preprocess the data set, and then uses the processed lung CT images to train the YOLOv5 model to detect whether the CT images contain nodules, and detects all real-time Dicom data of a patient according to the trained detection model, and detects 10 consecutive Dicom data with lung nodules as the data set for subsequent real-time classification of benign and malignant lung nodules, and records the detection results of these 10 images in a txt file. The information in the txt file can be used to calculate the position of the lung nodules in the CT image, and this part of the nodule area is cropped on the original Dicom data, and the cropped images are classified using the pre-trained lung nodule thirteen classification model EfficientNet, and the output category probability distribution is used as the category feature vector.

4. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The lung CT image segmentation system summarizes the lung CT images of all cases in the data set, and in order to avoid the interference of some text information in the lung nodule image on the detection and classification of nodules, the trained lung segmentation model TransUnet is used to segment the lung CT images to obtain the contour image of the lung CT image. The pixel values ​​in the contour image are 0 or 1, and the pixels in the contour and its interior are 1, and the remaining pixel values ​​are 0. The original image and the Mask image are multiplied pixel by pixel to obtain a preliminarily processed image, which is used for subsequent detection and classification tasks.

5. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The pulmonary nodule detection system uses the YOLOv5 algorithm. The YOLOv5 is a classic single-stage target detection algorithm with high detection accuracy and fast inference speed. It is now widely used in various target detection tasks. The YOLOv5 algorithm trains and detects nodules in lung CT images, and detects the original Dicom data based on the trained YOLOv5 model, and automatically selects the Dicom data with nodules. The pulmonary nodule target detection is divided into a training stage and an application stage.

6. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1, characterized in that: The lung nodule classification system is trained to automatically identify thirteen specific categories of lung nodules in medical images. The categories are precisely divided based on clinical experience to ensure the comprehensiveness and accuracy of the model classification. Subsequently, this classification model performs expert-level classification of the target detected nodule areas. The specific process of the thirteen classifications is divided into a training phase and an application phase.

7. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: Experimental parameter setting The system divides the entire data set in a ratio of 8:2, with 80% of the case data as the training set and 20% of the case data as the test set. In the upper and lower branch network modules, 10 pictures of each patient are spliced ​​in the channel dimension as picture features and input into the lower branch network. The category features of these 10 pictures are spliced ​​to obtain a category matrix and input into the upper branch network. The size features of the lung nodules are added before the final classifier. The sizes of various features are shown in Table 1. This experiment was completed on a server with two 1080 graphics cards, 8 CPU cores and 128G of running memory. The parameter configuration of the model during training is shown in Table 2: Table 1 Dimensions of input features Table 2 Training parameter settings 8. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1, characterized in that: The evaluation index system uses accuracy and F1 score as evaluation indicators to evaluate the classification effect of the designed classification model on real-time benign and malignant pulmonary nodules. The higher the ACC and F1 index values, the better the classification effect. The calculation formula is as follows: Where: TP represents that both the predicted value and the true value are positive, TN represents that both the predicted value and the true value are negative, and these two categories are predicted correctly, FN represents that the predicted value is negative and the true value is positive, and FP represents that the predicted value is positive and the true value is negative.

9. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1, characterized in that: The experimental setup and results system set up a set of control experiments to verify the necessity of integrating clinical experience in the task of classifying benign and malignant pulmonary nodules. Experiment one used the lower branch in the two-branch network for the experiment, and experiment two used the entire two-branch network for the experiment. Both experiments integrated the size features of the nodules in the fully connected layer.

Images