Deep learning model system for diagnosis of benign and malignant solid pulmonary nodules

Abstract

The present invention belongs to the technical field of model systems for diagnosing benign and malignant solid pulmonary nodules, and specifically relates to a deep learning model system for diagnosing benign and malignant solid pulmonary nodules, including an overall architecture system of the model, a data set introduction system, a data set preprocessing system, a lung CT image segmentation system, a lung nodule detection system, a lung nodule classification system, an experimental parameter setting system, an evaluation index system, and an experimental setting and result system. The present invention proposes a deep learning model system for diagnosing benign and malignant solid pulmonary nodules, and is mainly designed for the lower branch of the network. The lower branch network uses the Inception module to extract multi-scale feature information of the lung CT image, and introduces convolutional attention modules such as ECA and CBAM to enhance the extraction effect of the lower branch network on the position and shape features of the lung CT image.

Description

Technical Field

[0001] The present invention belongs to the technical field of model systems for diagnosing benign and malignant solid pulmonary nodules, and specifically relates to a deep learning model system for diagnosing benign and malignant solid pulmonary nodules. Background Art

[0002] The benign resection rate for lung nodules is generally expected to be around 10%, but the actual clinical benign resection rate is over 30%. Most benign resectable lesions appear as solid lesions on lung CT scans, making the excessively high benign resection rate for solid lung nodules an unavoidable clinical problem. The core of this problem is that the prognosis of solid lung cancer is far worse than that of subsolid lung cancer (ground-glass opacity). Physicians must avoid missed diagnoses while resisting the urge to perform unnecessary surgeries. The lack of precise tools and methods for the definitive diagnosis of lung cancer in clinical practice makes diagnostic surgery the most reliable option. The key to resolving this issue lies in identifying effective methods and tools for the definitive diagnosis of lung cancer, providing a valuable reference for preoperative decision-making. CT remains a clinically available, cost-effective, and effective tool for the definitive diagnosis of lung cancer. However, its ultimate performance in diagnosing benign and malignant solid lung nodules is unclear, and its ability to provide sufficient support for surgical decisions is even less clear.

[0003] Existing algorithms for diagnosing benign and malignant lung nodules are mainly divided into two categories: one is machine learning algorithms (ML) and the other is deep learning algorithms (DL). Machine learning algorithms are traditional, two-stage algorithms that require manual selection of features closely related to the task of diagnosing benign and malignant lung nodules. Then, appropriate machine learning algorithms are used to classify these features, thereby achieving the effect of diagnosing benign and malignant lung nodules. Warkentin et al. [1] used cross-validation and grid search methods to evaluate the ability of three machine learning algorithms, eXtreme Gradient Boosted Trees, random forest, and LASSO, to classify benign and malignant lung nodules. The results showed that the least absolute shrinkage and selection operator (LASSO) had the best classification effect and could accurately predict the benign and malignant nature of lung nodules. In response to the problem of poor performance of a single machine learning algorithm, Muzammil et al. proposed an ensemble learning algorithm based on the average of prediction scores and the maximum image. It can integrate the classification decisions of the two algorithms, SVM (support vector machine) and AdaBoostM2, and can more effectively predict the benign and malignant nature of lung nodules.

[0004] However, the quality of manually extracted features largely determines the performance of machine learning classification algorithms, requiring experts to spend considerable time manually extracting these key features. For example, in the diagnosis of benign and malignant lung nodules, the nodule's morphological and texture characteristics, size, and each patient's original data (age, gender, etc.) are all important artificial features, and obtaining these features requires significant labor costs. Furthermore, the classifiers used in machine learning algorithms are often relatively shallow, making it difficult to extract information from complex, high-dimensional data and prone to underfitting and overfitting issues.

[0005] As researchers continue to delve into artificial intelligence technology, deep learning algorithms have achieved significant success in fields such as computer vision, speech recognition, and natural language processing. From the remarkable success of the convolutional neural network AlexNet on the ImageNet dataset to the current large language model technology based on the Transformer architecture, deep neural network technology has become increasingly mature and its applications are becoming increasingly widespread. Its powerful feature extraction capabilities have enabled it to be widely used in many fields. End-to-end deep learning algorithms do not require manual feature extraction and can learn complex feature information. Consequently, a large number of studies have focused on deep learning-based methods to address the problem of diagnosing benign and malignant lung nodules. Yin et al. proposed a multimodal feature fusion network to classify benign and malignant lung nodules. Using MobileNetV3 as the backbone network, they designed a dual-branch network capable of effectively extracting features from both the original CT images of lung nodules and the extracted nodule images. This network not only extracts global information from the original CT images but also local feature information from the lung nodules. Song et al. [7] based on the multi-task learning method, by mining the potential common features between the tasks of lung nodule detection and lung nodule segmentation, enabled each task to complete training and learning efficiently, thus improving the effects of lung nodule detection and segmentation. Cao et al. proposed the MSM-ViT network based on CNN and MobileViT network. First, they used dilated convolution and ordinary convolution to extract the multi-scale local information of lung nodules to make up for the deficiency of the later ViT model in extracting local information. Then, they used the MovileViT module to extract the features of the global information of lung nodules. This model can effectively extract feature information at all scales and improve the accuracy of benign and malignant classification of lung nodules. Zhang et al. used the deep learning model of ResNet combined with convolutional attention module (CBAM) to classify the benign and malignant lung nodules based on graphics, morphological features, and clinical information, and compared it with the traditional machine learning algorithm NSDTCT-SVM. The results showed that the deep learning model combined with additional information is better than the traditional machine learning algorithm. Summary of the Invention

[0006] The purpose of this invention is to provide a deep learning model system for diagnosing benign and malignant solid pulmonary nodules. The system uses two independent branches to extract clinical experience and feature information from lung CT images, and effectively integrates the two to achieve more accurate predictions of benign and malignant pulmonary nodules. This combination not only fully utilizes the advantages of deep learning algorithms in feature extraction and classification, but also incorporates the clinical experience and professional knowledge of doctors, providing a more comprehensive and reliable solution for the diagnosis of pulmonary nodules.

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a deep learning model system for diagnosing benign and malignant solid pulmonary nodules, comprising an overall architecture system of the model, a data set introduction system, a data set preprocessing system, a lung CT image segmentation system, a lung nodule detection system, a lung nodule classification system, an experimental parameter setting system, an evaluation index system, and an experimental setting and result system. The overall architecture system of the model consists of an upper-half branch network module and a lower-half branch network module. The upper-half branch network module uses a deep neural network composed of a residual network Resnet and a self-attention mechanism to extract features from a category matrix. The lower-half branch network module uses a multi-scale deep neural network to extract features from an image composed of ten images, and extracts the location and shape information of the nodules. Finally, the nodule size features are normalized and input into the penultimate layer, where they are concatenated with the category features and the extracted nodule image features, and then input into the final fully connected layer with an activation function to obtain the predicted probability of malignancy of the lung nodules.

[0008] The lower half branch network module includes the Inception module, the ECA module, the CBAM module, and the Inception module is the core module of GoogLeNet. The Inception module performs multi-scale feature extraction on the input feature map by using convolution kernels of different sizes, and then splices these feature maps with multi-scale information in the channel dimension. Subsequently, further convolution operations can be performed to fuse multi-scale feature information. The Inception module uses a 1×1 convolution kernel to first perform a dimensionality reduction operation on the feature map, and then uses multiple small convolution kernels instead of large convolution kernels, which greatly reduces the network parameters and computational complexity and improves the efficiency of model training. In addition, residual connections are used in the lower branch of Inception, which can make the model converge faster and avoid the problem of gradient disappearance to a certain extent.

[0009] The ECA module is an improvement based on the defects of SENet. The ECA module replaces the two-layer fully connected operations in the SE module with a convolution operation. Compared with the fully connected operation, the convolution operation can share parameters, reduce the number of model parameters, and provide appropriate channel interaction.

[0010] The CBAM module is an advanced attention mechanism, and the CBAM module consists of two core parts: channel attention and spatial attention, which aim to refine the importance of features from the channel dimension and spatial dimension respectively, realize adaptive weighting of features, and thus improve the recognition accuracy and generalization ability of the network. The channel attention generates two channel feature vectors by performing global average pooling and global maximum pooling on the feature map. These two pooling operations can capture statistical information between channels and reflect the overall distribution and prominence of the features. Subsequently, these two feature vectors pass through a series of fully connected layers and a Sigmoid function to produce a normalized weight vector to indicate the importance of each channel. During the backpropagation process, these weights guide the network to pay more attention to those channels that are more discriminative for the task and effectively suppress irrelevant or noisy information. The spatial attention module focuses on the spatial dimension by applying two one-dimensional convolutions to each channel (along the width and height directions respectively) combined with the ReLU activation function to extract the correlation of spatial features. Afterwards, a two-dimensional weight score is generated by the Sigmoid function, which represents the contribution of each position in the feature map to the final decision.

[0011] Preferably, the data set introduction system uses a lung CT data set provided by Changhai Hospital containing 1,279 cases, of which 388 are benign cases and 899 are malignant cases. The data set contains real-time, continuous Dicom data, which can reflect the specific condition of the patient's lungs. The doctor selects multiple continuous lung CT images with nodules from the patient's Dicom data and puts them into a folder. The contents of the folder are used as key diagnostic data for the case. Each folder contains an original lung CT image and a labeled lung CT image annotated by the doctor. The diagnostic information of the case is given by experienced doctors through comprehensive evaluation of the lung CT images in the corresponding folder. Based on clinical experience, the doctor divides the lung nodules in the lung CT images into 13 categories, including clear and smooth, halo sign, clear and rough, lobed, spinous process, burr, pleural indentation, bronchial truncation, vascular bundle, cavitation, inflation, multi-nodule aggregation, and straight. All lung CT images of each case are labeled, and the nodule area is marked with a red circle.

[0012] Preferably, the data set preprocessing system uses the TransUnet network to preprocess the data set, and then uses the processed lung CT images to train the YOLOv5 model to detect whether the CT images contain nodules, and uses the trained detection model to detect all real-time Dicom data of a patient, and detects 10 consecutive Dicom data with lung nodules as a data set for subsequent real-time classification of benign and malignant lung nodules, and records the detection results of these 10 images in a txt file. The information in the txt file can be used to calculate the position of the lung nodules in the CT image, crop this part of the nodule area on the original Dicom data, and use the pre-trained lung nodule thirteen classification model EfficientNet to classify the cropped images, and output the category probability distribution as the category feature vector.

[0013] Preferably, the lung CT image segmentation system summarizes the lung CT images of all cases in the data set, and in order to avoid interference of some text information in the lung nodule image on the detection and classification of nodules, the trained lung segmentation model TransUnet is used to segment the lung CT image to obtain a contour image of the lung CT image. The pixel values ​​in the contour image are 0 or 1, and the pixels in the contour and its interior are 1, and the remaining pixel values ​​are 0. The original image and the mask image are multiplied pixel by pixel to obtain a preliminarily processed image, which is used for subsequent detection and classification tasks.

[0014] Preferably, the lung nodule detection system uses the YOLOv5 algorithm. The YOLOv5 is a classic single-stage target detection algorithm with high detection accuracy and fast inference speed. It is now widely used in various target detection tasks. The YOLOv5 algorithm trains and detects nodules in lung CT images, and detects the original Dicom data based on the trained YOLOv5 model, automatically selecting the Dicom data with nodules. The lung nodule target detection is divided into a training stage and an application stage.

[0015] Preferably, the lung nodule classification system is trained to automatically identify thirteen specific categories of lung nodules in medical images. The categories are precisely divided based on clinical experience to ensure the comprehensiveness and accuracy of the model classification. Subsequently, this classification model performs expert-level classification on the target detected nodule areas. The specific process of the thirteen classifications is divided into a training stage and an application stage.

[0016] Preferably, the experimental parameter setting system divides the entire data set into an 8:2 ratio, with 80% of the case data used as a training set and 20% of the case data used as a test set. In the upper and lower branch network modules, 10 images of each patient are spliced ​​in the channel dimension as image features and input into the lower branch network. The category features of these 10 images are spliced ​​to obtain a category matrix and input into the upper branch network. The size features of the lung nodules are added before the final classifier. The sizes of various features are shown in Table 1. This experiment was completed on a server with two 1080 graphics cards, 8 CPU cores and 128G of running memory. The parameter configuration of the model during training is shown in Table 2:

[0017] Table 1 Input feature dimensions

[0018]

[0019] Table 2 Training parameter settings

[0020]

[0021] Preferably, the evaluation index system uses accuracy and F1 score as evaluation indicators to evaluate the classification effect of the designed classification model on real-time benign and malignant pulmonary nodules. The higher the ACC and F1 index values, the better the classification effect. The calculation formula is as follows:

[0022]

[0023]

[0024]

[0025]

[0026] Where: TP represents that both the predicted value and the true value are positive, TN represents that both the predicted value and the true value are negative, and these two categories are correct predictions, FN represents that the predicted value is negative and the true value is positive, and FP represents that the predicted value is positive and the true value is negative.

[0027] Preferably, the experimental setting and results system set up a set of control experiments to verify the necessity of reintegrating clinical experience in the task of benign and malignant lung nodule classification. Experiment one used the lower branch in the two-branch network for the experiment, and experiment two used the entire two-branch network for the experiment. Both experiments integrated the size features of the nodules in the fully connected layer.

[0028] Compared with the existing technology, the beneficial effects of the present invention are as follows: the present invention proposes a deep learning model system for diagnosing benign and malignant solid lung nodules, and is mainly designed for the lower branch of the network. The lower branch network uses the Inception module to extract multi-scale feature information of lung CT images, and introduces convolutional attention modules such as ECA and CBAM to enhance the extraction effect of the lower branch network on the position and shape features of lung CT images. The upper branch network effectively extracts the category information of lung nodules through the Resnet and Self-Attention modules. Experimental results show that in the real-time classification task of benign and malignant lung nodules, the deep learning-based method based on the integration of clinical experience proposed in this paper can meet the requirements of clinical diagnosis in terms of evaluation indicators such as F1 and accuracy, and can be used as an auxiliary means to help doctors improve the accuracy of diagnosing benign and malignant lung nodules. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the overall architecture diagram of the dual-branch network integrating clinical experience of the present invention.

[0030] Figure 2 This is a schematic diagram of the network structure of the Inception module of the present invention.

[0031] Figure 3 This is a schematic diagram of the network structure of the ECA attention module of the present invention,

[0032] Figure 4 This is a schematic diagram of the network structure of the CBAM attention module of the present invention,

[0033] Figure 5 is a schematic diagram of the number of pictures contained in the case folder of the present invention,

[0034] Figure 6 is the overall flow chart of data processing of the present invention,

[0035] Figure 7 This is a schematic diagram of a picture obtained by merging the original picture and the segmented picture of the present invention.

[0036] Figure 8 Schematic diagram of different types of lung nodules obtained by cutting out the present invention.

[0037] Figure 9 Schematic diagram of the number of different nodule categories of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] Please see the attached Figure 1-9 , the present invention provides several embodiments:

[0040] A deep learning model system for diagnosing benign and malignant solid lung nodules, including the model's overall architecture system, dataset introduction system, dataset preprocessing system, lung CT image segmentation system, lung nodule detection system, lung nodule classification system, experimental parameter setting system, evaluation index system, and experimental setting and results system. The model's overall architecture system consists of an upper-branch network module and a lower-branch network module. The upper-branch network module uses a deep neural network composed of a residual network Resnet and a self-attention mechanism to extract features from the category matrix. The lower-branch network module uses a multi-scale deep neural network to extract features from an image composed of ten images, and extracts the location and shape information of the nodules. Finally, the nodule size features are normalized and input into the penultimate layer, where they are concatenated with the category features and the extracted nodule image features, and then input into the final fully connected layer with an activation function to obtain the predicted probability of malignancy of the lung nodules.

[0041] The lower half of the network includes the Inception module, the ECA module, and the CBAM module. The Inception module is the core module of GoogLeNet. The Inception module uses convolution kernels of different sizes to extract multi-scale features from the input feature map, and then splices these feature maps with multi-scale information in the channel dimension. Subsequently, further convolution operations can be performed to fuse multi-scale feature information. The Inception module uses a 1×1 convolution kernel to first reduce the dimension of the feature map, and then uses multiple small convolution kernels instead of large convolution kernels, which greatly reduces the network parameters and computational complexity and improves the efficiency of model training. In addition, the residual connection is used in the lower branch of Inception, which can make the model converge faster and avoid the problem of gradient disappearance to a certain extent.

[0042] The ECA module is an improvement based on the defects of SENet. The ECA module replaces the two-layer fully connected operations in the SE module with a convolution operation. Compared with the fully connected operation, the convolution operation can share parameters, reduce the number of model parameters, and provide appropriate channel interaction.

[0043] The CBAM module is an advanced attention mechanism, and the CBAM module consists of two core parts: channel attention and spatial attention, which aims to refine the importance of features from the channel dimension and spatial dimension respectively, realize adaptive weighting of features, and thus improve the recognition accuracy and generalization ability of the network. Channel attention generates two channel feature vectors by performing global average pooling and global maximum pooling on the feature map. These two pooling operations can capture statistical information between channels and reflect the overall distribution and prominence of the features. Subsequently, the two feature vectors pass through a series of fully connected layers and a Sigmoid function to produce a normalized weight vector to indicate the importance of each channel. In the backpropagation process, these weights guide the network to pay more attention to those channels that are more discriminative for the task, effectively suppressing irrelevant or noise information. The spatial attention module focuses on the spatial dimension, and extracts the correlation of spatial features by applying two one-dimensional convolutions to each channel (along the width and height directions respectively) combined with the ReLU activation function. Afterwards, a two-dimensional weight score is generated by the Sigmoid function. The score represents the contribution of each position in the feature map to the final decision. The structure of the entire CBAM is as follows Figure 4 As shown in Figure 2, through the channel attention and spatial attention mechanisms, the deep network model can more effectively extract important and key features from lung CT images.

[0044] Furthermore, the dataset introduction system uses a lung CT dataset provided by Changhai Hospital containing 1,279 cases, of which 388 are benign cases and 899 are malignant cases. The dataset contains real-time, continuous Dicom data, which can reflect the specific condition of the patient's lungs. The doctor selects multiple continuous lung CT images with nodules from the patient's Dicom data and puts them into a folder. The folder is used as the key diagnostic data of the case. Each folder contains an original lung CT image and a labeled lung CT image annotated by the doctor. The diagnostic information of the case is given by experienced doctors through comprehensive evaluation of the lung CT images in the corresponding folder. Based on clinical experience, the doctor divides the lung nodules in the lung CT images into 13 categories, including clear and smooth, halo sign, clear and rough, lobed, spinous process, burr, pleural retraction, bronchial truncation, vascular bundle, cavitation, inflation, multi-nodule aggregation, and straight. All lung CT images of each case are labeled, and the nodule area is marked with a red circle. The basic structure of the dataset is as follows Figure 5 shown.

[0045] Furthermore, the dataset preprocessing system uses the TransUnet network to preprocess the dataset, and then uses the processed lung CT images to train the YOLOv5 model to detect whether the CT images contain nodules, and uses the trained detection model to detect all real-time Dicom data of a patient, and detects 10 consecutive Dicom data with lung nodules as the dataset for subsequent real-time classification of benign and malignant lung nodules, and records the detection results of these 10 images in a txt file. The information in the txt file can be used to calculate the position of the lung nodules in the CT image, crop this part of the nodule area on the original Dicom data, and use the pre-trained lung nodule thirteen classification model EfficientNet to classify the cropped images, and output the category probability distribution as the category feature vector.

[0046] Furthermore, the lung CT image segmentation system summarizes the lung CT images of all cases in the dataset, and in order to avoid the interference of some text information in the lung nodule images on the detection and classification of nodules, the trained lung segmentation model TransUnet is used to segment the lung CT images to obtain the contour images of the lung CT images. The pixel values ​​in the contour image are 0 or 1, and the pixels within the contour and its interior are 1, and the remaining pixel values ​​are 0. The original image and the mask image are multiplied pixel by pixel to obtain the preliminarily processed image, which is used for subsequent detection and classification tasks.

[0047] Furthermore, the lung nodule detection system uses the YOLOv5 algorithm. YOLOv5 is a classic single-stage target detection algorithm with high detection accuracy and fast inference speed. It is now widely used in various target detection tasks. The YOLOv5 algorithm trains and detects nodules in lung CT images, and detects the original Dicom data based on the trained YOLOv5 model, automatically selecting Dicom data with nodules. Lung nodule target detection is divided into training and application stages:

[0048] Training phase: In this detection task, the goal is to determine whether a lung CT image contains nodules. Segmented lung nodule images are used as training and testing data for the YOLOv5 model. Labeling the original lung CT images with the corresponding annotated lung CT images in the original dataset is performed using LabelImg software. The only label used here is "nodule." Since the segmented image outlines are relatively precise, and most lung nodules are located within these outlines, the resulting labels can be used directly as labels for the segmented lung CT images. The YOLOv5 model is trained using this data.

[0049] Application Phase: Each patient's real-time DICOM data is preprocessed using the TransUNet network. This involves element-wise multiplication of the segmented image with the original image to obtain an image type suitable for YOLOv5 inference. The trained YOLOv5 model is then used to perform inference on each image. Because the patient's lung CT images containing lung nodules are continuous, we use data from images that appear consecutively for more than ten consecutive images as the dataset for subsequent real-time benign and malignant classification.

[0050] Furthermore, the lung nodule classification system has been trained to automatically identify thirteen specific categories of lung nodules in medical images. These categories are precisely divided based on clinical experience to ensure the comprehensiveness and accuracy of the model classification. Subsequently, this classification model performs expert-level classification on the target nodule areas detected. The specific process of the thirteen classifications is divided into the training phase and the application phase:

[0051] Training phase: Based on the label information obtained above, cropping is performed on the original lung CT image. Since each CT image corresponds to this expert-level nodule category label, the category label is used as the category label of the cropped nodule area. The cropped image has been labeled as follows: Figure 8 The dataset consisting of the cropped images and the corresponding category labels is used as the dataset for training the 13-classification model of lung nodules. The number of images in each category is preliminarily counted, as shown in the figure below. Figure 9 As shown in the figure, the number of nodules in the three categories of bronchial truncation, air-filled, and straight is relatively small. To enable the classification model to accurately learn the characteristics of each nodule type, we perform data augmentation on these minority categories by flipping the images horizontally and vertically. We selected EfficientNet as the classification model and trained the classification network using the augmented dataset.

[0052] Application phase: The dataset obtained from the detection task is processed, and the original image is cropped according to the detected area. The cropped image is inferred through the trained EfficientNet to obtain a 13-dimensional category probability distribution vector. The above operation is performed on all CT images of a patient. Finally, these category probability feature vectors are spliced ​​to obtain a category probability matrix for subsequent benign and malignant classification tasks.

[0053] Furthermore, the experimental parameter setting system divides the entire dataset into an 8:2 ratio, with 80% of the case data used as the training set and 20% of the case data used as the test set. In the upper and lower branch network modules, 10 images of each patient are spliced ​​in the channel dimension as image features and input into the lower branch network. The category features of these 10 images are spliced ​​to obtain a category matrix and input into the upper branch network. The size features of the lung nodules are added before the final classifier. The sizes of various features are shown in Table 1. This experiment was completed on a server with two 1080 graphics cards, 8 CPU cores and 128G of running memory. The parameter configuration of the model during training is shown in Table 2:

[0054] Table 1 Input feature dimensions

[0055]

[0056] Table 2 Training parameter settings

[0057]

[0058] Furthermore, the evaluation index system uses accuracy and F1 score as evaluation indicators to evaluate the classification effect of the designed classification model on real-time benign and malignant pulmonary nodules. The higher the ACC and F1 index values, the better the classification effect. The calculation formula is as follows:

[0059]

[0060]

[0061]

[0062]

[0063] Where: TP represents that both the predicted value and the true value are positive, TN represents that both the predicted value and the true value are negative, and these two categories are correct predictions, FN represents that the predicted value is negative and the true value is positive, and FP represents that the predicted value is positive and the true value is negative.

[0064] Furthermore, in order to verify the necessity of integrating clinical experience in the task of benign and malignant lung nodule classification, the experimental setting and results system set up a set of control experiments. Experiment one used the lower branch in the two-branch network for the experiment, and experiment two used the entire two-branch network for the experiment. Both experiments integrated the size features of the nodules in the fully connected layer.

[0065] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A deep learning model system for diagnosing benign and malignant solid pulmonary nodules, characterized by: The system includes an overall architecture system of the model, a data set introduction system, a data set preprocessing system, a lung CT image segmentation system, a lung nodule detection system, a lung nodule classification system, an experimental parameter setting system, an evaluation index system, and an experimental setting and result system. The overall architecture system of the model consists of an upper-branch network module and a lower-branch network module. The upper-branch network module uses a deep neural network composed of a residual network Resnet and a self-attention mechanism to extract features from the category matrix. The lower-branch network module uses a multi-scale deep neural network to extract features from an image composed of ten images, and extracts the location and shape information of the nodules. Finally, the nodule size features are normalized and input into the penultimate layer, where they are concatenated with the category features and the extracted nodule image features, and then input into the final fully connected layer with an activation function to obtain the predicted probability of malignancy of the lung nodules. The lower half branch network module includes the Inception module, the ECA module, the CBAM module, and the Inception module is the core module of GoogLeNet. The Inception module performs multi-scale feature extraction on the input feature map by using convolution kernels of different sizes, and then splices these feature maps with multi-scale information in the channel dimension. Subsequently, further convolution operations can be performed to fuse multi-scale feature information. The Inception module uses a 1×1 convolution kernel to first perform a dimensionality reduction operation on the feature map, and then uses multiple small convolution kernels instead of large convolution kernels, which greatly reduces the network parameters and computational complexity and improves the efficiency of model training. In addition, residual connections are used in the lower branch of Inception, which can make the model converge faster and avoid the problem of gradient disappearance to a certain extent. The ECA module is an improvement based on the defects of SENet. The ECA module replaces the two-layer fully connected operations in the SE module with a convolution operation. Compared with the fully connected operation, the convolution operation can share parameters, reduce the number of model parameters, and provide appropriate channel interaction. The CBAM module is an advanced attention mechanism, and the CBAM module consists of two core parts: Composition: Channel attention and spatial attention are designed to refine the importance of features from the channel dimension and spatial dimension respectively, realize adaptive weighting of features, and thus improve the recognition accuracy and generalization ability of the network. The channel attention generates two channel feature vectors by performing global average pooling and global maximum pooling on the feature map. These two pooling operations can capture statistical information between channels and reflect the overall distribution and prominent parts of the features. Subsequently, the two feature vectors pass through a series of fully connected layers and a Sigmoid function to generate a normalized weight vector to indicate the importance of each channel. In the backpropagation process, these weights guide the network to pay more attention to those channels that are more discriminative for the task and effectively suppress irrelevant or noise information. The spatial attention module focuses on the spatial dimension by applying two one-dimensional convolutions to each channel (along the width and height directions respectively) combined with the ReLU activation function to extract the correlation of spatial features. Afterwards, a two-dimensional weight score is generated by the Sigmoid function. This score represents the contribution of each position in the feature map to the final decision.

2. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The dataset introduction system uses a lung CT dataset provided by Changhai Hospital containing 1,279 cases, of which 388 are benign cases and 899 are malignant cases. The dataset contains real-time, continuous Dicom data, which can reflect the specific condition of the patient's lungs. The doctor selects multiple continuous lung CT images with nodules from the patient's Dicom data and puts them into a folder. The contents of the folder are used as the key diagnostic data of the case. Each folder contains an original lung CT image and a labeled lung CT image annotated by the doctor. The diagnostic information of the case is given by experienced doctors through comprehensive evaluation of the lung CT images in the corresponding folder. Based on clinical experience, the doctor divides the lung nodules in the lung CT images into 13 categories, including clear and smooth, halo sign, clear and rough, lobed, spinous process, burr, pleural indentation, bronchial truncation, vascular bundle, cavitation, inflation, multi-nodule aggregation, and straight. All lung CT images of each case are labeled, and the nodule area is marked with a red circle.

3. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The dataset preprocessing system uses the TransUnet network to preprocess the dataset, and then uses the processed lung CT images to train the YOLOv5 model to detect whether the CT images contain nodules, and uses the trained detection model to detect all real-time Dicom data of a patient, and detects 10 consecutive Dicom data with lung nodules as the dataset for subsequent real-time benign and malignant classification of lung nodules, and records the detection results of these 10 images in a txt file. The information in the txt file can be used to calculate the position of the lung nodules in the CT image, and this part of the nodule area is cropped on the original Dicom data. The cropped image is classified using the pre-trained lung nodule thirteen classification model EfficientNet, and the output category probability distribution is used as the category feature vector.

4. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The lung CT image segmentation system summarizes the lung CT images of all cases in the data set, and in order to avoid the interference of some text information in the lung nodule image on the detection and classification of nodules, the trained lung segmentation model TransUnet is used to segment the lung CT images to obtain the contour image of the lung CT image. The pixel values ​​in the contour image are 0 or 1, and the pixels within the contour and its interior are 1, and the remaining pixel values ​​are 0. The original image and the mask image are multiplied pixel by pixel to obtain a preliminarily processed image, which is used for subsequent detection and classification tasks.

5. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: The lung nodule detection system uses the YOLOv5 algorithm. The YOLOv5 is a classic single-stage target detection algorithm with high detection accuracy and fast inference speed. It is now widely used in various target detection tasks. The YOLOv5 algorithm trains and detects nodules in lung CT images, and detects the original Dicom data based on the trained YOLOv5 model, automatically selecting the Dicom data with nodules. The lung nodule target detection is divided into a training stage and an application stage.

6. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1, characterized in that: The lung nodule classification system is trained to automatically identify thirteen specific categories of lung nodules in medical images. The categories are precisely divided based on clinical experience to ensure the comprehensiveness and accuracy of the model classification. Subsequently, this classification model performs expert-level classification on the target detected nodule areas. The specific process of the thirteen classifications is divided into a training phase and an application phase.

7. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1 is characterized by: Experimental parameter setting: The system divides the entire dataset into an 8:2 ratio, with 80% of the case data used as the training set and 20% of the case data used as the test set. In the upper and lower branch network modules, 10 images of each patient are spliced ​​in the channel dimension as image features and input into the lower branch network. The category features of these 10 images are spliced ​​to obtain a category matrix and input into the upper branch network. The size features of the lung nodules are added before the final classifier. The sizes of various features are shown in Table 1. This experiment was completed on a server with two 1080 graphics cards, 8 CPU cores, and 128G of running memory. The parameter configuration of the model during training is shown in Table 2: Table 1 Input feature dimensions ； Table 2 Training parameter settings 。 8. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1, characterized in that: The evaluation index system uses accuracy and F1 score as evaluation indicators to evaluate the classification effect of the designed classification model on real-time benign and malignant pulmonary nodules. The higher the ACC and F1 index values, the better the classification effect. The calculation formula is as follows: ； ； ； ； Where: TP represents that both the predicted value and the true value are positive, TN represents that both the predicted value and the true value are negative, and these two categories are correct predictions, FN represents that the predicted value is negative and the true value is positive, and FP represents that the predicted value is positive and the true value is negative.

9. The deep learning model system for diagnosing benign and malignant solid pulmonary nodules according to claim 1, characterized in that: The experimental setup and results system was designed to verify the necessity of integrating clinical experience in the task of classifying benign and malignant lung nodules. A set of control experiments was set up. Experiment one used the lower branch in the two-branch network for the experiment, and experiment two used the entire two-branch network for the experiment. Both experiments integrated the size features of the nodules in the fully connected layer.

Images