Pulmonary nodule classification method based on SE-ResNet and LSTM fusion

Abstract

The invention belongs to the field of computer vision, and discloses a pulmonary nodule classification method based on SE-ResNet and LSTM fusion, and the method comprises the following steps: 1, a data input stage; 2, a data preprocessing stage; step 3, an SE-ResNet feature extraction stage; step 4, a ResNet-Inception optimization stage is carried out; step 5, an LSTM layer lifting network stage; 6, a classification output stage; and 7, a model verification test stage. According to the method, the classification accuracy of benign and malignant pulmonary nodules is improved, and the frequency of wrong diagnosis is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically is a lung nodule classification method based on SE-ResNet and LSTM fusion. Background Art

[0002] In modern medicine, the classification of benign and malignant pulmonary nodules is an important step in accurately assessing lung health. It is not only crucial for diagnosing diseases, but also has an important impact on developing individualized treatment plans and optimizing patient management. With the rapid advancement of computer technology, especially breakthroughs in medical imaging and artificial intelligence, the classification of benign and malignant pulmonary nodules has been greatly improved.

[0003] The application of computer-aided diagnosis systems in this field mainly relies on the in-depth analysis of lung CT images. The classification of benign and malignant nodules not only depends on the size and shape of the nodules, but also needs to comprehensively consider the growth rate, edge characteristics and other imaging characteristics of the nodules. Through advanced image analysis technology, pattern recognition and machine learning algorithms, computer-aided diagnosis systems can automatically extract the morphological characteristics, texture characteristics and density information of lung nodules, and compare them with a large amount of annotated clinical data, and finally provide high-precision predictions about the nature of nodules, thereby greatly improving the accuracy of early diagnosis and assisting doctors in formulating the most appropriate treatment plan. The application of computer-aided diagnosis systems makes the classification and management of lung nodules more scientific and accurate.

[0004] In deep learning, the attention mechanism can focus on specific elements to better handle complex tasks and improve the performance and expressiveness of the model. The SE attention mechanism (Squeeze-and-Excitation Attention) mainly includes two stages: "squeeze" and "excitation". It is a technology used to enhance the performance of deep neural networks. The mechanism models in the channel dimension and adjusts the weights to increase the network's attention to important features, thereby improving the performance of the model in all aspects.

[0005] The residual network (ResNet) introduces direct cross-layer connections, making it easier for the network to learn the residual part, that is, the difference between the input and output, rather than directly learning the entire mapping function, and can retain the low-resolution information in the high-level feature map. Residual connections make it easier to train and optimize the network, provide more information transmission paths, enhance the information flow and expression ability of the network, and effectively alleviate the problem of gradient disappearance. This method allows the network to learn the subtle changes and subtle properties of nodules during the learning process, thereby better capturing and expressing the key features of nodules and improving the sensitivity and accuracy of nodule recognition.

[0006] The ResNet-Inception layer aims to improve the performance and efficiency of deep neural networks by fusing the residual connections of ResNet and the multi-branch structure of Inception to achieve this goal. The introduction of residual connections optimizes the overall structure of the network, overcomes the problem of gradient vanishing or exploding, and deepens the depth of network learning. The multi-branch structure of Inception improves the representation ability of the network, enabling it to better capture features of different scales and levels. Combining these two advantages, the ResNet-Inception layer can improve model performance while keeping the network structure running efficiently.

[0007] Long Short-Term Memory (LSTM) effectively alleviates the drawback of traditional RNNs that they only have short-term memory capabilities. The forget gate, input gate, and output gate are three key gate controllers that control the transmission of data, allowing the network to selectively remember or forget previous information and learn long-term dependencies. When using LSTM for image classification, the network treats each pixel as a time step and each row or column in the image as an element of the sequence. By introducing the LSTM model in the row or column direction of the image, the model can naturally learn long-term related information in the image.

[0008] Since other existing models still have many misdiagnosis problems when facing complex lung lesions and diverse imaging features, the models have certain limitations for nodules with blurred edges or low contrast, and the accuracy of the classification model still needs to be improved. The present invention proposes a lung nodule classification method based on SE-ResNet and LSTM fusion. First, the SE attention mechanism is introduced into the residual neural network to extract and integrate lung nodule features. Secondly, the LSTM model is fused on the basis of SE-ResNet to extract the contextual information of the target area sequence and enhance the network's ability to capture information about lung nodule features. In addition, a ResNet-Inception layer is added after the last maximum pooling, and a residual network is used for connection to optimize the model performance and achieve more accurate classification results.

[0009] The present invention improves the accuracy of lung nodule classification, reduces the misdiagnosis rate, and ensures that accurate benign and malignant judgments can be provided in different types of lung nodules, thereby effectively improving the quality of early diagnosis of lung cancer and the formulation of individualized treatment plans. Summary of the invention

[0010] The purpose of the present invention is to provide a lung nodule classification method based on SE-ResNet and LSTM fusion to solve the problems raised in the above background technology.

[0011] In order to achieve the above object, the present invention provides the following technical solution: a pulmonary nodule classification method based on SE-ResNet and LSTM fusion, comprising the following steps:

[0012] Step 1: Data input stage;

[0013] Step 2: Data preprocessing stage;

[0014] Step 3: SE-ResNet feature extraction stage;

[0015] Step 4: ResNet-Inception optimization phase;

[0016] Step 5: LSTM layer boosting network stage;

[0017] Step 6: Classification output stage;

[0018] Step 7: Model validation testing phase.

[0019] Preferably, the data input stage in step 1 includes the following specific steps:

[0020] Step 1_1: This input stage mainly screens the lung CT images. There are 888 different patients' lung CT image data, which are stored in 10 files from subset0.zip to subset9.zip. Each CT scan image contains multiple slices, and some slices that do not contain important information need to be screened out, so that the entire lung area can be fully displayed;

[0021] Step 1_2: Consider a dataset X with N types of data,

[0022] 80% of them are used as training data, X t ,

[0023] Take 20% as test data X s ,

[0024] Preferably, the pre-processing stage in step 2 comprises the following specific steps:

[0025] Step 2_1: Get the nodule location and category information from the candidates.csv file, read the nodule coordinates, and extract a cubic area of ​​size (48, 48, 48) with the coordinates as the center as the candidate lung nodule image. Save the image as a .npy file according to the label value (0 or 1), and divide it into two categories: nodule data and non-nodule data;

[0026] Step 2_2: 1347 cases of pulmonary nodule image data were obtained. According to the random selection and stratified selection methods, 80% (1078 cases) of the data were used as training data and 20% (269 cases) were used as test data to ensure the independence and randomness of the image samples. 371857 non-pulmonary nodule images were obtained, and 20% (74371 cases) were randomly selected as non-pulmonary nodule data samples;

[0027] Step 2_3: Due to the large difference between positive and negative samples, the training and test data of lung nodules should be expanded by 40 times, and the data set should be enhanced by rotation, translation, scaling, and mirror flipping. 43,120 cases of lung nodule training data and 10,760 cases of test data were obtained, for a total of 53,880 lung nodule data samples;

[0028] Step 2_4: 74,371 non-pulmonary nodule data samples were divided into 80% training data (59,497 samples) and 20% test data (14,874 samples);

[0029] Step 2_5: Merge the training set and test set of the two types of data separately while maintaining their independence.

[0030] Preferably, the SE-ResNet feature extraction stage in step 3 comprises the following specific steps:

[0031] Step 3_1: Input the 48×48×48 image obtained in the preprocessing stage into the encoder of the SE-ResNet residual model in the form of an array. It is processed by two convolution blocks with a convolution kernel size of 3×3×3 to obtain a feature map with a constant size and 16 channels;

[0032] layer1=Conv(X,3×3×3, stride=1, padding=2)

[0033] Among them, X is the input feature map, stride is the sliding step size, padding is the boundary padding, and Conv is the convolution operation.

[0034] Step 3_2: The feature map processed in step 3_1 is sent to the SE attention module for learning in the channel dimension to determine the importance of each channel. First, in the squeeze stage, the input is transformed from [H, W, C] to the feature vector [1, 1, C] through a global average pooling; then, in the excitation stage, two FC layers and ReLU and Sigmoid functions are used to weight the feature map to generate channel weights; finally, in the scale stage, the weights are applied to each channel by element-by-element multiplication to achieve weight normalization;

[0035]

[0036] s=f excitation (z,w)=σ(W2δ(W1z))

[0037] layer2=f scale (x,s)=s·x

[0038] Among them, z represents the overall feature, f squeeze Indicates extrusion, X is the input parameter, H and W represent the height and width respectively, x i,j is the feature vector of the image point in the i-th row and j-th column. s represents the score vector, f excitation It is expressed as incentive, σ is the sigmoid function, and W1 is The weight array of row C column, W2 represents row C The weight array of the column, r represents the ratio, and δ is the ReLU activation function. scale Indicates zoom.

[0039] Step 3_3: Perform the maximum pooling operation on the feature map processed in step 3_2, using a 2×2×2 pooling window with a stride of 2, sliding in each dimension, and a border padding of 2 to halve the size of the feature map;

[0040] layer3=Conv(layer2,2×2×2,stride=2,padding=2)

[0041] Step 3_4: The entire SE-ResNet network structure from step 3_1 to step 3_3 is repeated four times to gradually extract higher-level features, and finally a feature map with a size of 3×3×3 and 128 channels is obtained.

[0042] Preferably, the ResNet-Inception optimization stage in step 4 comprises the following specific steps:

[0043] Step 4_1: Input the 3×3×3 feature map F(H×W×C) with 128 channels finally obtained by the SE-ResNet feature extraction stage in step 3 into the ResNet-Inception optimization layer, and divide the input feature map into two routes, one of which is to directly pass the processed feature map to the output layer to obtain branch1;

[0044] branch1=Result SE-ResNet

[0045] Step 4_2: ResNet-Inception optimization layer Another route has 3 branches, each branch is processed by a different convolution layer, and then at the end of the branch, the three branches are processed by a 1×1×1 convolution layer to adjust the number of channels to obtain branch2.

[0046] path1=Conv(Result SE-ResNet ,1×1×1)

[0047] path2=Conv(Conv(Conv(Result SE-ResNet ,1×1×1),3×3×3),3×3×3)

[0048] path3 = Conv(Conv(Conv(Result SE-ResNet ,1×1×1),5×5×5),5×5×5)

[0049] branch2=Conv(path1,path2,path3,1×1×1)

[0050] Finally, the results generated by these two paths are added together.

[0051] Result ResNet-Inception =branch1+branch2

[0052] Among them, Result SE-ResNet Represents the final feature map obtained in the SE-ResNet feature extraction stage, Result ResNet-Inception Represents the final feature map obtained in the ResNet-Inception optimization phase.

[0053] Preferably, in the LSTM layer upgrading network stage in step 5, the specific steps are:

[0054] Step 5_1: First, the result obtained in the ResNet-Inception optimization stage in step 4 is convolved twice again to obtain a feature map of size 3×3×3 and number of channels 256, and then a residual connection is performed with the feature map obtained in the SE-ResNet feature extraction stage in step 3;

[0055] path=Conv(Conv(Result ResNet-Inception ,3×3×3),3×3×3)

[0056] Step 5_2: Input the resulting feature map of step 5_1 to the fully connected layer for further feature extraction and transformation. Subsequently, the feature blocks obtained from the fully connected layer are sent to the LSTM units in sequence, that is, the first block to the first unit, the second block to the second unit, and so on. The three stacked LSTM layers generate relevant features between blocks;

[0057] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0058] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0059]

[0060] O t =σ(W o ·[h t-1 ,x t ]+b o )

[0061] h t =O t *tanh(C t )

[0062] Among them, t represents the current time t, f t 、i t , C t and O t All represent outputs, namely forget gate, input gate, tanh function, LSTM unit and output gate; h t represents the output of the hidden layer at the current time t, h t-1 represents the output of the hidden layer at the previous moment; σ is the sigmoid function; x t is the input vector; the weight matrix is ​​denoted as W, which are W f W i W c W o , the bias value is recorded as b, respectively b f b i b c b o , the subscripts represent the forget gate, input gate, tanh function and output gate respectively.

[0063] Step 5_3: Pass the output of the LSTM unit in step 5_2 to the fully connected layer again.

[0064] Preferably, the classification output stage in step 6 comprises the following specific steps:

[0065] Step 6_1: Input the output obtained from step 5 above into the Sigmoid function for classification to obtain the classification results of benign and malignant pulmonary nodules. The calculation formula is as follows:

[0066]

[0067] Where exp is the natural exponential function and x is the input value. The Sigmoid function has an S-shaped curve, which increases monotonically and changes slowly at both ends.

[0068] Preferably, the model verification test phase in step 7 includes the following specific steps:

[0069] Step 7_1: After the training set is trained, the test set is used for test evaluation. The present invention uses performance indicators for verification and retains the best value of the verification result;

[0070]

[0071] Accuracy (ACC) indicates the proportion of samples correctly classified by the model to the total number of samples, reflecting the ability of the model to correctly classify benign and malignant pulmonary nodules.

[0072]

[0073] Sensitivity (SE) indicates the proportion of correctly classified positive samples among all positive samples, reflecting the ability of the model to correctly identify malignant pulmonary nodules. The higher the value, the lower the missed detection rate, and vice versa.

[0074]

[0075] Specificity (SP) refers to the proportion of correctly classified negative samples among all negative samples, reflecting the ability of the model to correctly identify benign pulmonary nodules. The higher the specificity, the lower the false positive rate.

[0076]

[0077] The area under the ROC curve AUC (Area Under Curve), the horizontal axis represents the false positive rate (False Positive Rate, FPR), and the vertical axis represents the true positive rate (True Positive Rate, TPR), which is used to measure the classification accuracy of the model. The closer the AUC value is to 1, the better the model effect.

[0078] Step 7_2: After retaining the current best value, perform continuous iteration and verification until all iterations are completed and the best value is retained;

[0079] Step 7_3: Output the obtained performance index results and compare them with other models.

[0080] The beneficial effects of the present invention are as follows:

[0081] 1. The present invention introduces the SE (Squeeze and Excitation) attention mechanism into the residual neural network (Residual Network, ResNet) to extract and integrate lung nodule features, thereby improving the accuracy and robustness of feature extraction. The SE mechanism can dynamically adjust the weights of feature maps, highlight key features and suppress redundant information, thereby improving the network's sensitivity and ability to distinguish lung nodule details. In this way, the network can not only more accurately capture the morphological and texture features of nodules, but also effectively integrate these features for more reliable benign and malignant classification.

[0082] 2. The present invention adds the ResNet-Inception layer after the last maximum pooling of the SE-ResNet model, further improving the diversity and expressiveness of feature extraction. The introduction of residual connections optimizes the overall network structure and deepens the depth of network learning, while the Inception multi-branch structure improves the representation ability of the network, enabling it to better capture features of different scales and levels. Combining these two advantages, the ResNet-Inception layer can improve model performance while keeping the network structure running efficiently.

[0083] 3. The present invention integrates the LSTM model on the basis of SE-ResNet to extract the contextual information of the target area sequence and enhance the network's ability to capture information about lung nodule features. Specifically, SE-ResNet adaptively adjusts channel weights through its Squeeze-and-Excitation module to refine feature expression and extract key features in lung images. When processing these features, the LSTM model can capture the changing trends and contextual information of lung nodules in different slices or time points, enabling the model to perform feature analysis from both spatial and temporal dimensions, effectively solving the problem of feature information loss, and thus providing more accurate lung nodule detection and classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a flow chart of the present invention;

[0085] Figure 2 is a flow chart of the method of the present invention;

[0086] Figure 3 It is a specific execution diagram of the present invention;

[0087] Figure 4 This is a model architecture diagram of the present invention; DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] like Figures 1 to 4 As shown, the embodiment of the present invention provides a lung nodule classification method based on SE-ResNet and LSTM fusion, comprising the following steps:

[0090] Step 1: Data input stage;

[0091] Step 2: Data preprocessing stage;

[0092] Step 3: SE-ResNet feature extraction stage;

[0093] Step 4: ResNet-Inception optimization phase;

[0094] Step 5: LSTM layer boosting network stage;

[0095] Step 6: Classification output stage;

[0096] Step 7: Model validation testing phase.

[0097] In one embodiment, the data input stage in step 1 includes the following specific steps:

[0098] Step 1_1: This input stage mainly screens the lung CT images. There are 888 different patients' lung CT image data, which are stored in 10 files from subset0.zip to subset9.zip. Each CT scan image contains multiple slices, and some slices that do not contain important information need to be screened out, so that the entire lung area can be fully displayed;

[0099] Step 1_2: Consider a dataset X with N types of data,

[0100] 80% of them are used as training data, X t ,

[0101] Take 20% as test data X s ,

[0102] In one embodiment, the pre-processing stage in step 2 includes the following specific steps:

[0103] Step 2_1: Get the nodule location and category information from the candidates.csv file, read the nodule coordinates, and extract a cubic area of ​​size (48, 48, 48) with the coordinates as the center as the candidate lung nodule image. Save the image as a .npy file according to the label value (0 or 1), and divide it into two categories: nodule data and non-nodule data;

[0104] Step 2_2: 1347 cases of pulmonary nodule image data were obtained. According to the random selection and stratified selection methods, 80% (1078 cases) of the data were used as training data and 20% (269 cases) were used as test data to ensure the independence and randomness of the image samples. 371857 non-pulmonary nodule images were obtained, and 20% (74371 cases) were randomly selected as non-pulmonary nodule data samples;

[0105] Step 2_3: Due to the large difference between positive and negative samples, the training and test data of lung nodules should be expanded by 40 times, and the data set should be enhanced by rotation, translation, scaling, and mirror flipping. 43,120 cases of lung nodule training data and 10,760 cases of test data were obtained, for a total of 53,880 lung nodule data samples;

[0106] Step 2_4: 74,371 non-pulmonary nodule data samples were divided into 80% training data (59,497 samples) and 20% test data (14,874 samples);

[0107] Step 2_5: Merge the training set and test set of the two types of data separately while maintaining their independence.

[0108] In one embodiment, the SE-ResNet feature extraction stage in step 3 includes the following specific steps:

[0109] Step 3_1: Input the 48×48×48 image obtained in the preprocessing stage into the encoder of the SE-ResNet residual model in the form of an array. It is processed by two convolution blocks with a convolution kernel size of 3×3×3 to obtain a feature map with a constant size and 16 channels;

[0110] layer1=Conv(X,3×3×3, stride=1, padding=2)

[0111] Among them, X is the input feature map, stride is the sliding step size, padding is the boundary padding, and Conv is the convolution operation.

[0112] Step 3_2: The feature map processed in step 3_1 is sent to the SE attention module for learning in the channel dimension to determine the importance of each channel. First, in the squeeze stage, the input is transformed from [H, W, C] to the feature vector [1, 1, C] through a global average pooling; then, in the excitation stage, two FC layers and ReLU and Sigmoid functions are used to weight the feature map to generate channel weights; finally, in the scale stage, the weights are applied to each channel by element-by-element multiplication to achieve weight normalization;

[0113]

[0114] s=f excitation (z,w)=σ(W2δ(W1z))

[0115] layer2=f scale (x,s)=s·x

[0116] Among them, z represents the overall feature, f squeeze Indicates extrusion, X is the input parameter, H and W represent the height and width respectively, x i,j is the feature vector of the image point in the i-th row and j-th column. s represents the score vector, f excitation It is expressed as incentive, σ is the sigmoid function, and W1 is The weight array of row C column, W2 represents row C The weight array of the column, r represents the ratio, and δ is the ReLU activation function.scale Indicates zoom.

[0117] Step 3_3: Perform the maximum pooling operation on the feature map processed in step 3_2, using a 2×2×2 pooling window with a stride of 2, sliding in each dimension, and a border padding of 2 to halve the size of the feature map;

[0118] layer3=Conv(layer2,2×2×2,stride=2,padding=2)

[0119] Step 3_4: The entire SE-ResNet network structure from step 3_1 to step 3_3 is repeated four times to gradually extract higher-level features, and finally a feature map with a size of 3×3×3 and 128 channels is obtained.

[0120] In one embodiment, the ResNet-Inception optimization stage in step 4 includes the following specific steps:

[0121] Step 4_1: Input the 3×3×3 feature map F(H×W×C) with 128 channels finally obtained by the SE-ResNet feature extraction stage in step 3 into the ResNet-Inception optimization layer, and divide the input feature map into two routes, one of which is to directly pass the processed feature map to the output layer to obtain branch1;

[0122] branch1=Result SE-ResNet

[0123] Step 4_2: ResNet-Inception optimization layer Another route has 3 branches, each branch is processed by a different convolution layer, and then at the end of the branch, the three branches are processed by a 1×1×1 convolution layer to adjust the number of channels to obtain branch2.

[0124] path1=Conv(Result SE-ResNet ,1×1×1)

[0125] path2=Conv(Conv(Conv(Result SE-ResNet ,1×1×1),3×3×3),3×3×3)

[0126] path3 = Conv(Conv(Conv(Result SE-ResNet ,1×1×1),5×5×5),5×5×5)

[0127] branch2=Conv(path1,path2,path3,1×1×1)

[0128] Finally, the results generated by these two paths are added together.

[0129] Result ResNet-Inception =branch1+branch2

[0130] Among them, Result SE-ResNet Represents the final feature map obtained in the SE-ResNet feature extraction stage, Result ResNet-Inception Represents the final feature map obtained in the ResNet-Inception optimization phase.

[0131] In one embodiment, the LSTM layer boosting network stage in step 5 includes the following specific steps:

[0132] Step 5_1: First, the result obtained in the ResNet-Inception optimization stage in step 4 is convolved twice again to obtain a feature map of size 3×3×3 and number of channels 256, and then a residual connection is performed with the feature map obtained in the SE-ResNet feature extraction stage in step 3;

[0133] path=Conv(Conv(Result ResNet-Inception ,3×3×3),3×3×3)

[0134] Step 5_2: Input the resulting feature map of step 5_1 to the fully connected layer for further feature extraction and transformation. Subsequently, the feature blocks obtained from the fully connected layer are sent to the LSTM units in sequence, that is, the first block to the first unit, the second block to the second unit, and so on. The three stacked LSTM layers generate relevant features between blocks;

[0135] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0136] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0137]

[0138] O t =σ(W o ·[h t-1 ,x t ]+b o )

[0139] ht =O t *tanh(C t )

[0140] Among them, t represents the current time t, f t 、i t , C t and O t All represent outputs, namely forget gate, input gate, tanh function, LSTM unit and output gate; h t represents the output of the hidden layer at the current time t, h t-1 represents the output of the hidden layer at the previous moment; σ is the sigmoid function; x t is the input vector; the weight matrix is ​​denoted as W, which are W f W i W c W o , the bias value is recorded as b, respectively b f b i b c b o , the subscripts represent the forget gate, input gate, tanh function and output gate respectively.

[0141] Step 5_3: Pass the output of the LSTM unit in step 5_2 to the fully connected layer again.

[0142] In one embodiment, the classification output stage in step 6 includes the following specific steps:

[0143] Step 6_1: Input the output obtained from step 5 above into the Sigmoid function for classification to obtain the classification results of benign and malignant pulmonary nodules. The calculation formula is as follows:

[0144]

[0145] Where exp is the natural exponential function and x is the input value. The Sigmoid function has an S-shaped curve, which increases monotonically and changes slowly at both ends.

[0146] In one embodiment, the model verification test phase in step 7 includes the following specific steps:

[0147] Step 7_1: After the training set is trained, the test set is used for test evaluation. The present invention uses performance indicators for verification and retains the best value of the verification result;

[0148]

[0149] Accuracy (ACC) indicates the proportion of samples correctly classified by the model to the total number of samples, reflecting the ability of the model to correctly classify benign and malignant pulmonary nodules.

[0150]

[0151] Sensitivity (SE) indicates the proportion of correctly classified positive samples among all positive samples, reflecting the ability of the model to correctly identify malignant pulmonary nodules. The higher the value, the lower the missed detection rate, and vice versa.

[0152]

[0153] Specificity (SP) refers to the proportion of correctly classified negative samples among all negative samples, reflecting the ability of the model to correctly identify benign pulmonary nodules. The higher the specificity, the lower the false positive rate.

[0154]

[0155] The area under the ROC curve AUC (Area Under Curve), the horizontal axis represents the false positive rate (False Positive Rate, FPR), and the vertical axis represents the true positive rate (True Positive Rate, TPR), which is used to measure the classification accuracy of the model. The closer the AUC value is to 1, the better the model effect.

[0156] Step 7_2: After retaining the current best value, perform continuous iteration and verification until all iterations are completed and the best value is retained;

[0157] Step 7_3: Output the obtained performance index results and compare them with other models.

[0158] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0159] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A pulmonary nodule classification method based on SE-ResNet and LSTM fusion, characterized by: The SE attention mechanism is introduced into ResNet to extract lung nodule features, combined with LSTM to enhance context information, and a ResNet-Inception layer is added after the maximum pooling to optimize performance. The method includes the following steps: Step 1: Data input stage; Step 2: Data preprocessing stage; Step 3: SE-ResNet feature extraction stage; Step 4: ResNet-Inception optimization phase; Step 5: LSTM layer enhancement network stage; Step 6: Classification output stage; Step 7: Model validation testing phase.

2. The method for pulmonary nodule classification based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The data input stage in step 1 includes the following specific steps: Step 1_1: This input stage mainly screens the lung CT images. There are 888 different patients' lung CT image data, which are stored in 10 files from subset0.zip to subset9.zip. Each CT scan image contains multiple slices, and some slices that do not contain important information need to be screened out, so that the entire lung area can be fully displayed; Step 1_2: Consider a dataset X with N types of data, 80% of them are used as training data, X t , Take 20% as test data X s , 3. The lung nodule classification method based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The data preprocessing stage in step 2 includes the following specific steps: Step 2_1: Get the nodule location and category information from the candidates.csv file, read the nodule coordinates, and extract a cubic area of ​​size (48, 48, 48) with the coordinates as the center as the candidate lung nodule image. Save the image as a .npy file according to the label value (0 or 1), and divide it into two categories: nodule data and non-nodule data; Step 2_2: 1347 cases of pulmonary nodule image data were obtained. According to the random selection and stratified selection methods, 80% (1078 cases) of the data were used as training data and 20% (269 cases) were used as test data to ensure the independence and randomness of the image samples. 371857 non-pulmonary nodule images were obtained, and 20% (74371 cases) were randomly selected as non-pulmonary nodule data samples; Step 2_3: Due to the large difference between positive and negative samples, the training and test data of lung nodules should be expanded by 40 times, and the data set should be enhanced by rotation, translation, scaling, and mirror flipping. 43,120 cases of lung nodule training data and 10,760 cases of test data were obtained, for a total of 53,880 lung nodule data samples; Step 2_4: 74,371 non-pulmonary nodule data samples were divided into 80% training data (59,497 samples) and 20% test data (14,874 samples); Step 2_5: Merge the training set and test set of the two types of data separately while maintaining their independence.

4. The lung nodule classification method based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The SE-ResNet feature extraction stage in step 3 includes the following specific steps: Step 3_1: Input the 48×48×48 image obtained in the preprocessing stage into the encoder of the SE-ResNet residual model in the form of an array. It is processed by two convolution blocks with a convolution kernel size of 3×3×3 to obtain a feature map with a constant size and 16 channels; layer1=Conv(X,3×3×3, stride=1, padding=2) Among them, X is the input feature map, stride is the sliding step size, padding is the boundary padding, and Conv is the convolution operation. Step 3_2: The feature map processed in step 3_1 is sent to the SE attention module for learning in the channel dimension to determine the importance of each channel. First, in the squeeze stage, the input is transformed from [H, W, C] to the feature vector [1, 1, C] through a global average pooling; then, in the excitation stage, two FC layers and ReLU and Sigmoid functions are used to weight the feature map to generate channel weights; finally, in the scale stage, the weights are applied to each channel by element-by-element multiplication to achieve weight normalization; s=f excitation (z,w)=σ(W2δ(W1z)) layer2=f scale (x,s)=s·x Among them, z represents the overall feature, f squeeze Indicates extrusion, X is the input parameter, H and W represent the height and width respectively, x i,j is the feature vector of the image point in the i-th row and j-th column. s represents the score vector, f excitation represents the excitation, σ is the sigmoid function, and W1 represents The weight array of row C column, W2 represents row C The weight array of the column, r represents the ratio, and δ is the ReLU activation function. scale Indicates zoom. Step 3_3: Perform the maximum pooling operation on the feature map processed in step 3_2, using a 2×2×2 pooling window with a stride of 2, sliding in each dimension, and a border padding of 2 to halve the size of the feature map; layer3=Conv(layer2,2×2×2,stride=2,padding=2) Step 3_4: The entire SE-ResNet network structure from step 3_1 to step 3_3 is repeated four times to gradually extract higher-level features, and finally a feature map with a size of 3×3×3 and 128 channels is obtained.

5. The lung nodule classification method based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The ResNet-Inception optimization phase in step 4 includes the following specific steps: Step 4_1: Input the 3×3×3 feature map F(H×W×C) with 128 channels finally obtained by the SE-ResNet feature extraction stage in step 3 into the ResNet-Inception optimization layer, and divide the input feature map into two routes, one of which is to directly pass the processed feature map to the output layer to obtain branch1; branch1=Result SE-ResNet Step 4_2: ResNet-Inception optimization layer Another route has 3 branches, each branch is processed by a different convolution layer, and then at the end of the branch, the three branches are processed by a 1×1×1 convolution layer to adjust the number of channels to obtain branch2. path1=Conv(Result SE-ResNet ,1×1×1) path2=Conv(Conv(Conv(Result SE-ResNet ,1×1×1),3×3×3),3×3×3) path3=Conv(Conv(Conv(Result SE-ResNet ,1×1×1),5×5×5),5×5×5) branch2=Conv(path1,path2,path3,1×1×1) Finally, the results generated by these two paths are added together. Result ResNet-Inception =branch1+branch2 Among them, Result SE-ResNet Represents the final feature map obtained in the SE-ResNet feature extraction stage, Result ResNet-Inception Represents the final feature map obtained in the ResNet-Inception optimization phase.

6. The lung nodule classification method based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The LSTM layer in step 5 is used to improve the network stage. The specific steps are as follows: Step 5_1: First, the result obtained in the ResNet-Inception optimization stage in step 4 is convolved twice again to obtain a feature map of size 3×3×3 and number of channels 256, and then a residual connection is performed with the feature map obtained in the SE-ResNet feature extraction stage in step 3; path=Conv(Conv(Result ResNet-Inception ,3×3×3),3×3×3) Step 5_2: The resulting feature map of step 5_1 is input to the fully connected layer for further feature extraction and transformation. Subsequently, the feature blocks obtained from the fully connected layer are sent to the LSTM units in sequence, that is, the first block to the first unit, the second block to the second unit, and so on. The three stacked LSTM layers generate relevant features between blocks; f t =σ(W f ·[h t-1 ,x t ]+b f ) i t =σ(W i ·[h t-1 ,x t ]+b i ) The t =σ(W o ·[h t-1 ,x t ]+b o ) h t =O t *tanh(C t ) Among them, t represents the current time t, f t 、i t , C t and O t All represent outputs, namely forget gate, input gate, tanh function, LSTM unit and output gate; h t represents the output of the hidden layer at the current time t, h t-1 represents the output of the hidden layer at the previous moment; σ is the sigmoid function; x t is the input vector; the weight matrix is ​​denoted as W, which are W f W i W c W o , the bias value is recorded as b, respectively b f b i b c b o , the subscripts represent the forget gate, input gate, tanh function and output gate respectively. Step 5_3: Pass the output of the LSTM unit in step 5_2 to the fully connected layer again.

7. The lung nodule classification method based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The classification output stage in step 6 includes the following specific steps: Step 6_1: Input the output obtained from step 5 above into the Sigmoid function for classification to obtain the classification results of benign and malignant pulmonary nodules. The calculation formula is as follows: Where exp is the natural exponential function and x is the input value. The Sigmoid function has an S-shaped curve, which increases monotonically and changes slowly at both ends.

8. The method for pulmonary nodule classification based on SE-ResNet and LSTM fusion according to claim 1, characterized in that: The model verification test phase in step 7 includes the following specific steps: Step 7_1: After the training set is trained, the test set is used for test evaluation. The present invention uses performance indicators for verification and retains the best value of the verification result; Accuracy (ACC) indicates the proportion of samples correctly classified by the model to the total number of samples, reflecting the ability of the model to correctly classify benign and malignant pulmonary nodules. Sensitivity (SE) indicates the proportion of correctly classified positive samples among all positive samples, reflecting the ability of the model to correctly identify malignant pulmonary nodules. The higher the value, the lower the missed detection rate, and vice versa. Specificity (SP) refers to the proportion of correctly classified negative samples among all negative samples, reflecting the ability of the model to correctly identify benign pulmonary nodules. The higher the specificity, the lower the false positive rate. The area under the ROC curve AUC (Area Under Curve), the horizontal axis represents the false positive rate (FPR), and the vertical axis represents the true positive rate (TPR), which is used to measure the classification accuracy of the model. The closer the AUC value is to 1, the better the model effect. Step 7_2: After retaining the current best value, perform continuous iteration and verification until all iterations are completed and the best value is retained; Step 7_3: Output the obtained performance index results and compare them with other models.

Images