A method for automatically diagnosing benign and malignant nodules based on lung tomography images

Abstract

The present invention discloses an artificial intelligence method for automatically diagnosing benign and malignant nodules based on lung tomography images. The method mainly includes two parts: screening and diagnosis of lung nodules. In the screening process of lung nodules, a two-dimensional image of a suspected nodule is first obtained, and it is used as training data to complete the input of a convolutional neural network, and the nodules are classified by predicting the probability of true and false positive nodules; in the diagnosis process of lung nodules, the screened lung nodules are first segmented and reconstructed into a three-dimensional lung nodule image, and then the image features are extracted, and then the image features are input into a prediction model to obtain the prediction results and select the most relevant image features. The present invention can automatically diagnose based on a patient's CT sequence and the physician's annotation information on the suspicious area, and assist the physician in making a more objective and accurate diagnosis of the disease.

Description

Technical Field

[0001] The present invention relates to a method for automatically judging whether a nodule in an image is benign or malignant by using a lung tomography image, and is a computer program product related to the fields of artificial intelligence image recognition and biostatistics. Background Art

[0002] In 2011, the results of the National Lung Screening Trial (NLST) in the United States showed that low-dose computed tomography (LDCT) can reduce lung cancer mortality by 20%. Based on this research result, international academic organizations recommended the implementation of low-dose spiral CT screening in high-risk groups. my country also launched the screening of high-risk groups for lung cancer in the "Early Diagnosis and Treatment of Rural Cancer" project in 2009. However, in the NLST study, the false-positive rate of early screening based on CT was as high as 96.4%. Reducing lung cancer mortality requires not only the popularization of early screening for lung cancer, but also the improvement of the diagnostic accuracy of early screening for lung cancer.

[0003] In order to improve the accuracy of early screening for lung cancer and reduce the workload of professional radiologists, many scholars have proposed automatic diagnosis methods based on lung CT images, most of which are directly based on two-dimensional nodule tomographic images, or combined with some two-dimensional nodule image features to build deep learning models for prediction, achieving good prediction results. However, there are few studies in the fields of three-dimensional nodules and their image features, and there is still much room for improvement in prediction results. Summary of the invention

[0004] In order to improve the accuracy of diagnosis of benign and malignant lung nodules, the present invention provides an automatic diagnosis method for benign and malignant lung nodules, which combines lung CT data with deep learning (DeepLearning) and classical machine learning (MachineLearning) methods to perform artificial intelligence diagnosis of the malignancy of lung nodules. At the same time, the present invention also provides doctors with several image features most relevant to benign and malignant lung nodules for reference.

[0005] An automatic diagnosis method for benign and malignant pulmonary nodules according to the present invention comprises the following steps:

[0006] -Acquire 2D images of suspected nodules based on lung CT sequences and doctor-marked locations;

[0007] -Introduce the 2D image as input into the nodule classification model to distinguish nodules from non-nodules;

[0008] - The doctor-annotated coordinates of the classified nodules and the corresponding CT sequence of the nodules are used as input data for the diagnosis of benign and malignant pulmonary nodules;

[0009] -Segment the lung nodule area and obtain a three-dimensional lung nodule image;

[0010] -Extract 3D lung nodule image features;

[0011] -Introduce nodule image features as input data into the prediction model to diagnose benign or malignant pulmonary nodules.

[0012] The lung CT sequence is defined as all 2D CT images of the lungs of the diagnostic object, and one diagnostic object corresponds to a specific CT sequence number. All lung CT slice images in the same sequence have the same shape and size, the physical distance between adjacent pixels of all CT slice images (pixel spacing) is the same, and the distance between adjacent CT slice images (slice thickness) is the same. The lung CT sequence image data can be obtained by professional doctors with the help of CT equipment. The current internationally used medical image transmission protocol DICOM (Digital Imaging and Communications in Medicine) points out that the content of CT data storage, in addition to the patient's CT image data, also includes slice thickness, information about the machine that collects CT images, timestamps, basic patient information, etc.

[0013] Doctor's annotation information refers to the doctor's regional annotation of lung nodules, and the annotation information includes the boundary coordinates of the lung nodule area. The boundary coordinates must be continuous and closed. When observed on a 2D CT tomographic image, the boundary coordinates form a ring, with the suspected lung nodule inside the ring and normal lung tissue outside the ring; when combined with a CT sequence, the boundary coordinates form a closed surface, with the entire suspected lung nodule inside the surface and normal lung tissue outside the surface. Doctors can use computer-aided diagnosis equipment CAD (Computer-aided diagnosis) to annotate the nodule area and save this coordinate information.

[0014] The center position of the suspected nodule is averaged based on the nodule area boundary coordinates in the doctor's annotation information.

[0015] Segment the 2D image of the suspected nodule, take the lung CT sequence and the center position of the suspected nodule as input data, find the correspondence between the center position of the suspected nodule and a single slice in the CT tomographic image, and segment the image of 25 pixels in the positive and negative directions of the X-axis and Y-axis of the center position of the suspected nodule in the slice.

[0016] The HU value conversion of the CT image is to set the HU value pixels in the segmented 2D image data greater than 400 to 1, and less than -1000 to 0, and scale the middle range to between 0 and 1, and then convert it into a grayscale image.

[0017] The method for segmenting the lung nodule area takes the lung CT sequence of the object to be diagnosed and the screened doctor's annotation information as input parameters, finds the correspondence between the nodule boundary annotation information and the CT tomographic image, and segments the internal area of ​​the marked boundary in each CT tomographic image containing the annotation information. All segmented areas belong to the nodule to be diagnosed of the object to be diagnosed.

[0018] The method for obtaining a three-dimensional pulmonary nodule image is to use the segmented region of the CT tomographic image of the nodule to be diagnosed as input data, determine the order of the segmented region of each CT tomographic image, and then arrange the segmented regions in sequence to reconstruct a three-dimensional image of the pulmonary nodule. The ideal distance between adjacent voxels of a three-dimensional image is (SliceThickness=1, Pixelspacingx=1, Pixelspacingy=1), but the actual distance between adjacent voxels may not be so. Therefore, to obtain pulmonary nodules of uniform size, further resampling (resampling) is required. The resampling method is to scale the spacing between adjacent voxels according to the ratio (1 / SliceThickness, 1 / Pixelspacingx, 1 / Pixelspacingy), that is, the actual physical distance between adjacent voxels after scaling (1mm, 1mm, 1mm). The Python toolkit Pyradiomics and the scaling ratio (1 / SliceThickness, 1 / Pixelspacingx, 1 / Pixelspacingy) can be used for resampling to obtain the original three-dimensional pulmonary nodule image.

[0019] The features of the three-dimensional lung nodule image are extracted by a feature extractor (based on the Python toolkit Pyradiomics). The extracted features include different types of features such as first-order features (Firstorder), shape features (Shape) and texture features (Texture); the object of feature extraction can be the original three-dimensional lung nodule image, or it can be the result of processing with different filters (Filter) such as Laplacian filtering (Laplacian), wavelet filtering (Wavelet) and logarithmic filtering (Logarithm). Feature extraction uses the three-dimensional lung nodule image as input data to extract the features of the three-dimensional lung nodule image. As will be described in more detail in the following implementation scheme, each type of feature of the three-dimensional lung nodule image has its corresponding physical interpretation, and different filters play a corresponding role in the three-dimensional lung nodule image.

[0020] The prediction and important feature screening model based on machine learning is based on logistic regression (hereinafter referred to as logistic regression) and regularized using elastic network (ElasticNet). This model is used to predict the benign and malignant nature of lung nodules. The construction details of the prediction model can be seen in the implementation plan below. The AUC of the prediction model for the testing dataset reaches 96.57%, and the prediction accuracy reaches 92.55%. Using this prediction model to diagnose lung nodules can achieve more accurate diagnostic results, and it can also provide doctors with the most relevant image features for reference.

[0021] The classifier model of true and false positive nodules based on deep convolutional neural network uses multi-layer convolutional layers and pooling layers to obtain the image features of positive nodules, and uses Dropout for regularization. The details of the construction of the prediction model can be seen in the implementation scheme below. The model has an AUC of 97.1% for the test data and a prediction accuracy of 94.45%. This model is used to assist doctors in screening out positive nodules from suspected lung cancer nodules, reducing the interference of false positive nodules on the next step of benign and malignant nodule judgment. One implementation scheme is set up that the CT sequence and candidate nodule location data used to construct the nodule screening model are obtained by the following processing:

[0022] The 888 CT sequences used to train the nodule screening model come from the LUNA16 (Lung Nodule Analysis 16) public data whose parent is LIDC / IDRI (The Lung Image Database Consortium image collection). In the candidate file provided in LUNA16 for the challenge of reducing false positive nodules, 1351 of the 551065 candidate nodules are positive nodules, of which less than 3% are classified as non-nodules due to diameter <3mm. Each candidate nodule corresponds to a CT file name, world coordinates (x, y, z) and true and false positive labels;

[0023] The CT images provided by LUNA16 are stored through mdh and a raw file with the same name. The mdh file stores the header information of the CT image data, including the origin coordinates (Offset), pixel spacing (ElementSpacing), etc. The raw file stores the pixel information.

[0024] One implementation is set up as a method to preprocess the dataset for model building:

[0025] Get the actual position of the candidate nodule in the CT image. According to the world coordinates of the candidate nodule position and the origin coordinates and pixel distance in the CT image head information, the position of the nodule in the CT image coordinate system can be obtained. Nodule position = (world coordinates – origin coordinates) ÷ pixel distance;

[0026] Get the 2D image data of the candidate nodules, segment the image of 25 pixels in the positive and negative directions of the X-axis and Y-axis of the actual position of the candidate nodules, process the segmented images with the HU value of the CT image, set HU values ​​less than -1000 to 0, and greater than 400 to 1, and scale them between 0 and 1. Store the processed 2D images and the corresponding label data in the form of h5py files;

[0027] Since there is a huge imbalance between the positive and negative candidate nodules, the dataset randomly selects negative nodules that are 5 times the number of positive nodules as negative samples, and rotates the positive nodules 90° and 180° clockwise to increase the amount of data;

[0028] The processed positive and negative samples are randomly divided into three parts as training set, validation set and test set. Appropriate models are selected in training and validation, and the false positive rate of the model is evaluated by its performance on the test set.

[0029] One implementation scheme is set up such that the steps for training the convolutional neural network model are:

[0030] The convolutional neural network structure is constructed using the TFLearn open source framework. The structure uses a two-dimensional convolutional layer and a two-dimensional pooling layer, as well as a fully connected layer for nonlinear fitting, and a Dropout layer to prevent overfitting.

[0031] Before the input layer of the network structure, the samples are processed with zero-centering and normalized image data to improve the training speed of the neural network and accelerate the convergence of weight parameters;

[0032] Use the training set to train the neural network structure to classify true and false positive nodules, and compare the performance of multiple models on the validation set to obtain the optimal two-dimensional deep convolutional neural network model.

[0033] The trained network model is used to predict the probability of true and false positive nodules in the test set samples, and true and false positive nodules are distinguished according to the pre-set probability threshold, so as to obtain the accuracy of the model in the test set, and analyze the ROC curve of the prediction model in the test set samples. The accuracy of this model on the test set is 94.45%, and the AUC value is 97.1%.

[0034] In one embodiment, the original three-dimensional lung nodule image used for training the prediction model is obtained by the following process:

[0035] - The CT sequences of the 3D lung nodule images used to train the prediction model come from the LIDC-IDRI lung nodule public database, which contains 1,012 patients. The lung nodules in the CT sequences of each patient were independently annotated and diagnosed by four experienced professional doctors;

[0036] - The pulmonary nodule regions annotated by two different doctors may overlap. For the overlapping pulmonary nodule regions, whether the two labeled regions can be identified as the same pulmonary nodule is determined based on whether the ratio of the overlapping regions to the respective labeled regions reaches a set threshold (60% is used in the training of the present invention);

[0037] - For two labeled regions identified as the same lung nodule, the two labeled regions are merged into a new labeled region by finding their union, and further, all different labeled regions identified as the same lung nodule are merged into a new labeled region, i.e., the lung nodule region ultimately used to obtain the original three-dimensional lung nodule image;

[0038] -For the final diagnosis results of the lung nodule area, including the diagnostic opinions of different doctors, you can choose to take the average, median, highest frequency opinion, etc. to get the final diagnosis conclusion.

[0039] In one implementation, the original three-dimensional lung nodule image used to train the prediction model can be filtered using a Laplacian filter and a wavelet filter, and each of the two filters has the following functions:

[0040] -The mathematical form of the Laplace filter operator is:

[0041]

[0042] in represents the Laplace operator, is the second-order partial differential of the image gray value in the x direction, It is the second-order partial differential of the image grayscale value in the y direction. The Laplace operator makes the image grayscale value enhance the areas with drastic grayscale changes in the image and weaken the areas with slow grayscale changes. Superimposing the Laplace image on the original image can achieve the effect of sharpening the original image. When acting on a three-dimensional lung nodule image, it can sharpen the lung nodule image and enhance the details of image changes;

[0043] - Wavelet transform can be used to enhance image details like Laplace transform, and has the advantage of distinguishing noise. In the application, the wavelet function uses the first-order derivative of Gaussian pulse, and adjusts the scale according to the actual image grayscale change, detects the areas with drastic grayscale changes in the horizontal and vertical directions, and then sums and applies it to the original image to enhance the image details.

[0044] One implementation scheme is to use three-dimensional lung nodule image features for training a prediction model, including the following features:

[0045] -The first-order feature is a description of the voxel intensity distribution in the region of interest. It is the sum of the squares of the grayscale values ​​of all voxels. If there is indeed a difference in the intensity values ​​of benign nodules and malignant nodules in CT images, the energy feature can play a key role. It is larger in clear images with a wide distribution of grayscale than in blurred or single-tone images, and may effectively distinguish benign from malignant nodules;

[0046] -Shape features are a description of the shape of the region of interest, such as volume, surface area, maximum 3D diameter, etc., which can distinguish nodules with obvious differences in shape;

[0047] -Texture features are a description of the distribution of voxel intensity in the region of interest. Texture features are usually obtained indirectly based on texture matrices. The following are several types of texture matrices:

[0048] --Gray Level Co-occurrence Matrix GLCM is a matrix of voxel pairs arranged in accordance with a specific rule. For example, if voxel value m appears p times to the right of voxel value n in the image, then the value of row m and column n of the GLCM matrix is ​​p;

[0049] --GraylevelRunLengthMatrix GLRLM is a matrix of the lengths of consecutive voxels with the same grayscale value. For example, if the grayscale value in the image is m and the number of horizontally connected voxels with length n is p, then the value of the m-row and n-column of the GLRLM matrix is ​​p;

[0050] --GrayLevelSizeZone (GLSZM) is the number of regions formed by 26-connected voxels with the same gray value. For example, if the gray value in the image is m, and the number of 26-connected regions of size n is p, then the value of the mth row and nth column of the GLSZM matrix is ​​p.

[0051] --GrayLevelDependenceMatrix GLDM is the number of dependent voxels in the neighborhood of the central voxel, where the neighborhood can be optionally enlarged or reduced. If the absolute value of the difference between the grayscale value of the neighboring voxel and the grayscale value of the central voxel is less than the set threshold, it means that the neighborhood voxel depends on the central voxel. For example, when a voxel with a grayscale value of m is found in the image as the central voxel, the number of times there are n dependent voxels in the neighborhood is p, then the value of the mth row and nth column of the GLDM matrix is ​​p;

[0052] --Neighboring Gray Tone Difference Matrix NGTDM (Neighboring Gray Tone Difference Matrix) contains the number of voxels at each gray level, the probability, and the sum of the absolute values ​​of the difference with the neighborhood average gray value. For example, if the number of voxels with gray value m in the image is n, the probability is p, and the sum of the absolute values ​​of the difference with the neighborhood average gray value is s, then the mth row of NGTDM is [n, p, s];

[0053] -Texture features are features that are further extracted based on the texture matrix, such as the grayscale zone entropy (Zone Entropy), which represents the entropy value of the grayscale zone size matrix.

[0054] One implementation scheme is that the three-dimensional pulmonary nodule image feature matrix and pulmonary nodule diagnosis conclusion data used to train the prediction model can be further processed in the following ways:

[0055] - The same type of features need to be standardized, that is, the three-dimensional lung nodule image feature matrix is ​​standardized by column;

[0056] -Data from single or two lung nodules with fewer annotations by physicians can be selectively screened out;

[0057] - Lung nodule data with unclear diagnostic intent (the LIDC-IDRI dataset limits the degree of suspicion of malignancy of lung nodules to 1-5, where the smaller the value, the greater the possibility of benign nodules, the larger the value, the greater the possibility of malignant nodules, and the value of 3 means that the doctor cannot determine the attributes of the nodule) can be selectively screened out;

[0058] -Data with diagnostic conclusion marks outside the specified range need to be screened out.

[0059] One implementation scheme is set as the regularization method of the prediction model, which can choose Lasso, Ridge or ElasticNet, and their respective principles are as follows:

[0060] -The purpose of diagnosing pulmonary nodules is to determine whether the nodules are benign or malignant, which is a binary classification problem. Therefore, it is assumed that the relationship between the benign and malignant nature of pulmonary nodules and the image features of pulmonary nodules conforms to the law of the logistic regression model, that is:

[0061]

[0062] Among them, p i is the probability that lung nodule i is benign or malignant, x i is a vector of image features of lung nodule i, and is a coefficient vector of each image feature. Logistic regression models usually use the maximum likelihood method to estimate coefficients, and the likelihood function is as follows:

[0063]

[0064] Among them, y i is the corresponding probability p in logistic regression i The classification value of is 0 or 1, and n is the number of lung nodules. From this formula, it can be seen that the larger the likelihood function value l(β), the better the classification effect of the prediction model. Therefore, the problem of optimizing the prediction model can be transformed into the problem of maximizing the likelihood function value. Taking the logarithm of both sides of the above formula and then taking the opposite number, we can get:

[0065]

[0066] Substitute equation (2) into it and simplify it:

[0067]

[0068] Derivative of formula (5):

[0069]

[0070]

[0071] Let equation (6) be 0, and the optimal solution of the likelihood function is:

[0072] β=(X T X) -1 X T [lny-ln(1-y)]

[0073] Formula (7)

[0074] Since the rank of X always obeys: rank(X)≤min(n,p); at the same time, X T The rank of the matrix X is always subject to: rank(X T X)≤rank(X). So, the p×p dimensional matrix X TThe rank of X is always less than or equal to n. When p>n, X T The X matrix is ​​not rank sufficient and cannot be solved In other words, when p>n, L(β) has countless solutions, and we cannot simply minimize the likelihood function to determine which one is the best. At this time, one method is to add restrictions and determine the optimal solution within the restrictions, which is regularization.

[0075] -Lasso regularization is done by adding l1 regularization to the likelihood function (5), namely:

[0076]

[0077] In order to simplify the solution, consider the case of a single variable, that is, L L (β) = ln(1 + e xβ )-yxβ+λ||β||1, assuming that it can be differentiated for any β value, according to the property that the differential of the extreme point is 0, the optimal solution can be obtained as:

[0078]

[0079] That is, by adding the l1 regularization restriction, the coefficients of the features are penalized and constrained so that L(β) has an optimal solution;

[0080] The -Ridge regularization method is to add l2 regularization to the likelihood function (5), that is:

[0081]

[0082] Similar to the Lasso solution process, the optimal solution can be obtained as:

[0083]

[0084] That is, by adding the l2 regularization restriction, the coefficients of the features are penalized and constrained so that L(β) has an optimal solution.

[0085] -ElasticNet is a linear combination of l1 and l2 regularization, which is then added to the mean square error function:

[0086]

[0087]

[0088] Similar to the Lasso solution process, the optimal solution can be found as:

[0089]

[0090] That is, by adding the restriction conditions of ElasticNet regularization, the coefficients of the features are penalized and constrained, so that L(β) has an optimal solution, and the optimal solution changes with the changes of λ1 and λ2.

[0091] One implementation is set up such that the training step of the prediction model follows the following steps:

[0092] - The dataset used to build the prediction model includes 3D lung nodule image features and doctor diagnosis results, of which 70% are randomly selected as the training set for building the prediction model, and the remaining 30% are used as the test set;

[0093] -Select different regularization methods to build prediction models;

[0094] -For the same regularization method, different regularization penalty coefficients are input, and the optimal prediction model under the regularization method is determined by using the k-fold (10 folds in training) cross-validation method;

[0095] -The test set was input into the prediction model selected under each regularization method, and the optimal model was selected using the test set prediction accuracy and AUC. The prediction accuracy of the optimal model for the test set reached 92.55%, and the AUC reached 96.57%.

[0096] The beneficial effect of the present invention is to use the advantages of deep learning to pre-judge whether a suspected nodule is a pathological nodule, screen out false positive nodules, and improve the efficiency of further diagnosis of positive nodules. By combining image segmentation, feature extraction and machine learning methods, an automatic diagnosis method for lung nodules is proposed. The automatic diagnosis result has high accuracy and provides doctors with some relatively important image features, which can provide reliable reference opinions for doctors' diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 A flow chart for automated diagnosis of pulmonary nodules is shown.

[0098] Figure 2 The flowchart of the construction of the convolutional neural network model for screening lung nodules is shown. Figure A shows data extraction, locating the position of candidate nodules from CT tomographic images, and segmenting 50×50 fixed-size images; Figure B shows the preprocessing of the data, dividing all the data into three parts: training, verification, and testing, and fitting and verifying the training and verification data with the CNN model; Figure C shows the loss value and accuracy of the training and verification data sets during the training and verification process; Figure D shows the performance of the test set on the model. Figure E shows the construction of the convolutional neural network (Convolutional Neural Network) of this model.

[0099] Figure 3 The flowchart of diagnosing lung nodules is shown. Figure A shows the process of obtaining lung nodules, including segmenting lung nodules from two-dimensional images and reconstructing lung nodules into three-dimensional space; Figure B shows the process of feature extraction, including preprocessing of three-dimensional lung nodule images, filtering, i.e. feature extraction; Figure C shows the analysis of the features of three-dimensional lung nodule images. From the cluster heatmap, it can be seen that the features are roughly aggregated into two clusters, and the lung nodules are roughly aggregated into two clusters, with obvious differences; Figure D shows the ROC graph and AUC value of the prediction model's prediction results for the test set, and the prediction effect is very good.

[0100] Figure 4 The figure shows the change of the cross-validation error bar (Errorbar) with respect to the penalty coefficient when the model uses ElasticNet. DETAILED DESCRIPTION

[0101] Figure 1 The specific implementation of the present invention is shown to include two steps:

[0102] -The first step is to screen for lung nodules;

[0103] -The second step is to diagnose lung nodules.

[0104] like Figure 2 As shown in A, firstly, the center position of the area marked by the doctor's suspected nodule is matched with the CT sequence, the center position coordinates are converted into voxel coordinates and matched to the 3D CT image, and then the suspected nodule is cut out into a 2D image of 50×50 size according to the center position coordinates of the suspected nodule.

[0105] Convert the 2D image to HU, set HU greater than 400 to 1, less than -1000 to 0, and scale the intermediate values ​​between 0 and 1. Then convert the pixels to grayscale image pixels.

[0106] like Figure 2 As shown in B, the sample is zero-centered and normalized, and then input into the trained convolutional neural network prediction model to obtain the true and false positive probabilities of the sample, and distinguish true and false positive nodules according to the pre-set probability threshold. The CT image and the corresponding nodule doctor annotation information are recorded and used as the input for the second step of nodule diagnosis.

[0107] like Figure 3 As described in A, after screening the lung nodules, the lung nodules are first segmented using the doctor's annotation information that confirms the lung nodules combined with the CT sequence, which specifically includes the following steps:

[0108] -Link the pulmonary nodule boundary annotated by the doctor with the CT tomographic image, that is, correspond the z coordinate of the pulmonary nodule boundary coordinates with the z coordinate of the CT tomographic image to ensure that the point depicted by the boundary is located on the correct CT tomographic image;

[0109] - Connect the boundary coordinates of the lung nodules marked by the doctor with the pixel points on the CT tomographic image, and use the image processing morphological expansion method to obtain the boundary coordinates and all the pixel points inside, so as to obtain the coordinates of the entire lung nodule boundary and inside and the grayscale values ​​corresponding to the coordinates;

[0110] - When multiple doctors annotate the same CT sequence or the areas annotated by the same doctor overlap, the repetition of the coordinates of the overlapping areas is calculated (the ratio of the number of coordinates in the overlapping areas to the number of coordinates in the annotated areas of each lung nodule). If the repetition is greater than the set threshold (60%), the two lung nodules are merged into the same lung nodule for processing. Otherwise, if the repetition is less than the set threshold, they are processed separately.

[0111] -The obtained lung nodule coordinates and the grayscale values ​​corresponding to each lung nodule coordinate are distributed into three-dimensional space (three-dimensional matrix). The values ​​at the coordinate points marked by the doctor are their corresponding grayscale values, and the remaining points are filled with 0. Then, a three-dimensional image containing only one lung nodule can be obtained.

[0112] like Figure 3 As described in B, after segmenting and obtaining the three-dimensional lung nodule image, the image features are extracted according to the following steps:

[0113] -First, the 3D pulmonary nodule image is resampled. The purpose of resampling is to scale the distance between adjacent voxels to the same size, reduce the interference of the CT image acquisition device on the segmentation of the pulmonary nodule size, and make the pulmonary nodule as realistic as possible;

[0114] -Cut off the 0-filled area around the lung nodules to reduce the three-dimensional matrix containing the lung nodules to the minimum volume. When the 0-filled area is large, although it will not affect the extracted image features, it will greatly increase the computational complexity of feature extraction, slow down the computer's running speed, and even cause the computer to crash due to insufficient running memory;

[0115] - Use Laplace filter and wavelet filter to filter the three-dimensional lung nodule image. Both Laplace filter and wavelet filter are based on Gaussian function. The Gaussian function σ value must be set to be greater than 0. The size of any dimension of the three-dimensional lung nodule image must be greater than the σ value.

[0116] -The feature extractor is used to extract relevant features of the three-dimensional lung nodule image, including first-order features, shape features and texture features. However, it is not necessary to extract shape features from images after Laplace filtering and wavelet filtering, because the shape and size of the filtered image will not change relative to the original image. If it is extracted again, the data will be repeated.

[0117] After the features are extracted, the pulmonary nodules are diagnosed using the 3D pulmonary nodule image features. By inputting the pulmonary nodules into the prediction model, the probability of the pulmonary nodules being benign or malignant can be obtained. Note that different benign and malignant thresholds can be set according to the actual situation to classify the pulmonary nodules into benign and malignant types. For example, when predicting the probability of benignity, if the probability of fewer false positives (False-positive rate, a benign nodule is diagnosed as a malignant nodule) is expected to be small, the threshold can be appropriately increased to ensure that more pulmonary nodules are not misdiagnosed as benign.

[0118] Although the present invention has been further shown and described in detail through preferred embodiments, the present invention is not limited to the disclosed examples, and other variations may be derived by those skilled in the art without departing from the scope of protection of the present invention.

Claims

1. An automatic diagnosis method for pulmonary nodules, characterized in that: include: Segment the suspected nodule area and obtain a two-dimensional nodule image; The two-dimensional nodule images were introduced into the neural network prediction model to distinguish true and false positive nodules; Segment the lung nodule area and obtain a three-dimensional lung nodule image; Extract three-dimensional lung nodule image features; The nodule image features are introduced as input parameters into the prediction model to obtain the probability that the lung nodule is benign or malignant; The feature extractor is used to extract features from the three-dimensional lung nodule image, and the extracted three-dimensional lung nodule image features include but are not limited to the following three categories: The first-order features of 3D lung nodule images include the mean, variance, and entropy of image grayscale values; Shape characteristics of 3D lung nodule images, including image volume, surface area, and sphericity; Texture features of 3D lung nodule images, including image grayscale co-occurrence matrix derived features, image grayscale level length matrix derived features, image grayscale region size matrix derived features, image grayscale dependency matrix derived features, and image neighborhood grayscale difference matrix derived features; The training set used to train the prediction model is the nodule image features and the corresponding diagnosis results, which include the results of processing the annotation information of multiple doctors. The processing of the training set includes selectively excluding the results with unclear diagnosis results or the results with less annotation information from doctors and uncertainty. Among them, the pulmonary nodule image features are input into the prediction model to obtain the probability that the pulmonary nodule image features belong to benign or malignant, and different benign and malignant thresholds are set according to the actual situation to classify the pulmonary nodules into benign and malignant. The prediction model is built based on the logistic regression model, and the regularization method is the elastic network. The regularization method for training the prediction model is selected from L1 regularization, L2 regularization and elastic network based on the best prediction AUC and accuracy; The optimal solution of the likelihood function of the prediction model is: Where β is the coefficient vector of each image feature; n is the number of lung nodules; y is the classification value corresponding to probability p in logistic regression; λ and λ2 represent regularization coefficients; The training steps of the prediction model include: Select different regularization methods to establish prediction models; for the same regularization method, input different regularization penalty coefficients, and use the k-fold cross-validation method to determine the optimal prediction model under the regularization method; input the test set into the prediction model selected under each regularization method, and select the optimal model using the prediction accuracy and AUC of the test set; The method for segmenting the lung nodule area is to perform segmentation based on the physician's annotation information of the lung nodule area in combination with the patient's CT sequence. The physician's annotation information specifically refers to the boundary coordinates of the nodule area, and the boundary coordinates of each CT tomographic image correspond to the CT tomographic image. Among them, the method for obtaining a three-dimensional lung nodule image is to distribute the grayscale values ​​corresponding to the coordinates of the lung nodule boundary and its interior marked by the physician into a three-dimensional matrix, and the obtained three-dimensional matrix is ​​the three-dimensional lung nodule image. Among them, when there is more than one annotating physician and the annotated nodule areas overlap, in this case, the nodule overlapping areas are divided according to the occupancy of the nodule overlapping areas. Whether the ratio of the domain is greater than the set threshold, it is judged whether the two marked nodule areas belong to the same nodule. If they belong to the same nodule, the two nodules are synthesized into a new nodule by taking the union method. If they do not belong to the same nodule, they are processed separately according to their respective nodule areas. If they belong to the same nodule, the two nodules are synthesized into a new nodule by taking the union method, including: for two marked areas identified as the same lung nodule, the two marked areas are merged into a new marked area by taking the union of the two marked areas, and all different marked areas identified as the same lung nodule are merged into a new marked area; The preprocessing for extracting the features of three-dimensional lung nodules images includes resampling, shearing and filtering of three-dimensional lung nodule images; resampling is to scale the distance between adjacent voxels of three-dimensional lung nodule images to the same size, ensuring that all three-dimensional lung nodule images have the same size ratio as the real lung nodules; shearing is to scale the three-dimensional lung nodule images to the smallest cube containing the lung nodules, minimizing the volume of the three-dimensional matrix; filtering refers to convolving the original three-dimensional nodule image with a Laplacian filter and a wavelet filter; The three-dimensional lung nodule image feature matrix was standardized by column; the data obtained from a single or two lung nodules with fewer annotations by doctors were selectively screened out; the lung nodule data with the suspicion level of malignancy of lung nodules limited to 3 in the LIDC-IDRI dataset were selectively screened out; and the data with diagnostic conclusion labels outside the limited range were screened out.

2. The method according to claim 1, characterized in that The method for segmenting the suspected nodule area is to calculate the center position of the annotated area of ​​a single slice in the CT image based on the physician's annotation information of the lung nodule area and the corresponding CT sequence, and then take 25 pixels in the positive and negative directions of the X-axis and Y-axis as the image information of the suspected nodule to be detected according to the center position, perform HU value processing on the image information of the CT image, and convert it into a grayscale image.

3. The method according to claim 1, characterized in that: Introducing the two-dimensional nodule image into the neural network model is to zero-center and normalize the processed two-dimensional image, and then use it as the input of the trained network model to predict the probability of true and false positive nodules, and distinguish true and false positive nodules according to the set threshold.

Images