Multi-attribute Classification Method, System, Medium and Device for Pulmonary Nodules

Abstract

The present invention provides a multi-attribute classification method, system, medium and device for pulmonary nodules, including: obtaining a computed tomography image, establishing a CT data representation method, processing the computed tomography image using the CT data representation method to obtain a de-differenced computed tomography image with the differences in slice thickness and pixel pitch eliminated; extracting the de-differenced computed tomography image based on a lung contour extraction technique to obtain a lung contour image; denoising the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system; extracting the coordinates and sizes of pulmonary nodules from the denoised lung contour image to obtain pulmonary nodule images; extracting image features from the pulmonary nodule images based on a multi-attribute classification model of deep learning, and performing attribute classification on the image features. The present invention is used to classify the attributes presented by pulmonary nodules on CT images, so as to predict the degree of attribute presentation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a multi-attribute classification method, system, medium and device for lung nodules. Background Art

[0002] Lung cancer is one of the malignant tumors with the fastest growth rate of incidence and mortality, which has a significant impact on the physical health of all mankind. Therefore, the early diagnosis of lung lesions is of great significance for the diagnosis and treatment of lung cancer. The early form of lung cancer is lung nodules, and computerized tomography (CT) technology is clinically used to examine lung nodule lesions. The imaging features of lung nodules are diverse. Manual screening not only has a large workload, but also the judgment of features by observers is subjective. Accurately quantifying the manifestation features of lung nodules is of great significance for doctors to judge whether lung nodules will grow into malignant tumors.

[0003] Therefore, it is hoped to solve the problem of how to better extract image features from lung nodule images and perform attribute classification on the image features. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a multi-attribute classification method, system, medium and device for lung nodules, which are used to solve the problem of how to better extract image features from lung nodule images and perform attribute classification on the image features in the prior art.

[0005] To achieve the above object and other related objects, the present invention provides a multi-attribute classification method for lung nodules, including the following steps: obtaining a computerized tomography (CT) image, establishing a CT data representation method, using the CT data representation method to process the CT image to obtain a de-differenced CT image with the differences in slice thickness and pixel spacing eliminated; extracting the de-differenced CT image based on lung contour extraction technology to obtain a lung contour image; denoising the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system; extracting the coordinates and sizes of lung nodules from the denoised lung contour image to obtain lung nodule images; extracting image features from the lung nodule images based on a multi-attribute classification model of deep learning and performing attribute classification on the image features.

[0006] In an embodiment of the present invention, the CT data representation method includes: truncating the CT image within the range of -1200 to 600 Hounsfield units to obtain a CT image within the range of -1200 to 600 Hounsfield units, and linearly converting it to the range of 0 to 255 to obtain a de-differenced CT image; using an interpolation method to unify the pixel spacing and slice thickness of the de-differenced CT image to 1 pixel per cubic millimeter.

[0007] In one embodiment of the present invention, the extraction of the differential computed tomography image based on the lung contour extraction technology to obtain a lung contour image includes: setting a gray value threshold based on the difference between the gray values of lung air and lung wall tissue, and obtaining data higher than the gray value threshold based on the gray value threshold to obtain a preliminary lung contour image; removing the edge-connected regions of the preliminary lung contour image to obtain a lung contour image with air regions removed; extracting the two largest connected components of the lung contour image with air regions removed as a lung region image; separating the lung lesions attached to blood vessels from the blood vessels to obtain a lung region image without blood vessels; filling the cavities caused by lung nodules attached to the lung wall to obtain a lung region image with cavities filled; and filling the holes in the lung region image with cavities filled to obtain a complete and smooth lung contour image.

[0008] In one embodiment of the present invention, the denoising of the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system includes: using a three-dimensional convex hull algorithm to extract the convex hull of the lung for the lung contour image, and filling the region outside the convex hull with a preset gray value; performing three-dimensional dilation on the lung contour image to obtain the intralung region, and obtaining the data outside the intralung region and inside the convex hull as the data that needs to remove bone data; filling the image corresponding to the data that needs to remove bone data with a preset gray value; calculating a bounding box according to the lung contour, and removing the region filled with the preset gray value to obtain a lung contour image without bone; and converting the positions of the lung nodules in the lung contour image without bone to the relative coordinates of the lung to obtain a denoised lung contour image with a unified relative coordinate system.

[0009] In one embodiment of the present invention, the extraction of the coordinates and sizes of lung nodules from the denoised lung contour image to obtain a lung nodule image includes: calculating the circumscribed circle of the lung nodules in the denoised lung contour image, and taking the radius of the circumscribed circle as the size of the lung nodules, calculating the geometric centroid of the lung nodules, and taking the geometric centroid (x0, y0, z0) as the position coordinates of the lung nodules; and converting the position coordinates of the lung nodules to world coordinates to obtain a lung nodule image.

[0010] In one embodiment of the present invention, the feature extraction of the lung nodule image based on a multi-attribute classification model of deep learning to obtain image features and the attribute classification of the image features include:

[0011] The multi-attribute classification model based on deep learning includes a feature extraction module, and the feature extraction module includes a residual structure and a dense connection structure;

[0012] The residual structure is defined by the following formula:

[0013] x l= H l (x l-1 ) + x l-1

[0014] where the output x of the l-th layer l is calculated from the features x of the (l - 1)-th layer l-1 input to the computational unit H l (·), and added to the original features x l-1 to obtain;

[0015] The dense connection structure is defined by the following formula:

[0016] x l = H l ([x0, x1,..., x l-1 )

[0017] where the output x of the l-th layer l is calculated from the concatenation of the features from the 0-th layer to the (l - 1)-th layer as the input to the computational unit H l (·);

[0018] The multi-attribute classification model includes a multi-scale feature extraction module, which is defined by the following formula:

[0019]

[0020] where x encoder and x decoder represent the features of the encoder and decoder respectively, and the subscript l indicates the layer number of the encoder or decoder; concat(.) is the concatenation operation function between features, and upsample(.) is the upsampling function; the decoder features of the (l - 1)-th layer are restored to the same resolution as the encoder features of the l-th layer through a learnable upsampling layer, and then they are concatenated and features are extracted through a two-stream fusion module to obtain the decoder features of the l-th layer;

[0021] The multi-attribute classification model includes 8 fully convolutional classifiers and 1 fully connected classifier;

[0022] The loss function of the multi-attribute classification model is a weighted cross-entropy loss function, and the cross-entropy loss function is defined by the following formula:

[0023]

[0024] where exp(·) is the natural exponential function; x k represents the predicted value of the k-th class; x jRepresents the predicted value of the j-th category; k belongs to j; Softmax maps the category prediction value to the interval (0, 1) by dividing the exponent of the original prediction value of this category by the sum of the exponents of the prediction values of all categories, and makes the sum of the prediction values of all categories equal to 1.

[0025] The loss function of exp(·) is defined by the following formula:

[0026]

[0027] Where K represents the total number of categories of a certain attribute, there are a preset number of categories for a certain attribute, k represents a specific category of a certain attribute, k ∈ [1, K]; y k Represents whether the predicted category is the same as the sample category, 1 if the same, 0 if different; p k Represents the probability that the sample belongs to category k;

[0028] Substitute the Softmax(x k ) function into the loss(p, y) function to obtain the cross-entropy loss function formula for the class-th category of category k;

[0029]

[0030] The cross-entropy loss function formula can be split into two parts, which consists of the negative value of the probability value that category k is predicted as the class-th category and the logarithm of the sum of the probability exponents that category k is predicted as each category; Represents the probability value that category k is predicted as the class-th category; Represents the probability value that category k is predicted as the j-th category when class is j;

[0031] Add the loss function values of each attribute to obtain the final loss value function:

[0032]

[0033] Where, Is the probability value that all categories k are predicted as the class-th category; The category weights consist of the normalized values of the reciprocals of the sample ratios:

[0034]

[0035] Where, Represents the sample size of the class class of category k, Represents the weight of the class-th category of category k; The weight of the class-th category of k is composed of the reciprocal of the ratio of its sample size to the total number of samples;

[0036] The calculated class weights are used to weight the loss function values of each class to obtain:

[0037]

[0038] Taking the average of the loss function values of each class for 10 iterations respectively, the loss function values loss of each class can be obtained k ; represents the weight of the class of the k-th class;

[0039] The loss function weight values of each class are composed of the normalized values of the reciprocals of the class loss function value ratios. The loss function after weight adjustment is defined by the following formula:

[0040]

[0041] where is the weight of the j-th class of the k-th class, w k is the weight of the k-th class.

[0042] To achieve the above object, the present invention also provides a multi-attribute classification system for pulmonary nodules, including: an acquisition module, an extraction module, a denoising module, an obtaining module, and a classification module; the acquisition module is used to acquire a computed tomography image, establish a CT data representation method, and use the CT data representation method to process the computed tomography image to obtain a de-differenced computed tomography image that eliminates the differences in slice thickness and pixel spacing; the extraction module is used to extract the de-differenced computed tomography image based on the lung contour extraction technology to obtain a lung contour image; the denoising module is used to denoise the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system; the obtaining module is used to extract the coordinates and sizes of the pulmonary nodules from the denoised lung contour image to obtain a pulmonary nodule image; the classification module is used to extract image features from the pulmonary nodule image based on a multi-attribute classification model of deep learning and perform attribute classification on the image features.

[0043] In an embodiment of the present invention, the CT data representation method includes: truncating the computed tomography image within -1200 to 600 Hounsfield units to obtain a computed tomography image within -1200 to 600 Hounsfield units, and linearly converting it to the range of 0 to 255 to obtain a de-differenced computed tomography image; using the interpolation method to unify the pixel spacing and slice thickness of the de-differenced computed tomography image to 1 pixel / cubic millimeter.

[0044] To achieve the above object, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned multi-attribute classification methods for pulmonary nodules is implemented.

[0045] To achieve the above object, the present invention further provides a multi-attribute classification device for pulmonary nodules, including: a processor and a memory; the memory is used for storing a computer program; the processor is connected to the memory and is used for executing the computer program stored in the memory, so that the multi-attribute classification device for pulmonary nodules executes any of the above-mentioned multi-attribute classification methods for pulmonary nodules.

[0046] As described above, a multi-attribute classification method, system, medium and device for pulmonary nodules of the present invention have the following beneficial effects: being used for classifying the attributes presented by pulmonary nodules on CT images, so as to predict the degree of attribute presentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1a Shown is a flowchart of the multi-attribute classification method for pulmonary nodules of the present invention in an embodiment;

[0048] Figure 1b Shown is a structural diagram of a feature extraction module of the multi-attribute classification method for pulmonary nodules of the present invention in an embodiment;

[0049] Figure 1c Shown is a structural diagram of a multi-attribute classification model for pulmonary nodules of the present invention in an embodiment;

[0050] Figure 2 Shown is a schematic structural diagram of the multi-attribute classification system for pulmonary nodules of the present invention in an embodiment;

[0051] Figure 3 Shown is a schematic structural diagram of the multi-attribute classification device for pulmonary nodules of the present invention in an embodiment.

[0052] Description of Component Labels

[0053] 21 Acquisition Module

[0054] 22 Extraction Module

[0055] 23 Denoising Module

[0056] 24 Obtaining Module

[0057] 25 Classification Module

[0058] 31 Processor

[0059] 32 Memory DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The following specific examples illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0061] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present invention. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The types, quantities, and proportions of the components in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0062] The multi-attribute classification method, system, medium, and device for lung nodules of the present invention are used to classify the attributes presented by lung nodules on CT images, so as to predict the degree of attribute presentation.

[0063] As shown in FIG. 1, in one embodiment, the multi-attribute classification method for lung nodules of the present invention includes the following steps:

[0064] Step S11: Obtain a computed tomography image, establish a CT data representation method, and use the CT data representation method to process the computed tomography image to obtain a de-differenced computed tomography image that eliminates the differences in slice thickness and pixel pitch.

[0065] Specifically, the CT data representation method includes: truncating the computed tomography image within the range of -1200 to 600 Hounsfield units to obtain a computed tomography image within the range of -1200 to 600 Hounsfield units, and linearly converting it to the range of 0 to 255 to obtain a de-differenced computed tomography image. The measurement unit of the gray value of the computed tomography image is the Hounsfield unit, and the gray value range of the lung tomography image is -1024 to 31738 Hounsfield units. Most lesions in the lung image are composed of soft tissues, and the gray value range of the soft tissues is about 20 to 70 Hounsfield units. This value range accounts for a very small proportion in the total range of the lung image values, which is not conducive to the stability of the model. Truncating the computed tomography image within the range of -1200 to 600 Hounsfield units makes the proportion of valid data larger. Combining the window width and window level used by doctors in the diagnosis of lung diseases, using a window width of 900 and a window level of 300, that is, a value range of -1200 to 600 Hounsfield units to perform threshold truncation on the original image, and linearly transforming the range to 0 to 255 for subsequent steps.

[0066] The pixel pitch and slice thickness of the de-differenced computed tomography images are unified to 1 pixel per cubic millimeter using an interpolation method. After scale scaling, the differences in the scales of the three axes are unified, and all images have the same-scale image space, eliminating the differences caused by device parameters.

[0067] Step S12: Extract the differential computed tomography images based on the lung contour extraction technique to obtain lung contour images.

[0068] Specifically, the extracting the differential computed tomography images based on the lung contour extraction technique to obtain lung contour images includes:

[0069] Set a gray value threshold based on the difference between the gray values of lung air and lung wall tissue, and obtain the data higher than the gray value threshold to get a preliminary lung contour image. That is, utilize the difference between the gray values of lung air and lung wall tissue, and use the gray value threshold to preliminarily separate the lungs in the image from outside the lungs to obtain a preliminary lung contour image.

[0070] Remove the edge-connected regions of the preliminary lung contour image to obtain a lung contour image with the air regions removed.

[0071] Extract the two largest connected components of the lung contour image with the air regions removed as the lung region image; to remove noises such as trachea outside the lungs.

[0072] Perform an erosion operation on the lung region image using a disk with a radius of 2 pixels to separate the lung lesions attached to the blood vessels from the blood vessels, and obtain a lung region image without blood vessels.

[0073] Perform a closing operation on the lung region image without blood vessels using a disk with a radius of 10 pixels to fill the cavities caused by lung nodules attached to the lung wall, and obtain a lung region image with the cavities filled.

[0074] Fill the holes in the lung region image with the cavities filled to obtain a complete and smooth lung contour image. The holes refer to the relatively small holes other than the cavities caused by lung nodules.

[0075] Step S13: Denoise the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system.

[0076] Specifically, the denoising the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system includes: using a three-dimensional convex hull algorithm to extract the convex hull of the lungs from the lung contour image, and filling the region outside the convex hull with a preset gray value; the preset gray value includes: the gray value of water.

[0077] The three-dimensional dilation of the lung contour image is performed to obtain the intralung region, and the data that is within the convex hull and outside the intralung region is the data for bone removal. That is, a three-dimensional dilation is performed using the lung contour to obtain a region, which is regarded as the intralung region. The data that is within the convex hull and outside the intralung region is the data for bone removal.

[0078] Fill the image corresponding to the data for bone removal with a preset gray value; for example, fill the image corresponding to the data for bone removal with the gray value of water.

[0079] Calculate the bounding box based on the lung contour, and remove the region filled with the preset gray value to obtain the deboned lung contour image. After extraction, a clean lung region without bone can be obtained.

[0080] Convert the positions of the lung nodules in the deboned lung contour image to the relative coordinates of the lung to obtain a denoised lung contour image with a unified relative coordinate system. Filter the noise caused by the overlapping data value ranges inside and outside the lung, and unify the coordinate system of the data inside the lung.

[0081] Step S14: Extract the coordinates and sizes of the lung nodules from the denoised lung contour image to obtain the lung nodule image.

[0082] Specifically, the extracting the coordinates and sizes of the lung nodules from the denoised lung contour image to obtain the lung nodule image includes: calculating the circumscribed circle of the lung nodules in the denoised lung contour image, and taking the radius of the circumscribed circle as the size of the lung nodules, calculating the geometric centroid of the lung nodules, and taking the geometric centroid (x0, y0, z0) as the position coordinates of the lung nodules; converting the position coordinates of the lung nodules to the world coordinates to obtain the lung nodule image.

[0083] Specifically, calculate the circumscribed circle of the lung nodule, and take the radius d of the circumscribed circle as the size of the lung nodule. Calculate the geometric centroid of the lung nodule, and take the geometric centroid (x0, y0, z0) as the position coordinates of the lung nodule. Since the shapes of the lung nodules are diverse and irregular, directly using the center coordinates of the circumscribed circle may cause the position of the lung nodule to deviate from the lesion, thus bringing interference to the detection result. Convert the image coordinates to the world coordinates. Use formula (1) to convert the coordinates of the lung nodules, the image coordinate c v Convert to the world coordinate c through the pixel pitch s of the device parameter and the image origin o w . Due to the differences in machine parameters and individual differences, all lung images cannot be aligned to a unified world coordinate system. Therefore, a relatively unified coordinate space is needed;

[0084] c w =c v *s + o (1)

[0085] Convert the world coordinates to the same coordinate space. Normalize the coordinates using the origin coordinates and the length, width, and height of the processed lungs to obtain relative coordinates.

[0086] Step S15: Use a multi-attribute classification model based on deep learning to extract image features from the lung nodule image and perform attribute classification on the image features.

[0087] Specifically, using a multi-attribute classification model based on deep learning to extract image features from the lung nodule image and perform attribute classification on the image features includes:

[0088] The multi-attribute classification model based on deep learning includes a feature extraction module. As shown in Figure 1, the feature extraction module includes a residual structure and a dense connection structure; the multi-attribute classification model based on deep learning has multiple basic feature extraction modules, that is, a feature extraction module that fuses the residual structure and the dense connection structure. Among them, the residual structure is based on the concept of identity mapping, and the features are fused by skip-layer addition to obtain features x with the same number of channels as before. l 。

[0089] The residual structure is defined by the following formula:

[0090] x l =H l (x l-1 )+x l-1

[0091] where the output x l of the l-th layer is calculated by the input calculation unit H l-1 (·) of the feature x l of the (l-1)-th layer and added to the original feature x l-1 ; thus, the learning objective of the model changes from learning features to learning the residual part of the features.

[0092] The dense connection structure is defined by the following formula:

[0093] x l =H l ([x0,x1,…,x l-1 )

[0094] where the output x l of the l-th layer is calculated by using the concatenation of the features from the 0-th layer to the (l-1)-th layer as the input of the calculation unit H l (·); the dense connection structure uses a large number of shallow-depth convolutions to fuse the features by skip-layer concatenation to obtain features x l。The densely connected structure enables features to be reused at a high density, further reducing the number of parameters while efficiently extracting features. By combining the advantages of the residual structure and the densely connected structure, a two-stream fusion feature extraction module with relatively low number of parameters and computational cost and excellent performance can be obtained. This module can not only utilize the advantages of the long skip layer connection of the residual model in terms of features and gradient transfer, but also be compatible with the advantages of the densely connected module in terms of rapid feature iteration and high reusability.

[0095] The multi-attribute classification model includes a multi-scale feature extraction module, which is defined by the following formula:

[0096]

[0097] where x encoder and x decoder represent the features of the encoder and the decoder respectively, and the subscript l indicates the layer number of the encoder or decoder; concat(.) is a concatenation operation function between features, and upsample(.) is an upsampling function; the features of the (l - 1)-th layer decoder are restored to the same resolution as the features of the l-th layer encoder through a learnable upsampling layer, and then they are concatenated and passed through a two-stream fusion module to extract features, obtaining the features of the l-th layer decoder; the entire formula indicates that the features of the (l - 1)-th layer decoder are restored to the same resolution as the features of the l-th layer encoder through a learnable upsampling layer, and then they are concatenated and passed through a two-stream fusion module to extract features, obtaining the features of the l-th layer decoder. The encoder-decoder structure completes the fusion and extraction of abstract information and representational information by fusing high-resolution shallow features and low-resolution deep features multiple times.

[0098] The multi-attribute classification model includes 8 fully convolutional classifiers and 1 fully connected classifier; it is responsible for classifying the extracted multi-scale features into specific attributes.

[0099] The loss function of the multi-attribute classification model is a weighted cross-entropy loss function, and the cross-entropy loss function is defined by the following formula: The cross-entropy loss function used in this model is a combination of the Softmax non-linear function and the loss function of the logistic regression model. The Softmax function is a non-linear mapping function that maps a set of model output results to predicted class probability values.

[0100] Suppose an attribute has a total of K categories.

[0101]

[0102] where exp(·) is the natural exponential function; x k represents the predicted value of the k-th category; x jDenote the predicted value of the j-th category; k belongs to j; Softmax divides the exponent of the original predicted value of this category by the sum of the exponents of the predicted values of all categories, maps the category predicted value to the interval (0, 1), and makes the sum of the predicted values of all categories equal to 1. The multinomial logistic regression model for each category can be used for multi-class classification, and its loss function is as follows.

[0103] The loss function of the exp(·) is defined by the following formula:

[0104]

[0105] Among them, K represents the total number of categories of a certain attribute. There are a preset number of categories for a certain attribute. k represents a specific category of a certain attribute, and k ∈ [1, K]; y k represents whether the predicted category is the same as the sample category. If it is the same, it is 1, and if it is different, it is 0; p k represents the probability that the sample belongs to category k;

[0106] Substitute the Softmax(x k ) function into the loss(p, y) function to obtain the cross-entropy loss function formula for the class-th class of category k;

[0107]

[0108] The cross-entropy loss function formula can be split into two parts, which consists of the negative value of the probability value that category k is predicted as the class-th category and the logarithm of the sum of the probability exponents that category k is predicted as each category; represents the probability value that category k is predicted as the class-th category; represents the probability value that category k is predicted as the j-th category when class is j;

[0109] Add the loss function values of each attribute to obtain the final loss value function: as Figure 1c shown. At this time, there are a total of 9 attributes, which are: lesion type, margin, spiculation, lobulation, calcification, cavity, vascular convergence, pleural retraction.

[0110]

[0111] Among them, is the probability value that all categories k are predicted as the class-th category; the category weight consists of the normalized value of the reciprocal of the sample ratio: obtain the final loss value of the model. Use this loss function value for backpropagation to optimize the model parameters, and then the multi-attribute classification model can be learned. The category weight within the attribute consists of the normalized value of the reciprocal of the sample ratio:

[0112]

[0113] Among them, represents the sample size of class class belonging to category k, represents the weight of the class class of category k; the weight of the class class of k is composed of the reciprocal of the ratio of its sample size to the total number of samples; the larger the weight value, the smaller the sample size of this category. By adjusting the loss value of the class within the attribute with weights, the imbalance problem caused by the sample ratio can be basically solved.

[0114] The calculated class weights are used to weight the loss function values of each class to obtain:

[0115]

[0116] Taking the average of the loss function values of each class for 10 iterations respectively, the loss function value loss of each class can be obtained k ; represents the weight of the class class of category k;

[0117] The loss function weight values of each class are composed of the normalized values of the reciprocals of the class loss function value ratios. The loss function after weight adjustment is defined by the following formula:

[0118]

[0119] Among them, is the weight of the jth class of the kth class, w k is the weight of the kth class.

[0120] Specifically, the present invention is used to extract features from pulmonary nodule images to obtain multi-scale fused image features, and uses a multi-attribute classifier to perform multi-attribute classification on the multi-scale image features.

[0121] A multi-branch multi-attribute classifier for multi-attribute classification of pulmonary nodule images.

[0122] The present invention is used for the classification of six kinds of abnormal lung tissues such as pulmonary masses, nodules, ground glass, mutations, pleural effusions, and cavity lesions, as well as the classification of eight common imaging sign attributes such as the location of the lesion, the clarity of the edge, the degree of lobulation, the degree of spiculation, the degree of calcification, whether there is a cavity in the tissue, whether there is a vascular bundle, and whether there is an association with the pleura. This system can provide rich reference information for doctors to diagnose lung diseases.

[0123] Such as Figure 2 shown, in an embodiment, the multi-attribute classification system for pulmonary nodules of the present invention includes an acquisition module 21, an extraction module 22, a denoising module 23, an obtaining module 24, and a classification module 25.

[0124] The obtaining module 21 is configured to obtain a computed tomography image, establish a CT data representation method, and process the computed tomography image using the CT data representation method to obtain a de-differenced computed tomography image with the layer thickness and pixel pitch differences eliminated.

[0125] The extraction module 22 is configured to extract the de-differenced computed tomography image based on a lung contour extraction technique to obtain a lung contour image.

[0126] The denoising module 23 is configured to denoise the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system.

[0127] The obtaining module 24 is configured to extract the coordinates and sizes of lung nodules from the denoised lung contour image to obtain a lung nodule image;

[0128] The classification module 25 is configured to extract image features from the lung nodule image based on a multi-attribute classification model of deep learning and perform attribute classification on the image features.

[0129] It should be noted that the structures and principles of the obtaining module 21, the extraction module 22, the denoising module 23, the obtaining module 24, and the classification module 25 correspond one by one to the steps in the above multi-attribute classification method for lung nodules, so they will not be elaborated here.

[0130] It should be noted that it should be understood that the division of each module of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by a processing element; they can also all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device to perform the functions of the above x module. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or can be independently implemented. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit in the processor element or the instruction in the form of software.

[0131] For example, the above-mentioned modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or, one or more Micro Processor Units (MPUs), or, one or more Field Programmable Gate Arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processors that can call program code. Again, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0132] In an embodiment of the present invention, the present invention further includes a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements any one of the above-mentioned multi-attribute classification methods for pulmonary nodules.

[0133] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to a computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program code.

[0134] As Figure 3 As shown, in an embodiment, the acquisition module 21, extraction module 22, denoising module 23, obtaining module 24, and classification module 25 of the present invention include: a processor 31 and a memory 32; the memory 32 is used to store a computer program; the processor 31 is connected to the memory 32 and is used to execute the computer program stored in the memory 32, so that the acquisition module 21, extraction module 22, denoising module 23, obtaining module 24, and classification module 25 execute any one of the above-mentioned multi-attribute classification methods for pulmonary nodules.

[0135] Specifically, the memory 32 includes: various media such as ROM, RAM, magnetic disks, USB flash drives, memory cards, or optical discs that can store program code.

[0136] Preferably, the processor 31 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0137] In summary, the multi-attribute classification method, system, medium, and device for pulmonary nodules of the present invention are used to classify the attributes of pulmonary nodules shown in CT images, so as to predict the degree of attribute manifestation. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0138] The above embodiments are only illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A multi-attribute classification method for pulmonary nodules, characterized in that, Including the following steps: Obtain a computed tomography (CT) image, establish a CT data representation method, and process the CT image using the CT data representation method to obtain a de-differenced CT image with the layer thickness and pixel pitch differences eliminated; wherein, the CT data representation method includes: Perform truncation on the CT image within the range of -1200 to 600 Hounsfield units, obtain a CT image within the range of -1200 to 600 Hounsfield units, and linearly convert it to the range of 0 to 255 to obtain a de-differenced CT image; Use the interpolation method to unify the pixel pitch and layer thickness of the de-differenced CT image to 1 pixel per cubic millimeter; Extract the CT image with differences based on the lung contour extraction technique to obtain a lung contour image; Denoise the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system; Extract the coordinates and sizes of lung nodules from the denoised lung contour image to obtain a lung nodule image; Extract image features from the lung nodule image based on a deep learning multi-attribute classification model and perform attribute classification on the image features; wherein, the deep learning multi-attribute classification model includes a feature extraction module, a multi-scale feature extraction module, 8 fully convolutional classifiers, and 1 fully connected classifier.

2. The multi-attribute classification method for pulmonary nodules according to claim 1, wherein, The extracting the CT image with differences based on the lung contour extraction technique to obtain a lung contour image includes: Set a gray value threshold based on the difference between the gray values of lung air and lung wall tissue, and obtain data higher than the gray value threshold based on the gray value threshold to obtain a preliminary lung contour image; Remove the edge-connected regions of the preliminary lung contour image to obtain a lung contour image with the air region removed; Extract the two largest connected components of the lung contour image with the air region removed as a lung region image; Separate the lung lesions attached to blood vessels from the blood vessels to obtain a lung region image without blood vessels; Fill the cavities caused by lung nodules attached to the lung wall to obtain a lung region image with cavities filled; Fill the holes in the lung region image with cavities filled to obtain a complete and smooth lung contour image.

3. The multi-attribute classification method for pulmonary nodules according to claim 1, characterized in that, The denoising the lung contour image to obtain a denoised lung contour image with a unified relative coordinate system includes: Use the three-dimensional convex hull algorithm to extract the convex hull of the lung from the lung contour image, and fill the region outside the convex hull with a preset gray value; Perform three-dimensional dilation on the lung contour image to obtain the lung internal region, and obtain the data outside the lung internal region and inside the convex hull as the data that needs to remove bones; Fill the image corresponding to the data that needs to remove bones with a preset gray value; Calculate the bounding box according to the lung contour, and remove the region filled with the preset gray value to obtain a lung contour image without bones; Convert the positions of the lung nodules in the lung contour image without bones to the relative coordinates of the lung to obtain a denoised lung contour image with a unified relative coordinate system.

4. The multi-attribute classification method for pulmonary nodules according to claim 1, characterized in that, The extracting the coordinates and sizes of lung nodules from the denoised lung contour image to obtain a lung nodule image includes: Calculate the circumcircle of the pulmonary nodules in the denoised pulmonary contour image, and use the radius of the circumcircle as the size of the pulmonary nodules. Calculate the geometric centroid of the pulmonary nodules, and use the geometric centroid (x0, y0, z0) as the position coordinates of the pulmonary nodules; Convert the position coordinates of the pulmonary nodules into world coordinates to obtain a pulmonary nodule image.

5. The multi-attribute classification method for pulmonary nodules according to claim 1, wherein, Based on a deep learning multi-attribute classification model, extract image features from the pulmonary nodule image. The attribute classification of the image features includes: The feature extraction module includes a residual structure and a dense connection structure; the residual structure is defined by the following formula: x l = H l (x l-1 ) + x l-1 Among them, the output x of the l-th layer l is obtained by inputting the feature x of the (l - 1)-th layer l-1 into the calculation unit H l (·) for calculation and adding it to the original feature x l-1 ; The dense connection structure is defined by the following formula: x l = H l ([x0, x1, …, x l-1 ) Among them, the output x of the l-th layer l is obtained by using the feature concatenation from the 0-th layer to the (l-1)-th layer as the computing unit H l for input calculation of (·); The multi-scale feature extraction module is defined by the following formula: where x encoder and x decoder represent the features of the encoder and the decoder respectively, and the subscript l indicates the layer number where the encoder or decoder is located; concat(.) is a concatenation operation function for features, and upsample(.) is an upsampling function; the decoder features of the (l - 1)-th layer are restored to the same resolution as the encoder features of the l-th layer through a learnable upsampling layer, and then they are concatenated, and features are extracted through a two-stream fusion module to obtain the decoder features of the l-th layer; The loss function of the multi-attribute classification model is a weighted cross-entropy loss function, and the cross-entropy loss function is defined by the following formula: where exp(·) is the natural exponential function; x k represents the predicted value of the k-th category; x j represents the predicted value of the j-th category; k belongs to j; Softmax divides the exponent of the original predicted value of this category by the sum of the exponents of the predicted values of all categories, maps the category predicted value to the interval (0, 1), and makes the sum of the predicted values of all categories equal to 1; The loss function of exp(·) is defined by the following formula: Among them, K represents the total number of categories of a certain attribute. There are a preset number of categories for a certain attribute. k represents a specific category of a certain attribute, and k ∈ [1, K]; y k represents whether the predicted category is the same as the sample category. If it is the same, it is 1; if it is different, it is 0; p k represents the probability that the sample belongs to category k; Bring the Softmax(x k ) function into the loss(p, y) function to obtain the cross-entropy loss function formula for the class of the k-th class of the class The cross-entropy loss function formula can be split into two parts, which consists of the negative of the probability value that class k is predicted as the class-th class and the logarithm of the sum of the probability exponents that class k is predicted as each class; represents the probability value that class k is predicted as the class-th class; represents the probability value that class k is predicted as the j-th class when class is j; Add the loss function values of each attribute to obtain the final loss value function: Among them, is the probability value that all categories k are predicted as the class-th category; the category weights consist of the normalized values of the reciprocals of the sample ratios: Among them, represents the sample size of class class representing category k, represents the weight of the class class of category k; the weight of the class class of k is composed of the reciprocal of the proportion of its sample size in the total number of samples; The calculated class weights are used to weight the loss function values of each class to obtain: The loss function values of each category for 10 iterations are averaged respectively to obtain the loss function value loss of each category k ; represents the weight of the class of the k-th category The loss function weight values of each class consist of the normalized values of the reciprocals of the class loss function value ratios. The loss function after weight adjustment is defined by the following formula: Among them, is the weight of the j-th category of the k-th category, w k is the weight of the k-th category.

6. A multi-attribute classification system for pulmonary nodules, characterized in that, Include: An acquisition module, an extraction module, a denoising module, an obtaining module, and a classification module; The acquisition module is used to acquire computed tomography images, establish a CT data representation method, and use the CT data representation method to process the computed tomography images to obtain de-differenced computed tomography images that eliminate differences in slice thickness and pixel spacing; wherein, the CT data representation method includes: Perform truncation on the computed tomography images within the range of -1200 to 600 Hounsfield units to obtain computed tomography images within the range of -1200 to 600 Hounsfield units, and linearly convert them to the range of 0 to 255 to obtain de-differenced computed tomography images; use the interpolation method to unify the pixel spacing and slice thickness of the de-differenced computed tomography images to 1 pixel per cubic millimeter; The extraction module is used to extract the computed tomography images with differences based on pulmonary contour extraction technology to obtain pulmonary contour images; The denoising module is used to denoise the pulmonary contour images to obtain denoised pulmonary contour images with a unified relative coordinate system; The obtaining module is used to extract the coordinates and sizes of pulmonary nodules from the denoised pulmonary contour images to obtain pulmonary nodule images; The classification module is used to extract image features from the pulmonary nodule images based on a deep learning multi-attribute classification model and perform attribute classification on the image features; wherein, the deep learning-based multi-attribute classification model includes a feature extraction module, a multi-scale feature extraction module, 8 fully convolutional classifiers, and 1 fully connected classifier.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement the multi-attribute classification method of pulmonary nodules according to any one of claims 1 to 5.

8. A multi-attribute classification device for pulmonary nodules, characterized in that, Include: A processor and a memory; The memory is used to store a computer program; The processor is connected to the memory and is configured to execute the computer program stored in the memory, so that the multi-attribute classification device for pulmonary nodules executes the multi-attribute classification method for pulmonary nodules according to any one of claims 1 to 5.

Images