Lung lesion classification method, device and equipment based on convolutional neural network

Abstract

The invention provides a lung lesion classification method, device and equipment based on a convolutional neural network. The classification method comprises the following steps: acquiring a CT image of a lung of a patient; inputting the CT image into a lung lesion classification model to obtain a classification result for the lung lesion of the patient; wherein the lung lesion classification model comprises a preprocessing network, an index extraction network, a convolutional neural network, a feature fusion network and a first classification network. By adopting the technical scheme provided by the invention, the model generalization ability can be improved, and the accuracy of CT image classification in practical application is improved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image classification, and in particular to a method, device and equipment for classifying lung lesions based on a convolutional neural network. Background Art

[0002] For lung lesions, such as chronic obstructive pulmonary disease (COPD), which is a chronic bronchitis and / or emphysema with airflow obstruction, it can further develop into common chronic diseases such as cor pulmonale and respiratory failure, with high disability and mortality rates. CT imaging is an important means for screening COPD (Chronic Obstructive Pulmonary Disease), and CT imaging can effectively evaluate the damage to the lung parenchyma.

[0003] At present, convolutional neural networks can be used to extract features from CT images. Based on the rules between features and symptoms learned during training, features can be classified into negative and positive categories to assist diagnosis. However, due to the scarcity of sample data and the single data dimension, this method has poor generalization ability of the network, which affects the accuracy of CT image classification in practical applications. Therefore, how to improve the accuracy of lung lesion classification in CT images has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, device and equipment for classifying lung lesions based on convolutional neural networks, which can classify CT images through a lung lesion classification model composed of a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network, and obtain the classification results of the lung lesions of the patient. By using the indicator extraction network, the convolutional neural network and the feature fusion network, the generalization ability of the model is improved, and the accuracy of CT image classification in practical applications is improved.

[0005] This application mainly includes the following aspects:

[0006] In a first aspect, an embodiment of the present application provides a method for classifying lung lesions based on a convolutional neural network, the method comprising:

[0007] Obtain CT images of the patient's lungs;

[0008] The CT image is input into a lung lesion classification model to obtain a classification result of the patient's lung lesions; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

[0009] Furthermore, the step of inputting the CT image into a lung lesion classification model to obtain a classification result of the lung lesions of the patient includes:

[0010] Inputting the CT image into the preprocessing network for preprocessing to obtain a preprocessed image output by the preprocessing network;

[0011] Inputting the preprocessed image into the convolutional neural network for feature extraction to obtain feature data output by the convolutional neural network;

[0012] Inputting the CT image into the index extraction network to obtain index data output by the index extraction network;

[0013] Inputting the feature data and the indicator data into the feature fusion network for splicing to obtain fusion features output by the feature fusion network;

[0014] The fused features are input into the first classification network for classification, and a classification result of the lung lesions of the patient output by the first classification network is obtained.

[0015] Furthermore, the step of inputting the CT image into the index extraction network to obtain the index data output by the index extraction network includes:

[0016] Inputting the CT image into an index extraction network to obtain preset indexes and formulas corresponding to each preset index;

[0017] For each preset indicator, based on the CT image, determine the parameter value of the parameter required in the formula corresponding to the preset indicator, substitute the determined parameter value into the formula corresponding to the preset indicator, and obtain the indicator data of the preset indicator;

[0018] Obtaining the maximum value and the minimum value of each preset indicator in the sample data in the training set, and normalizing the indicator data of the corresponding preset indicator based on the maximum value and the minimum value of each preset indicator to obtain the normalized indicator data of the corresponding preset indicator;

[0019] The indicator data of each preset indicator is updated to the normalized indicator data of the corresponding preset indicator, so as to obtain the indicator data of each preset indicator output by the indicator extraction network.

[0020] Furthermore, the step of inputting the CT image into a preprocessing network for preprocessing to obtain a preprocessed image output by the preprocessing network includes:

[0021] Inputting the CT image into a preprocessing network, extracting a lung region mask from the CT image by an image processing method, and obtaining a lung region mask frame;

[0022] intercepting the CT image according to the lung region mask frame to obtain a lung region mask and the intercepted CT image;

[0023] Resampling the lung region mask to a preset size to obtain a lung region binary image;

[0024] Resampling the intercepted CT image to a preset size to obtain a target CT image;

[0025] Acquire a target pixel point with a pixel value of zero in the binary image of the lung region, and update the tissue density value of the pixel point corresponding to the target pixel point in the target CT image to a preset threshold value to obtain an updated target CT image;

[0026] Acquire a window width value and a window level value, and determine a region of interest from the updated target CT image based on the window width value and the window level value;

[0027] The tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are less than the lower limit of the tissue density value of the region of interest are updated to the lower limit of the tissue density value, and the tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are greater than the upper limit of the tissue density value of the region of interest are updated to the upper limit of the tissue density value, to obtain the target image;

[0028] Normalizing the target image to obtain a normalized target image, and processing the target CT image to obtain an emphysema characteristic image;

[0029] The normalized target image and the emphysema characteristic image are stacked to obtain a preprocessed image output by the preprocessing network.

[0030] Furthermore, the step of inputting the fusion feature into the first classification network for classification to obtain a classification result of the lung lesions of the patient output by the first classification network includes:

[0031] Inputting the fused features into the first classification network for classification to obtain a positive score and a negative score of the fused features;

[0032] The positive class score is converted into a first probability that the fused feature belongs to the positive class by a normalization function, and the negative class score is converted into a second probability that the fused feature belongs to the negative class by the normalization function;

[0033] Determine whether the first probability that the fused feature belongs to the positive class meets a preset condition; wherein the preset condition is that the first probability that the fused feature belongs to the positive class is greater than the second probability that the fused feature belongs to the negative class, or the first probability that the fused feature belongs to the positive class is greater than the preset probability;

[0034] If yes, then taking the indicator mark of lung lesions as the classification result of the lung lesions of the patient output by the first classification network;

[0035] If not, the indicator that the lungs do not have lesions is used as the classification result of the lung lesions for the patient output by the first classification network.

[0036] Furthermore, after inputting the fused features into the first classification network for classification to obtain the positive class score and the negative class score of the fused features, the classification method further includes:

[0037] Inputting the index data output by the index extraction network into the second classification network to obtain negative weights and positive weights;

[0038] Multiplying the positive weight by the positive class score to obtain first data, and converting the first data into a third probability that the fused feature belongs to the positive class through the normalization function;

[0039] The negative weight is multiplied by the negative class score to obtain second data, and the second data is converted into a fourth probability that the fused feature belongs to the negative class through the normalization function.

[0040] Furthermore, when the preset indicator is the volume ratio of the trachea to the tracheal wall, the step of determining the parameter value of the parameter required in the formula corresponding to the preset indicator based on the CT image, substituting the determined parameter value into the formula corresponding to the preset indicator to obtain the indicator data of the preset indicator includes:

[0041] Extracting the lung trachea in the CT image from the CT image to obtain the trachea volume;

[0042] Based on the CT image and the lung trachea, extract the tracheal wall in the CT image to obtain the tracheal wall volume;

[0043] The ratio of the trachea volume to the tracheal wall volume is determined as the index data of the volume ratio of the trachea to the tracheal wall.

[0044] In a second aspect, the embodiment of the present application further provides a classification device for lung lesions based on a convolutional neural network, the classification device comprising:

[0045] An acquisition module, used for acquiring a CT image of a patient's lungs;

[0046] A classification module is used to input the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

[0047] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned method for classifying lung lesions based on convolutional neural networks are performed.

[0048] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the above-mentioned method for classifying lung lesions based on a convolutional neural network.

[0049] The embodiments of the present application provide a method, device and equipment for classifying lung lesions based on a convolutional neural network. The classification method includes: obtaining a CT image of a patient's lungs; inputting the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a classification network.

[0050] In this way, the technical solution provided in the present application can classify CT images through a lung lesion classification model composed of a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network, and obtain a classification result of the lung lesions of the patient. By using the indicator extraction network, the convolutional neural network and the feature fusion network, the generalization ability of the model is improved, and the accuracy of CT image classification in practical applications is improved.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flow chart of a method for classifying lung lesions based on a convolutional neural network provided in an embodiment of the present application is shown;

[0054] Figure 2 A flow chart of another method for classifying lung lesions based on a convolutional neural network provided in an embodiment of the present application is shown;

[0055] Figure 3 A schematic diagram of the structure of a lung lesion classification device based on a convolutional neural network provided in an embodiment of the present application is shown;

[0056] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0058] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0059] In order to enable those skilled in the art to use the contents of this application, the following implementation is given in combination with the specific application scenario "Classification of lung lesions based on convolutional neural networks". For those skilled in the art, the general principles defined here can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application.

[0060] The following methods, devices, electronic devices or computer-readable storage media of the embodiments of the present application can be applied to any scenario that requires classification of lung lesions based on a lung lesion classification model. The embodiments of the present application are not limited to specific application scenarios. Any solution using a lung lesion classification method, device, electronic device and storage medium based on a convolutional neural network provided by the embodiments of the present application is within the scope of protection of the present application.

[0061] It is worth noting that for lung diseases, such as chronic obstructive pulmonary disease (COPD), which is a chronic bronchitis and / or emphysema with airflow obstruction, it can further develop into common chronic diseases such as cor pulmonale and respiratory failure, with high disability and mortality rates. CT imaging is an important means for screening COPD (Chronic Obstructive Pulmonary Disease), and CT imaging can effectively evaluate the damage to the lung parenchyma.

[0062] At present, convolutional neural networks can be used to extract features from CT images. Based on the rules between features and symptoms learned during training, features can be classified into negative and positive categories to assist diagnosis. However, due to the scarcity of sample data and the single data dimension, this method has poor generalization ability of the network, which affects the accuracy of CT image classification in practical applications. Therefore, how to improve the accuracy of lung lesion classification in CT images has become an urgent problem to be solved.

[0063] Based on this, the present application proposes a classification method, device and equipment for lung lesions based on convolutional neural networks. The classification method includes: obtaining a CT image of the patient's lungs; inputting the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

[0064] In this way, the technical solution provided in the present application can classify CT images through a lung lesion classification model composed of a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a classification network, and obtain a classification result of the lung lesions of the patient. By using the indicator extraction network, the convolutional neural network and the feature fusion network, the generalization ability of the model is improved, and the accuracy of CT image classification in practical applications is improved.

[0065] To facilitate understanding of the embodiments of the present application, a method for classifying lung lesions based on a convolutional neural network disclosed in the embodiments of the present application is first introduced in detail.

[0066] See also Figure 1 , Figure 1 A flow chart of a method for classifying lung lesions based on a convolutional neural network provided in an embodiment of the present application, such as Figure 1 As shown in , the classification methods include:

[0067] S101. Obtain a CT image of the patient's lungs.

[0068] S102: Input the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions.

[0069] In this step, the lung lesion classification model includes a preprocessing network, an index extraction network, a convolutional neural network, a feature fusion network and a first classification network.

[0070] Please note that Figure 2 , Figure 2 A flowchart of another method for classifying lung lesions based on a convolutional neural network provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the step of inputting the CT image into the lung lesion classification model to obtain the classification result of the lung lesion of the patient includes:

[0071] S201, inputting the CT image into the preprocessing network for preprocessing, and obtaining a preprocessed image output by the preprocessing network.

[0072] It should be noted that the step of inputting the CT image into the preprocessing network for preprocessing to obtain the preprocessed image output by the preprocessing network includes:

[0073] S2011. Input the CT image into a preprocessing network, extract the lung region mask from the CT image through an image processing method, and obtain a lung region mask frame.

[0074] In this step, the CT image includes a background area and a desired lung area. The background area can be understood as an unwanted area, which here refers to the extra-pulmonary area. In order to eliminate the interference of the extra-pulmonary area as much as possible, an image processing method can be used to extract the lung area mask in the CT image and generate a box of the mask.

[0075] S2012: intercepting the CT image according to the lung region mask frame to obtain a lung region mask and a intercepted CT image.

[0076] In this step, the mask in the CT image is intercepted through the box to obtain the lung area mask and the intercepted CT image.

[0077] S2013. Resample the lung region mask to a first preset size to obtain a lung region binary image.

[0078] In this step, since the size of each person's lungs is different, resampling can standardize the image, so the lung area mask is resampled to a first preset size. The first preset size here refers to the size of a unified input image. Here, as an example, the lung area mask can be resampled to a preset size by the nearest collar interpolation method, and the specific algorithm for resampling is not specifically limited in this application.

[0079] S2014: resample the captured CT image to a second preset size to obtain a target CT image.

[0080] In this step, the intercepted CT image is resampled to the same size as the lung region binary image. For example, the lung region mask and the intercepted CT image can be resampled to a preset size [300, 300, 300] or [224, 224, 224]. As an example, the intercepted CT image can be resampled to a preset size by trilinear interpolation. The specific algorithm for resampling is not specifically limited in this application.

[0081] S2015. Obtain a target pixel point with a pixel value of zero in the binary image of the lung region, and update the tissue density value of the pixel point corresponding to the target pixel point in the target CT image to a preset threshold value to obtain an updated target CT image.

[0082] In this step, the grayscale value of the pixel in the binary image of the lung region is either 1 or 0. If the grayscale value of the corresponding pixel in the binary image of the lung region is 0, the HU value (tissue density value) at the corresponding pixel position of the resampled CT image is set to a preset threshold (e.g., -1024). The specific value of the preset threshold here is set so that the background area becomes 0 after normalization.

[0083] S2016, obtaining a window width value and a window level value, and determining a region of interest from the updated target CT image based on the window width value and the window level value.

[0084] In this step, the window position and window width are set for the lung tissue. The window position is the center position representing the grayscale of the image during image display, and the window width is the span of the grayscale of the image. According to the window width and window position settings, a region can be obtained. For example, the window width value is 1324 and the window width value is -362, and the region of interest can be determined from the target CT image.

[0085] S2017. Update the tissue density values ​​of the pixel points in the updated target CT image whose tissue density values ​​are less than the lower limit of the tissue density value of the region of interest to the lower limit of the tissue density value, and update the tissue density values ​​of the pixel points in the updated target CT image whose tissue density values ​​are greater than the upper limit of the tissue density value of the region of interest to the upper limit of the tissue density value, to obtain the target image.

[0086] In this step, the lower limit value and the upper limit value of the tissue density value of the region of interest are determined, and then the tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are less than the lower limit value of the tissue density value are updated to the lower limit value of the tissue density value, and the tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are greater than the upper limit value of the tissue density value are updated to the upper limit value of the tissue density value, so as to obtain the target image. Continuing with the above embodiment, when the tissue density value range in the region of interest is [-1024, 300], the HU values ​​of the pixels in the region where the tissue density value in the target CT image is less than -1024 can be set to -1024 through this region, and the HU values ​​of the pixels in the region where the HU value is greater than 300 can be set to 300, so as to realize the division of the background region and the lung region.

[0087] S2018. Normalize the target image to obtain a normalized target image, and process the target CT image to obtain an emphysema characteristic image.

[0088] In this step, the CT value of the target image is first normalized to the grayscale range (0 to 1) to obtain a normalized target image. Then, the target CT image is processed to obtain an emphysema characteristic image. In a specific implementation, according to the target CT image that is cropped and resampled using the lung region mask, the value of the area in the lung region mask of the target CT image with a threshold value less than -950 is set to 1, and the value of other areas is set to 0, to obtain an emphysema characteristic image.

[0089] S2019. Stacking the normalized target image and the emphysema feature image to obtain a preprocessed image output by the preprocessing network.

[0090] In this step, the normalized target image and the emphysema feature image are stacked and processed as the preprocessed image output by the preprocessing network.

[0091] S202: Input the preprocessed image into the convolutional neural network for feature extraction to obtain feature data output by the convolutional neural network.

[0092] In this step, the convolutional neural network can be a conventional convolutional neural network. In some embodiments, the convolutional neural network can be one of efficientnet, resnet, ViT, densenet, LeNet, AlexNet, ZFNeet, VGGNet, ResNet, GoogLeNet, Swin-transformer or multi-view. Preferably, the convolutional neural network is Swin-transformer.

[0093] Here, firstly, the above step S202 is described in detail by taking Swin-transformer as an embodiment of a convolutional neural network:

[0094] The network uses a multi-channel strategy. The preprocessed image is obtained by stacking the normalized target image and the emphysema feature image. Therefore, the input channel includes the processed CT image and its corresponding emphysema feature as the image input of the convolutional neural network. These inputs are stacked into a four-dimensional tensor with a size of (2, 224, 224, 224), where the first channel is the preprocessed CT image and the second channel is the corresponding emphysema feature. Feature extraction is performed through the Swin-transformer network to obtain feature data.

[0095] Here, the above step S202 is described in detail by taking a multi-view network as an embodiment of a convolutional neural network:

[0096] When the convolutional neural network is a multi-view network, the preprocessed image is input into the multi-view network for multi-view division to obtain multiple division units, and features are extracted for each division unit, and the features corresponding to each division unit are spliced ​​to obtain feature data output by the multi-view network.

[0097] Here, the multi-view network uses a multi-view algorithm for feature extraction. Through the multi-view network, the consistency and complementarity between different views can be used in practical applications to achieve better clustering quality, that is, multiple features are merged into a comprehensive feature. Specifically, the preprocessed image is input into the multi-view network, and the preprocessed image is first divided to obtain multiple division units. Then, a convolutional neural network with shared parameters is used to extract features from each division unit, and the features of each division unit are spliced ​​to obtain feature data output by the multi-view network. As an example, the convolutional neural network can be one of LeNet, AlexNet, ZFNeet, VGGNet, ResNet, or GoogLeNet. Preferably, the convolutional neural network is ResNet.

[0098] In this embodiment, the division of the preprocessed image refers to dividing the image of a preset size into multiple division units. For example, for a preprocessed image with a preset size of [300, 300, 300], an equal division method can be adopted, such as dividing it into 8 division units of a size of [150, 150, 150], 27 division units of a size of [100, 100, 100], or 64 division units of a size of [75, 75, 75]. This application does not specifically limit the division method of the preprocessed image in the multi-view network.

[0099] S203, inputting the CT image into the index extraction network to obtain index data output by the index extraction network.

[0100] It should be noted that the step of inputting the CT image into the index extraction network to obtain the index data output by the index extraction network includes:

[0101] S2031. Input the CT image into an index extraction network to obtain preset indexes and formulas corresponding to each preset index.

[0102] In this step, the preset indicators can be the proportion of airway lesion area, the proportion of emphysema area, and the volume ratio of trachea to tracheal wall, etc. Each preset indicator has a corresponding calculation formula, for example, the calculation formula corresponding to the proportion of airway lesion area is: the ratio of the number of pixels in the lung area with HU value less than the first threshold (for example, -856) to the number of all pixels in the lung area, that is, the proportion of the area in the lung area with HU value less than -856; the calculation formula corresponding to the proportion of emphysema area is: the ratio of the number of pixels in the lung area with HU value less than the second threshold (for example, -950) to the number of all pixels in the lung area, that is, the proportion of the area in the lung area with HU value less than -950; the calculation formula corresponding to the volume ratio of trachea to tracheal wall is: the ratio of tracheal volume to tracheal wall volume.

[0103] S2032. For each preset indicator, based on the CT image, determine the parameter value of the parameter required in the formula corresponding to the preset indicator, substitute the determined parameter value into the formula corresponding to the preset indicator, and obtain the indicator data of the preset indicator.

[0104] It should be noted that when the preset indicator is the volume ratio of the trachea to the tracheal wall, the step of determining the parameter value of the parameter required in the formula corresponding to the preset indicator based on the CT image, substituting the determined parameter value into the formula corresponding to the preset indicator, and obtaining the indicator data of the preset indicator includes:

[0105] 1) Extracting the lung trachea in the CT image from the CT image to obtain the tracheal volume.

[0106] 2) Based on the CT image and the lung trachea, extract the tracheal wall in the CT image to obtain the tracheal wall volume.

[0107] 3) Determine the ratio of the tracheal volume to the tracheal wall volume as index data of the volume ratio of the trachea to the tracheal wall.

[0108] The above steps 1) to 3) are specific steps for determining the index data of the volume ratio of the trachea to the tracheal wall. After obtaining the calculation formula corresponding to the volume ratio of the trachea to the tracheal wall, it is necessary to determine the parameter values ​​of the required parameters in the formula, that is, the parameter value of the tracheal volume and the parameter value of the tracheal wall volume. First, the pulmonary trachea in the CT image is extracted from the CT image to obtain the tracheal volume. As an example, the pulmonary trachea segmentation algorithm in the index extraction network can be used to extract the pulmonary trachea in the CT image. Here, any method that can determine the pulmonary trachea in the CT image is acceptable without specific limitation. Then, based on the CT image and the above-extracted pulmonary trachea, the tracheal wall in the CT image is extracted to obtain the tracheal wall volume. As an example, the half-peak full-moment algorithm can be used to extract the tracheal wall in the CT image. Here, any method that can determine the tracheal wall in the CT image is acceptable without specific limitation. Finally, the ratio of the tracheal volume to the tracheal wall volume is determined as the index data of the volume ratio of the trachea to the tracheal wall.

[0109] S2033. Obtain the maximum value and the minimum value of each preset indicator in the sample data in the training set, and normalize the indicator data corresponding to the preset indicator based on the maximum value and the minimum value of each preset indicator to obtain the normalized indicator data of the corresponding preset indicator.

[0110] In this step, for each preset indicator, the preset indicator is normalized according to the maximum and minimum values ​​of the preset indicator in the training set, and the normalization formula is: (indicator data-minimum value) / (maximum value-minimum value).

[0111] S2034. Update the indicator data of each preset indicator to the normalized indicator data of the corresponding preset indicator, and obtain the indicator data of each preset indicator output by the indicator extraction network.

[0112] S204: Input the feature data and the indicator data into the feature fusion network for splicing to obtain fusion features output by the feature fusion network.

[0113] In this step, each obtained indicator data is concatenated with the feature data and input as new feature data (i.e., fused features) into the first classification network for classification.

[0114] S205: Input the fused features into the first classification network for classification, and obtain a classification result of the lung lesions of the patient output by the first classification network.

[0115] The first classification network here is a classification network including a fully connected layer (FC), and the lung lesion results are classified by the first classification network. In this step, the fusion features are input into the first classification network for classification, and the classification results of the lung lesions of the patient output by the first classification network are obtained.

[0116] It should be noted that the step of inputting the fusion feature into the first classification network for classification to obtain the classification result of the lung lesions of the patient output by the first classification network includes:

[0117] S2051. Input the fused features into the first classification network for classification to obtain a positive score and a negative score of the fused features.

[0118] Here, the positive class can be understood as the lungs having lesions, and the negative class can be understood as the lungs not having lesions. In this step, the fused features are input into the first classification network to obtain the positive class score and the negative class score of the fused features.

[0119] S2052. Convert the positive class score into a first probability that the fused feature belongs to the positive class through a normalization function, and convert the negative class score into a second probability that the fused feature belongs to the negative class through the normalization function.

[0120] In this step, after obtaining the positive class score and the negative class score of the fused feature, the positive class score is converted into the first probability that the fused feature belongs to the positive class through the normalization function, and the negative class score is converted into the second probability that the fused feature belongs to the negative class through the normalization function. As an example, the normalization function can be one of the Min-Max function, the Z-Score function, the Softmax function, the L1 normalization function or the L2 normalization function. Preferably, the normalization function is the Softmax function, which can convert each element of the input vector into a value between 0 and 1, and the sum of all elements is 1, so that the probability distribution can be represented. Specifically, the positive class score is converted into the first probability that the fused feature belongs to the positive class through the Softmax function, and the negative class score is converted into the second probability that the fused feature belongs to the negative class through the Softmax function.

[0121] S2053: Determine whether the first probability that the fused feature belongs to the positive class meets a preset condition.

[0122] In this step, the preset condition may be that the first probability that the fused feature belongs to the positive class is greater than the second probability that the fused feature belongs to the negative class, or that the first probability that the fused feature belongs to the positive class is greater than the preset probability.

[0123] S2054: If yes, use the indicator of lung lesions as the classification result of the lung lesions of the patient output by the first classification network.

[0124] In this step, the indicator mark indicating that the lung has a lesion can be recorded as 1, and the output classification result is 1, indicating that the patient's lung has a lesion.

[0125] S2055: If not, taking the indicator that the lungs do not have lesions as the classification result of the lung lesions of the patient output by the first classification network.

[0126] In this step, the indicator mark indicating that the lungs do not have lesions can be recorded as 0, and the output classification result is 0, indicating that the patient's lungs do not have lesions.

[0127] As an optional embodiment, after inputting the fused feature into the first classification network for classification in the above step S2051 to obtain the positive class score and the negative class score of the fused feature, the classification method further includes:

[0128] A: Input the index data output by the index extraction network into the second classification network to obtain negative weights and positive weights.

[0129] Here, the second classification network outputs the positive and negative class weights obtained by normalization function processing. The second classification network is a classification network including a fully connected layer (FC). In this step, the indicator data output by the indicator extraction network is input into the second classification network to obtain negative weights and positive weights.

[0130] B: multiplying the positive weight by the positive class score to obtain first data, and converting the first data into a third probability that the fused feature belongs to the positive class through the normalization function.

[0131] C: multiplying the negative weight by the negative class score to obtain second data, and converting the second data into a fourth probability that the fused feature belongs to the negative class through the normalization function.

[0132] In the above steps B-C, the positive weight is multiplied by the positive class score output by the first classification network to obtain the first data, and the first data is converted into the third probability that the fusion feature belongs to the positive class through the normalization function. The negative weight is multiplied by the negative class score output by the first classification network to obtain the second data, and the second data is converted into the fourth probability that the fusion feature belongs to the negative class through the normalization function. Further, the final classification result can be determined based on the third probability that the fusion feature belongs to the positive class and the fourth probability that the fusion feature belongs to the negative class. As an optional embodiment, the sofmax function can also be used to correct the third probability and the fourth probability, and then the final classification result is obtained by the corrected third probability and the corrected fourth probability.

[0133] The classification method provided in this embodiment uses CT images to screen lung lesions such as COPD. Specifically, a lung lesion classification model is used to extract relevant lesion features in various areas of the lung, and feature fusion is performed by combining the indicator data of multiple indicators. Finally, the task of negative and positive classification of lung lesions is achieved through a fully connected layer and weight multiplication, thereby improving the generalization ability of the model and the accuracy of classification.

[0134] An embodiment of the present application provides a method for classifying lung lesions based on a convolutional neural network, the classification method comprising: obtaining a CT image of a patient's lungs; inputting the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions; wherein the lung lesion classification model comprises a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a classification network.

[0135] In this way, the technical solution provided in the present application can classify CT images through a lung lesion classification model composed of a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network, and obtain a classification result of the lung lesions of the patient. By using the indicator extraction network, the convolutional neural network and the feature fusion network, the generalization ability of the model is improved, and the accuracy of CT image classification in practical applications is improved.

[0136] Based on the same inventive concept, the embodiment of the present application also provides a classification device for lung lesions based on a convolutional neural network corresponding to the above-mentioned classification method for lung lesions based on a convolutional neural network. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0137] See also Figure 3 , Figure 3 A schematic diagram of the structure of a lung lesion classification device based on a convolutional neural network provided in an embodiment of the present application is shown in FIG. Figure 3As shown in , the classification device 310 includes:

[0138] An acquisition module 311 is used to acquire a CT image of the patient's lungs;

[0139] The classification module 312 is used to input the CT image into the lung lesion classification model to obtain the classification result of the lung lesions of the patient; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

[0140] Optionally, the classification module 312 is specifically used for:

[0141] Inputting the CT image into the preprocessing network for preprocessing to obtain a preprocessed image output by the preprocessing network;

[0142] Inputting the preprocessed image into the convolutional neural network for feature extraction to obtain feature data output by the convolutional neural network;

[0143] Inputting the CT image into the index extraction network to obtain index data output by the index extraction network;

[0144] Inputting the feature data and the indicator data into the feature fusion network for splicing to obtain fusion features output by the feature fusion network;

[0145] The fused features are input into the first classification network for classification, and a classification result of the lung lesions of the patient output by the first classification network is obtained.

[0146] Optionally, when the classification module 312 is used to input the CT image into the index extraction network to obtain the index data output by the index extraction network, the classification module 312 is specifically used to:

[0147] Inputting the CT image into an index extraction network to obtain preset indexes and formulas corresponding to each preset index;

[0148] For each preset indicator, based on the CT image, determine the parameter value of the parameter required in the formula corresponding to the preset indicator, substitute the determined parameter value into the formula corresponding to the preset indicator, and obtain the indicator data of the preset indicator;

[0149] Obtaining the maximum value and the minimum value of each preset indicator in the sample data in the training set, and normalizing the indicator data of the corresponding preset indicator based on the maximum value and the minimum value of each preset indicator to obtain the normalized indicator data of the corresponding preset indicator;

[0150] The indicator data of each preset indicator is updated to the normalized indicator data of the corresponding preset indicator, so as to obtain the indicator data of each preset indicator output by the indicator extraction network.

[0151] Optionally, when the classification module 312 is used to input the CT image into a preprocessing network for preprocessing to obtain a preprocessed image output by the preprocessing network, the classification module 312 is specifically used to:

[0152] Inputting the CT image into a preprocessing network, extracting a lung region mask from the CT image by an image processing method, and obtaining a lung region mask frame;

[0153] intercepting the CT image according to the lung region mask frame to obtain a lung region mask and the intercepted CT image;

[0154] Resampling the lung region mask to a preset size to obtain a lung region binary image;

[0155] Resampling the intercepted CT image to a preset size to obtain a target CT image;

[0156] Acquire a target pixel point with a pixel value of zero in the binary image of the lung region, and update the tissue density value of the pixel point corresponding to the target pixel point in the target CT image to a preset threshold value to obtain an updated target CT image;

[0157] Acquire a window width value and a window level value, and determine a region of interest from the updated target CT image based on the window width value and the window level value;

[0158] The tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are less than the lower limit of the tissue density value of the region of interest are updated to the lower limit of the tissue density value, and the tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are greater than the upper limit of the tissue density value of the region of interest are updated to the upper limit of the tissue density value, to obtain the target image;

[0159] Normalizing the target image to obtain a normalized target image, and processing the target CT image to obtain an emphysema characteristic image;

[0160] The normalized target image and the emphysema characteristic image are stacked to obtain a preprocessed image output by the preprocessing network.

[0161] Optionally, the classification module 312 inputs the fusion feature into the first classification network for classification to obtain a classification result of the lung lesions of the patient output by the first classification network, including:

[0162] Inputting the fused features into the first classification network for classification to obtain a positive score and a negative score of the fused features;

[0163] The positive class score is converted into a first probability that the fused feature belongs to the positive class by a normalization function, and the negative class score is converted into a second probability that the fused feature belongs to the negative class by the normalization function;

[0164] Determine whether the first probability that the fused feature belongs to the positive class meets a preset condition; wherein the preset condition is that the first probability that the fused feature belongs to the positive class is greater than the second probability that the fused feature belongs to the negative class, or the first probability that the fused feature belongs to the positive class is greater than the preset probability;

[0165] If yes, then taking the indicator mark of lung lesions as the classification result of the lung lesions of the patient output by the first classification network;

[0166] If not, the indicator that the lungs do not have lesions is used as the classification result of the lung lesions for the patient output by the first classification network.

[0167] Optionally, after inputting the fused feature into the first classification network for classification to obtain a positive score and a negative score of the fused feature, the classification module 312 is further used to:

[0168] Inputting the index data output by the index extraction network into the second classification network to obtain negative weights and positive weights;

[0169] Multiplying the positive weight by the positive class score to obtain first data, and converting the first data into a third probability that the fused feature belongs to the positive class through the normalization function;

[0170] The negative weight is multiplied by the negative class score to obtain second data, and the second data is converted into a fourth probability that the fused feature belongs to the negative class through the normalization function.

[0171] Optionally, when the preset indicator is the volume ratio of the trachea to the tracheal wall, the classification module 312 is used to determine the parameter value of the parameter required in the formula corresponding to the preset indicator based on the CT image, and substitute the determined parameter value into the formula corresponding to the preset indicator to obtain the indicator data of the preset indicator. The classification module 312 is specifically used to:

[0172] Extracting the lung trachea in the CT image from the CT image to obtain the trachea volume;

[0173] Based on the CT image and the lung trachea, extract the tracheal wall in the CT image to obtain the tracheal wall volume;

[0174] The ratio of the trachea volume to the tracheal wall volume is determined as the index data of the volume ratio of the trachea to the tracheal wall.

[0175] An embodiment of the present application provides a lung lesion classification device based on a convolutional neural network, the classification device comprising: an acquisition module for acquiring a CT image of a patient's lungs; a classification module for inputting the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions; wherein the lung lesion classification model comprises a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

[0176] In this way, the technical solution provided in the present application can classify CT images through a lung lesion classification model composed of a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network, and obtain a classification result of the lung lesions of the patient. By using the indicator extraction network, the convolutional neural network and the feature fusion network, the generalization ability of the model is improved, and the accuracy of CT image classification in practical applications is improved.

[0177] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0178] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 to Figure 2 The steps of the method for classifying lung lesions based on a convolutional neural network in the method embodiment shown, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0179] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 to Figure 2 The steps of the method for classifying lung lesions based on a convolutional neural network in the method embodiment shown, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0184] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0185] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for classifying lung lesions based on convolutional neural networks, characterized in that: The classification method includes: Obtain CT images of the patient's lungs; The CT image is input into a lung lesion classification model to obtain a classification result of the patient's lung lesions; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

2. The classification method according to claim 1, characterized in that: The step of inputting the CT image into the lung lesion classification model to obtain a classification result of the lung lesion of the patient comprises: Inputting the CT image into the preprocessing network for preprocessing to obtain a preprocessed image output by the preprocessing network; Inputting the preprocessed image into the convolutional neural network for feature extraction to obtain feature data output by the convolutional neural network; Inputting the CT image into the index extraction network to obtain index data output by the index extraction network; Inputting the feature data and the indicator data into the feature fusion network for splicing to obtain fusion features output by the feature fusion network; The fused features are input into the first classification network for classification, and a classification result of the lung lesions of the patient output by the first classification network is obtained.

3. The classification method according to claim 2, characterized in that: The step of inputting the CT image into the index extraction network to obtain the index data output by the index extraction network includes: Inputting the CT image into an index extraction network to obtain preset indexes and formulas corresponding to each preset index; For each preset indicator, based on the CT image, determine the parameter value of the parameter required in the formula corresponding to the preset indicator, substitute the determined parameter value into the formula corresponding to the preset indicator, and obtain the indicator data of the preset indicator; Obtaining the maximum value and the minimum value of each preset indicator in the sample data in the training set, and normalizing the indicator data of the corresponding preset indicator based on the maximum value and the minimum value of each preset indicator to obtain the normalized indicator data of the corresponding preset indicator; The indicator data of each preset indicator is updated to the normalized indicator data of the corresponding preset indicator, so as to obtain the indicator data of each preset indicator output by the indicator extraction network.

4. The classification method according to claim 2, characterized in that: The step of inputting the CT image into a preprocessing network for preprocessing to obtain a preprocessed image output by the preprocessing network comprises: Inputting the CT image into a preprocessing network, extracting a lung region mask from the CT image by an image processing method, and obtaining a lung region mask frame; intercepting the CT image according to the lung region mask frame to obtain a lung region mask and the intercepted CT image; Resampling the lung region mask to a preset size to obtain a lung region binary image; Resampling the intercepted CT image to a preset size to obtain a target CT image; Acquire a target pixel point with a pixel value of zero in the binary image of the lung region, and update the tissue density value of the pixel point corresponding to the target pixel point in the target CT image to a preset threshold value to obtain an updated target CT image; Acquire a window width value and a window level value, and determine a region of interest from the updated target CT image based on the window width value and the window level value; The tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are less than the lower limit of the tissue density value of the region of interest are updated to the lower limit of the tissue density value, and the tissue density values ​​of the pixels whose tissue density values ​​in the updated target CT image are greater than the upper limit of the tissue density value of the region of interest are updated to the upper limit of the tissue density value, to obtain the target image; Normalizing the target image to obtain a normalized target image, and processing the target CT image to obtain an emphysema characteristic image; The normalized target image and the emphysema characteristic image are stacked to obtain a preprocessed image output by the preprocessing network.

5. The classification method according to claim 2, characterized in that: The step of inputting the fusion feature into the first classification network for classification to obtain a classification result of the lung lesions of the patient output by the first classification network includes: Inputting the fused features into the first classification network for classification to obtain a positive score and a negative score of the fused features; The positive class score is converted into a first probability that the fused feature belongs to the positive class by a normalization function, and the negative class score is converted into a second probability that the fused feature belongs to the negative class by the normalization function; Determine whether the first probability that the fused feature belongs to the positive class satisfies a preset condition; wherein the preset condition is that the first probability that the fused feature belongs to the positive class is greater than the second probability that the fused feature belongs to the negative class, or the first probability that the fused feature belongs to the positive class is greater than the preset probability; If yes, then taking the indicator mark of lung lesions as the classification result of the lung lesions of the patient output by the first classification network; If not, the indicator that the lungs do not have lesions is used as the classification result of the lung lesions for the patient output by the first classification network.

6. The classification method according to claim 5, characterized in that: After inputting the fused features into the first classification network for classification to obtain the positive class score and the negative class score of the fused features, the classification method further includes: Inputting the index data output by the index extraction network into the second classification network to obtain negative weights and positive weights; Multiplying the positive weight by the positive class score to obtain first data, and converting the first data into a third probability that the fused feature belongs to the positive class through the normalization function; The negative weight is multiplied by the negative class score to obtain second data, and the second data is converted into a fourth probability that the fused feature belongs to the negative class through the normalization function.

7. The classification method according to claim 3, characterized in that: When the preset indicator is the volume ratio of the trachea to the tracheal wall, the step of determining the parameter value of the parameter required in the formula corresponding to the preset indicator based on the CT image, substituting the determined parameter value into the formula corresponding to the preset indicator to obtain the indicator data of the preset indicator includes: Extracting the lung trachea in the CT image from the CT image to obtain the trachea volume; Based on the CT image and the lung trachea, extract the tracheal wall in the CT image to obtain the tracheal wall volume; The ratio of the trachea volume to the tracheal wall volume is determined as the index data of the volume ratio of the trachea to the tracheal wall.

8. A lung lesion classification device based on convolutional neural network, characterized in that: The classification device comprises: An acquisition module, used for acquiring a CT image of a patient's lungs; A classification module is used to input the CT image into a lung lesion classification model to obtain a classification result for the patient's lung lesions; wherein the lung lesion classification model includes a preprocessing network, an indicator extraction network, a convolutional neural network, a feature fusion network and a first classification network.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are run by the processor, the steps of the method for classifying lung lesions based on a convolutional neural network as described in any one of claims 1 to 7 are executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for classifying lung lesions based on a convolutional neural network as described in any one of claims 1 to 7 are executed.

Images