Lung ct image classification method based on multi-view multi-task feature learning

Abstract

The application discloses a lung CT image classification method based on multi-view multi-task feature learning, comprising the following steps: 1. obtaining a lung CT image dataset and performing standardization and normalization preprocessing; 2. constructing a multi-view multi-task feature learning network for lung CT image classification; 3. designing a network overall loss function for training; 4. training the network in an iterative optimization manner, and classifying the lung CT image to obtain a corresponding classification result. The application can fully mine the common information among multi-views by relieving the conflict between the consistency learning task and the reconstruction task, retain the view common features related to the lung CT image classification in each view, and complete the interference removal task to filter out the view specific features containing noise and other interference information, thereby effectively improving the performance of the lung CT image classification.

Description

Technical Field

[0001] This invention belongs to the field of pathological image analysis technology, and in particular relates to a lung CT image classification method based on multi-view, multi-task feature learning. Background Technology

[0002] The goal of lung CT image classification is to analyze patient CT images using artificial intelligence. By clarifying the image classification results, doctors can develop targeted treatment plans, which has broad clinical application value. Early lung CT image classification mainly employed traditional machine learning methods. The core principle of these methods is to input manually extracted CT image features into a machine learning model to achieve classification. However, these methods are highly dependent on expert experience and knowledge, the extraction process is cumbersome and complex, and they fail to comprehensively reflect CT image information, resulting in significant limitations of traditional machine learning methods.

[0003] In recent years, with the rapid development of deep learning, significant progress has been made in the clinical analysis of CT images, providing new opportunities to solve the aforementioned problems. For example, Chaunzwa et al. used the VGG-16 deep convolutional neural network model to classify lung CT images (Chaunzwa TL, Hosny A, Xu Y, et al. Deep learning classification of lung cancer histology using CT images[J]. Scientific reports,2021,11(1):5471.): By inputting the preprocessed CT images into the VGG-16 model, its ability to automatically learn features was utilized to extract lung features from the images, and the lung features were then input into a subsequent classifier for output. Experimental results show that deep learning methods can achieve good classification results. However, the above methods only consider a single view and ignore the rich spatial information of the lungs in CT images. To address the aforementioned issues, Guo et al. designed a 3D deep neural network, ProNet (Guo Y, Song Q, Jiang M, et al. Histological subtypes classification of lung cancers on CT images using 3D deep learning and radiomics[J]. Academic radiology, 2021, 28(9):e258-e266.): by inputting CT images into a 3D deep neural network, the three-dimensional spatial information of the lungs is mined, thereby effectively enhancing the classification effect of lung CT images. Although the above method captures rich spatial information, it requires expensive computational resources and suffers from severe overfitting. To alleviate these problems, using a multi-view representation learning method is a natural and intuitive approach. Li et al. designed an attention-based end-to-end multi-view classification method (Li C, Xu J, Liu Q, et al. Multi-view mammographic density classification by dilated and attention-guided residual learning[J].IEEE / ACM transactions on computationalbiology and bioinformatics,2020,18(3):1003-1013.): By inputting the axial, coronal, and sagittal views of CT images into a multi-view fusion model, the features of each view are extracted and fused for downstream classification tasks.

[0004] However, existing multi-view classification methods suffer from a conflict between consistency learning and reconstruction tasks when learning discriminative common view features. Specifically, existing methods rely on consistency learning tasks on latent features to fully explore common view information, while simultaneously introducing reconstruction tasks to comprehensively extract both view-specific and common view information from the same latent features, thus avoiding overfitting caused by the scarcity of medical image data. Furthermore, existing multi-view methods typically employ only a general encoder to learn common view features, neglecting the inevitable encoding of some noise information unrelated to lung CT image classification, resulting in unsatisfactory predictive performance in real-world scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a lung CT image classification method based on multi-view multi-task feature learning. This method aims to eliminate noise information irrelevant to lung CT image classification while fully exploring the common information among multiple views, thereby achieving lung CT image classification in a highly accurate and efficient manner.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The lung CT image classification method based on multi-view multi-task feature learning of the present invention is characterized by the following steps:

[0008] Step 1: Data Collection and Preprocessing

[0009] Data from N cases with lung CT image classification labels were obtained. A trilinear interpolation algorithm was then used to resample all lung CT image data to standardize voxel size, resulting in a set of lung CT slices for each case and its corresponding category label, denoted as . Among them, X i This represents a lung CT scan of the i-th case, y i X represents the lung CT slice of the i-th case. i Corresponding lung CT image classification labels; i = 1, 2, ..., N; N represents the number of cases;

[0010] With X i Taking the center of the abnormal location as the origin, image blocks of dimension H×W pixels are cropped along the axial view direction av, the coronal view direction cv, and the sagittal view direction sv, respectively, to obtain the lung CT image I of the i-th case in the axial view direction av. i，av Lung CT images in the coronal view CV direction i，cv Lung CT images in the sagittal view direction (SV) i，sv ;

[0011] The axial lung CT image of the i-th case I i，av Coronary lung CT images I i，cv and sagittal lung CT images I i，sv The multi-view data of the i-th case {I i，n |n = av, cv, sv}; where I i，n This represents the lung CT image of the i-th case in the viewing direction n;

[0012] Step 2: Construct a lung CT image classification network, including: a general encoder (UE). n Consistency-aware nonlocal module (CANB), reconstruction module, classification module;

[0013] Step 2.1: Construct a ResNet-based general encoder UE n It consists of C convolutional layers, D pooling layers and L fully connected layers. The stride of all convolutional layers is set to k. The activation function of all convolutional layers and the first L-1 fully connected layers is set to ReLU. The activation function of the Lth fully connected layer is Softmax.

[0014] The multi-view data of the i-th case {I i，n |n = av, cv, sv} is input into the lung CT image classification network and passes through the universal encoder UE. n After processing, low-level features {L} containing semantic information are obtained. i，n |n = av, cv, sv}, L i，n This represents the low-level features of the i-th case in the view direction n;

[0015] Step 2.2: The Consistency Aware Nonlocal Module (CANB) processes the low-level features to obtain the common features of the view. and view features

[0016] Step 2.3: The reconstruction module processes low-level features {L} i,n The image reconstruction data is obtained by processing |n=av,cv,sv}.

[0017] Step 2.4: The classification module classifies lung CT images I i,n Classify;

[0018] Step 2.4.1: Use equation (1) to obtain the joint features of the i-th case.

[0019]

[0020] In equation (1), This indicates a splicing operation.

[0021] Step 2.4.2: Utilize a global average pooling layer to process the joint features. After processing, the pooled features are input into the first fully connected layer, where the ReLU activation function is used for non-linear processing to obtain the feature vector. This vector then passes through the second fully connected layer, and the Softmax function is used to output I. i,n Lung CT image classification prediction probability P i ;

[0022] Step 3: Construct the loss function;

[0023] Step 3.1: Construct the loss function L for multi-view reconstruction using equation (2). rec :

[0024]

[0025] In equation (2), This represents the L2 norm.

[0026] Step 3.2: Construct the view using equation (3) Figure 1 Consistency loss function L con :

[0027]

[0028] In equation (3), 1 [j≠i] Let 1 represent an indicator function, where 1 is set if j ≠ i. [j≠i] Set it to 1; otherwise, set it to 1. [j≠i] =0; The j-th case represents the view-specific features in the view direction n; d represents the cosine similarity.

[0029] Step 3.3: Construct the orthogonal loss function L using equation (4). orth :

[0030]

[0031] In equation (4), Let Frobenius norm be represented by (·). T This represents the matrix transpose operation;

[0032] Step 3.4: Construct the interference loss function L using equation (5). ia :

[0033]

[0034] In equation (5), KL represents the KL divergence, y unif Indicates a uniform distribution;

[0035] Step 3.5 Constructs the binary cross-entropy classification loss function L using equation (6). cla :

[0036] L cla =-y i log(P i )-(1-y i log(1-P) i (6)

[0037] Step 3.6: Use equation (7) to obtain the total loss function L. total :

[0038] L total =L cla +αL rec +βL con +γL orth +ηL ia (7)

[0039] In equation (7), α, β, γ and η are four weighting coefficients;

[0040] Step 4: Train the lung CT image classification network using the Adam optimizer and calculate the total loss function L. total To update network parameters, a linear decay strategy is used to adjust the learning rate during training until the maximum number of training iterations or the total loss function L is reached. total The process continues until convergence, thus obtaining a trained lung CT image classification model, which is used to process lung CT image data to obtain classification results.

[0041] The lung CT image classification method based on multi-view multi-task feature learning described in this invention is also characterized in that step 2.2 includes the following steps:

[0042] Step 2.2.1: The Consistency Aware Nonlocal Module (CANB) includes: Query branch, Key branch, and Value branch;

[0043] Low-level features {L i，n The inputs of |n=av,cv,sv} into the Consistency Aware Nonlocal Module (CANB) are processed through convolutional layers in three branches to obtain the dimensionality-reduced query features {F}. iq，n |n = av, cv, sv}, the reduced key features {F ik，n |n = av, cv, sv} and the value features after dimensionality reduction {F iv，n |n=av,cv,sv}, where F iq，n F represents the query feature of the i-th case in the view direction n.ik，n F represents the key features of the i-th case in the view direction n. iv，n This represents the value characteristic of the i-th case in the view direction n;

[0044] Step 2.2.2, the Key branch pair {F ik，n Spatial pyramid pooling is performed on |n = av, cv, sv} to obtain the pooled key features. in, This represents the key features of the i-th case after pooling in the view direction n;

[0045] The Value branch will deal with {F} iv，n Perform spatial pyramid pooling operation on |n=av,cv,sv} to obtain the pooled value features. in, This represents the pooled value feature of the i-th case in the view direction n;

[0046] Calculate {F iq，n |n = av, cv, sv} and Correlation matrix between in, This represents the correlation matrix of the i-th case in the view direction n;

[0047] Step 2.2.3: Use the Softmax function to process the above correlation matrix. Perform a normalization operation to obtain the weight vector {v} of the common features of the view. i，n |n = av, cv, sv}, where v i，n This represents the weight vector of the i-th case in the view direction n;

[0048] Step 2.2.4, set {v i，n |n = av, cv, sv} and After multiplication, the common features of the view of the i-th case are obtained. in, This represents the common view feature of the i-th case in the view direction n;

[0049] Step 2.2.5: Transfer the low-level features {L} i,n |n = av, cv, sv} minus the common features of the views Then, the specific features of the view of the i-th case are obtained. in, This represents the view-specific feature of the i-th case in the view direction n.

[0050] Step 2.3 includes the following steps:

[0051] The reconstruction module utilizes a C1-layer convolutional neural network to analyze {L} i,n The convolutional feature vector {F} is obtained by processing |n=av,cv,sv}. i,n |n=av,cv,sv}, then apply a K1-sized convolution kernel to {F i,n Upsampling is performed on |n=av,cv,sv} to generate an upsampled feature vector {F' i,n |n=av,cv,sv}, then {F' i,n |n=av,cv,sv} are sequentially input into two layers of convolutional neural networks and one layer of upsampling neural networks for processing, resulting in a quadratic upsampling feature vector {F”. i,n |n=av,cv,sv}, finally for {F” i,n After performing a convolution operation on |n=av,cv,sv}, we obtain the reconstructed multi-view data for the i-th case. Among them, F i,n F' represents the convolutional feature of the i-th case in the view direction n. i,n F represents the upsampled feature of the i-th case in the view direction n. i,n This represents the quadratic upsampling feature of the i-th case in the view direction n. This represents the reconstructed multi-view data of the i-th case in the view direction n.

[0052] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the lung CT image classification method, and the processor is configured to execute the program stored in the memory.

[0053] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the lung CT image classification method.

[0054] Compared with the prior art, the positive effects of the present invention are as follows:

[0055] 1. This invention proposes a hierarchical feature learning network; this network generates low-level features and high-level features sequentially for each view. The low-level features can comprehensively learn common information and specific information of the view in order to better complete the reconstruction task; at the same time, high-level common features of the view flexibly selected from the low-level features complete the consistency learning task; this can effectively solve the conflict between the consistency learning task and the reconstruction task, improve the learning efficiency of common representations of the view, and thus effectively improve the accuracy of lung CT image classification.

[0056] 2. This invention proposes a Consistency-Aware Nonlocal Module (CANB). This module can fully learn common view information from low-level features by using feature selection, and filter out view-specific features containing noise and other interference information to complete the interference removal task. This alleviates the interference caused by a large amount of background noise in CT image data, and improves the predictive performance of lung CT image classification. Attached Figure Description

[0057] Figure 1 This is a flowchart of a lung CT image classification method based on multi-view multi-task feature learning according to the present invention;

[0058] Figure 2 These are axial, coronal, and sagittal views of the lung CT image data after preprocessing according to the present invention.

[0059] Figure 3 This is a diagram of the lung CT image classification network structure of the present invention;

[0060] Figure 4 This is a diagram of a consistency-aware nonlocal module in the network of this invention. Detailed Implementation

[0061] In this embodiment, a lung CT image classification method based on multi-view, multi-task feature learning is described, such as... Figure 1 As shown, the procedure is as follows:

[0062] Step 1: Data Collection and Preprocessing

[0063] Data from N cases with lung CT image classification labels were obtained. A trilinear interpolation algorithm was then used to resample all lung CT image data to standardize voxel size, resulting in a set of lung CT slices for each case and its corresponding category label, denoted as . Among them, X i This represents a lung CT scan of the i-th case, y i X represents the lung CT slice of the i-th case. i Corresponding lung CT image classification labels; i = 1, 2, ..., N; N represents the number of cases;

[0064] like Figure 2 As shown, with X i Taking the center of the abnormal location as the origin, image blocks of dimension H×W pixels are cropped along the axial view direction av, the coronal view direction cv, and the sagittal view direction sv, respectively, to obtain the lung CT image I of the i-th case in the axial view direction av. i，av Lung CT images in the coronal view CV direction i，cv Lung CT images in the sagittal view direction (SV)i，sv ;

[0065] The axial lung CT image of the i-th case I i，av Coronary lung CT images I i，cv and sagittal lung CT images I i，sv The multi-view data of the i-th case {I i，n |n = av, cv, sv}; where I i，n This represents the lung CT image of the i-th case in the viewing direction n;

[0066] In this embodiment, there are 325 eligible cases, including 146 positive cases and 179 negative cases.

[0067] Step 2: Construct a lung CT image classification network, such as... Figure 3 As shown, it includes: a general encoder UE n Consistency-aware nonlocal module (CANB), reconstruction module, classification module;

[0068] Step 2.1: In order to extract low-level features containing both rich view common information and view specific information from the preprocessed image data, it is necessary to construct a ResNet-based general encoder UE. n It consists of C convolutional layers, D pooling layers and L fully connected layers. The stride of all convolutional layers is set to k. The activation functions of all convolutional layers and the first L-1 fully connected layers are ReLU functions, and the activation function of the Lth fully connected layer is Softmax function.

[0069] Multi-view data of the i-th case {I i，n |n = av, cv, sv} are input into the lung CT image classification network and passed through the general encoder UE. n After processing, low-level features {L} containing semantic information are obtained. i，n |n = av, cv, sv}, L i，n This represents the low-level features of the i-th case in the view direction n.

[0070] Step 2.2: The Consistency Aware Nonlocal Module (CANB) processes the low-level features to obtain the common features of the view. and view features like Figure 4As shown, CT images contain a large amount of noise and interference in multiple views, and the features related to lung CT image classification are not evenly distributed across different views. Existing methods often fuse multi-view features into a unified feature representation, making it difficult to effectively remove noise and interference from the data. Therefore, this invention employs a consistency-aware nonlocal module based on attention feature selection to flexibly select view-common features with common information and view-specific features unrelated to lung CT image classification from low-level features.

[0071] Step 2.2.1: The Consistency-Aware Nonlocal Module (CANB) includes: Query branch, Key branch, and Value branch;

[0072] Low-level features {L i，n The input |n = av, cv, sv} is processed by the Consistency Aware Nonlocal Module (CANB) and undergoes convolutional processing through three branches to obtain the dimensionality-reduced query features {F}. iq，n |n = av, cv, sv}, the reduced key features {F ik，n |n = av, cv, sv} and the value features after dimensionality reduction {F iv，n |n=av,cv,sv}, where F iq，n F represents the query feature of the i-th case in the view direction n. ik，n F represents the key features of the i-th case in the view direction n. iv，n This represents the value characteristic of the i-th case in the view direction n.

[0073] Step 2.2.2: The Consistency-Aware Nonlocal Module (CANB) includes: Query branch, Key branch, and Value branch;

[0074] Low-level features {L i，n The input |n = av, cv, sv} is processed by the Consistency Aware Nonlocal Module (CANB) and undergoes convolutional processing through three branches to obtain the dimensionality-reduced query features {F}. iq，n |n = av, cv, sv}, the reduced key features {F ik，n |n = av, cv, sv} and the value features after dimensionality reduction {F iv，n |n=av,cv,sv}, where F iq，n F represents the query feature of the i-th case in the view direction n. ik，n F represents the key features of the i-th case in the view direction n. iv，n This represents the value feature of the i-th case in the view direction n; through this multiple pooling operation, spatial feature information of different sizes can be effectively extracted, greatly reducing the storage and computational overhead of feature dimensions and modules, thereby greatly improving the training efficiency of the model.

[0075] To effectively capture the contextual information of CT images, {F} is calculated. iq，n |n = av, cv, sv} and Correlation matrix between in, Let represent the correlation matrix of the i-th case in the view direction n; this matrix can fully mine common information in the view, thereby promoting the generation of features related to lung CT image classification by the consistent perception nonlocal module.

[0076] Step 2.2.3: Use the Softmax function to process the above correlation matrix. Perform a normalization operation to obtain the weight vector {v} of the common features of the view. i,n |n=av,cv,sv}, where v i,n This represents the weight vector of the i-th case in the view direction n;

[0077] Step 2.2.4, set {v i,n |n=av,cv,sv} and After multiplication, the common features of the view of the i-th case are obtained. in, This represents the common view feature of the i-th case in the view direction n;

[0078] Step 2.2.5: Transfer the low-level features {L} i,n |n = av, cv, sv} minus the common features of the views Then, the specific features of the view of the i-th case are obtained. in, This represents the view-specific feature of the i-th case in the view direction n.

[0079] Step 2.3: The reconstruction module processes low-level features {L} i,n The image reconstruction data is obtained by processing |n=av,cv,sv}.

[0080] To avoid overfitting caused by the scarcity of medical imaging data and to improve the classification performance of lung CT images, a reconstruction task is introduced. The reconstruction module utilizes a C1-layer convolutional neural network to classify {L} i,n The convolutional feature vector {F} is obtained by processing |n=av,cv,sv}. i,n |n=av,cv,sv}, then apply a K1-sized convolution kernel to {F i,n Upsampling is performed on |n=av,cv,sv} to generate an upsampled feature vector {F' i,n |n=av,cv,sv}, then {F'i,n |n=av,cv,sv} are sequentially input into two layers of convolutional neural networks and one layer of upsampling neural networks for processing, resulting in a quadratic upsampling feature vector {F”. i,n |n=av,cv,sv}, finally for {F” i,n After performing a convolution operation on |n=av,cv,sv}, we obtain the reconstructed multi-view data for the i-th case. Among them, F i,n F' represents the convolutional feature of the i-th case in the view direction n. i,n F represents the upsampled feature of the i-th case in the view direction n. i,n This represents the quadratic upsampling feature of the i-th case in the view direction n. This represents the reconstructed multi-view data of the i-th case in the view direction n.

[0081] Step 2.4: The classification module classifies lung CT images I i,n Classify;

[0082] Step 2.4.1: Use equation (1) to obtain the joint features of the i-th case.

[0083]

[0084] In equation (1), This indicates a splicing operation.

[0085] Step 2.4.2: Utilize a global average pooling layer to process the joint features. The pooled features are processed and then fed into the first fully connected layer. A ReLU activation function is then used for non-linear processing to obtain the feature vector. This feature vector is then passed through a second fully connected layer, where the Softmax function is used to output I. i,n Lung CT image classification prediction probability P i Among them, P i This represents the classification prediction probability of the i-th case.

[0086] Step 3: Construct the loss function;

[0087] Step 3.1: To effectively reconstruct the input multi-view data and provide more ideal regularization constraints during model training, thereby avoiding overfitting due to insufficient medical image data, a multi-view reconstruction loss function is used to measure the difference between the input CT image data and the reconstructed CT image data for each view. The loss function L for multi-view reconstruction is constructed using equation (2). rec :

[0088]

[0089] In equation (2), This represents the L2 norm.

[0090] Step 3.2: To address the issue of uneven distribution of features related to lung CT image classification across different views, this invention designs a visual... Figure 1 The consistency loss function constrains the CANB module to better capture common features of the views, thereby fusing interactive multi-view common information to improve the classification ability of lung CT images. The view is constructed using equation (3). Figure 1 Consistency loss function L con :

[0091]

[0092] In equation (3), 1 [j≠i] Let 1 represent an indicator function, where 1 is set if j ≠ i. [j≠i] Set it to 1; otherwise, set it to 1. [j≠i] =0; d represents the view-specific features of the j-th case in the view direction n; d represents the cosine similarity.

[0093] Step 3.3: To ensure that the Consistency Awareness Module (CANB) can effectively separate view-specific features from view-common features, this invention designs an orthogonal loss function to remove noise information in CT image data that is irrelevant to lung CT image classification. The orthogonal loss function L is constructed using equation (4). orth :

[0094]

[0095] In equation (4), Let Frobenius norm be (·). T This represents the matrix transpose operation.

[0096] Step 3.4: To explicitly capture noise information in CT image data using view-specific features, this invention designs an interference loss function. The classification results of the view-specific features are uniformly distributed across all categories using a subtype classifier, minimizing the KL divergence with the uniform distribution. Thus, view-specific features can explicitly capture noise information unrelated to CT image classification. The interference loss function L is constructed using equation (5). ia :

[0097]

[0098] In equation (5), KL represents the KL divergence, y unif It indicates a uniform distribution.

[0099] Step 3.5: To predict lung CT images from multiple views, the joint features stitched together from the common features of the views are input into a multilayer perceptron to calculate the error between the true subtype and the classification prediction result. A binary cross-entropy classification loss function L is constructed using equation (6). cla :

[0100] L cla =-y i log(P i )-(1-y i log(1-P) i (6)

[0101] Step 3.6: Perform a linear combination of the loss functions obtained above to obtain the total loss function, and use equation (7) to obtain the total loss function L. total :

[0102] L total =L cla +αL rec +βL con +γL orth +ηL ia (7)

[0103] In equation (7), α, β, γ and η are four weighting coefficients.

[0104] Step 4: Train the lung CT image classification network using the Adam optimizer and calculate the total loss function L. total The network parameters are updated accordingly, with the corresponding parameters β1 = 0.9, β2 = 0.999, and ε = 10. -8 The entire training cycle consists of 20 epochs, with an initial learning rate of 0.00001. A linear decay strategy is used to adjust the learning rate during training until the maximum number of training iterations or the total loss function L is reached. total To achieve convergence, and to accelerate model training while reducing computational resources, a mini-batch training strategy was adopted with a batch size of 20. This resulted in the best-performing lung CT image classification model, which was then used to process lung CT image data to obtain classification results.

[0105] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0106] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0107] The following evaluation metrics were used to assess the classification results of lung CT images:

[0108]

[0109]

[0110]

[0111] In this system, TP (true positive) represents the number of correctly classified positive examples, TN (true negative) represents the number of correctly classified negative examples, FP (false positive) represents the number of incorrectly classified negative examples, and FN (false negative) represents the number of incorrectly classified positive examples. Acc represents the ratio between the number of correctly classified examples and the total number of classifications, while Spe represents the proportion of correctly classified negative examples, and Sen represents the proportion of correctly classified positive examples. Furthermore, by calculating the area under the receiver operating characteristic (ROC) curve with the horizontal axis, an important performance metric, AUC (Area Under Curve), can be obtained, whose value represents the classification performance.

[0112] To quantitatively evaluate the performance of the proposed method, this embodiment presents the performance comparison results of the proposed OURS method with the RF method, the traditional SVM machine learning method, the single-view deep learning method, and the multi-view deep learning method on the same dataset, as shown in Table 1.

[0113] Table 1: Performance comparison of the proposed method with other methods on the same dataset

[0114]

[0115] In Table 1, single-view deep learning methods include: VGGCNN method (ChaunzwaT L, HosnyA, XuY, et al. Deep learning classification of lung cancer histology using CT images [J]. Scientific reports, 2021, 11 (1): 5471.), Inception-LSTM ( Marentakis P, KaraiskosP, Kouloulias V, et al. Lung cancer histology classification from CT images based on radiomics and deep learning models[J].Medical&BiologicalEngineering&Computing,2021,59:215-226.), 3DCNN method (Yanagawa M, Niioka H, ​​Kusumoto M, et al. Diagnostic performance for pulmonary adenocarcinoma on CT: comparison of radiologists with and without three-dimensional convolutionalneural network[J].European (Radiology, 2021, 31: 1978-1986.) and the ProNet method

[0116] Multi-view deep learning methods include: the MVCNN method (Su H, Maji S, Kalogerakis E, et al. Multi-view convolutional neural networks for 3D shape recognition[C] / / Proceedings of the IEEE International Conference on Computer Vision. 2015: 945-953.), the GVCNN method (Feng Y, Zhang Z, Zhao X, et al. GVCNN: Group-view convolutional neural networks for 3D shape recognition[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 264-272.), the WuCNN method (Wu X, Hui H, Niu M, et al. Deep learning-based multi-view fusion model for screening 2019 novel coronavirus pneumonia: a multicentre study[J]. European Journal of Radiology, 2020, 128: 109041.), and the LiCNN method (Li C, Xu J, Liu Q, et al. Multi-view mammographic density classification by dilated and attention-guided residual learning[J]. IEEE / ACM Transactions on Computational Biology and Bioinformatics, 2020, 18(3): 1003-1013.)

[0117] Table 1 shows that the deep learning method significantly outperforms traditional machine learning methods in lung CT image classification, indicating that deep learning can learn more comprehensive lung CT image information. Furthermore, the multi-view deep learning method generally outperforms the single-view method, demonstrating that it can more fully extract the rich spatial information of CT images. Finally, Table 1 shows that the proposed method significantly outperforms all other methods, proving that the hierarchical feature network can generate low-level and high-level features for each view to complete the reconstruction and consistency learning tasks respectively, greatly alleviating the conflict between the two tasks, avoiding overfitting due to insufficient medical image data, and fully extracting common information between views. Simultaneously, it shows that the consistency learning nonlocal module can effectively and flexibly select features relevant to lung CT image classification from low-level representations through feature selection, and remove noise information from CT images. In conclusion, the method of this invention improves the classification performance of lung CT images.

Claims

1. A lung CT image classification method based on multi-view, multi-task feature learning, characterized in that, Includes the following steps: Step 1: Data Collection and Preprocessing Data from N cases with lung CT image classification labels were obtained. A trilinear interpolation algorithm was then used to resample all lung CT image data to standardize voxel size, resulting in a set of lung CT slices for each case and its corresponding category label, denoted as . Among them, X i This represents a lung CT scan of the i-th case, y i X represents the lung CT slice of the i-th case. i Corresponding lung CT image classification labels; i = 1, 2, ..., N; N represents the number of cases; With X i Taking the center of the abnormal location as the origin, image blocks of dimension H×W pixels are cropped along the axial view direction av, the coronal view direction cv, and the sagittal view direction sv, respectively, to obtain the lung CT image I of the i-th case in the axial view direction av. i，av Lung CT images in the coronal view CV direction i，cv Lung CT images in the sagittal view direction (SV) i，sv ; The axial lung CT image of the i-th case I i，av Coronary lung CT images I i，cv sagittal lung CT images I i，sv The multi-view data of the i-th case {I i，n |n = av, cv, sv}; where I i，n This represents the lung CT image of the i-th case in the viewing direction n; Step 2: Construct a lung CT image classification network, including: a general encoder (UE). n Consistency-aware nonlocal module (CANB), reconstruction module, classification module; Step 2.1: Construct a ResNet-based general encoder UE n It consists of C convolutional layers, D pooling layers and L fully connected layers. The stride of all convolutional layers is set to k. The activation function of all convolutional layers and the first L-1 fully connected layers is set to ReLU. The activation function of the Lth fully connected layer is Softmax. The multi-view data of the i-th case {I i，n The input of |n = av, cv, sv} to the lung CT image classification network and the input of the universal encoder UE are as follows: n After processing, low-level features {L} containing semantic information are obtained. i，n |n = av, cv, sv}, L i，n This represents the low-level features of the i-th case in the view direction n; Step 2.2: The Consistency Aware Nonlocal Module (CANB) processes the low-level features to obtain the common features of the view. and view features Step 2.3: The reconstruction module processes low-level features {L} i，n The image reconstruction data is obtained by processing |n = av, cv, sv}. Step 2.4: The classification module classifies lung CT images I i，n Classify; Step 2.4.1: Use equation (1) to obtain the joint features of the i-th case. In equation (1), This indicates a splicing operation. Step 2.4.2: Utilize a global average pooling layer to process the joint features. After processing, the pooled features are input into the first fully connected layer, where the ReLU activation function is used for non-linear processing to obtain the feature vector. This vector then passes through the second fully connected layer, and the Softmax function is used to output I. i，n Lung CT image classification prediction probability P i ; Step 3: Construct the loss function; Step 3.1: Construct the loss function L for multi-view reconstruction using equation (2). rec : In equation (2), Represents the L2 norm; Step 3.2: Construct the loss function L for view consistency using equation (3). con : In equation (3), 1 [j≠i] Let 1 represent an indicator function, where 1 is set if j ≠ i. [j≠i] Set it to 1; otherwise, set it to 1. [j≠i] =0; The j-th case represents the view-specific features in the view direction n; d represents the cosine similarity. Step 3.3: Construct the orthogonal loss function L using equation (4). orth : In equation (4), Let Frobenius norm be represented by (·). T This represents the matrix transpose operation; Step 3.4: Construct the interference loss function L using equation (5). ia : In equation (5), KL represents the KL divergence, y unif Indicates a uniform distribution; Step 3.5 Constructs the binary cross-entropy classification loss function L using equation (6). vla : L cla =-y i log(P i )-(1-y i )log(1-P i ) (6) Step 3.6: Use equation (7) to obtain the total loss function L. total : L total =L cla +αL rec +βL con +γL orth +ηL ia (7) In equation (7), α, β, γ and η are four weighting coefficients; Step 4: Train the lung CT image classification network using the Adam optimizer and calculate the total loss function L. total To update network parameters, a linear decay strategy is used to adjust the learning rate during training until the maximum number of training iterations or the total loss function L is reached. total The process continues until convergence, thus obtaining a trained lung CT image classification model, which is used to process lung CT image data to obtain classification results.

2. The lung CT image classification method based on multi-view multi-task feature learning according to claim 1, characterized in that, Step 2.2 includes the following steps: Step 2.2.1: The Consistency Aware Nonlocal Module (CANB) includes: Query branch, Key branch, and Value branch; Low-level features {L i，n The inputs |n = av, cv, sv} into the Consistency Aware Nonlocal Module (CANB) are processed through convolutional layers in three branches to obtain the dimensionality-reduced query features {F}. iq，n |n = av, cv, sv}, the reduced key features {F ik，n |n = av, cv, sv} and the value features after dimensionality reduction {F iv，n |n=av,cv,sv}, where F iq，n F represents the query feature of the i-th case in the view direction n. ik，n F represents the key features of the i-th case in the view direction n. iv，n This represents the value characteristic of the i-th case in the view direction n; Step 2.2.2, the Key branch pair {F ik，n Spatial pyramid pooling is performed on |n = av, cv, sv} to obtain the pooled key features. in, This represents the key features of the i-th case after pooling in the view direction n; The Value branch will deal with {F} iv，n Perform spatial pyramid pooling on |n=av,cv,sv} to obtain the pooled value features. in, This represents the pooled value characteristic of the i-th case in the view direction n; Calculate {F iq，n |n = av, cv, sv} and Correlation matrix between in, This represents the correlation matrix of the i-th case in the view direction n; Step 2.2.3: Use the Softmax function to process the above correlation matrix. Perform a normalization operation to obtain the weight vector {v} of the common features of the view. i，n |n = av, cv, sv}, where v i，n This represents the weight vector of the i-th case in the view direction n; Step 2.2.4, set {v i，n |n = av, cv, sv} and After multiplication, the common features of the view of the i-th case are obtained. in, This represents the common view feature of the i-th case in the view direction n; Step 2.2.5: Transfer the low-level features {L} i，n |n = av, cv, sv} minus the common features of the views Then, the specific features of the view of the i-th case are obtained. in, This represents the view-specific feature of the i-th case in the view direction n.

3. The lung CT image classification method based on multi-view multi-task feature learning according to claim 2, characterized in that, Step 2.3 includes the following steps: The reconstruction module utilizes a C1-layer convolutional neural network to analyze {L} i，n The convolutional feature vector {F} is obtained by processing |n=av,cv,sv}. i，n |n = av, cv, sv}, then apply a K1-sized convolution kernel to {F i，n Upsampling is performed on |n=av,cv,sv} to generate an upsampled feature vector {F′ i，n |n=av,cv,sv}, then {F′ i，n |n=av,cv,sv} are sequentially input into two layers of convolutional neural networks and one layer of upsampling neural networks for processing, resulting in a quadratic upsampling feature vector {F″}. i，n |n = av, cv, sv}, finally set {F″ i，n After performing a convolution operation on |n=av,cv,sv}, we obtain the reconstructed multi-view data for the i-th case. Among them, F i，n F′ represents the convolutional feature of the i-th case in the view direction n. i，n F″ represents the upsampled feature of the i-th case in the view direction n. i，n This represents the quadratic upsampling feature of the i-th case in the view direction n. This represents the reconstructed multi-view data of the i-th case in the view direction n.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the lung CT image classification methods of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the lung CT image classification methods described in claims 1-3.

Images