A Lung CT Image Classification Method Based on Multi-View and Multi-Slice

Through the multi-view and multi-slice lung CT image classification method, combined with feature decomposition and spatial correlation capture, the problem of insufficient utilization of view and slice information in the prior art is solved, and the accuracy of lung CT image classification is improved.

CN117036814BActive Publication Date: 2025-07-25UNIV OF SCI & TECH OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311038445.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-17
Publication Date
2025-07-25
Estimated Expiration
2043-08-17

AI Technical Summary

Technical Problem

Existing lung CT image classification methods ignore abnormal position characteristics in other views other than axial views and other sections other than central sections, resulting in limited classification capabilities.

Method used

The multi-view multi-slicing method is used to use the central slices of the axial, coronal and sagittal views and multiple adjacent slices, combining feature coding networks, feature decomposition networks, image reconstruction networks and recurrent neural networks, and separating common and unique features through attention mechanisms, and using multi-view multi-slicing cross-reconstruction loss function and feature similarity loss function to capture the spatial correlation between slices.

Benefits of technology

The classification accuracy of lung CT images was significantly improved, and the abnormal position morphological structure information in different views and slices were fully explored, which improved the model's discernment ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117036814B_ABST
    Figure CN117036814B_ABST
Patent Text Reader

Abstract

The present invention discloses a lung CT image classification method based on multi-view and multi-slice, including: 1. Obtaining CT image data and performing preprocessing; 2. Constructing a classification model composed of a feature encoding network, a feature decomposition network, a recurrent neural network, an image reconstruction network, and a classifier; 3. Jointly using the feature encoding network and the attention mechanism-based feature decomposition network to extract tumor features and separate background information from CT image samples; 4. Using the recurrent neural network to learn the spatial correlation information of multi-slice and performing weighted fusion; 5. Constructing a loss function and training the classification model to obtain a trained classification model. The present invention can make full use of the tumor morphological structure information contained in the multi-view and multi-slice data of CT images and capture the spatial correlation information of CT continuous slices, thereby improving the accuracy of lung CT image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of lung CT image classification, and more specifically, it is a method for classifying lung CT images based on multi-view and multi-slice. Background Art

[0002] The goal of the lung CT image classification task is to accurately classify the abnormalities in lung CT images through computer-aided diagnosis, that is, to give the abnormal category to which the CT image belongs. Thus, it is of great significance in helping doctors formulate targeted treatment plans and prolonging the survival time of patients. For example, for tumor abnormalities, in clinical practice, methods such as bronchoscopy and thoracentesis are usually used to perform histological typing diagnosis on patients with non-small cell lung cancer. However, these invasive diagnostic methods will cause trauma to the patient's body. Therefore, a large number of researchers in the field of medical imaging use CT images to conduct research on the classification of non-small cell lung cancer subtypes.

[0003] However, the existing methods for classifying abnormal regions in lung based on CT images still have certain problems. On the one hand, the method of extracting features of abnormal positions only using the axial view of CT images ignores the abnormal position features contained in other views; on the other hand, only using the central slice of the abnormal position in CT images for model training without considering the morphological structure information of the abnormal positions contained in other slices severely restricts the classification ability of the model. Summary of the Invention

[0004] In order to solve the deficiencies of the above-mentioned existing technologies, the present invention proposes a method for classifying lung CT images based on multi-view and multi-slice, hoping to fully explore the morphological structure information of abnormal regions contained in different slices of CT images, fully capture and utilize the spatial correlation information contained in continuous slices, so as to improve the accuracy of classifying lung CT images.

[0005] In order to achieve the above-mentioned invention purpose, the present invention adopts the following technical solutions:

[0006] A method for classifying lung CT images based on multi-view and multi-slice of the present invention is characterized by including the following steps:

[0007] Step 1: Obtain lung CT image data and perform preprocessing;

[0008] Obtain the lung CT image data of N cases with category labels, and use the trilinear interpolation algorithm to resample the lung CT image data to adjust the voxel size, so as to obtain the set of lung CT slices of each case and its corresponding category label, denoted as where X t represents the set of lung CT slices of the t-th case, and X t ={x t,j|j = 1, 2, …, J t}, x t,j represents the j-th lung image of the t-th case, and J t represents the number of lung images in the set of lung CT slices of the t-th case, and y t represents the set of CT slices X of the t-th case t corresponding class label; t = 1, 2 …, N; N represents the number of cases;

[0009] Taking the t middle th lung image in X as the central slice, where, represents rounding up. Taking the abnormal position center of the central slice as the origin, along the axial view direction av, coronal view direction sv, and sagittal view direction cv, sample n lung images adjacent to the left and right of the central slice respectively, so as to obtain the lung images of the t-th case in 2n + 1 axial view directions av, 2n + 1 coronal view directions sv, and 2n + 1 sagittal view directions cv. Denote the set of lung images of the t-th case in any view direction i as Denote the l-th lung image in as C, H, and W respectively represent the number of channels, height, and width of the lung image; l ∈ {t - n, t, t + n};

[0010] Step 2, construct a CT image classification network, including: a feature encoding network, a feature decomposition network based on an attention mechanism, an image reconstruction network, a recurrent neural network, and an image classification network;

[0011] Step 2.1, the feature encoding network processes to output an encoded feature map

[0012] Step 2.2, the feature decomposition network based on an attention mechanism processes to obtain the l-th abnormal common feature and the background specific feature

[0013] Step 2.3, the image reconstruction network processes and to correspondingly obtain the cross-reconstructed image in the axial view direction av the cross-reconstructed image in the coronal view direction sv the cross-reconstructed image in the sagittal view direction cv

[0014] Step 2.4. The gated recurrent unit (GRU) processes to obtain the fused feature of the \(t\)-th case in any view direction \(i\)

[0015] Step 2.5. The image classification network is composed of \(F\) fully connected layers. The activation function of the first \(F - 1\) fully connected layers is set as the Relu function, and the activation function of the \(F\)-th fully connected layer is set as the Softmax function.

[0016] The multi-view multi-slice fused feature of the \(t\)-th case is obtained using Equation (5)

[0017]

[0018] In Equation (5), Cat represents the concatenation operation. represents the fused feature of the \(t\)-th case in the axial view direction \(av\). represents the fused feature of the \(t\)-th case in the coronal view direction \(sv\). represents the fused feature of the \(t\)-th case in the sagittal view direction \(cv\).

[0019] The multi-view multi-slice fused feature is input into the image classification network to obtain the classification result of

[0020] Step 3. Training of the CT image classification network.

[0021] Step 3.1. The classification loss function \(L\) of the multi-view multi-slice of the \(t\)-th case is constructed using Equation (6) t,cla ;

[0022]

[0023] Step 3.2. The multi-view abnormal common feature of the \(t\)-th case in any view direction \(i\) is obtained using Equation (7)

[0024]

[0025] Step 3.3. The KL divergence between and is obtained using Equation (8)

[0026]

[0027] In Equation (8), Denote the multi-view abnormal common features of the \(t\)-th case in any view direction \(j\), where \(j\in\{av, sv, cv\}\) and \(i\neq j\);

[0028] Calculate the feature distribution difference between the abnormal common features of the \(t\)-th case in any view directions \(i\) and \(j\) by using (9)

[0029]

[0030] Step 3.4: Construct the similarity loss function \(L\) of the multi-view multi-slice features of the \(t\)-th case by using Equation (10) t,mfs ;

[0031]

[0032] In Equation (10), denotes the feature distribution difference between the abnormal common features of the \(t\)-th case in the axial view direction \(av\) and the sagittal view direction \(cv\), denotes the feature distribution difference between the abnormal common features of the \(t\)-th case in the axial view direction \(av\) and the coronal view direction \(sv\), denotes the feature distribution difference between the abnormal common features of the \(t\)-th case in the sagittal view direction \(cv\) and the coronal view direction \(sv\);

[0033] Step 3.5: Construct the image cross-reconstruction loss function of the \(l\)-th view slice of the \(t\)-th case in any view direction \(i\) by using Equation (11)

[0034]

[0035] In Equation (11), \(N\) is the number of pixel points included, and \(\|\cdot\|_1\) represents the \(L1\) norm;

[0036] Step 3.6: Construct the average value of the loss function of the \(t\)-th case in any view direction \(i\) by using Equation (12)

[0037]

[0038] Construct the cross-reconstruction loss function \(L\) of the multi-view multi-slice of the \(t\)-th case by using Equation (13) t,mcr ;

[0039]

[0040] In Equation (13), denotes the average value of the loss function of the \(t\)-th case in the axial view direction \(av\), Denote the average value of the loss function of the \(t\)-th case in the coronal view direction \(sv\). Denote the average value of the loss function of the \(t\)-th case in the sagittal view direction \(cv\);

[0041] Step 3.7: Use Equation (14) to construct the overall loss function \(L\) of the multi-view and multi-slice of the \(t\)-th case t,total ;

[0042] \(L\) t,total \(=\) \(L\) t,cta \(+\) \(\alpha L\) t,mcr \(+\) \(\beta L\) t,mfs (14)

[0043] In Equation (14), \(\alpha\) represents the weight coefficient of \(L\) t,mcr and \(\beta\) represents the weight coefficient of \(L\) t,mfs ;

[0044] Step 3.8: Based on the labeled set of lung images Train the CT image classification network according to the initial learning rate \(l1\), and calculate the overall loss function to update the network parameters. When the number of training steps reaches \(t1\), continue to train the CT image classification network according to the learning rate \(l2\) until the total number of training steps \(T\) is reached, so as to obtain a trained CT image classification model for realizing the classification of lung CT images.

[0045] The feature encoding network in Step 2.1 of the multi-view and multi-slice based lung CT image classification method of the present invention is also characterized in that it is successively composed of \(C1\) shallow convolutional layers, \(I1\) Inception layers and \(P1\) max pooling layers; among them, each Inception layer is composed of \(I2\) Inception-v2 modules; set the activation function of all convolutional layers to the Relu function;

[0046] Input into the feature encoding network, and after being successively processed by \(C1\) shallow convolutional layers, obtain the convolutional feature vector After being successively processed by \(I1\) Inception layers and \(P1\) max pooling layers, output the encoded feature map

[0047] The feature decomposition network in Step 2.2 is composed of \(C2\) convolutional layers, \(P1\) global max pooling layers, \(P1\) global average pooling layers, \(M\) feature splicing layers and a background separation layer; set the activation function of all convolutional layers to the Relu function;

[0048] Input Input into the feature decomposition network based on the attention mechanism, and after being processed by P1 global average pooling and P1 global max pooling respectively, the global average feature map is obtained accordingly and the global max feature map

[0049] and After being processed by M feature concatenation layers, the concatenated feature map is obtained

[0050] After being processed by C2 convolutional layers in sequence, then batch normalization operation is carried out, and the feature vector is obtained

[0051] Use the Sigmoid function to perform a non-linear transformation on the feature vector so as to obtain the l-th spatial weight feature map of the t-th case in any view direction i

[0052] The background separation layer respectively obtains the l-th abnormal common feature of the t-th case in any view direction i by using formula (1) and formula (2) and the background specific feature

[0053]

[0054]

[0055] The image reconstruction network in step 2.3 consists of C3 convolutional modules and a cross reconstruction layer; among them, the first C3 - 1 convolutional modules are composed of C4 convolutional layers in cascade, and the C3-th convolutional module is composed of C5 convolutional layers in cascade; set the activation function of the convolutional layers in the first C3 - 1 convolutional modules to be the Relu function; the activation function of the convolutional layers in the C3-th convolutional module is the Sigmoid function; the cross reconstruction layer includes: P2 max pooling layers and T1 transposed convolutional layers

[0056] The background specific feature in any view direction i After being processed by C3 convolutional modules, the background specific feature vector in any view direction i is obtained

[0057] The l-th background specific feature of the t-th case in the axial view direction av The l-th abnormal common feature of the t-th case in the coronal view direction sv The l-th abnormal common feature of the t-th case in the sagittal view direction cv into the input cross-reconstruction layer, so as to obtain the cross-reconstructed image in the axial view direction av by using Equation (3).

[0058]

[0059] Input the l-th background-specific feature of the t-th case in the coronal view direction sv the l-th abnormal common feature in the axial view direction av and the l-th abnormal common feature in the sagittal view direction cv into the input cross-reconstruction layer, so as to obtain the cross-reconstructed image in the coronal view direction sv

[0060] Input the l-th background-specific feature of the t-th case in the sagittal view direction cv the l-th abnormal common feature in the coronal view direction sv and the l-th abnormal common feature in the axial view direction av into the input cross-reconstruction layer, so as to obtain the cross-reconstructed image in the sagittal view direction cv

[0061] The gated recurrent unit GRU in step 2.4 is composed of a gated recurrent unit and a weighted fusion layer;

[0062] Step 2.4.1. After performing global average pooling on the dimension-reduced abnormal common feature is obtained wherein, represents the l-th dimension-reduced abnormal common feature of the t-th case in any view direction i;

[0063] Input it into the gated recurrent unit for processing to obtain the spatial correlation information of the t-th case in any view direction i wherein, represents the l-th spatial correlation information of the t-th case in any view direction i;

[0064] Step 2.4.2. The weighted fusion layer obtains the fusion feature of the t-th case in any view direction i by using Equation (4)

[0065]

[0066] In Equation (4), represents the l-th weight parameter of the t-th case in any view direction i.

[0067] An electronic device of the present invention includes a memory and a processor, characterized in that the memory is used to store a program for supporting the processor to execute the lung CT image classification method, and the processor is configured to execute the program stored in the memory.

[0068] A computer-readable storage medium of the present invention, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the steps of the lung CT image classification method.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] 1. First, the method of the present invention uses the central slice and multiple adjacent slices of the abnormal positions in the axial, coronal, and sagittal views of the CT image as multi-view multi-slice inputs to fully mine the morphological structure information of the abnormal positions contained in different views and different slices of the CT image, and prompts the classification model to learn more discriminative feature representations.

[0071] 2. On this basis, in order to separate the background information in different slices, the present invention designs a feature decomposition module to separate the common and specific features of multiple views through an attention mechanism, and proposes a multi-view multi-slice cross-reconstruction loss function to ensure the effectiveness of feature decomposition, and a multi-view multi-slice feature similarity loss function to further enhance the similarity between the common features of multiple views.

[0072] 3. While performing global calculations, the present invention further uses a recurrent neural network to fully capture and utilize the spatial correlation information contained in continuous slices, significantly improving the performance of lung CT image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 is a schematic flow chart of the present invention;

[0074] Figure 2 is a schematic diagram of the classification model of the present invention;

[0075] Figure 3 is a ROC curve diagram of the classification result of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0076] In this embodiment, a method for classifying lung CT images based on multi-view multi-slice, as Figure 1 shown, is carried out according to the following steps:

[0077] Step 1, obtain lung CT image data and perform preprocessing;

[0078] Obtain the lung CT image data of N cases with class labels, and use the trilinear interpolation algorithm to resample the lung CT image data to adjust the voxel size, so as to obtain the set of lung CT slices of each case and its corresponding class label, denoted as Among them, X t represents the set of lung CT slices of the t-th case, and X t ={x t,j |j = 1, 2, …, J t}, x t,j represents the j-th lung image of the t-th case, J t represents the number of lung images in the set of lung CT slices of the t-th case, y t represents the class label corresponding to the set of CT slices X t of the t-th case; t = 1, 2…, N; N represents the number of cases; in this embodiment, the CT images of 499 non-small cell lung cancer patients are collected as a sample set, the CT images are resampled to adjust their voxel size to 1mm×1mm×1mm, and their pixel intensities are clipped to -1000~400HU.

[0079] Taking the middle t rd lung image in X as the central slice, where, represents rounding up. Taking the abnormal position center of the central slice as the origin, along the axial view direction av, coronal view direction sv, and sagittal view direction cv, n adjacent lung images on the left and right of the central slice are sampled respectively, so as to obtain the lung images of the t-th case in 2n + 1 axial view directions av, 2n + 1 coronal view directions sv, and 2n + 1 sagittal view directions cv. Denote the set of lung images of the t-th case in any view direction i as Denote any l-th lung image in as C, H, and W represent the number of channels, height, and width of the lung image respectively; l ∈ {t - n, t, t + n}; in this embodiment, n takes 2, H and W both take 128, and C takes 1; the pixel value range of the CT image is adjusted to [0, 1] through mean normalization processing, and then data augmentation is performed on the training set images using random flipping (including flipping up and down and left and right) and random rotation {90°, 180°, 270°, 360°}.

[0080] Step 2, as Figure 2 shown, construct a CT image classification network, which includes: a feature encoding network, a feature decomposition network based on the attention mechanism, an image reconstruction network, a recurrent neural network, and an image classification network;

[0081] Step 2.1: Processing of the feature encoding network;

[0082] The feature encoding network consists of C1 shallow convolutional layers, I1 Inception layers, and P1 max pooling layers in sequence; among them, each Inception layer consists of I2 Inception-v2 modules; in this embodiment, C1 is taken as 2, P1 is taken as 4, and I1 is taken as 9; the activation function of all convolutional layers is set as the Relu function;

[0083] Input into the feature encoding network, and after being processed by C1 shallow convolutional layers in sequence, a convolutional feature vector is obtained After being processed by I1 Inception layers and P1 max pooling layers in sequence, an encoded feature map is output

[0084] Step 2.2: Processing of the feature decomposition network based on the attention mechanism;

[0085] The feature decomposition network consists of C2 convolutional layers, P1 global max pooling layers, P1 global average pooling layers, M feature splicing layers, and a background separation layer; in this embodiment, C2 is taken as 1, P1 is taken as 1, and M is taken as 1; the activation function of all convolutional layers is set as the Relu function;

[0086] Input into the feature decomposition network based on the attention mechanism, and after being processed by P1 global average pooling and P1 global max pooling respectively, a global average feature map and a global max feature map

[0087] and are obtained respectively. After being spliced by M feature splicing layers, a spliced feature map

[0088] After being processed by C2 convolutional layers in sequence, batch normalization operation is performed, and a feature vector

[0089] The Sigmoid function is used to perform a non-linear transformation on the feature vector to obtain the l-th spatial weight feature map of the t-th case in the i-th view direction

[0090] The background separation layer uses equations (1) and (2) to obtain the l-th abnormal common feature and the background specific feature

[0091]

[0092]

[0093] In this embodiment, the feature map with a size of 8×8×1024 is input into the feature decomposition module. First, the global average pooling and global max pooling are used to obtain the global average feature map with a size of 8×8×1 and the global max feature map Then and After the splicing process by 1 feature splicing layer, the spliced feature map with a size of 8×8×2 is obtained Subsequently Further, a feature vector is obtained through a 5×5 convolutional layer and batch normalization operation And the Sigmoid function is used to perform a non-linear transformation on the feature vector Thereby, a spatial weight feature map with a size of 8×8×1 can be obtained Finally is multiplied by to obtain the multi-view common feature The remaining part of after removing the common feature is regarded as the view-specific feature

[0094] Step 2.3. Processing of the image reconstruction network;

[0095] The image reconstruction network consists of C3 convolutional modules and a cross-reconstruction layer; among them, the first C3-1 convolutional modules are composed of C4 convolutional layers in cascade, and the C3rd convolutional module is composed of C5 convolutional layers in cascade; in this embodiment, C3 is taken as 5, C4 is taken as 2, and C5 is taken as 1; the activation function of the convolutional layers in the first C3-1 convolutional modules is set as the Relu function; the activation function of the convolutional layers in the C3rd convolutional module is the Sigmoid function; the cross-reconstruction layer includes: P2 max pooling layers and T1 transposed convolutional layers; in this embodiment, P1 is taken as 1, T1 is taken as 4, the stride of all convolutional layers is 1, and the stride of the transposed convolutional layer is set as 2;

[0096] The background-specific feature in any view direction i After being processed by C3 convolutional modules, the background-specific feature vector in any view direction i is obtained

[0097] The l-th background-specific feature of the t-th case in the axial view direction av The l-th common abnormal feature of the t-th case in the coronal view direction sv The l-th common abnormal feature of the t-th case in the sagittal view direction cv Input into the cross-reconstruction layer, and then use Equation (3) to obtain the cross-reconstructed image in the axial view direction av

[0098]

[0099] The l-th background-specific feature of the t-th case in the coronal view direction sv The l-th common abnormal feature in the axial view direction av And the l-th common abnormal feature in the sagittal view direction cv Input into the cross-reconstruction layer, so as to obtain the cross-reconstructed image in the coronal view direction sv

[0100] The l-th background-specific feature of the t-th case in the sagittal view direction cv The l-th common abnormal feature in the coronal view direction sv And the l-th common abnormal feature in the axial view direction av Input into the cross-reconstruction layer, so as to obtain the cross-reconstructed image in the sagittal view direction cv

[0101] Step 2.4. Processing by the gated recurrent unit GRU;

[0102] The gated recurrent unit GRU consists of R gated recurrent units and E weighted fusion layers; in this embodiment, R is taken as 3 and E is taken as 1;

[0103] Step 2.4.1. After performing global average pooling on the common abnormal features after dimensionality reduction are obtained where, represents the l-th common abnormal feature after dimensionality reduction of the t-th case in any view direction i;

[0104] In this embodiment, for the multi-view common features these features with a size of 8×8×1024 are dimensionally reduced to 1024-dimensional common abnormal features by using global average pooling operation

[0105] Input into the gated recurrent unit for processing to obtain the spatial correlation information of the t-th case in any view direction i where, represents the l-th spatial correlation information of the t-th case in any view direction i.

[0106] Step 2.4.2: The weighted fusion layer obtains the fused feature of the \(t\)-th case in any view direction \(i\) using Equation (4).

[0107]

[0108] In Equation (4), represents the \(l\)-th weight parameter of the \(t\)-th case in any view direction \(i\).

[0109] Step 2.5: Processing by the image classification network;

[0110] The image classification network consists of \(F\) fully connected layers; the activation function of the first \(F - 1\) fully connected layers is set as the Relu function; the activation function of the \(F\)-th fully connected layer is the Softmax function; in this embodiment, \(F\) takes 2.

[0111] The multi-view multi-slice fused feature of the \(t\)-th case is obtained using Equation (5). In a specific implementation, the multi-slice fused features of the axial, sagittal, and coronal views and are integrated through feature concatenation to obtain a 768-dimensional multi-view multi-slice fused feature \(V\). mv ;

[0112]

[0113] In Equation (5), Cat represents the concatenation operation, represents the fused feature of the \(t\)-th case in the axial view direction \(av\), represents the fused feature of the \(t\)-th case in the coronal view direction \(sv\), represents the fused feature of the \(t\)-th case in the sagittal view direction \(cv\);

[0114] The multi-view multi-slice fused feature is input into the image classification network to obtain the classification result.

[0115] In this embodiment, the ROC curve of the classification result is as Figure 3 shown. In the figure, the '×-shaped' curve corresponds to the classification result of the present invention, and the closer the curve is to the upper left corner, the better the classification performance. In addition, to quantitatively evaluate the performance of the proposed method, this embodiment shows the performance comparison results of the proposed method with the lung cancer subtype classification methods of Chaunzwa et al., Marentakis et al., and Guo et al. on a collected dataset of 499 non-small cell lung cancer cases, as shown in Table 1.

[0116] Comparison of Table 1 with Other Methods

[0117]

[0118] In the experiment of the present invention, first, the central slices of multiple views and multiple adjacent slices are jointly utilized for model training, fully mining the rich tumor morphological structure features in CT images, and the background information in different slices is separated through a feature decomposition module. In addition, the multi-view multi-slice cross-reconstruction loss function and the multi-view multi-slice feature similarity loss function are also used to ensure the effectiveness of feature decomposition. On this basis, a recurrent neural network is further utilized to capture slice changes and learn the spatial correlation information contained in consecutive slices, improving the classification performance of the network.

[0119] Step 3: Training of the CT image classification network;

[0120] Step 3.1: Construct the classification loss function L of the multi-view multi-slice of the t-th case using Equation (6) t,cla ;

[0121]

[0122] Step 3.2: Obtain the multi-view abnormal common features of the t-th case in any view direction i using Equation (7)

[0123]

[0124] Step 3.3: Obtain the KL divergence between and using Equation (8)

[0125]

[0126] In Equation (8), represents the multi-view abnormal common features of the t-th case in any view direction j, j ∈ {av, sv, cv}, and i ≠ j;

[0127] Calculate the feature distribution difference between the abnormal common features of the t-th case in any view directions i and j using (9)

[0128]

[0129] Step 3.4: Construct the similarity loss function L of the multi-view multi-slice features of the t-th case using Equation (10) t,mfs ;

[0130]

[0131] In formula (10), represents the feature distribution difference between the common features of abnormalities of the t-th case in the axial view direction av and the sagittal view direction cv, represents the feature distribution difference between the common features of abnormalities of the t-th case in the axial view direction av and the coronal view direction sv, represents the feature distribution difference between the common features of abnormalities of the t-th case in the sagittal view direction cv and the coronal view direction sv.

[0132] Step 3.5. Use formula (11) to construct the image cross-reconstruction loss function of the l-th view slice of the t-th case in any view direction i

[0133]

[0134] In formula (11), N is the number of pixel points it contains, and ||·||1 represents the L1 norm.

[0135] Step 3.6. Use formula (12) to construct the average value of the loss function of the t-th case in any view direction i

[0136]

[0137] Use formula (13) to construct the cross-reconstruction loss function L of the multi-view and multi-slice of the t-th case t,mcr ;

[0138]

[0139] In formula (13), represents the average value of the loss function of the t-th case in the axial view direction av, represents the average value of the loss function of the t-th case in the coronal view direction sv, represents the average value of the loss function of the t-th case in the sagittal view direction cv.

[0140] Step 3.7. Use formula (14) to construct the overall loss function L of the multi-view and multi-slice of the t-th case t,total ;

[0141] L t,total = L t,cta + αL t,mcr + βL t,mfs (14)

[0142] In formula (14), α represents the weight coefficient of L t,mcr and β represents the weight coefficient of L t,mfsweight coefficients; in this embodiment, α and β are respectively 0.4 and 0.6;

[0143] Step 3.8. Based on the labeled lung image set Train the CT image classification network according to the initial learning rate l1, and calculate the overall loss function L t,total to update the network parameters. When the number of training steps reaches t1, continue to train the CT image classification network according to the learning rate l2 until the total number of training steps T is reached, so as to obtain a trained CT image classification model for realizing the classification of lung CT images. In this embodiment, the number of non-small cell lung cancer patient samples sampled each time during training is set to 8; l1 is 10 -4 ; t1 is 60, and l2 is 10 -5 ; T is 100.

[0144] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0145] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is run by a processor, it executes the steps of the above method.

Claims

1. A lung CT image classification method based on multi-view and multi-slice, characterized in that, Including the following steps: Step 1, obtain lung CT image data and perform preprocessing; Obtain the lung CT image data of N cases with class labels, and use the trilinear interpolation algorithm to resample the lung CT image data to adjust the voxel size, so as to obtain the lung CT slice set of each case and its corresponding class label, denoted as Among them, X t represents the lung CT slice set of the t-th case, and X t ={x t,j |j = 1, 2,..., J t}, x t,j represents the j-th lung image of the t-th case, J t represents the number of lung images in the lung CT slice set of the t-th case, y t represents the class label corresponding to the CT slice set X t of the t-th case; t = 1, 2,..., N; N represents the number of cases; Take X t The middle lung image as the central slice, where represents rounding up. Taking the center of the abnormal position of the central slice as the origin, along the axial view direction av, coronal view direction sv, and sagittal view direction cv, sample n lung images adjacent to the left and right of the central slice respectively, so as to obtain the lung images of the t-th case on 2n + 1 axial view directions av, 2n + 1 coronal view directions sv, and 2n + 1 sagittal view directions cv. Denote the set of lung images of the t-th case in any view direction i as i ∈ {av, sv, cv}; Denote any l-th lung image in C, H, and W represent the number of channels, height, and width of the lung image respectively; l ∈ {t - n, t, t + n}; Step 2, construct a CT image classification network, including: a feature encoding network, a feature decomposition network based on an attention mechanism, an image reconstruction network, a recurrent neural network, and an image classification network; Step 2.1, the feature encoding network processes to output an encoded feature map Step 2.

2. The feature decomposition network based on the attention mechanism processes to obtain the l-th abnormal common feature of the t-th case in any view direction i and the background-specific feature Step 2.3, the image reconstruction network processes and to correspondingly obtain the cross-reconstructed image in the axial view direction av the cross-reconstructed image in the coronal view direction sv the cross-reconstructed image in the sagittal view direction cv Step 2.4, the gated recurrent unit (GRU) processes to obtain the fused feature of the t-th case in any view direction i Step 2.5, the image classification network is composed of F fully connected layers; set the activation function of the first F - 1 fully connected layers to the Relu function; the activation function of the Fth fully connected layer is the Softmax function; The multi-view multi-slice fusion features of the t-th case are obtained using Equation (5). In formula (5), Cat represents the splicing operation, represents the fused feature of the t-th case in the axial view direction av, represents the fused feature of the t-th case in the coronal view direction sv, represents the fused feature of the t-th case in the sagittal view direction cv; Input the multi-view multi-slice fusion features into the image classification network to obtain the classification result Step 3, training of the CT image classification network; Step 3.

1. Construct the classification loss function \(L\) of multi-view and multi-slice for the \(t\)-th case by using Equation (6). t,cla ; Step 3.2: Obtain the multi-view abnormal common features of the t-th case in any view direction i by using Equation (7) Step 3.3, obtain the KL divergence between and using Equation (8) In formula (8), represents the multi-view abnormal common feature of the t-th case in any view direction j, where j ∈ {av, sv, cv}, and i ≠ j; Use (9) to calculate the feature distribution difference between the abnormal common features of the t-th case in any view directions i and j Step 3.

4. Construct the similarity loss function \(L\) of the multi-view multi-slice features of the \(t\)-th case by using Equation (10). t,mfs ; In formula (10), represents the feature distribution difference between the common features of abnormalities in the axial view direction av and the sagittal view direction cv for the t-th case, represents the feature distribution difference between the common features of abnormalities in the axial view direction av and the coronal view direction sv for the t-th case, represents the feature distribution difference between the common features of abnormalities in the sagittal view direction cv and the coronal view direction sv for the t-th case; Step 3.

5. Construct the image cross-reconstruction loss function of the l-th view slice of the t-th case in any view direction i by using Equation (11). In Equation (11), N is the number of pixel points included, and ||·||1 represents the L1 norm; Step 3.

6. Construct the average loss function of the \(t\)-th case in any view direction \(i\) using Equation (12). Construct the cross - reconstruction loss function \(L\) of multi - view and multi - slice for the \(t\) - th case using Equation (13). t,mcr ; In formula (13), represents the average value of the loss function of the t-th case in the axial view direction av, represents the average value of the loss function of the t-th case in the coronal view direction sv, represents the average value of the loss function of the t-th case in the sagittal view direction cv; Step 3.

7. Construct the overall loss function \(L\) of the multi-view and multi-slice of the \(t\)-th case by using Equation (14). t,total ; L t,total = L t,cla + αL t,mcr + βL t,mfs (14) In formula (14), α represents the weight coefficient of L t,mcr , and β represents the weight coefficient of L t,mfs ; Step 3.8: Based on the labeled lung image set Train the CT image classification network according to the initial learning rate l1, and calculate the overall loss function to update the network parameters. When the number of training steps reaches t1, continue to train the CT image classification network according to the learning rate l2 until the total number of training steps T is reached, so as to obtain a trained CT image classification model for realizing the classification of lung CT images.

2. The method for classifying lung CT images based on multi-view multi-slice according to claim 1, characterized in that, The feature encoding network in Step 2.1 is successively composed of C1 shallow convolutional layers, I1 Inception layers, and P1 max pooling layers; among them, each Inception layer is composed of I2 Inception-v2 modules; set the activation function of all convolutional layers to the Relu function; Input it into the input feature encoding network, and after being processed by C1 shallow convolutional layers in sequence, a convolutional feature vector is obtained After being processed by I1 Inception layers and P1 max pooling layers in sequence, an encoded feature map is output 3. The method for classifying lung CT images based on multi-view multi-slice according to claim 2, wherein, The feature decomposition network in Step 2.2 is composed of C2 convolutional layers, P1 global max pooling layer, P1 global average pooling layer, M feature splicing layers, and a background separation layer; set the activation function of all convolutional layers to the Relu function; Input it into the feature decomposition network based on the attention mechanism. After being processed by P1 global average pooling and P1 global maximum pooling respectively, the global average feature map and the global maximum feature map are obtained accordingly and After the splicing process of M feature splicing layers, a spliced feature map is obtained After being processed by C2 convolutional layers in sequence, batch normalization is then performed to obtain feature vectors Use the Sigmoid function to perform a non-linear transformation on the feature vector to obtain the l-th spatial weight feature map of the t-th case in any view direction i The background separation layer uses Equation (1) and Equation (2) to obtain the l-th abnormal common feature of the t-th case in any view direction i and the background-specific feature 4. The method for classifying lung CT images based on multi-view multi-slice according to claim 3, characterized in that, The image reconstruction network in Step 2.3 is composed of C3 convolutional modules and a cross-reconstruction layer; among them, the first C3 - 1 convolutional modules are cascaded by C4 convolutional layers, and the C3rd convolutional module is cascaded by C5 convolutional layers; set the activation function of the convolutional layers in the first C3 - 1 convolutional modules to the Relu function; the activation function of the convolutional layers in the C3rd convolutional module is the Sigmoid function; the cross-reconstruction layer includes: P2 max pooling layers and T1 transposed convolutional layers; Background-specific features in any view direction i After being processed by C3 convolutional modules, a background-specific feature vector in any view direction i is obtained the l-th background-specific feature of the t-th case in the axial view direction av the l-th abnormality-common feature of the t-th case in the coronal view direction sv the l-th abnormality-common feature of the t-th case in the sagittal view direction cv into the input cross-reconstruction layer, so as to obtain the cross-reconstruction image in the axial view direction av by using Equation (3) the l-th background-specific feature of the t-th case in the coronal view direction sv the l-th abnormal common feature in the axial view direction av and the l-th abnormal common feature in the sagittal view direction cv are input into the cross-reconstruction layer to obtain a cross-reconstruction image in the coronal view direction sv The l-th background-specific feature of the t-th case in the sagittal view direction cv The l-th abnormal common feature in the coronal view direction sv And the l-th abnormal common feature in the axial view direction av Input into the cross-reconstruction layer to obtain a cross-reconstruction image in the sagittal view direction cv 5. The lung CT image classification method based on multi-view multi-slice according to claim 4, characterized in that, The recurrent neural network GRU in Step 2.4 is composed of a gated recurrent unit and a weighted fusion layer; Step 2.4.

1. After performing global average pooling on the dimensionality-reduced abnormal common features are obtained. Among them, represents the l-th dimensionality-reduced abnormal common feature of the t-th case in any view direction i. Processed in the input gated recurrent unit to obtain the spatial correlation information of the t-th case in any view direction i wherein, represents the l-th spatial correlation information of the t-th case in any view direction i; Step 2.4.2: The weighted fusion layer obtains the fusion feature of the \(t\)-th case in any view direction \(i\) using Equation (4). In formula (4), represents the l-th weight parameter of the t-th case in any view direction i.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program for supporting the processor to execute any one of the lung CT image classification methods in claims 1 - 5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it executes the steps of any one of the lung CT image classification methods in claims 1 - 5.

Citation Information

Patent Citations

  • Lung X-ray image classification method based on K-means clustering and GAN

    CN113222072A

  • Deep learning-based lung CT image segmentation method and imaging method

    CN116468732A