A CT image classification method based on image segmentation and RU classifier and application thereof

Abstract

The application relates to the technical field of medical images, and discloses a CT image classification method based on image segmentation and an RU classifier and application thereof, which comprises the following specific steps: collecting CT images, performing pretreatment, converting the pretreated CT images into two-dimensional images, and obtaining a CT image set; a segmentation network model comprising a symmetrical encoder, a decoder and a feature fusion layer is constructed; a prediction segmentation image of the CT image set is obtained through the trained segmentation network model; an imageomics feature of the prediction segmentation image is extracted; an RU classifier is constructed; the RU classifier comprises a branch comprising a binary classification prediction model and a branch comprising an auxiliary function in parallel; a CT image to be predicted is acquired and pretreated, and a survival classification result is generated through the trained segmentation network model and the RU classifier in sequence. The application solves the problems of lack of robustness, overfitting and insufficient generalization ability of the existing survival prediction technology, and has the characteristics of high prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical images, more particularly, to a CT image classification method based on image segmentation and RU classifier and application thereof. BACKGROUND

[0002] Lung cancer is one of the most common malignant tumors in the world. Among them, non-small cell lung cancer accounts for about 80% of all lung cancers, and about 75% of patients are found to be in the middle and late stages, with a very low 5-year survival rate. Non-small cell lung cancer includes squamous cell carcinoma (squamous carcinoma), adenocarcinoma, and large cell carcinoma. Compared with small cell carcinoma, non-small cell lung cancer cells grow and divide slowly, and metastasis is relatively late, but it may not be easily detected in the early stage, and diagnosis is usually made through imaging examinations (such as CT scans) and the like. Treatment options depend on the severity of the condition and the overall health of the patient, and the prognosis of non-small cell lung cancer is of great significance for treatment options and patient management.

[0003] Medical analysis of lung cancer mainly applies Radiomics to extract imaging features, and then realizes survival prediction of patients through machine learning methods. Current feature extraction techniques rely on doctors manually labeling CT images of patients, which requires a lot of time and resources, so implementing deep learning methods can automatically identify features in medical images and further realize survival prediction, thereby reducing the burden of manual analysis by doctors, helping doctors to make more accurate and faster medical image diagnoses, and thereby improving the quality and efficiency of medical services.

[0004] The characteristics of lung tumor images are low contrast, unclear edges between tissues, varying lesion sizes and shapes, and variable lesion locations. Feature extraction in medical images generally uses image segmentation methods, and current medical image segmentation methods are mainly divided into traditional methods and deep learning methods.

[0005] The prior art has:

[0006] 1) Threshold-based segmentation method:

[0007] This method segments the image according to the relationship between the pixel gray value and the preset threshold, and is simple to operate and easy to implement, such as the maximum inter-class variance method (Otsu method). However, since the pixel difference between lung tumors and tissues is not large, the threshold-based segmentation method is not suitable for segmenting lung tumors.

[0008] 2) Edge-based segmentation method:

[0009] This method detects the edges in the image through differential operators for segmentation. Common ones are Sobel operator, Canny edge detection, etc. However, these methods are sensitive to contours, and the edges of lung tumors are not clear, so the segmentation effect is not good.

[0010] 3) Region-based segmentation methods:

[0011] This method is divided into different regional blocks according to the similarity criterion of the image. There are mainly seed region growing method, region splitting and merging method, watershed method, etc.

[0012] With the improvement of computing power and the rapid development of deep learning methods, in more complex task scenarios, deep learning-based methods are more superior to traditional segmentation methods. Deep learning methods can learn deeper image features through a large amount of labeled data, so as to more accurately obtain the segmentation target. Among them, UNet is a classic convolution-based neural network, which is widely used in medical image segmentation tasks. The main structure of UNet network includes encoder and decoder, and a jump connection method is designed to make the network can fuse multi-scale information at different levels. Most of the existing medical segmentation networks are improved on the basis of UNet structure. However, due to the small receptive field problem of UNet convolution structure, it may cause difficulties in processing some segmentation tasks of global context information, especially for larger targets or image structures with long-range dependencies. Vision Transformer (ViT) is a structure based on Transformer, which regards the image as a two-dimensional sequence and takes each pixel or image block as an element in the sequence. By serializing the image, the self-attention mechanism of the Transformer model can be used to capture the relationship and context information between pixels, which effectively solves the problem of convolution receptive field. However, the parameter quantity of ViT model is huge, and it cannot show its advantage compared with convolution structure network under the condition of not having a large amount of data. In this case, due to the lack of data to support the large-scale training of the ViT model, it may cause the model to overfit or lack of generalization ability.

[0013] Different radiomics features have a significant impact on the survival prediction of patients. Machine learning methods such as Random Forest (RF) are often used to achieve classification prediction of patient survival status within a specified time. Studies have shown that combining radiomics and patient clinical features can effectively improve prediction performance. Existing prediction methods are mainly white-box algorithms, including Logistic, Support Vector Machine (SVM), etc., which can provide good interpretability but often lack performance. Weighted and ensemble methods have certain potential to improve model performance, but still lack robustness.

[0014] Because the radiomics features obtained after lung tumor image segmentation have the characteristics of small sample and high dimension, black box algorithms can usually achieve higher prediction accuracy on such data sets. Generally speaking, neural networks (ANN) can capture complex patterns and associations in data, automatically learn features in data, and thus achieve more accurate classification prediction. However, in small sample data sets, uneven classification and data sampling bias can cause more serious underfitting or overfitting.

[0015] In summary, how to combine the calculation method in Rockafellar-Uryasev (RU) loss to solve the problems of lack of robustness, overfitting, and insufficient generalization ability of prediction technology is a technical problem that needs to be solved in the technical field. SUMMARY

[0016] To solve the problems of lack of robustness, overfitting, and insufficient generalization ability of existing survival prediction technology, the application provides a CT image classification method based on image segmentation and RU classifier and application thereof, which has the characteristics of high prediction accuracy.

[0017] To achieve the above-mentioned purposes of the application, the technical solutions adopted are as follows:

[0018] A CT image classification method based on image segmentation and RU classifier, comprising the following specific steps:

[0019] Collecting CT images of a plurality of patients, pre-processing the CT images, and converting the pre-processed CT images into two-dimensional images to obtain a CT image set;

[0020] Constructing a segmentation network model including a symmetric encoder, a decoder, and a feature fusion layer;

[0021] Inputting the CT image set into the segmentation network model for training;

[0022] Obtaining a prediction segmentation map of the CT image set through the trained segmentation network model; extracting radiomics features of the prediction segmentation map;

[0023] Constructing an RU classifier using Rockafellar-Uryasev loss; the RU classifier includes a branch including a binary classification prediction model in parallel and a branch including an auxiliary function;

[0024] Inputting the radiomics features into the RU classifier for training;

[0025] Obtaining and pre-processing a CT image to be predicted, and sequentially passing through the trained segmentation network model and the RU classifier to generate a survival classification result.

[0026] ​​Preferably, the CT image is preprocessed, and the preprocessed CT image is converted into a two-dimensional image to obtain a CT image set, specifically:

[0027] The GTV labels of each patient in the CT image are merged, and the size and interval of each slice in the CT image are made the same to obtain the CT image set;

[0028] The CT image set is divided into a training set and a test set; the three-dimensional data in the training set and the test set are cut into two-dimensional images according to the cross section; the window level and window width of the two-dimensional image are set to highlight the cancer region in the image; and the picture and label of the two-dimensional image are center cropped;

[0029] The data in the training set is randomly subjected to horizontal flipping, small-angle rotation, and random size scaling.

[0030] Further, the encoding layer comprises n encoding convolution layers provided with a down-sampling layer and 1 intermediate layer; each encoding convolution layer is composed of the same convolution structure, including a basic convolution module and a ResCBAM module;

[0031] The basic convolution module CBG comprises a cascaded 3x3 convolution, a batch normalization BN, and an activation function layer; and the ResCBAM module comprises a CBAM module and a residual connection module.

[0032] In each encoding convolution layer, the output of the basic convolution module is connected to the input of the ResCBAM module; and the output of the ResCBAM module is connected to the down-sampling layer.

[0033] The down-sampling layer of the 1st to (n-1)th convolution layers is connected to the input of the basic convolution module of the next convolution layer; and the down-sampling layer of the nth convolution layer is connected to the input of the intermediate layer.

[0034] Further, the decoder comprises n decoding convolution layers provided with an up-sampling layer and a feature fusion layer corresponding to the n encoding convolution layers, and 1 1x1 convolution; each decoding convolution layer is composed of the same convolution structure, including a basic convolution module and a ResCBAM module.

[0035] The basic convolution module CBG comprises a cascaded 3x3 convolution, a batch normalization BN, and an activation function layer; the ResCBAM module comprises two basic convolution modules, a CBAM module, and a residual connection module connected in sequence; and the feature fusion layer comprises a 1x1 convolution and an up-sampling layer.

[0036] In each decoding convolution layer, the output of the basic convolution module is connected to the input of the ResCBAM module; the input of the basic convolution module is connected to the up-sampling layer; and the output of the ResCBAM module is connected to the input of the corresponding feature fusion layer.

[0037] The up-sampling layer of the first to the (n-1)-th convolutional layer is respectively connected with the output of the ResCBAM module of the corresponding encoding convolutional layer, the output of the ResCBAM module of the up convolutional layer; the up-sampling layer of the n-th convolutional layer is respectively connected with the output of the ResCBAM module of the n-th encoding convolutional layer, the output of the intermediate layer; the n feature fusion layers output After being spliced, the final output is obtained through 1x1 convolution .

[0038] Further, the CBAM module is a convolutional layer attention module; after the feature is input into the ResCBAM module, the specific processing steps are as follows:

[0039] Suppose the dimension of the data input into the ResCBAM module is (C, H, W), the feature input is processed through two basic convolutional modules in turn to obtain the input feature;

[0040] The input feature is input into the CBAM module, and the attention mechanism operation is performed on the channel dimension and the spatial dimension in turn:

[0041] In the channel dimension, the input feature is changed from the (C, H, W) dimension to the (C, 1, 1) dimension; two features output by the maximum pooling layer and the average pooling layer are spliced together to obtain a spliced feature with a dimension of (2, H, W); the spliced feature is input into the MLP layer to obtain a feature with a result dimension of (1, H, W), the MLP layer is a 1x1 convolution for compressing channels, an activation function, and a 1x1 convolution for expanding channels; the Sigmoid function is used to convert it into a probability on the spatial level, and it is multiplied by the input feature to obtain a feature adjusted by channel attention;

[0042] In the spatial dimension, the feature adjusted by channel attention is changed from the (C, H, W) dimension to the (C, 1, 1) dimension through two parallel maximum pooling layers and average pooling layers; two features output by the maximum pooling layer and the average pooling layer are respectively input into the shared MLP layer to keep the output and the input dimension consistent; the two features output by the MLP layer are added, and the dimension is (C, 1, 1), the MLP layer is a convolution operation with an output channel of 1, a convolution kernel of 7, and a padding of 3; the Sigmoid function is used to convert it into a probability on the channel level, and it is multiplied by the feature adjusted by channel attention to obtain a feature adjusted by spatial attention; the feature adjusted by spatial attention is taken as the output of the CBAM module;

[0043] The residual connection module adds the data input into the ResCBAM module and the output of the CBAM module to obtain the output of the ResCBAM module.

[0044] Further, when the CT image set is input into the segmentation network model for training, a deep supervision method is introduced for training.

[0045] In the training, the total loss function used is:

[0046]

[0047] wherein, is the weight assigned to the corresponding .. Loss function:

[0048]

[0049] wherein, X is the predicted probability map , Y is the true label, H is the input picture length, and W is the input picture width.

[0050] After training, the output results of the trained segmentation network model are tested using the test set , and the evaluation index is:

[0051]

[0052] Set greater than 0.5 as foreground and less than 0.5 as background.

[0053] Further, the radiomics features of the predicted segmentation map are extracted, specifically: the predicted segmentation map is stacked to restore a three-dimensional nii data, and the radiomics features of each patient are extracted to obtain three-dimensional data.

[0054] Further, the branch and branch each include a first hidden layer, a ReLU activation function, and a second hidden layer.

[0055] The radiomics features are input into the RU classifier for training, and the specific steps are:

[0056] Let the input layer of the RU classifier include the radiomics features and the corresponding binary classification labels , perform linear transformation on the input layer.

[0057] In the branch, the linearly transformed The first hidden layer is processed by applying the ReLU activation function to its output and learning the algorithm. The output of the first hidden layer is then used as input, undergoing a linear transformation and outputting to the second hidden layer. The output of the second hidden layer is then processed by applying the ReLU activation function to introduce non-linearity and learning the algorithm. This learning process produces an output layer consisting of a linear transformation and a Sigmoid activation function. The linear transformation maps to a single neuron, meaning the output layer dimension is 1. The Sigmoid function scales the output to the range [0,1], representing the predicted probability for the binary classification task. A probability greater than 0.5 is predicted as "death," and a probability less than or equal to 0.5 is predicted as "survival."

[0058] exist In the branch, the linearly transformed The first hidden layer is processed by applying the ReLU activation function to its output and learning the process. The output of the first hidden layer is then used as input, a linear transformation is performed, and the result is fed into the second hidden layer. The output of the second hidden layer is then processed by applying the ReLU activation function to introduce non-linearity and learning the process. This learning process produces an output layer consisting of linear transformations, where each linear transformation maps to a single neuron, meaning the output layer has a dimension of 1.

[0059] Branches and Each branch outputs the predicted probability. and parameters Then, according to , and binary labels The Rockafellar-Uryasev loss is calculated based on the value of the loss; then, iterative training is performed with the goal of minimizing the loss, and finally the optimal prediction value is obtained.

[0060] Furthermore, the Rockafellar-Uryasev loss specifically refers to:

[0061]

[0062] in, For the first The radiomics feature vector of each sample, To represent correspondence The true binary classification labels, It is a non-negative hyperparameter. For the first one sample The prediction results of the branches, For the first one sample The output parameters of the branch, Defined as binary cross-entropy loss:

[0063]

[0064] In addition, is a rectified linear unit, used to improve the joint expression capacity of the two prediction branches in the RU classifier, specifically:

[0065]

[0066] In addition, is a rectified linear unit, used to improve the joint expression capacity of the two prediction branches in the RU classifier, specifically:

[0067]

[0068] Let the learning goal be ; when the expectation of the intra-loss of all samples reaches the minimum, it means that the classification prediction performance reaches the best:

[0069] .

[0070] The application of a CT image classification method based on image segmentation and RU classifier is used for lung cancer survival classification.

[0071] The beneficial effects of the present application are as follows:

[0072] The application discloses a CT image classification method based on image segmentation and RU classifier, and first proposes a lung cancer image targeted segmentation network model for CT images of lung cancer patients, assigns different weights to each position of the feature map, enhances the global information acquisition capability and feature expression capability of the network, and the residual connection and deep supervision method makes the network more easily converge, thereby enhancing the generalization capability of the network and improving the segmentation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 is a flowchart of the CT image classification method based on image segmentation and RU classifier.

[0074] Figure 2 is the image obtained after the two-dimensional image is labeled in example 2.

[0075] Figure 3 is the image obtained after the two-dimensional image is windowed in example 2.

[0076] Figure 4 is a schematic diagram of a segmentation network model.

[0077] Figure 5 is a schematic diagram of a channel attention dimension of a CBAM module.

[0078] Figure 6 is a schematic diagram of a spatial attention dimension of a CBAM module.

[0079] Figure 7 is a schematic diagram of a CBAM module in embodiment 2.

[0080] Figure 8 is a schematic diagram of an intermediate layer in embodiment 2.

[0081] Figure 9 is a schematic diagram of a ResCBAM module in embodiment 2.

[0082] Figure 10 is a schematic diagram of a basic convolution module in embodiment 2.

[0083] Figure 11 is a schematic diagram of a RU classifier in embodiment 3.

[0084] Figure 12 is a two-dimensional image of lung cancer in embodiment 4.

[0085] Figure 13 is a preprocessed two-dimensional image of lung cancer in embodiment 4.

[0086] Figure 14 is a predicted segmentation map in embodiment 4.

[0087] Figure 15 is a ROC curve visualization result obtained after the RU classifier in embodiment 4 is applied to the test set. DETAILED DESCRIPTION

[0088] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0089] Embodiment 1

[0090] As shown in Figure 1 , a CT image classification method based on image segmentation and RU classifier includes the following specific steps:

[0091] Collect lung cancer CT images of a plurality of patients, pre-process the lung cancer CT images, and convert the pre-processed CT images into two-dimensional images to obtain a CT image set;

[0092] Construct a segmentation network model including a symmetric encoder, a decoder, and a feature fusion layer;

[0093] The CT image set is input into the segmentation network model for training;

[0094] Predicted segmentation maps of CT image sets are obtained through a trained segmentation network model; radiomics features of the predicted segmentation maps are extracted.

[0095] Construct an RU classifier using Rockafellar-Uryasev loss; the RU classifier includes a parallel binary classification prediction model. Branches and including auxiliary functions Branches;

[0096] The radiomics features are input into the RU classifier for training;

[0097] The lung cancer CT images to be predicted are acquired and preprocessed, and survival classification results are generated sequentially through a trained segmentation network model and an RU classifier.

[0098] Example 2

[0099] More specifically, in one embodiment, the CT images are preprocessed, and the preprocessed CT images are converted into two-dimensional images to obtain a CT image set, specifically as follows:

[0100] Merge the GTV tags of each patient in the CT images; make the size and spacing of each slice in the CT images the same to obtain the CT image set;

[0101] The CT image set is divided into a training set and a test set; the three-dimensional data in the training set and the test set are divided into two-dimensional images according to cross-sections; the window position and window width of the two-dimensional images are set to make the cancer area stand out in the image; the images and labels of the two-dimensional images are center-cropped.

[0102] The data in the training set is randomly flipped horizontally, rotated at a small angle, and scaled to a random size.

[0103] In this embodiment, as Figure 2 , Figure 3 As shown, this invention uses GTV labels from 3D CT images of non-minimum cell lung cancer patients. For patients with multiple GTVs, they are merged together. The total slice size of the dataset is (512, 512), and the spacing is (0.977, 0.977, 3.0).

[0104] Patients were divided into training and testing sets at an 8:2 ratio. The 3D data was then segmented into 2D images based on cross-sections. Since the CT values ​​of the image data ranged from 3000 to 1000, a window level of 0 and a window width of 1000 were set to make the cancerous areas stand out more clearly in the images and to eliminate unnecessary interference. Figure 2 An example of adding labels to the original image.Figure 3 The windowed image.

[0105] In this embodiment, in order to reduce redundant calculation, the picture and the label are center cropped, and the size after cropping is (256, 256). The input image is the single-channel image described above, and in order to increase sample diversity, the training set data is randomly horizontally flipped, small-angle rotated and randomly scaled. After network prediction, a single-channel probability map is output, and values greater than 0.5 are foreground and values less than 0.5 are background.

[0106] In one specific embodiment, as shown in Figure 4 The encoding layer includes n encoding convolution layers provided with a down-sampling layer and 1 intermediate layer; each encoding convolution layer is composed of the same convolution structure, including a basic convolution module and a ResCBAM module;

[0107] The basic convolution module CBG includes a 3x3 convolution, a batch normalization BN and an activation function layer; and the ResCBAM module includes a CBAM module and a residual connection module.

[0108] In each encoding convolution layer, the output of the basic convolution module is connected to the input of the ResCBAM module, and the output of the ResCBAM module is connected to the down-sampling layer.

[0109] The down-sampling layer of the 1st to (n-1)th convolution layers is connected to the input of the basic convolution module of the next convolution layer, and the down-sampling layer of the nth convolution layer is connected to the input of the intermediate layer.

[0110] In this embodiment, in order to alleviate the problem of neuron death and improve the expression ability of the model, a GELU activation function is adopted.

[0111] In one specific embodiment, the decoder includes n decoding convolution layers provided with an up-sampling layer and a feature fusion layer corresponding to the n encoding convolution layers, and 1 1x1 convolution; each decoding convolution layer is composed of the same convolution structure, including a basic convolution module and a ResCBAM module.

[0112] As shown in Figure 10 The basic convolution module CBG (Conv) includes a 3x3 convolution, a batch normalization BN and an activation function layer GELU; as shown in Figure 9 The ResCBAM module includes two basic convolution modules, a CBAM module and a residual connection module connected in sequence; and the feature fusion layer includes a 1x1 convolution and an up-sampling layer.

[0113] In this embodiment, as shown in Figure 8 The intermediate layer (Mid Conv) includes two basic convolution modules connected in sequence.

[0114] In each decoding convolutional layer, the output of the basic convolutional module is connected with the input of the ResCBAM module; the input of the basic convolutional module is connected with the up-sampling layer; the output of the ResCBAM module is connected with the input of the corresponding feature fusion layer;

[0115] The up-sampling layer of the 1st to (n-1)th convolutional layers is respectively connected with the output of the ResCBAM module of the corresponding encoding convolutional layer, the output of the ResCBAM module of the previous convolutional layer; the up-sampling layer of the nth convolutional layer is respectively connected with the output of the ResCBAM module of the nth encoding convolutional layer, the output of the intermediate layer; the outputs of the n feature fusion layers , n, and the final output is obtained after 1x1 convolution .

[0116] In one specific embodiment, in order to enable the convolution to learn the feature area with cancer more focusedly, the present application introduces a CBAM module; the CBAM module is as shown in Figure 7 The CBAM module is an attention module at the level of convolutional layer; after the feature is input into the ResCBAM module, the specific processing steps are as follows:

[0117] Suppose the dimension of the data input into the ResCBAM module is (C, H, W), the input feature is processed by two basic convolutional modules in turn to obtain the input feature;

[0118] The input feature is input into the CBAM module, and the attention mechanism is operated from the channel dimension and the spatial dimension in turn:

[0119] As shown in Figure 5 In the channel dimension (Channel Attention), the input feature is changed from the dimension (C, H, W) to the dimension (C, 1, 1); two features output by the maximum pooling layer and the average pooling layer are spliced together to obtain the spliced feature with the dimension (2, H, W); the spliced feature is input into the MLP layer to obtain the feature with the result dimension (1, H, W), the MLP layer is a 1x1 convolution for compressing the channel, an activation function, and a 1x1 convolution for expanding the channel; the Sigmoid function is used to convert it into a probability on the spatial level, and the input feature is multiplied to obtain the feature adjusted by the channel attention;

[0120] In this embodiment, the MLP layer is a convolutional operation with 1 output channel, 7 kernels, and 3 padding. The input result dimension is (1, H, W), which is converted into spatial probabilities using the Sigmoid function and multiplied with the input features to achieve spatial attention adjustment. Due to the excessive number of convolutional operations, to prevent network degradation and gradient vanishing, this invention introduces residual connections from the ResNet network, adding the input and the output of the ResCBAM module to make the network easier to optimize. All downsampling layers are convolutions with a kernel size of 2 and a stride of 2.

[0121] like Figure 6 As shown, in the spatial dimension (Spacial Attention), after passing through two parallel max pooling and average pooling layers, the channel attention-adjusted features are transformed from (C,H,W) dimension to (C,1,1) dimension. The two features output from the max pooling and average pooling layers are respectively input into a shared MLP layer, keeping the output and input dimensions consistent. The two features output from the MLP layer are added together, resulting in a dimension of (C,1,1). The MLP layer is a convolution operation with 1 output channel, 7 kernels, and 3 padding. The sigmoid function is used to convert it into channel-level probabilities, which are then multiplied with the channel attention-adjusted features to obtain the spatial attention-adjusted features. The spatial attention-adjusted features are then used as the output of the CBAM module.

[0122] In this embodiment, the MLP layer is a convolution operation with 1 output channel, 7 kernels, and 3 padding. The channel compression and expansion ratios are both 16 times, maintaining the consistency between the output and input dimensions. The output is the sum of the two results from the MLP layer, with dimensions (C, 1, 1), which is then converted into channel-level probabilities using the Sigmoid function and multiplied by the input features to achieve spatial attention adjustment.

[0123] The residual connection module adds the data input to the ResCBAM module and the output of the CBAM module to obtain the output of the ResCBAM module.

[0124] In this embodiment, the specific parameters of the several coding layers are as follows:

[0125] 1) The first layer input image data x(1,256,256) is transformed into (64,256,256) after passing through a convolutional module, and then enters the ResCBAM module to output feature map h1(64,256,256);

[0126] 2) The second layer input feature map x(64,128,128) becomes (128,128,128) after passing through a convolutional module, and then enters the ResCBAM module to output feature map h2(128,128,128);

[0127] 3) The third layer input feature map x(128, 64, 64) becomes (256, 64, 64) after a convolution module, and then enters the ResCBAM module to output the feature map h3(256, 64, 64). The down-sampling layer: down-samples h3 to output the feature map x(256, 32, 32).

[0128] 4) The fourth layer input feature map x(256, 32, 32) becomes (512, 32, 32) after a convolution module, and then enters the ResCBAM module to output the feature map h4(512, 32, 32).

[0129] The down-sampling layer: down-samples h4 to output the feature map x(512, 16, 16).

[0130] The intermediate layer is two basic convolution blocks, and the first convolution block realizes the expansion of the channel by 2 times. The detailed process of the intermediate layer is: the input feature map x(512, 16, 16) becomes (1024, 16, 16).

[0131] In this embodiment, the specific parameters in the decoding convolution layer are:

[0132] 1) The input feature map x(1024, 16, 16) is up-sampled to become (512, 32, 32), spliced with the fourth layer feature map h4 to obtain the feature map (1024, 32, 32), and then becomes (512, 32, 32) after a convolution module, and then enters the ResCBAM module to output the feature map de4(512, 32, 32).

[0133] 2) The input feature map de4(512, 32, 32) is up-sampled to become (256, 64, 64), spliced with the third layer feature map h3 to obtain the feature map (512, 64, 64), and then becomes (256, 64, 64) after a convolution module, and then enters the ResCBAM module to output the feature map de3(256, 64, 64).

[0134] 3) The input feature map x(256, 64, 64) is up-sampled to become (128, 128, 128), spliced with the second layer feature map h2 to obtain the feature map (256, 128, 128), and then becomes (128, 128, 128) after a convolution module, and then enters the ResCBAM module to output the feature map de2(128, 128, 128).

[0135] 4) Input feature map x(128,128,128), upsample x to (64,256,256), concatenate it with the first layer feature map h1 to obtain feature map (128,256,256), pass through a convolution module to become (64,256,256), and then enter the ResCBAM module to output feature map de1(64,256,256).

[0136] Each feature fusion layer is a 1×1 convolution with 1 output channel, followed by an upsampling operation with a scaling factor of 256 divided by the feature map size. Then, the outputs of each layer are... After concatenation, a 1×1 convolution is performed to obtain the final output. .

[0137] In this embodiment, the specific parameters of the feature fusion layer are as follows:

[0138] 1) The fourth layer feature map de4(512,32,32) is convolved with a 1×1 convolution to obtain (1,32,32), and then transposed convolution is performed to obtain... (1,256,256).

[0139] 2) The third layer feature map de3(256,64,64) is convolved by 1×1 to obtain (1,64,64), and then transposed convolution is performed to obtain... (1,256,256).

[0140] 3) The second layer feature map de2(128,128,128) is convolved by 1×1 to obtain (1,128,128), and then transposed convolution is performed to obtain... (1,256,256).

[0141] 4) The first layer feature map de1(64,256,256) is convolved by 1×1 to obtain (1,256,256), and then transposed convolution is performed to obtain... (1,256,256).

[0142] 5) The concatenated feature map (4,256,256) is then used to obtain the final output using a 1×1 convolution. (1,256,256).

[0143] In one specific embodiment, when the CT image set is input into the segmentation network model for training, a deep supervision method is introduced for training.

[0144] During training, the total loss function used is:

[0145]

[0146] wherein, corresponding to the weight assigned to n; loss function:

[0147]

[0148] wherein, X is a predicted probability map , Y is a real label, H is the length of an input picture, and W is the width of an input picture;

[0149] After training, the output results of the trained segmentation network model are tested by using a test set, and the evaluation index is:

[0150]

[0151] greater than 0.5 is set as a foreground, and less than 0.5 is set as a background.

[0152] In one specific embodiment, the radiomics features of the predicted segmentation map are extracted, specifically: the predicted segmentation map is stacked to restore a three-dimensional nii data, and the radiomics features of each patient are extracted to obtain three-dimensional data.

[0153] The present application proposes an end-to-end lung cancer patient survival prediction method. For the CT image of a lung cancer patient, an original network structure specific to the lung cancer image is proposed. Spatial attention and channel attention are introduced in the convolution process, different weights are given to each position of the feature map, and the ability of the network to obtain global information and the expression ability of the features are enhanced. The residual connection and the deep supervision method make the network more easily converge, thereby enhancing the generalization ability of the network and improving the accuracy of segmentation.

[0154] Embodiment 3

[0155] More specifically, in this embodiment, the present application takes the t-year survival status of a patient from receiving treatment to death as the prediction target. This is a binary classification problem, that is, to judge whether the survival time of the patient exceeds t years, the result is "survival" (representing "0" class) or "death" (representing "1" class). The data set of the prediction part is still divided in the ratio of 8:2.

[0156] In one specific embodiment, the branch and branch each include a first hidden layer, a ReLU activation function, and a second hidden layer.

[0157] As Figure 11 shown, the radiomics features are input into the RU classifier for training, and the specific steps are:

[0158] ​The true distribution of the research target may be biased from the distribution of the given data set, so some new parameters are needed to be introduced to reduce these biases, thereby increasing the flexibility of the model so that it can adapt to a wider or biased data set. Based on this, the application designs a binary classifier RU classifier with Rockafellar-Uryasev loss; the input layer of the RU classifier includes imaging features and the corresponding binary classification labels , the input layer is linearly transformed;

[0159] In the branch, the linearly transformed is output to the first hidden layer, the output of the first hidden layer is used ReLU activation function and learning; the output of the first hidden layer is taken as input, linearly transformed and output to the second hidden layer, and the output of the second hidden layer is used ReLU activation function to introduce nonlinearity and learning; learning produces an output layer composed of linear transformation and Sigmoid activation function, where the linear transformation maps to a single neuron, i.e. the output layer dimension is 1, and the Sigmoid function scales the output to the range [0, 1], representing the prediction probability of the binary classification task, and the prediction greater than 0.5 is "death", and the prediction less than or equal to 0.5 is "survival";

[0160] In the branch, the linearly transformed is output to the first hidden layer, the output of the first hidden layer is used ReLU activation function and learning; the output of the first hidden layer is taken as input, linearly transformed and output to the second hidden layer, and the output of the second hidden layer is used ReLU activation function to introduce nonlinearity and learning; learning produces an output layer composed of linear transformation, where the linear transformation maps to a single neuron, i.e. the output layer dimension is 1;

[0161] branch and branch output prediction probability and parameter ; then, according to the values of , and binary classification label , the Rockafellar-Uryasev loss is calculated; next, the training is iterated with the goal of minimizing the loss, and finally the optimal prediction value is obtained.

[0162] In this embodiment, branch is a conventional prediction model for obtaining the decision result of the model; branch is an auxiliary function of model decision, which is used to provide more parameter space for biased data.

[0163] In one specific embodiment, the Rockafellar-Uryasev loss is specifically:

[0164]

[0165] wherein, is the radiomics feature vector of the i-th sample, is the true binary label of the i-th sample, is a non-negative hyper-parameter, is the prediction result of the i-th sample in the j-th branch, is the output parameter of the i-th sample in the j-th branch, is defined as the binary cross-entropy loss: In addition, is a rectified linear unit used to improve the joint expression capability of the two prediction branches in the RU classifier, specifically:

[0166]

[0167] In addition, is a rectified linear unit used to improve the joint expression capability of the two prediction branches in the RU classifier, specifically:

[0168]

[0169] In addition, is a rectified linear unit used to improve the joint expression capability of the two prediction branches in the RU classifier, specifically:

[0170]

[0171] Let the learning goal be ; when the expectation of the intra-sample loss of all samples reaches the minimum, it means that the classification prediction performance reaches the best:

[0172] .

[0173] During the model training process, the RU loss is used to measure the relationship between the prediction output and the auxiliary output . This provides more predictable parameter space compared to the original loss function , thereby enhancing the robustness of the model to biased sample datasets. It is emphasized that the present application does not need to explore the true distribution of the research target, and the method can be flexibly adjusted according to the characteristics of the dataset under different sampling distributions, and through cyclic iteration to find the optimal hyper-parameter suitable for the current dataset ​​​​, to get better experimental results or adapt to specific applications in reality.

[0174] The present application extracts radiomics features from the segmentation results. Further, for the obtained quantitative radiomics features, the present application proposes a survival classification prediction model for patients. The calculation method in the Rockafellar-Uryasev loss is introduced into the loss function, thereby increasing the degree of freedom of the model, to a certain extent, solving the underfitting and overfitting problems of the neural network caused by uneven classification and data sampling bias in small sample data sets, and improving the reliability of prediction.

[0175] In this embodiment, the present application realizes an end-to-end process for survival prediction of non-small cell lung cancer patients. It includes two main steps: image segmentation and classification prediction.

[0176] The segmentation part of the present application combines multiple attention structures, residual connections, deep supervision methods and output feature fusion modules, thereby improving the robustness and generalization ability of the network. For comparison, as shown in Table 1;

[0177] Table 1

[0178] Method Dice U-Net 60.0% U2Net 62.6% Method of the invention 65.6%

[0179] The prediction part of the present application adopts the Rockafellar-Uryasev loss, which provides more predictable parameter space than a single loss function, thereby enhancing the effectiveness and stability of the model for biased sample data sets. For comparison, as shown in Table 2;

[0180] Table 2

[0181]

[0182] Embodiment 4

[0183] An application of a CT image classification method based on image segmentation and RU classifier is used for lung cancer survival classification, and the specific application method is: obtaining and preprocessing the lung CT image of the patient to be predicted, and sequentially passing through the trained segmentation network model and the RU classifier to generate a survival classification result.

[0184] In this embodiment, a CT image classification method based on image segmentation and RU classifier is also used for prediction, and the test is performed.

[0185] The data set of this example is derived from NSCLC-Radiomics in the Cancer Imaging Archive (TCIA) public data set, and the input data is a three-dimensional CT image in nii format, a total of 422 cases. First, the three-dimensional data is divided into two-dimensional images, for example,Figure 12 as shown, and unified to the size of (256, 256) by windowing and center cropping, as shown. Figure 13

[0186] The images are sequentially input into the trained segmentation network model to predict the segmentation map, as shown. Figure 14

[0187] After the model outputs the prediction result, the mask is pasted back to the image of size (512, 512) according to the original position, and finally stacked to restore the nii data.

[0188] Radiomics is used to extract quantitative features. In this example, a total of 107 radiomic features quantified after image segmentation are used for classification testing.

[0189] In 422 nii data, some patients with missing clinical data were excluded, and a total of 352 cases were used in the classification prediction part of this example. In this case, the 1-year survival status of the patient from receiving treatment to death is taken as the prediction target. The number ratio of the results of "survival" and "death" is 233:119.

[0190] In this example, the Pytorch library and Adam optimization are used to realize the iterative calculation of the neural network, with a learning rate of 0.01, and 150 epochs are set, and 10% of the training data of each epoch is used for verification.

[0191] The RU classifier obtains good results in the AUC index. Compared with the ANN model of a single neural network, the RU classifier with an auxiliary network is superior to the original scheme in various indicators, indicating that the method of the present application can capture valuable information that other machine learning models have not learned.

[0192] The ROC curve visualization result of the RU classifier on the test set is shown in Figure 15

[0193] As shown in Table 3, The larger the data set allows the sampling deviation to be larger. The example results prove that the data sampling deviation has a significant impact on the prediction result, and the effect of the present application is reflected in different highlight the importance of data sampling distribution, and the performance of the RU classifier when is more matched with the example data set.

[0194] Table 3

[0195]

[0196] ​​​Obviously, the above-mentioned embodiments of the present application are only examples for clearly illustrating the present application, but not limitation on the embodiments of the present application. Any modification, equivalent replacement and improvement etc. within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A CT image classification method based on image segmentation and RU classifier, characterized in that: The method comprises the following specific steps: Collecting CT images of a plurality of patients, pre-processing the CT images, and converting the pre-processed CT images into two-dimensional images to obtain a CT image set; The split network model comprises a symmetrical encoder, a decoder and a feature fusion layer; the decoder comprises n decoding convolutional layers provided with an upsampling layer and a feature fusion layer corresponding to n encoding convolutional layers and a 1*1 convolution; each decoding convolutional layer is composed of the same convolutional structure, including a basic convolutional module and a ResCBAM module; the ResCBAM module comprises two basic convolutional modules, a CBAM module and a residual connection module connected in sequence; the feature fusion layer comprises a 1*1 convolutional layer and an upsampling layer; n feature fusion layers output n, and the final output is obtained through 1*1 convolution after splicing ​ The output of the ResCBAM module is connected to the input of the corresponding feature fusion layer; The CT image set is input into the segmentation network model for training; The prediction segmentation image of the CT image set is obtained through the trained segmentation network model; and the radiomics features of the prediction segmentation image are extracted; constructing a RU classifier with a Rockafellar-Uryasev loss; the RU classifier comprising in parallel a binary classification prediction model branches and a branch comprising a helper function branches; the Rockafellar-Uryasev loss being in particular: wherein, is the imageomic feature vector of the i-th sample, is the true binary label of the i-th sample, is a non-negative hyperparameter, is the prediction result of the j-th branch of the i-th sample, is the output parameter of the j-th branch of the i-th sample, is the output parameter of the j-th branch of the i-th sample, is defined as binary cross-entropy loss:​​​​​ Furthermore, is a rectified linear unit, used to improve the joint representation capability for two prediction branches in the RU classifier, in particular: The radiomics features are input into the RU classifier for training; The CT image to be predicted is obtained and pre-processed, and the survival classification result is generated by sequentially passing through the trained segmentation network model and the RU classifier.

2. The CT image classification method based on image segmentation and RU classifier according to claim 1, characterized in that: The CT images are pre-processed, and the pre-processed CT images are converted into two-dimensional images to obtain a CT image set, specifically as follows: The GTV labels of each patient in the CT images are merged; the size and interval of each slice in the CT images are made the same to obtain the CT image set; The CT image set is divided into a training set and a test set; the three-dimensional data in the training set and the test set are cut into two-dimensional images according to the cross section; the window level and the window width of the two-dimensional images are set to highlight the cancer region in the images; and the pictures and labels of the two-dimensional images are center cropped; The data in the training set is randomly subjected to horizontal flipping, small-angle rotation, and random size scaling.

3. The CT image classification method based on image segmentation and RU classifier according to claim 1, characterized in that: The encoder comprises n encoding convolutional layers provided with down-sampling layers and 1 intermediate layer; each encoding convolutional layer is composed of the same convolutional structure, including a basic convolutional module and a ResCBAM module; The basic convolutional module CBG comprises a 3*3 convolution, a batch normalization BN, and an activation function layer connected in cascade; and the ResCBAM module comprises a CBAM module and a residual connection module; In each encoding convolutional layer, the output of the basic convolutional module is connected to the input of the ResCBAM module; and the output of the ResCBAM module is connected to the down-sampling layer. The down-sampling layer of the 1st to (n-1)th convolutional layers is connected to the input of the basic convolutional module of the next convolutional layer; and the down-sampling layer of the nth convolutional layer is connected to the input of the intermediate layer.

4. The CT image classification method based on image segmentation and RU classifier according to claim 3, characterized in that: The basic convolutional module CBG comprises a 3*3 convolution, a batch normalization BN, and an activation function layer connected in cascade; In each decoding convolutional layer, the output of the basic convolutional module is connected to the input of the ResCBAM module; and the input of the basic convolutional module is connected to the up-sampling layer; The up-sampling layer of the 1st to (n-1)th convolutional layers is respectively connected to the output of the ResCBAM module of the corresponding encoding convolutional layer, the output of the ResCBAM module of the next convolutional layer; and the up-sampling layer of the nth convolutional layer is respectively connected to the output of the ResCBAM module of the nth encoding convolutional layer and the output of the intermediate layer.

5. The CT image classification method based on image segmentation and RU classifier of claim 4, wherein: The CBAM module is a convolutional layer attention module; and after the feature is input into the ResCBAM module, the specific processing steps are as follows: The dimensions of the data input into the ResCBAM module are (C, H, W), the feature input is processed by two basic convolution modules in turn to obtain an input feature; The input feature is input into the CBAM module, and the attention mechanism is operated in turn from the channel dimension and the spatial dimension: In the channel dimension, the input feature is changed from (C, H, W) dimension to (C, 1, 1) dimension; two features output by the maximum pooling layer and the average pooling layer are spliced together to obtain a spliced feature with a dimension of (2, H, W); the spliced feature is input into the MLP layer to obtain a feature with a result dimension of (1, H, W), the MLP layer is a 1x1 convolution for compressing channels, an activation function, and a 1x1 convolution for expanding channels; the Sigmoid function is used to convert it into a probability on the spatial level, and it is multiplied by the input feature to obtain a feature adjusted by channel attention; In the spatial dimension, the feature adjusted by channel attention is changed from (C, H, W) dimension to (C, 1, 1) dimension by two parallel maximum pooling layers and average pooling layers; two features output by the maximum pooling layer and the average pooling layer are input into the shared MLP layer respectively, and the output and input dimensions are kept consistent; the two features output by the MLP layer are added, and the dimension is (C, 1, 1), the MLP layer is a convolution operation with an output channel of 1, a convolution kernel of 7, and a padding of 3; the Sigmoid function is used to convert it into a probability on the channel level, and it is multiplied by the feature adjusted by channel attention to obtain a feature adjusted by spatial attention; the feature adjusted by spatial attention is taken as the output of the CBAM module; The residual connection module adds the data input into the ResCBAM module and the output of the CBAM module to obtain the output of the ResCBAM module.

6. The CT image classification method based on image segmentation and RU classifier according to claim 5, characterized in that: When the CT image set is input into the segmentation network model for training, the method of deep supervision is introduced for training; In the training, the total loss function used is: wherein, corresponding to a weight assigned to n; loss function: wherein X is a prediction probability map Y is a true label, H is a length of an input picture, and W is a width of an input picture. After training, the test set is used to test the output results of the trained segmentation network model Test and evaluation index: Set greater than 0.5 as foreground, and less than 0.5 as background.

7. The CT image classification method based on image segmentation and RU classifier of claim 1, wherein: Extract the radiomics features of the predicted segmentation image, specifically: stack the predicted segmentation image to restore it to a three-dimensional nii data, use Radiomics to extract the radiomics features of each patient, and obtain three-dimensional data.

8. The CT image classification method based on image segmentation and RU classifier of claim 1, wherein: The described Branches and Branches each include a first hidden layer, a ReLU activation function, a second hidden layer; Input the radiomics features into the RU classifier for training, the specific steps are: The input layer of the RU classifier comprises radiomics features and their corresponding binary classification labels performing a linear transformation on the input layer; In In the branch, the linearly changed to the first hidden layer, the ReLU activation function is used for the output of the first hidden layer and learning is performed; The output of the first hidden layer is taken as the input, linearly transformed and output to the second hidden layer, and the ReLU activation function is used to introduce nonlinearity and learn the output of the second hidden layer; The learning produces an output layer consisting of a linear transformation and a sigmoid activation function where the linear transformation maps to a single neuron, i.e. the output layer has dimension 1, and the sigmoid function scales the output to the range [0, 1] representing the predicted probability for a binary classification task, predicting "death" for predictions greater than 0.5 and "survival" for predictions less than or equal to 0.5; In In the branch, the linearly changed to the first hidden layer, the ReLU activation function is used for the output of the first hidden layer and learning is performed; The output of the first hidden layer is taken as the input, linearly transformed and output to the second hidden layer, and the ReLU activation function is used to introduce nonlinearity and learn the output of the second hidden layer; The learning produces an output layer consisting of linear transformations where the linear transformations map to a single neuron, i.e. the output layer has dimension 1 ; According to , and binary classification labels the Rockafellar-Uryasev loss is computed; next, cyclic iterative training is performed with the goal of minimizing the loss, resulting in the optimal prediction value.

9. The CT image classification method based on image segmentation and RU classifier according to claim 8, characterized in that: During training, let the goal of learning be ; when the expectation of the intra-sample loss of all samples reaches the minimum, it means that the classification prediction performance reaches the best.

10. Use of a CT image classification method based on image segmentation and RU classifier according to any one of claims 1 to 9, characterized in that: Used for lung cancer survival classification.

Images